---
title: "How I browse the web from Claude Code"
author: "Stan Voo"
url: https://www.99rabbits.com/blog/how-i-browse-the-web-from-claude-code
published: 2026-10-05
updated: 2026-10-27
summary: "Claude Code has seven ways to read a web page, and they break in different places. I climb from the cheapest rung, stop at the first one that returns real content, and keep anything with consequences off the top two, which run inside my own browser. Measured on eleven bot-walled sites: plain fetching read one, a fresh automated browser none, my own signed-in Chrome through Claude in Chrome eight. A crawler API, added later, read Upwork's job search with no browser at all."
---

# How I browse the web from Claude Code

Claude Code has at least seven ways to read a web page, and they break in different places. I use a ladder: start at the cheapest rung, climb only when the rung below returns nothing real, and keep anything with consequences off the top two, which run inside my own browser.

On 2 October I measured it against eleven sites that wall agents off. Plain fetching read one. A fresh automated browser read none. My own signed-in Chrome, driven by Anthropic's Claude in Chrome extension, reached real content on eight. On 27 October I added a rung I had skipped: Exa, a search API that fetches pages with its own crawler. It read four of the eleven, Upwork's job search among them, with no browser and nobody at the Mac.

## The ladder

| Rung | Tool | Climb to it when | It breaks when |
| - | - | - | - |
| 1 | WebFetch, WebSearch (built in) | you need one fact | you need the whole page: WebFetch returns a small model's summary |
| 2 | `defuddle` | you need the whole page as Markdown | the site has a bot wall |
| 3 | Exa, a search API with its own crawler | the site answers with a 403 or a challenge page | the page needs a login or JavaScript, or the wall stops Exa's crawler too |
| 4 | `agent-browser`, fresh session | the page needs JavaScript, clicks or screenshots | the site checks for automation |
| 5 | `agent-browser` on a dedicated profile | you need a login, or the task changes something | the site checks the browser itself |
| 6 | Claude in Chrome, in your own browser | the wall checks the browser, or the data lives in your session | the extension refuses the domain, or the page renders only on screen |
| 7 | Apple Events into your own Chrome (macOS) | the extension refuses the domain | you need it to run unattended |

Three rules sit over all seven. **Stop at the first rung that returns real content**: the listings, the thread, the signed-in view, not an HTTP 200 wrapped around a challenge page. **Before you climb past a blank page, check that the page works at all**: if the request that loads its content fails, no rung will read it. **Keep anything that clicks, types, uploads or spends on rung 5.**

## Rungs 1 and 2: fetch the page

WebFetch hands the page to a small model and returns its summary, so details go missing. WebSearch won't take some domains as a filter, Reddit and X among them. For the whole page, try defuddle. It strips the navigation and returns the article as local Markdown, links intact:

```sh
npx defuddle parse https://code.claude.com/docs/en/chrome --markdown
```

Neither gets past a bot wall: on the eleven walled sites, `curl` with a desktop Chrome user agent read one, Product Hunt. The rest returned a 403, Cloudflare's "Just a moment...", an empty JavaScript shell or a login redirect.

## Rung 3: someone else's crawler

Exa is a search API built for agents. It keeps its own index and fetches pages with its own crawler, so the request that meets the wall comes from Exa's machines, not mine. Its contents endpoint takes a URL and returns the page as text:

```sh
curl -s https://api.exa.ai/contents -H "x-api-key: $EXA_API_KEY" -H "Content-Type: application/json" -d '{"urls": ["https://www.upwork.com/nx/search/jobs/?q=ai%20video%20editing"], "text": true}'
```

On Upwork's job search, where curl got a 403 and a captcha, it returned ten job cards with rates, descriptions and how long ago each was posted. It did the same with `maxAgeHours: 0`, which forces a fresh fetch instead of a stored copy. Why Upwork lets this crawler in and not curl, I don't know. My guess is that a site that wants its jobs found in search treats an indexing crawler more kindly than an unknown client.

The stored copy cuts both ways. An Etsy listing came back with its price where curl and defuddle got a 403, but a fresh fetch of the same page timed out, so what I read was Exa's copy from some earlier day. Check the date before trusting a price.

It read nothing behind a login, and nothing from Reddit, Instagram, Etsy's search or Rozetka's search grid. A page costs about a tenth of a cent and a search under a cent, and the free tier resets your balance to $10 every month. I call it through a small script, and Claude Code reaches for it whenever defuddle gets a 403.

## Rung 4: an automated browser

For JavaScript-built pages, or clicking through, drive a browser from the terminal. I use Vercel's `agent-browser`: open the page, snapshot the accessibility tree, act on an element by its ref.

```sh
agent-browser open https://example.com
agent-browser snapshot
agent-browser click @e3
```

A fresh session read none of the eleven. Six sites showed Cloudflare's challenge, Etsy DataDome's captcha, Reddit a block page, and the signed-in sites loaded as for a stranger: a login screen, a public page, a signed-out studio. (If you script screenshots, check the files: several failure modes print a success line while writing the wrong file, no file, or an image blank below the fold.)

When a page comes back blank, `agent-browser network requests` lists every call it made, with its status. On 27 October a session climbed all the way to my own Chrome for a knowledge-base article that was blank for everyone. The request that loaded its text was returning 502: the site was down, and no rung could have read it.

## Rung 5: a dedicated logged-in profile

For a login, or a task with consequences like generating video on a paid account, give the automated browser its own Chrome: a separate profile directory and a debug port. Log in once by hand, then point `agent-browser` at the port:

```sh
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" --user-data-dir="$HOME/.cache/agent-chrome" --remote-debugging-port=9333 --no-first-run
agent-browser --cdp 9333 open https://higgsfield.ai
```

The login survives restarts and nothing touches your everyday browser; recent Chrome refuses a debug port on its default data directory anyway. It won't get you past a wall that checks the browser, though. In September I copied my real profile onto a debug port and got into Upwork, until the clearance cookie the copy carried expired. After that no profile got in, copy or original.

## Rung 6: your own browser, through Claude in Chrome

Claude Code drives the extension through its `mcp__claude-in-chrome__*` tools. It opens a tab group inside your real, signed-in Chrome, so a page load is your own browser loading it, cookies and history included.

Install the extension from the Chrome Web Store and sign in to Claude Code with a claude.ai account; an API key won't connect. In the terminal, start with `claude --chrome`, or run `/chrome` and choose "Enabled by default". In the VS Code panel, type `@browser` in a message, or turn the setting on: since Claude Code 2.1.287 the panel then connects at the start of every session.

Each browser call then asks for permission in VS Code, whatever mode the session runs in, bypass included: the connection pins the extension to ask every time. An allow rule gets past the pin, so I list the seven tools that never click or type in `~/.claude/settings.json`:

```json
{
  "permissions": {
    "allow": [
      "mcp__claude-in-chrome__tabs_context_mcp",
      "mcp__claude-in-chrome__tabs_create_mcp",
      "mcp__claude-in-chrome__tabs_close_mcp",
      "mcp__claude-in-chrome__navigate",
      "mcp__claude-in-chrome__get_page_text",
      "mcp__claude-in-chrome__read_page",
      "mcp__claude-in-chrome__find"
    ]
  }
}
```

If you already have a `permissions` block, merge the list into it. One caveat: `navigate` loads whatever URL it's handed, in a browser signed in everywhere, so it's only as harmless as the link.

The same eleven sites, the same afternoon:

| Site | curl | Fresh agent-browser | Claude in Chrome | Exa, 27 October |
| - | - | - | - | - |
| Upwork job search | 403 challenge | "Just a moment..." | Signed in, job cards, two pages | Ten job cards |
| ChatGPT | 403 | "Just a moment..." | Signed in | Public landing page |
| Skool classroom | Bounced to public page | Bounced to public page | Signed in, course list | Public preview, courses locked |
| Kasta search | 403 | "Just a moment..." | 24 of 24 listings with prices | Listings with prices |
| Rozetka search | 403 | "Just a moment..." | Past the wall; results on some pages only | Empty |
| Instagram keyword search | Login redirect | Login redirect | Signed in, first screen of posts | Nothing |
| Product Hunt | Readable | "Just a moment..." | Readable | Readable |
| Higgsfield studio | Navigation only | Signed out | Signed in, full staging flow | Nothing |
| Reddit | Empty shell | Block page | Refused by the extension | Empty |
| Patreon | 403 | "Just a moment..." | Refused by the extension | Creator page readable |
| Etsy search | 403 | DataDome captcha | Refused by the extension | Nothing; single listings from a stored copy |

One or two loads per site, sessions warm from my ordinary browsing: a field note, not a benchmark. The Exa column comes from a re-run on 27 October, on the same kind of page; by then Kasta and Patreon let plain curl through as well. Walls move.

Before the first run:

- **Click by element reference, never by screenshot coordinates.** Six of six ref clicks landed and none of three coordinate clicks did: the page is 2,305 CSS pixels wide and the screenshot shows it at 1,530.
- **Check the DOM, not the screenshot.** Two of eight screenshots came back wrong while reporting success; one tiled the page into a 2×2 grid.
- **The tab stays in the background, and the page knows it.** `document.visibilityState` reads `hidden` for the whole run, so scroll-triggered content never renders. Rozetka's search grid sat at 70 grey placeholders and Instagram's infinite scroll loaded nothing. Server-rendered pages read fine.
- **Everything off the allow list still asks.** Clicks, typing, scripts and `browser_batch`, which can carry any of them, ask for permission on every call. I tested the list on 13 October, on Claude Code 2.1.292, in a fresh tab: opening a page and reading its text went through with no permission prompt, and reading the console, which I'd left off the list as a control, asked. One run, three calls. Typing `@browser` lifts the pin altogether and auto-allows every browser tool for the session, clicks and uploads included, without auto mode's own check. I keep the permission prompts for anything that acts. (Test a new rule from a fresh tab. The session that wrote mine had its next browser call refused by auto mode's classifier as self-modification: an agent granting itself a permission, then using it. Fair enough.)
- **It brings its own wall.** Reddit and Patreon got "This site is not allowed due to safety restrictions", a category check inside the extension. Etsy got "Navigation to this domain is not allowed", from its per-site permissions. Rung 7 is for these.

On Higgsfield's studio, which I drive for models its API doesn't offer, the extension picked the model, uploaded a still and read the price off the Generate button. The Generate click stays on rung 5.

**Why this rung passes when rung 4 doesn't, as far as I can tell.** Two weeks ago I noted that a browser driven over CDP, the Chrome DevTools Protocol, gets detected before a challenge renders. The extension attaches Chrome's debugger, which speaks CDP, on every navigation, and Cloudflare let it through on five sites. So CDP isn't the tell, or not the only one. I suspect the walls grade the browser and its history: months of cookies, a matching fingerprint, a record of passing challenges as me. A fresh session has none of that, and a copied profile only until its clearance cookie expires.

## Rung 7: Apple Events, for what the extension refuses

On a Mac, Chrome will run JavaScript sent by a local script once you allow it: View → Developer → Allow JavaScript from Apple Events. Then one line reads the page you're on:

```sh
osascript -e 'tell application "Google Chrome" to execute active tab of front window javascript "document.title"'
```

Set the URL, wait, read the DOM, repeat. It took me through more than a hundred Upwork search pages in September with zero challenges, and it's now my route for Reddit and Patreon. The cost: while the setting is on, any local process with automation rights can run JavaScript in your tabs, so switch it off when you're done. And it needs you at the Mac.

## Where the ladder runs out

- **Volume.** Upwork tolerated Apple Events for a hundred pages. Whether it tolerates a debugger-attached tab that long, or Exa's crawler, I haven't measured.
- **Pages that render only on screen,** from a background tab. Bringing the tab forward would probably fix it, at the price of your screen.
- **Anything unattended that needs you.** Rungs 6 and 7 need you at the machine. A scheduled job gets every rung below them, and since rung 3 that includes some walled pages, Upwork's job search among them. It still gets nothing only your own browser can see.

## Changes since publication

- **13 October 2026.** Added an allow rule that lets Claude in Chrome's seven tools that never click or type run without a permission prompt, and how I tested it. It replaces "Expect a prompt per call in VS Code, even in auto mode", which no longer holds once the rule is in.
- **14 October 2026.** Two words corrected. Chrome refuses a debug port on its default data directory, which holds every profile, not only the default one. And what expired on the copied profile was Cloudflare's clearance cookie, `cf_clearance`, not a token.
- **27 October 2026.** Added rung 3, Exa's crawler, and an Exa column in the table from a re-run that day; the rungs above it moved up by one. Added a third rule, on checking that a blank page works before climbing. Rewrote the limit on unattended jobs: they now reach some walled pages.
