Claude Code has at least seven ways to read a web page, and they break in different places. I use a ladder: start at the cheapest rung, climb only when the rung below returns nothing real, and keep anything with consequences off the top two, which run inside my own browser.
On 2 October I measured it against eleven sites that wall agents off. Plain fetching read one. A fresh automated browser read none. My own signed-in Chrome, driven by Anthropic’s Claude in Chrome extension, reached real content on eight. On 27 October I added a rung I had skipped: Exa, a search API that fetches pages with its own crawler. It read four of the eleven, Upwork’s job search among them, with no browser and nobody at the Mac.
| Rung | Tool | Climb to it when | It breaks when |
|---|---|---|---|
| 1 | WebFetch, WebSearch (built in) | you need one fact | you need the whole page: WebFetch returns a small model’s summary |
| 2 | defuddle | you need the whole page as Markdown | the site has a bot wall |
| 3 | Exa, a search API with its own crawler | the site answers with a 403 or a challenge page | the page needs a login or JavaScript, or the wall stops Exa’s crawler too |
| 4 | agent-browser, fresh session | the page needs JavaScript, clicks or screenshots | the site checks for automation |
| 5 | agent-browser on a dedicated profile | you need a login, or the task changes something | the site checks the browser itself |
| 6 | Claude in Chrome, in your own browser | the wall checks the browser, or the data lives in your session | the extension refuses the domain, or the page renders only on screen |
| 7 | Apple Events into your own Chrome (macOS) | the extension refuses the domain | you need it to run unattended |
Three rules sit over all seven. Stop at the first rung that returns real content: the listings, the thread, the signed-in view, not an HTTP 200 wrapped around a challenge page. Before you climb past a blank page, check that the page works at all: if the request that loads its content fails, no rung will read it. Keep anything that clicks, types, uploads or spends on rung 5.
WebFetch hands the page to a small model and returns its summary, so details go missing. WebSearch won’t take some domains as a filter, Reddit and X among them. For the whole page, try defuddle. It strips the navigation and returns the article as local Markdown, links intact:
npx defuddle parse https://code.claude.com/docs/en/chrome --markdown
Neither gets past a bot wall: on the eleven walled sites, curl with a desktop Chrome user agent read one, Product Hunt. The rest returned a 403, Cloudflare’s “Just a moment…”, an empty JavaScript shell or a login redirect.
Exa is a search API built for agents. It keeps its own index and fetches pages with its own crawler, so the request that meets the wall comes from Exa’s machines, not mine. Its contents endpoint takes a URL and returns the page as text:
curl -s https://api.exa.ai/contents -H "x-api-key: $EXA_API_KEY" -H "Content-Type: application/json" -d '{"urls": ["https://www.upwork.com/nx/search/jobs/?q=ai%20video%20editing"], "text": true}'
On Upwork’s job search, where curl got a 403 and a captcha, it returned ten job cards with rates, descriptions and how long ago each was posted. It did the same with maxAgeHours: 0, which forces a fresh fetch instead of a stored copy. Why Upwork lets this crawler in and not curl, I don’t know. My guess is that a site that wants its jobs found in search treats an indexing crawler more kindly than an unknown client.
The stored copy cuts both ways. An Etsy listing came back with its price where curl and defuddle got a 403, but a fresh fetch of the same page timed out, so what I read was Exa’s copy from some earlier day. Check the date before trusting a price.
It read nothing behind a login, and nothing from Reddit, Instagram, Etsy’s search or Rozetka’s search grid. A page costs about a tenth of a cent and a search under a cent, and the free tier resets your balance to $10 every month. I call it through a small script, and Claude Code reaches for it whenever defuddle gets a 403.
For JavaScript-built pages, or clicking through, drive a browser from the terminal. I use Vercel’s agent-browser: open the page, snapshot the accessibility tree, act on an element by its ref.
agent-browser open https://example.com
agent-browser snapshot
agent-browser click @e3
A fresh session read none of the eleven. Six sites showed Cloudflare’s challenge, Etsy DataDome’s captcha, Reddit a block page, and the signed-in sites loaded as for a stranger: a login screen, a public page, a signed-out studio.If you script screenshots, check the files: several failure modes print a success line while writing the wrong file, no file, or an image blank below the fold.
When a page comes back blank, agent-browser network requests lists every call it made, with its status. On 27 October a session climbed all the way to my own Chrome for a knowledge-base article that was blank for everyone. The request that loaded its text was returning 502: the site was down, and no rung could have read it.
For a login, or a task with consequences like generating video on a paid account, give the automated browser its own Chrome: a separate profile directory and a debug port. Log in once by hand, then point agent-browser at the port:
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" --user-data-dir="$HOME/.cache/agent-chrome" --remote-debugging-port=9333 --no-first-run
agent-browser --cdp 9333 open https://higgsfield.ai
The login survives restarts and nothing touches your everyday browser; recent Chrome refuses a debug port on its default data directory anyway. It won’t get you past a wall that checks the browser, though. In September I copied my real profile onto a debug port and got into Upwork, until the clearance cookie the copy carried expired. After that no profile got in, copy or original.
Claude Code drives the extension through its mcp__claude-in-chrome__* tools. It opens a tab group inside your real, signed-in Chrome, so a page load is your own browser loading it, cookies and history included.
Install the extension from the Chrome Web Store and sign in to Claude Code with a claude.ai account; an API key won’t connect. In the terminal, start with claude --chrome, or run /chrome and choose “Enabled by default”. In the VS Code panel, type @browser in a message, or turn the setting on: since Claude Code 2.1.287 the panel then connects at the start of every session.
Each browser call then asks for permission in VS Code, whatever mode the session runs in, bypass included: the connection pins the extension to ask every time. An allow rule gets past the pin, so I list the seven tools that never click or type in ~/.claude/settings.json:
{
"permissions": {
"allow": [
"mcp__claude-in-chrome__tabs_context_mcp",
"mcp__claude-in-chrome__tabs_create_mcp",
"mcp__claude-in-chrome__tabs_close_mcp",
"mcp__claude-in-chrome__navigate",
"mcp__claude-in-chrome__get_page_text",
"mcp__claude-in-chrome__read_page",
"mcp__claude-in-chrome__find"
]
}
}
If you already have a permissions block, merge the list into it. One caveat: navigate loads whatever URL it’s handed, in a browser signed in everywhere, so it’s only as harmless as the link.
The same eleven sites, the same afternoon:
| Site | curl | Fresh agent-browser | Claude in Chrome | Exa, 27 October |
|---|---|---|---|---|
| Upwork job search | 403 challenge | ”Just a moment…” | Signed in, job cards, two pages | Ten job cards |
| ChatGPT | 403 | ”Just a moment…” | Signed in | Public landing page |
| Skool classroom | Bounced to public page | Bounced to public page | Signed in, course list | Public preview, courses locked |
| Kasta search | 403 | ”Just a moment…“ | 24 of 24 listings with prices | Listings with prices |
| Rozetka search | 403 | ”Just a moment…” | Past the wall; results on some pages only | Empty |
| Instagram keyword search | Login redirect | Login redirect | Signed in, first screen of posts | Nothing |
| Product Hunt | Readable | ”Just a moment…” | Readable | Readable |
| Higgsfield studio | Navigation only | Signed out | Signed in, full staging flow | Nothing |
| Empty shell | Block page | Refused by the extension | Empty | |
| Patreon | 403 | ”Just a moment…” | Refused by the extension | Creator page readable |
| Etsy search | 403 | DataDome captcha | Refused by the extension | Nothing; single listings from a stored copy |
One or two loads per site, sessions warm from my ordinary browsing: a field note, not a benchmark. The Exa column comes from a re-run on 27 October, on the same kind of page; by then Kasta and Patreon let plain curl through as well. Walls move.
Before the first run:
document.visibilityState reads hidden for the whole run, so scroll-triggered content never renders. Rozetka’s search grid sat at 70 grey placeholders and Instagram’s infinite scroll loaded nothing. Server-rendered pages read fine.browser_batch, which can carry any of them, ask for permission on every call. I tested the list on 13 October, on Claude Code 2.1.292, in a fresh tab: opening a page and reading its text went through with no permission prompt, and reading the console, which I’d left off the list as a control, asked. One run, three calls. Typing @browser lifts the pin altogether and auto-allows every browser tool for the session, clicks and uploads included, without auto mode’s own check. I keep the permission prompts for anything that acts.Test a new rule from a fresh tab. The session that wrote mine had its next browser call refused by auto mode’s classifier as self-modification: an agent granting itself a permission, then using it. Fair enough.On Higgsfield’s studio, which I drive for models its API doesn’t offer, the extension picked the model, uploaded a still and read the price off the Generate button. The Generate click stays on rung 5.
Why this rung passes when rung 4 doesn’t, as far as I can tell. Two weeks ago I noted that a browser driven over CDP, the Chrome DevTools Protocol, gets detected before a challenge renders. The extension attaches Chrome’s debugger, which speaks CDP, on every navigation, and Cloudflare let it through on five sites. So CDP isn’t the tell, or not the only one. I suspect the walls grade the browser and its history: months of cookies, a matching fingerprint, a record of passing challenges as me. A fresh session has none of that, and a copied profile only until its clearance cookie expires.
On a Mac, Chrome will run JavaScript sent by a local script once you allow it: View → Developer → Allow JavaScript from Apple Events. Then one line reads the page you’re on:
osascript -e 'tell application "Google Chrome" to execute active tab of front window javascript "document.title"'
Set the URL, wait, read the DOM, repeat. It took me through more than a hundred Upwork search pages in September with zero challenges, and it’s now my route for Reddit and Patreon. The cost: while the setting is on, any local process with automation rights can run JavaScript in your tabs, so switch it off when you’re done. And it needs you at the Mac.
cf_clearance, not a token.