Claude Code has at least six ways to read a web page, and they break in different places. I use a ladder: start at the cheapest rung, climb only when the rung below returns nothing real, and keep anything with consequences off the top two, which run inside my own browser.
On 2 October I measured it against eleven sites that wall agents off. Plain fetching read one. A fresh automated browser read none. My own signed-in Chrome, driven by Anthropic’s Claude in Chrome extension, reached real content on eight.
| Rung | Tool | Climb to it when | It breaks when |
|---|---|---|---|
| 1 | WebFetch, WebSearch (built in) | you need one fact | you need the whole page: WebFetch returns a small model’s summary |
| 2 | defuddle | you need the whole page as Markdown | the site has a bot wall |
| 3 | agent-browser, fresh session | the page needs JavaScript, clicks or screenshots | the site checks for automation |
| 4 | agent-browser on a dedicated profile | you need a login, or the task changes something | the site checks the browser itself |
| 5 | Claude in Chrome, in your own browser | the wall checks the browser, or the data lives in your session | the extension refuses the domain, or the page renders only on screen |
| 6 | Apple Events into your own Chrome (macOS) | the extension refuses the domain | you need it to run unattended |
Two rules sit over all six. Stop at the first rung that returns real content: the listings, the thread, the signed-in view, not an HTTP 200 wrapped around a challenge page. Keep anything that clicks, types, uploads or spends on rung 4.
WebFetch hands the page to a small model and returns its summary, so details go missing. WebSearch won’t take some domains as a filter, Reddit and X among them. For the whole page, try defuddle. It strips the navigation and returns the article as local Markdown, links intact:
npx defuddle parse https://code.claude.com/docs/en/chrome --markdown
Neither gets past a bot wall: on the eleven walled sites, curl with a desktop Chrome user agent read one, Product Hunt. The rest returned a 403, Cloudflare’s “Just a moment…”, an empty JavaScript shell or a login redirect.
For JavaScript-built pages, or clicking through, drive a browser from the terminal. I use Vercel’s agent-browser: open the page, snapshot the accessibility tree, act on an element by its ref.
agent-browser open https://example.com
agent-browser snapshot
agent-browser click @e3
A fresh session read none of the eleven. Six sites showed Cloudflare’s challenge, Etsy DataDome’s captcha, Reddit a block page, and the signed-in sites loaded as for a stranger: a login screen, a public page, a signed-out studio.If you script screenshots, check the files: several failure modes print a success line while writing the wrong file, no file, or an image blank below the fold.
For a login, or a task with consequences like generating video on a paid account, give the automated browser its own Chrome: a separate profile directory and a debug port. Log in once by hand, then point agent-browser at the port:
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" --user-data-dir="$HOME/.cache/agent-chrome" --remote-debugging-port=9333 --no-first-run
agent-browser --cdp 9333 open https://higgsfield.ai
The login survives restarts and nothing touches your everyday browser; recent Chrome refuses a debug port on your default profile anyway. It won’t get you past a wall that checks the browser, though. In September I copied my real profile onto a debug port and got into Upwork, until the clearance token the copy carried expired. After that no profile got in, copy or original.
Claude Code drives the extension through its mcp__claude-in-chrome__* tools. It opens a tab group inside your real, signed-in Chrome, so a page load is your own browser loading it, cookies and history included.
Install the extension from the Chrome Web Store and sign in to Claude Code with a claude.ai account; an API key won’t connect. In the terminal, start with claude --chrome, or run /chrome and choose “Enabled by default”. In the VS Code panel, type @browser in a message, or turn the setting on: since Claude Code 2.1.287 the panel then connects at the start of every session.
The same eleven sites, the same afternoon:
| Site | curl | Fresh agent-browser | Claude in Chrome |
|---|---|---|---|
| Upwork job search | 403 challenge | ”Just a moment…” | Signed in, job cards, two pages |
| ChatGPT | 403 | ”Just a moment…” | Signed in |
| Skool classroom | Bounced to public page | Bounced to public page | Signed in, course list |
| Kasta search | 403 | ”Just a moment…“ | 24 of 24 listings with prices |
| Rozetka search | 403 | ”Just a moment…” | Past the wall; results on some pages only |
| Instagram keyword search | Login redirect | Login redirect | Signed in, first screen of posts |
| Product Hunt | Readable | ”Just a moment…” | Readable |
| Higgsfield studio | Navigation only | Signed out | Signed in, full staging flow |
| Empty shell | Block page | Refused by the extension | |
| Patreon | 403 | ”Just a moment…” | Refused by the extension |
| Etsy search | 403 | DataDome captcha | Refused by the extension |
One or two loads per site, sessions warm from my ordinary browsing: a field note, not a benchmark.
Before the first run:
document.visibilityState reads hidden for the whole run, so scroll-triggered content never renders. Rozetka’s search grid sat at 70 grey placeholders and Instagram’s infinite scroll loaded nothing. Server-rendered pages read fine.@browser lifts the pin and auto-allows every browser tool for the session, clicks and uploads included, without auto mode’s own check. I keep the prompts and batch calls.On Higgsfield’s studio, which I drive for models its API doesn’t offer, the extension picked the model, uploaded a still and read the price off the Generate button. The Generate click stays on rung 4.
Why this rung passes when rung 3 doesn’t, as far as I can tell. Two weeks ago I noted that a browser driven over CDP, the Chrome DevTools Protocol, gets detected before a challenge renders. The extension attaches Chrome’s debugger, which speaks CDP, on every navigation, and Cloudflare let it through on five sites. So CDP isn’t the tell, or not the only one. I suspect the walls grade the browser and its history: months of cookies, a matching fingerprint, a record of passing challenges as me. A fresh session has none of that, and a copied profile only until its token expires.
On a Mac, Chrome will run JavaScript sent by a local script once you allow it: View → Developer → Allow JavaScript from Apple Events. Then one line reads the page you’re on:
osascript -e 'tell application "Google Chrome" to execute active tab of front window javascript "document.title"'
Set the URL, wait, read the DOM, repeat. It took me through more than a hundred Upwork search pages in September with zero challenges, and it’s now my route for Reddit and Patreon. The cost: while the setting is on, any local process with automation rights can run JavaScript in your tabs, so switch it off when you’re done. And it needs you at the Mac.