How I browse the web from Claude Code

Claude Code has at least seven ways to read a web page, and they break in dif­fer­ent places. I use a ladder: start at the cheap­est rung, climb only when the rung below returns noth­ing real, and keep any­thing with con­se­quences off the top two, which run inside my own browser.

On 2 October I mea­sured it against eleven sites that wall agents off. Plain fetch­ing read one. A fresh auto­mated browser read none. My own signed-in Chrome, driven by Anthropic’s Claude in Chrome exten­sion, reached real con­tent on eight. On 27 October I added a rung I had skipped: Exa, a search API that fetches pages with its own crawler. It read four of the eleven, Upwork’s job search among them, with no browser and nobody at the Mac.

The ladder

RungToolClimb to it whenIt breaks when
1WebFetch, WebSearch (built in)you need one factyou need the whole page: WebFetch returns a small model’s summary
2defuddleyou need the whole page as Markdownthe site has a bot wall
3Exa, a search API with its own crawlerthe site answers with a 403 or a challenge pagethe page needs a login or JavaScript, or the wall stops Exa’s crawler too
4agent-browser, fresh sessionthe page needs JavaScript, clicks or screenshotsthe site checks for automation
5agent-browser on a dedicated profileyou need a login, or the task changes somethingthe site checks the browser itself
6Claude in Chrome, in your own browserthe wall checks the browser, or the data lives in your sessionthe extension refuses the domain, or the page renders only on screen
7Apple Events into your own Chrome (macOS)the extension refuses the domainyou need it to run unattended

Three rules sit over all seven. Stop at the first rung that returns real con­tent: the list­ings, the thread, the signed-in view, not an HTTP 200 wrapped around a chal­lenge page. Before you climb past a blank page, check that the page works at all: if the request that loads its con­tent fails, no rung will read it. Keep any­thing that clicks, types, uploads or spends on rung 5.

Rungs 1 and 2: fetch the page

WebFetch hands the page to a small model and returns its sum­mary, so details go missing. WebSearch won’t take some domains as a filter, Reddit and X among them. For the whole page, try defuddle. It strips the nav­i­ga­tion and returns the arti­cle as local Markdown, links intact:

Terminal
npx defuddle parse https://code.claude.com/docs/en/chrome --markdown

Neither gets past a bot wall: on the eleven walled sites, curl with a desk­top Chrome user agent read one, Product Hunt. The rest returned a 403, Cloudflare’s “Just a moment…”, an empty JavaScript shell or a login redirect.

Rung 3: someone else’s crawler

Exa is a search API built for agents. It keeps its own index and fetches pages with its own crawler, so the request that meets the wall comes from Exa’s machines, not mine. Its con­tents end­point takes a URL and returns the page as text:

Terminal
curl -s https://api.exa.ai/contents -H "x-api-key: $EXA_API_KEY" -H "Content-Type: application/json" -d '{"urls": ["https://www.upwork.com/nx/search/jobs/?q=ai%20video%20editing"], "text": true}'

On Upwork’s job search, where curl got a 403 and a captcha, it returned ten job cards with rates, descrip­tions and how long ago each was posted. It did the same with maxAgeHours: 0, which forces a fresh fetch instead of a stored copy. Why Upwork lets this crawler in and not curl, I don’t know. My guess is that a site that wants its jobs found in search treats an index­ing crawler more kindly than an unknown client.

The stored copy cuts both ways. An Etsy list­ing came back with its price where curl and defud­dle got a 403, but a fresh fetch of the same page timed out, so what I read was Exa’s copy from some ear­lier day. Check the date before trust­ing a price.

It read noth­ing behind a login, and noth­ing from Reddit, Instagram, Etsy’s search or Rozetka’s search grid. A page costs about a tenth of a cent and a search under a cent, and the free tier resets your bal­ance to $10 every month. I call it through a small script, and Claude Code reaches for it when­ever defud­dle gets a 403.

Rung 4: an automated browser

For JavaScript-built pages, or click­ing through, drive a browser from the terminal. I use Vercel’s agent-browser: open the page, snap­shot the acces­si­bil­ity tree, act on an ele­ment by its ref.

Terminal
agent-browser open https://example.com
agent-browser snapshot
agent-browser click @e3

A fresh ses­sion read none of the eleven. Six sites showed Cloudflare’s chal­lenge, Etsy DataDome’s captcha, Reddit a block page, and the signed-in sites loaded as for a stranger: a login screen, a public page, a signed-out studio.If you script screenshots, check the files: several failure modes print a success line while writing the wrong file, no file, or an image blank below the fold.

When a page comes back blank, agent-browser network requests lists every call it made, with its status. On 27 October a ses­sion climbed all the way to my own Chrome for a knowledge-base arti­cle that was blank for everyone. The request that loaded its text was return­ing 502: the site was down, and no rung could have read it.

Rung 5: a dedicated logged-in profile

For a login, or a task with con­se­quences like gen­er­at­ing video on a paid account, give the auto­mated browser its own Chrome: a sep­a­rate pro­file direc­tory and a debug port. Log in once by hand, then point agent-browser at the port:

Terminal
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" --user-data-dir="$HOME/.cache/agent-chrome" --remote-debugging-port=9333 --no-first-run
agent-browser --cdp 9333 open https://higgsfield.ai

The login sur­vives restarts and noth­ing touches your every­day browser; recent Chrome refuses a debug port on its default data direc­tory anyway. It won’t get you past a wall that checks the browser, though. In September I copied my real pro­file onto a debug port and got into Upwork, until the clear­ance cookie the copy car­ried expired. After that no pro­file got in, copy or original.

Rung 6: your own browser, through Claude in Chrome

Claude Code drives the exten­sion through its mcp__claude-in-chrome__* tools. It opens a tab group inside your real, signed-in Chrome, so a page load is your own browser load­ing it, cook­ies and his­tory included.

Install the exten­sion from the Chrome Web Store and sign in to Claude Code with a claude.ai account; an API key won’t connect. In the ter­mi­nal, start with claude --chrome, or run /chrome and choose “Enabled by default”. In the VS Code panel, type @browser in a mes­sage, or turn the set­ting on: since Claude Code 2.1.287 the panel then con­nects at the start of every session.

Each browser call then asks for per­mis­sion in VS Code, what­ever mode the ses­sion runs in, bypass included: the con­nec­tion pins the exten­sion to ask every time. An allow rule gets past the pin, so I list the seven tools that never click or type in ~/.claude/settings.json:

JSON
{
  "permissions": {
    "allow": [
      "mcp__claude-in-chrome__tabs_context_mcp",
      "mcp__claude-in-chrome__tabs_create_mcp",
      "mcp__claude-in-chrome__tabs_close_mcp",
      "mcp__claude-in-chrome__navigate",
      "mcp__claude-in-chrome__get_page_text",
      "mcp__claude-in-chrome__read_page",
      "mcp__claude-in-chrome__find"
    ]
  }
}

If you already have a permissions block, merge the list into it. One caveat: navigate loads what­ever URL it’s handed, in a browser signed in every­where, so it’s only as harm­less as the link.

The same eleven sites, the same afternoon:

SitecurlFresh agent-browserClaude in ChromeExa, 27 October
Upwork job search403 challenge”Just a moment…”Signed in, job cards, two pagesTen job cards
ChatGPT403”Just a moment…”Signed inPublic landing page
Skool classroomBounced to public pageBounced to public pageSigned in, course listPublic preview, courses locked
Kasta search403”Just a moment…“24 of 24 listings with pricesListings with prices
Rozetka search403”Just a moment…”Past the wall; results on some pages onlyEmpty
Instagram keyword searchLogin redirectLogin redirectSigned in, first screen of postsNothing
Product HuntReadable”Just a moment…”ReadableReadable
Higgsfield studioNavigation onlySigned outSigned in, full staging flowNothing
RedditEmpty shellBlock pageRefused by the extensionEmpty
Patreon403”Just a moment…”Refused by the extensionCreator page readable
Etsy search403DataDome captchaRefused by the extensionNothing; single listings from a stored copy

One or two loads per site, ses­sions warm from my ordi­nary brows­ing: a field note, not a benchmark. The Exa column comes from a re-run on 27 October, on the same kind of page; by then Kasta and Patreon let plain curl through as well. Walls move.

Before the first run:

On Higgsfield’s studio, which I drive for models its API doesn’t offer, the exten­sion picked the model, uploaded a still and read the price off the Generate button. The Generate click stays on rung 5.

Why this rung passes when rung 4 doesn’t, as far as I can tell. Two weeks ago I noted that a browser driven over CDP, the Chrome DevTools Protocol, gets detected before a chal­lenge renders. The exten­sion attaches Chrome’s debug­ger, which speaks CDP, on every nav­i­ga­tion, and Cloudflare let it through on five sites. So CDP isn’t the tell, or not the only one. I sus­pect the walls grade the browser and its his­tory: months of cook­ies, a match­ing fin­ger­print, a record of pass­ing chal­lenges as me. A fresh ses­sion has none of that, and a copied pro­file only until its clear­ance cookie expires.

Rung 7: Apple Events, for what the extension refuses

On a Mac, Chrome will run JavaScript sent by a local script once you allow it: View → Developer → Allow JavaScript from Apple Events. Then one line reads the page you’re on:

Terminal
osascript -e 'tell application "Google Chrome" to execute active tab of front window javascript "document.title"'

Set the URL, wait, read the DOM, repeat. It took me through more than a hun­dred Upwork search pages in September with zero chal­lenges, and it’s now my route for Reddit and Patreon. The cost: while the set­ting is on, any local process with automa­tion rights can run JavaScript in your tabs, so switch it off when you’re done. And it needs you at the Mac.

Where the ladder runs out

Changes since publication