Home / Guides / Scrape Sites That Change

Web Scraping Sites That Keep Changing Their HTML

Last updated: July 17, 2026

Every scraper author knows the cycle: the target site ships a redesign, your CSS paths die, and you spend an evening reading minified markup to find where the price moved. You can't stop sites from changing, but you can build scrapers where a redesign is a log line instead of an outage. Three strategies, in order of preference.

Strategy 1: skip the HTML, intercept the API

Most modern sites are client-rendered: the HTML you fight with is just a projection of a JSON API the frontend calls anyway. Instead of parsing the projection, capture the source. Run the page in a real browser and use Chrome DevTools Protocol network interception to grab the site's own API responses as they happen.

The payoff is durability: frontend redesigns churn constantly, but the internal API contract changes rarely, because the site's own frontend depends on it. LumaBrowser exposes this as a network watcher, you declare a URL pattern and a webhook, and matched responses are forwarded to your endpoint with no proxy setup or root certificates:

POST http://localhost:3000/api/watchers
{
  "urlPattern": "*/api/products*",
  "sendTo": "https://your-backend.com/webhook"
}

Strategy 2: write selectors that age well

When you do need the DOM, most selector rot is self-inflicted. Selectors copied from DevTools' "Copy selector" are maximally brittle: deep positional paths through generated class names. Prefer, in order:

• Semantic attributes: [itemprop=price], [data-product-id], ARIA roles. Sites keep these for their own SEO and accessibility.
• Stable anchors + short hops: find the one element with a reliable id, then navigate one or two steps, not ten.
• Text landmarks: "the cell after the header containing 'Price'". Copy changes less often than markup.
• Never generated class names: anything like .css-8xk2m9 or a ten-segment Tailwind chain is a countdown timer, one deploy from breaking.

Strategy 3: let an LLM resolve what breaks anyway

Good selectors reduce breakage; nothing eliminates it. The remaining failures share a shape: the element is still on the page, still obvious to a human, just addressed differently. That's exactly the gap an LLM fallback fills. Alongside each strict selector you keep a plain-English description; extraction tries the selector first at normal speed, and only on a miss does the LLM read the rendered page and resolve the description to the new element.

The run continues, the fallback is logged, and you update the strict path on your own schedule, a log line instead of a panicked debug session. LumaBrowser builds this into its scraping API, callable from Python, Node, Go, or curl, and the same mechanism serves Puppeteer and Playwright scripts attached over CDP.

Putting it together

1. Open DevTools' Network tab on your target page. If the data arrives as JSON, intercept the API and skip DOM parsing entirely.
2. Where you must parse the DOM, write semantic, anchor-based selectors, never generated class names.
3. Give every selector a plain-English fallback description, and monitor the audit log for fallbacks instead of waiting for crashes.
4. For pages you only need to watch rather than scrape, a change monitor with webhooks is cheaper than a polling scraper.