Skip to content

Scheduled web collection

Agent scraper workflows that run every day without surprises

An agent scraper is a pipeline: public URLs in, clean Markdown out, your own AI step after, in n8n, Make, Zapier or cron. It is the collection half, not a point-and-click extractor.

See the no-code recipesRequest access

Only Business, Enterprise and Custom sign-ups are reviewed before checkout. Failed fetches cost 0 credits.

Route on the per-URL status
{
  "results": [
    { "id": "https://example.com/changelog", "title": "Changelog",
      "markdown": "# Changelog\n\n## 2.4.0\n..." }
  ],
  "statuses": [
    { "id": "https://example.com/changelog", "status": "success",
      "source": "livecrawled", "credits": 1, "error": null },
    { "id": "https://example.com/app", "status": "error",
      "source": null, "credits": 0,
      "error": { "tag": "CRAWL_NOT_RENDERED", "message": "..." } }
  ]
}

Where runs break

Why most AI scraper jobs fail quietly

The failures are rarely dramatic. A job looks green in the run history while the output slowly fills with gaps.

  • The loop stops at item 40

    A 500-row list runs into an execution time limit or a 429 from the API. The run ends, the Sheet looks half done, and nothing tells you which rows were skipped.

  • Green runs with empty pages

    Many fetch APIs report a failed page inside a 200 response. If the workflow branches on the HTTP code alone, error pages and blank bodies land in your results as if they were content.

  • One black box, two bills

    An all-in-one AI scraper fetches and prompts in a single step. When the output is wrong you cannot tell which half failed, and the fetch cost and the model cost arrive as one number.

  • Nobody read the terms

    The job was scheduled before anyone checked robots.txt, the site's terms or whether the pages hold personal data. That question gets harder to ask once the workflow is in production.

How it works

How an AI scraper for automation works, step by step

Four plain steps. Each one has a single job, so each one can fail visibly and be fixed on its own.

  1. Step 1: Keep the list somewhere boring

    Put public URLs in a Google Sheet, a CSV file or an Airtable base, one row per URL, with an empty status column.

  2. Step 2: Fetch at your rate limit

    Send one URL per item and pace items to your plan's rate limit, which counts URLs, not requests. Store fetchedAt per URL so repeat runs skip pages that are still fresh.

  3. Step 3: Route on the per-URL status

    Each URL gets status success or error with a tag. Retry 429, 5xx and timeouts up to 3 times; retry nothing else in the same run; stop on 402.

  4. Step 4: Hand clean text to your model

    Send the Markdown to your own LLM to summarize, label or extract fields, then write one JSONL line or one row per page.

On the line

What an AI web scraper pipeline is good for

All of these run on public pages and a list you already have. None of them needs a crawler or a browser.

  • Daily page summaries in a Sheet

    An n8n Schedule Trigger reads URLs from Google Sheets, fetches each page and writes a short model summary and a status back to the same row.

    Read the guide

  • Make or Zapier scenarios

    Iterate over a list with the generic HTTP step, filter on the status field and send the clean text on to the next module.

    Read the guide

  • Nightly JSONL batches

    Read a CSV of a few thousand URLs, fetch with retries by error type and write one JSON line per page for an internal search index.

    Read the guide

  • Change detection on public pages

    Re-collect your own docs, public changelogs or public price pages on a schedule, within each site's robots.txt, terms and the AUP, and alert when a page changes.

    Read the guide

  • LLM labels after the fetch

    Classify each collected page as relevant or not, or pull a few fields with a JSON schema prompt, in a model step you control.

    Read the guide

  • A compliance check before launch

    Add a robots.txt check step to the job and read each site's terms and personal data questions before the first scheduled run. Not legal advice.

    Read the post

Copy and paste

An AI scraper pipeline starts with one HTTP step

The same request works in n8n, Make, Zapier or a cron job. Only the field names around it change, and the retry rules stay the same.

Node: HTTP Request
Method: POST
URL: https://livewebdata.com/api/v1/contents
Authentication: Generic Credential Type > Bearer Auth
Send Body: on
Body Content Type: JSON
Specify Body: Using JSON
JSON: {"urls": ["{{ $json.url }}"]}
Options > Batching:
  Items per Batch: your plan's rate limit (URLs/s)
  Batch Interval (ms): 1000

Next node: IF {{ $json.statuses[0].status }} is "success"

Native n8n, Make and Zapier integrations are announced as coming soon. Until then, use the generic HTTP step shown here.

Honest comparison

All-in-one AI scraper or plain fetch plus your model?

Three common ways to turn a list of pages into AI-ready output, and where each one is the better fit.

Fetch + your AI step

Cost visibility
Fetch credits and model tokens billed apart
Failure visibility
Per-URL status and error tag
Swap the model
Any model, any prompt
Fixed list, every day
Good fit: plain loop at a known rate
Structured fields
Your LLM step, your JSON schema
JavaScript-heavy pages
Rendered on request (render or renderFallback)
Finding new URLs
No: bring your own list

All-in-one AI scraper

Cost visibility
One combined price per page or run
Failure visibility
Varies by tool
Swap the model
Usually the vendor's choice
Fixed list, every day
Good fit if the fields are standard
Structured fields
Built in
JavaScript-heavy pages
Often rendered
Finding new URLs
Some tools crawl

Agent re-navigating a list

Cost visibility
Model tokens for every step the agent takes
Failure visibility
Buried in the agent trace
Swap the model
Any model, but prompts drive navigation
Fixed list, every day
Slow and costly for the same pages
Structured fields
Built in through prompting
JavaScript-heavy pages
Rendered in a real browser
Finding new URLs
Yes, by navigating

AgentScraper is run by Teneo Labs, which makes Live Web Data API (the first column). The other columns describe tool categories, not named products, checked in October 2026 against public documentation of those categories.

Guides

Long-form guides for every part of the pipeline

Each guide covers one stage end to end, with settings taken from the tools' own documentation.

All guides

FAQ

Questions about agent scrapers and AI scrapers

What is an agent scraper?

Usually it means an LLM agent that opens pages, decides where to go next and extracts what it finds. That fits pages you do not know in advance. For a fixed list of public URLs collected on a schedule, a pipeline is cheaper and easier to debug: fetch each page as clean Markdown at a known rate, route failures where you can see them, then run your own model on the text. This site covers that pipeline.

Can AI scrape websites by itself?

A language model does not fetch pages. Something has to download the page and hand the text to the model. AI scrapers combine those two jobs in one tool. Keeping them separate, a fetch step and then your own model, makes costs and failures easier to see and lets you change the model without rebuilding the workflow.

How many URLs can an agent scraper handle per run?

The pipeline has no fixed cap; your plan's rate limit and your credit balance set the pace. The rate limit counts URLs per second, and one request takes 1 to 10 URLs. At 2 URLs per second, 3,000 URLs take at least 25 minutes. Check GET /v1/account before a large run, and split long lists so each n8n, Make or Zapier execution stays short.

Does it work on JavaScript-heavy pages?

Yes, when you ask for it. Set render: true to read a page in a browser, or renderFallback: true to render only when the normal fetch is blocked or finds no visible text. Without it, such pages come back with a THIN_CONTENT warning or the error tag CRAWL_NOT_RENDERED (0 credits), so you can route them.

What happens when a run stops halfway through the list?

Nothing is lost if every URL gets its own output line or sheet row with a status. On the next run, skip the URLs that already succeeded or failed for good, and send only unfinished URLs and retryable errors. A 402 stops the job cleanly; top up, then resume from the same file.

What does a failed page cost?

Nothing. One credit is charged per page that comes back successfully. Timeouts, missing pages, unsupported content types and restricted targets cost 0 credits. See the pricing page for plans and how credits are counted.

Who can request access?

Sign-up is for businesses only. Only Business, Enterprise and Custom sign-ups are reviewed before checkout, usually within 1 business day. The review checks the intended use against the acceptable-use policy. On Lite, Starter and Developer you can check out right after you confirm your email. Once your plan is active, you create an API key in the dashboard.

Blog

Latest notes from the workshop

Short, sourced posts on one workflow problem each: a node setting, a retry rule, a schedule that drifted.

All posts

Build your first scheduled collection job

Sign up as a business. Once your plan is active, create an API key and start with the n8n, Make or Zapier recipe or the batch scripts for cron.

Request accessAutomate requests