Blueprints BP-05

High-concurrency crawl and freshness pipeline

Models need current data from many changing external sources.

Data pipelinesLogisticsSupply chainFinanceLast reviewed · November 2026

01 / Fit

Use it when

  • Models or retrieval depend on data held by many external sources.
  • Those sources change at different rates and offer no reliable feed.
  • Stale or silently wrong data would mislead a decision downstream.
  • There is engineering time to maintain parsers as sources change.

Avoid it when

  • The source offers a licensed feed or an API that covers the need. Use that first.
  • The source’s terms do not permit automated access.
  • The data changes rarely and a periodic manual export is enough.

02 / The diagram

Run it. Then run the failure path.

Select a node to see what it does, how it fails and what we measure.

Idle
drift detectedage of published datareorderSCHEDULECrawl planFETCHRate-limitedPARSETyped recordsVALIDATESchema, driftDIFFChanges onlyPUBLISHFresh recordsFRESHNESSLag per sourceQUARANTINEHold the batchENGINEERFix the parser
  1. Schedule (Crawl plan). Decides which source to crawl next, from its change rate, its priority and its freshness lag. Fails when: Every source is crawled at the same interval, so fast-changing ones go stale. Safeguard: Per-source intervals that adapt to how often the content changes.
  2. Fetch (Rate-limited). Fetches pages and API responses in parallel, within each source’s rate limits and access terms. Fails when: A source throttles, blocks or changes its response format. Safeguard: Per-source concurrency limits, backoff and a circuit breaker. A blocked source is paused, not retried harder.
  3. Parse (Typed records). Turns raw responses into typed records and keeps the raw response beside them. Fails when: The page layout changes and the parser returns empty or shifted fields. Safeguard: Parsers are versioned per source and tested against saved pages.
  4. Validate (Schema, drift). Checks each batch against the schema and against the usual shape of that source. Fails when: Records pass the schema and are still wrong: a value in the wrong field, or half the usual volume. Safeguard: Drift checks on volume, fill rate and value ranges. A failing batch is held, not published.
  5. Diff (Changes only). Compares new records with the last published version and keeps what changed. Fails when: Unstable fields make every record look new. Safeguard: Records are fingerprinted on stable fields only.
  6. Publish (Fresh records). Writes changed records to the store that models and retrieval read, with crawl time and source. Fails when: A consumer reads a half-written batch. Safeguard: Batches are published atomically. Every record carries its crawl time.
  7. Freshness (Lag per source). Tracks how old the published data is for every source and feeds that back to the schedule. Fails when: The average looks healthy while a few important sources are far behind. Safeguard: Lag is tracked and alerted per source, not as one average.
  8. Quarantine (Hold the batch). Holds the failing batch with its raw responses. The last good data stays live and is marked with its age. Fails when: Stale data stays live longer than anyone realises. Safeguard: An alert on the age of the last good batch, per source.
  9. Engineer (Fix the parser). An engineer sees the failing check beside the raw page, fixes the parser and replays the batch. Fails when: The same source breaks again soon after. Safeguard: Every fix adds the page that broke to the parser tests.
Tuned for · Freshness lag, success rate per sourceReference design
DataGuardHumanActive pathFallback path
Read the diagram as text
  1. Schedule (Crawl plan). Decides which source to crawl next, from its change rate, its priority and its freshness lag. Fails when: Every source is crawled at the same interval, so fast-changing ones go stale. Safeguard: Per-source intervals that adapt to how often the content changes.
  2. Fetch (Rate-limited). Fetches pages and API responses in parallel, within each source’s rate limits and access terms. Fails when: A source throttles, blocks or changes its response format. Safeguard: Per-source concurrency limits, backoff and a circuit breaker. A blocked source is paused, not retried harder.
  3. Parse (Typed records). Turns raw responses into typed records and keeps the raw response beside them. Fails when: The page layout changes and the parser returns empty or shifted fields. Safeguard: Parsers are versioned per source and tested against saved pages.
  4. Validate (Schema, drift). Checks each batch against the schema and against the usual shape of that source. Fails when: Records pass the schema and are still wrong: a value in the wrong field, or half the usual volume. Safeguard: Drift checks on volume, fill rate and value ranges. A failing batch is held, not published.
  5. Diff (Changes only). Compares new records with the last published version and keeps what changed. Fails when: Unstable fields make every record look new. Safeguard: Records are fingerprinted on stable fields only.
  6. Publish (Fresh records). Writes changed records to the store that models and retrieval read, with crawl time and source. Fails when: A consumer reads a half-written batch. Safeguard: Batches are published atomically. Every record carries its crawl time.
  7. Freshness (Lag per source). Tracks how old the published data is for every source and feeds that back to the schedule. Fails when: The average looks healthy while a few important sources are far behind. Safeguard: Lag is tracked and alerted per source, not as one average.
  8. Quarantine (Hold the batch). Holds the failing batch with its raw responses. The last good data stays live and is marked with its age. Fails when: Stale data stays live longer than anyone realises. Safeguard: An alert on the age of the last good batch, per source.
  9. Engineer (Fix the parser). An engineer sees the failing check beside the raw page, fixes the parser and replays the batch. Fails when: The same source breaks again soon after. Safeguard: Every fix adds the page that broke to the parser tests.

03 / Components

Every component, and how it fails.

ComponentJobTypical choicesFails whenSafeguard
ScheduleDecides which source to crawl next, from its change rate, its priority and its freshness lag.Priority scheduler with per-source intervals and a source registryEvery source is crawled at the same interval, so fast-changing ones go stale.Per-source intervals that adapt to how often the content changes.
FetchFetches pages and API responses in parallel, within each source’s rate limits and access terms.Distributed fetch workers with per-source rate limiting, backoff and circuit breakersA source throttles, blocks or changes its response format.Per-source concurrency limits, backoff and a circuit breaker. A blocked source is paused, not retried harder.
ParseTurns raw responses into typed records and keeps the raw response beside them.Versioned per-source parsers with saved-page tests, and raw response storageThe page layout changes and the parser returns empty or shifted fields.Parsers are versioned per source and tested against saved pages.
ValidateChecks each batch against the schema and against the usual shape of that source.Schema validation and statistical drift checks per sourceRecords pass the schema and are still wrong: a value in the wrong field, or half the usual volume.Drift checks on volume, fill rate and value ranges. A failing batch is held, not published.
DiffCompares new records with the last published version and keeps what changed.Content fingerprinting and change detection on stable fieldsUnstable fields make every record look new.Records are fingerprinted on stable fields only.
PublishWrites changed records to the store that models and retrieval read, with crawl time and source.Atomic batch writes to the serving store, with crawl timestampsA consumer reads a half-written batch.Batches are published atomically. Every record carries its crawl time.
FreshnessTracks how old the published data is for every source and feeds that back to the schedule.Per-source freshness metrics and alertingThe average looks healthy while a few important sources are far behind.Lag is tracked and alerted per source, not as one average.
QuarantineHolds the failing batch with its raw responses. The last good data stays live and is marked with its age.Held-batch store with raw responses and the failing checkStale data stays live longer than anyone realises.An alert on the age of the last good batch, per source.
EngineerAn engineer sees the failing check beside the raw page, fixes the parser and replays the batch.On-call queue with parser test tooling and batch replayThe same source breaks again soon after.Every fix adds the page that broke to the parser tests.

04 / Budgets and evals

What the design is tuned for.

Freshness lag, success rate per source. These are design targets, not results.

StageDesign target
ScheduleCrawl capacity goes to the sources that change most and matter most.
FetchStays within each source’s limits. Politeness comes before throughput.
ValidateRule and statistics based, so it runs on every batch without model cost.
PublishShort time from a change at the source to a record a model can read.

Eval plan 01

Golden set

Saved raw pages and responses per source with the records they should produce. Pages that broke a parser in the past are kept as permanent cases.

Eval plan 02

Release gates

Parse success and field fill rate per source on the saved pages. Drift checks must flag the known-bad batches and pass the known-good ones.

Eval plan 03

Catching regressions

Parser changes replay the saved pages before release. In production, per-source volume, fill rate and freshness lag are compared with their own history.

05 / Design notes

Where the whiteboard and production differ.

  1. Sources change their layout without notice, and the usual symptom is empty or shifted fields, not an error.
  2. A successful response is not a successful crawl. Block pages and placeholders are returned with a success status.
  3. Freshness averaged across sources looks healthy long after the sources that matter have fallen behind.
  4. Most sources change far less often than they are crawled. Change-rate scheduling frees more capacity than adding workers.
  5. Keeping the raw response beside the parsed record makes every parser fix replayable.

06 / Deploy

Deploy this with us.

A pod can take BP-05 from this page to your stack. Tell us the workflow and we will tell you what changes for your systems.

You will talk to an engineer

Book a scoping call

Start with one workflow.

Tell us where the work piles up. You will talk to an engineer, not a salesperson.

What do you need?

After the call you get a one-page scoping note: what we would build first, how we would measure it, and what it takes.