Blueprints BP-05
High-concurrency crawl and freshness pipeline
Models need current data from many changing external sources.
01 / Fit
Use it when
- Models or retrieval depend on data held by many external sources.
- Those sources change at different rates and offer no reliable feed.
- Stale or silently wrong data would mislead a decision downstream.
- There is engineering time to maintain parsers as sources change.
Avoid it when
- The source offers a licensed feed or an API that covers the need. Use that first.
- The source’s terms do not permit automated access.
- The data changes rarely and a periodic manual export is enough.
02 / The diagram
Run it. Then run the failure path.
Select a node to see what it does, how it fails and what we measure.
- Schedule (Crawl plan). Decides which source to crawl next, from its change rate, its priority and its freshness lag. Fails when: Every source is crawled at the same interval, so fast-changing ones go stale. Safeguard: Per-source intervals that adapt to how often the content changes.
- Fetch (Rate-limited). Fetches pages and API responses in parallel, within each source’s rate limits and access terms. Fails when: A source throttles, blocks or changes its response format. Safeguard: Per-source concurrency limits, backoff and a circuit breaker. A blocked source is paused, not retried harder.
- Parse (Typed records). Turns raw responses into typed records and keeps the raw response beside them. Fails when: The page layout changes and the parser returns empty or shifted fields. Safeguard: Parsers are versioned per source and tested against saved pages.
- Validate (Schema, drift). Checks each batch against the schema and against the usual shape of that source. Fails when: Records pass the schema and are still wrong: a value in the wrong field, or half the usual volume. Safeguard: Drift checks on volume, fill rate and value ranges. A failing batch is held, not published.
- Diff (Changes only). Compares new records with the last published version and keeps what changed. Fails when: Unstable fields make every record look new. Safeguard: Records are fingerprinted on stable fields only.
- Publish (Fresh records). Writes changed records to the store that models and retrieval read, with crawl time and source. Fails when: A consumer reads a half-written batch. Safeguard: Batches are published atomically. Every record carries its crawl time.
- Freshness (Lag per source). Tracks how old the published data is for every source and feeds that back to the schedule. Fails when: The average looks healthy while a few important sources are far behind. Safeguard: Lag is tracked and alerted per source, not as one average.
- Quarantine (Hold the batch). Holds the failing batch with its raw responses. The last good data stays live and is marked with its age. Fails when: Stale data stays live longer than anyone realises. Safeguard: An alert on the age of the last good batch, per source.
- Engineer (Fix the parser). An engineer sees the failing check beside the raw page, fixes the parser and replays the batch. Fails when: The same source breaks again soon after. Safeguard: Every fix adds the page that broke to the parser tests.
Read the diagram as text
- Schedule (Crawl plan). Decides which source to crawl next, from its change rate, its priority and its freshness lag. Fails when: Every source is crawled at the same interval, so fast-changing ones go stale. Safeguard: Per-source intervals that adapt to how often the content changes.
- Fetch (Rate-limited). Fetches pages and API responses in parallel, within each source’s rate limits and access terms. Fails when: A source throttles, blocks or changes its response format. Safeguard: Per-source concurrency limits, backoff and a circuit breaker. A blocked source is paused, not retried harder.
- Parse (Typed records). Turns raw responses into typed records and keeps the raw response beside them. Fails when: The page layout changes and the parser returns empty or shifted fields. Safeguard: Parsers are versioned per source and tested against saved pages.
- Validate (Schema, drift). Checks each batch against the schema and against the usual shape of that source. Fails when: Records pass the schema and are still wrong: a value in the wrong field, or half the usual volume. Safeguard: Drift checks on volume, fill rate and value ranges. A failing batch is held, not published.
- Diff (Changes only). Compares new records with the last published version and keeps what changed. Fails when: Unstable fields make every record look new. Safeguard: Records are fingerprinted on stable fields only.
- Publish (Fresh records). Writes changed records to the store that models and retrieval read, with crawl time and source. Fails when: A consumer reads a half-written batch. Safeguard: Batches are published atomically. Every record carries its crawl time.
- Freshness (Lag per source). Tracks how old the published data is for every source and feeds that back to the schedule. Fails when: The average looks healthy while a few important sources are far behind. Safeguard: Lag is tracked and alerted per source, not as one average.
- Quarantine (Hold the batch). Holds the failing batch with its raw responses. The last good data stays live and is marked with its age. Fails when: Stale data stays live longer than anyone realises. Safeguard: An alert on the age of the last good batch, per source.
- Engineer (Fix the parser). An engineer sees the failing check beside the raw page, fixes the parser and replays the batch. Fails when: The same source breaks again soon after. Safeguard: Every fix adds the page that broke to the parser tests.
03 / Components
Every component, and how it fails.
| Component | Job | Typical choices | Fails when | Safeguard |
|---|---|---|---|---|
| Schedule | Decides which source to crawl next, from its change rate, its priority and its freshness lag. | Priority scheduler with per-source intervals and a source registry | Every source is crawled at the same interval, so fast-changing ones go stale. | Per-source intervals that adapt to how often the content changes. |
| Fetch | Fetches pages and API responses in parallel, within each source’s rate limits and access terms. | Distributed fetch workers with per-source rate limiting, backoff and circuit breakers | A source throttles, blocks or changes its response format. | Per-source concurrency limits, backoff and a circuit breaker. A blocked source is paused, not retried harder. |
| Parse | Turns raw responses into typed records and keeps the raw response beside them. | Versioned per-source parsers with saved-page tests, and raw response storage | The page layout changes and the parser returns empty or shifted fields. | Parsers are versioned per source and tested against saved pages. |
| Validate | Checks each batch against the schema and against the usual shape of that source. | Schema validation and statistical drift checks per source | Records pass the schema and are still wrong: a value in the wrong field, or half the usual volume. | Drift checks on volume, fill rate and value ranges. A failing batch is held, not published. |
| Diff | Compares new records with the last published version and keeps what changed. | Content fingerprinting and change detection on stable fields | Unstable fields make every record look new. | Records are fingerprinted on stable fields only. |
| Publish | Writes changed records to the store that models and retrieval read, with crawl time and source. | Atomic batch writes to the serving store, with crawl timestamps | A consumer reads a half-written batch. | Batches are published atomically. Every record carries its crawl time. |
| Freshness | Tracks how old the published data is for every source and feeds that back to the schedule. | Per-source freshness metrics and alerting | The average looks healthy while a few important sources are far behind. | Lag is tracked and alerted per source, not as one average. |
| Quarantine | Holds the failing batch with its raw responses. The last good data stays live and is marked with its age. | Held-batch store with raw responses and the failing check | Stale data stays live longer than anyone realises. | An alert on the age of the last good batch, per source. |
| Engineer | An engineer sees the failing check beside the raw page, fixes the parser and replays the batch. | On-call queue with parser test tooling and batch replay | The same source breaks again soon after. | Every fix adds the page that broke to the parser tests. |
04 / Budgets and evals
What the design is tuned for.
Freshness lag, success rate per source. These are design targets, not results.
| Stage | Design target |
|---|---|
| Schedule | Crawl capacity goes to the sources that change most and matter most. |
| Fetch | Stays within each source’s limits. Politeness comes before throughput. |
| Validate | Rule and statistics based, so it runs on every batch without model cost. |
| Publish | Short time from a change at the source to a record a model can read. |
Eval plan 01
Golden set
Saved raw pages and responses per source with the records they should produce. Pages that broke a parser in the past are kept as permanent cases.
Eval plan 02
Release gates
Parse success and field fill rate per source on the saved pages. Drift checks must flag the known-bad batches and pass the known-good ones.
Eval plan 03
Catching regressions
Parser changes replay the saved pages before release. In production, per-source volume, fill rate and freshness lag are compared with their own history.
05 / Design notes
Where the whiteboard and production differ.
- Sources change their layout without notice, and the usual symptom is empty or shifted fields, not an error.
- A successful response is not a successful crawl. Block pages and placeholders are returned with a success status.
- Freshness averaged across sources looks healthy long after the sources that matter have fallen behind.
- Most sources change far less often than they are crawled. Change-rate scheduling frees more capacity than adding workers.
- Keeping the raw response beside the parsed record makes every parser fix replayable.
06 / Deploy
Deploy this with us.
A pod can take BP-05 from this page to your stack. Tell us the workflow and we will tell you what changes for your systems.
You will talk to an engineer