A signals pipeline has one job: surface the few events that matter from the many that happen. The failure mode is always the same. The pipeline works in the demo, then production traffic arrives and the output becomes a firehose nobody reads. Once a user learns to ignore your notifications, no ranking improvement wins them back.
Precision is the product
Recall feels like the goal because missed events are visible in retros. But in practice, users forgive a missed signal and never forgive a noisy feed. We set an explicit precision budget: if fewer than roughly seven of ten surfaced events are genuinely useful, the feature is failing regardless of what else it catches.
The pipeline's real output isn't events. It's the user's continued attention.
Score at the edge, filter at the center
Our first version filtered late: scrape everything, enrich everything, then decide relevance at the end. That meant paying enrichment costs (LLM calls, entity resolution, embedding) on garbage. The fix was a cheap relevance gate immediately after ingestion: a small classifier plus hard rules that discard the obvious noise before anything expensive runs. Two thirds of raw events die at the gate, and the expensive stages run on the survivors.
Enrichment is where trust is won
An event without context is homework for the user. "Company X raised a Series B" is noise; "your client's biggest competitor raised a Series B two weeks before your renewal conversation" is a signal. The enrichment stage joins events against what we know about the user's world: their accounts, their contacts, their timing. This is also where LLMs earn their keep, not in detection but in explanation.
Make degradation boring
Scrapers break constantly; that's their nature. The design question is what happens when they do. Our rules: every source has a freshness SLO, stale sources are dropped from scoring rather than served stale, and a broken source pages nobody at 3 a.m. because the pipeline treats missing data as normal weather, not an outage. Alerting only fires when total coverage drops below a floor.
What I'd tell past me
- Define the precision budget before writing the first scraper.
- Put the cheapest possible filter as early as possible.
- Spend LLM tokens on explaining events, not finding them.
- Design for sources being broken, because some always are.
- Instrument "did the user act on this?" from day one. It's the only metric that matters.