Spectrum Data Research Pipeline: Multi-Source News Aggregation & Claude Analysis

The Problem

A large organization needed daily intelligence on regulatory and market developments spanning 60+ countries and dozens of topics. At that scale, manual research just isn't an option — you'd need a team the size of a newsroom to keep up. What they actually needed was automated discovery from regulatory bodies, trade press, and news aggregators, consistent structure applied to whatever came back, a way to surface the stories that actually mattered rather than raw search noise, ingestion that could scale across multiple search providers, and enough discipline about dates and sources that credibility didn't get lost along the way.

The goal, then: a research pipeline that runs on its own, fetches content at scale, and enriches everything it finds with AI-driven analysis.

---

What the Pipeline Does

At a high level, a search phase kicks things off by issuing 200+ parameterized searches across countries and topics, using whichever search provider is configured — Perplexity, SerpAPI, or Tavily. A fetch phase then downloads URLs in parallel, ranks the extracted content by relevance with BM25, and caches PDFs locally so they don't get re-downloaded. Running alongside that, an enrichment phase proactively fetches known regulatory sources — regulator websites, government portals, operator newsrooms — and injects them as high-signal results before Claude ever sees the data. Claude then analyzes the ranked results and structures findings using rule-based tiering (primary/secondary/low-relevance). Report generation formats everything as markdown and CSV, grouped by country and topic and ordered by source tier, and a backup-and-resume layer saves search results and Claude's progress along the way so re-runs and mid-pipeline restarts don't cost much.

Retry logic, token counting, and error tracking run underneath all of it, end to end.

---

Tech Stack

---

The Journey

Searching at scale without hitting a wall

The naive approach — one API call per search term — holds up fine for 10 terms and falls apart at 200+. Getting past that meant a few things working together: fan-out with rate limiting so searches spread across countries and topics without tripping API limits (delay_between_searches handles the pacing), a provider abstraction so Perplexity, SerpAPI, and Tavily are swappable without touching the orchestrator, automatic fallback queries when a search comes back thin (below MIN_RESULTS triggers a retry with a broader term — "SMP regulation" becomes "market dominance," then "competition"), and backups saved between stages so a restart can skip whatever's already done.

Early runs blew through rate limits anyway. The fix was reading the API tier straight from config and auto-adjusting max_search_calls so the pipeline stays within whatever the daily limit actually is.

Fetching in parallel without getting blocked

Fetching 200+ search results one at a time took hours — not a surprise, but still too slow to be useful. Parallelizing it was necessary, but risky in its own way: hammer a server with too many concurrent requests and you get rate-limited or outright blocked.

I ended up with a thread pool fetcher built around a few safeguards: URL deduplication via a hash-based cache so nothing gets fetched twice across search items, concurrency tuned per environment (8 workers locally, adaptive in the cloud), content extraction split between BeautifulSoup for HTML and pdfplumber for PDFs with pypdf as a fallback, a compressed local PDF cache so re-runs don't re-download anything, and configurable timeouts with retry-and-backoff for anything flaky.

The one gotcha that actually bit me: PDFs with corrupted metadata would crash pypdf outright. I added quality flags — has_corrupted_pdf, oversized, and similar — that feed into how Claude treats that document downstream, instead of letting a bad PDF take down the whole run.

Ranking pages with BM25 once the content's in hand

Search results give you snippets, not full content, so the real text gets fetched separately — but once you have a full page, which parts of it actually matter? That's where per-URL BM25 ranking comes in: keywords get scored in three tiers (primary, secondary, tertiary, each weighted differently), the keyword sets themselves are topic-aware and pulled from config, regex boost patterns add extra signal for things like currency amounts and regulator names, and each URL's pages get ranked with only the top 3 surviving.

The end result is that Claude sees the 3 most relevant pages from a URL, not whatever raw search happened to return.

Making sure regulators don't lose to news aggregators

Search results skew toward news aggregators almost by default. Regulators and operator newsrooms are the official sources, but they tend to rank lower — search engines just aren't optimized for "who actually made this decision."

So I added a proactive fetch path on top of search: a config-driven mapping of known regulatory and operator websites to their country scope, topic matching that connects a search topic to the right sources (searching "SMP regulations" pulls in TRAI, PTA, BTRC directly), injection of those matched sources as high-signal results, and index expansion so list-page URLs like /pages/notices get parsed out into individual article links instead of being treated as one page.

That's what keeps official sources in the mix even on days when news aggregators are crowding out everything else in raw search.

Setting clear rules before Claude sees anything

Going from raw search results straight to Claude extraction without any structure in between gets messy fast. So I built a rule-based extraction schema first:

Relevance hierarchy:

• PRIMARY — Spectrum auctions, operator licensing, regulator enforcement (always CONFIRMED if in date range)
• SECONDARY — Broadband policy, cybersecurity rules (CONFIRMED only if mobile operator named)
• LOW RELEVANCE — Fixed ISP licensing, admin notices (always MIGHT_BE_IMPORTANT)

Date handling:

• CONFIRMED — Event within the reporting period
• CONTEXT — Event before period but had in-period consequences
• MIGHT_BE_IMPORTANT — Unclear date or indirect relevance
• BORDERLINE — Source URL is generic index with no article slug

Claude receives this as XML in the prompt, then outputs structured entries with tier, date, country, and citations.

Turning a flat list of 500 entries into something a stakeholder can use

Raw Claude output is just a flat list of 500+ entries, which is exactly as unhelpful to read as it sounds. So I grouped everything into a three-level hierarchy: country first (India, the EU, wherever), then topic or bucket (SMP, spectrum, M&A), then source tier, ordered by credibility so regulatory sources lead, news follows, and trade press comes last.

Within each tier, entries get deduplicated — same URL and headline collapses to one entry — and sorted newest-first by date.

Making failures cheap to recover from

Full runs took 4 to 6 hours, and any failure along the way — an API timeout, a malformed PDF, a Claude error — used to mean starting the whole thing over. That got old fast, so I added two levels of checkpoints:

• Search backup — After every 5 searches, save SearchItem objects (pickle + JSON). On restart, load and skip completed items.
• Claude progress — After every 5 Claude calls, save processed_ids and results. On restart, skip finished searches and resume.

A run that fails at 80% completion now resumes in about 20 minutes instead of 4 hours.

The same path bug, again

The config loader used Path("config.toml").parent to find the project root, then built the .env path as parent / ".env". Relative config paths broke this — .parent just returns ., the current directory, not the actual project root — and .env ended up getting looked up in the wrong place, which broke AWS Bedrock auth.

The fix, same as elsewhere: Path(config_path).resolve().parent, which converts the relative path to absolute before pulling out the parent.

---

Deployment & Operations

The pipeline runs both on-demand and nightly. On-demand runs get triggered by stakeholder requests when someone needs a rapid re-run or a topic update. The nightly cron job pulls the latest news, updates the report, and emails it out to stakeholders. Configuration is TOML plus .env, validated before any API calls go out — better to fail fast on a bad config than burn API quota on a run that was doomed from the start.

What I actually track: search success rate (the percentage returning more than MIN_RESULTS), fetch efficiency (URLs returning content versus timing out or 404ing), Claude token usage per country, and extraction yield per country.

---

What I'd Do Differently

Streaming extraction. Claude calls are batched right now. Real streaming with stream=True would show progress as it happens and let the pipeline stop early once it hits quota, instead of finding out after the fact.

Provider auto-selection. The provider gets configured upfront today. A smarter version would pick based on topic — news topics to SerpAPI, research topics to Perplexity — or fall back automatically if a provider starts failing.

Distributed fetching. ThreadPoolExecutor only scales to one machine. Past 10,000 URLs, distributed workers — Lambda, a task queue, something like that — would be both faster and cleaner.

Real-time indexing. Search results get re-fetched daily right now, in full. Incremental indexing that only touches new URLs would cut bandwidth and speed up every re-run.