Regulatory News Extraction Pipeline: Multi-Country Intelligence & Claude-Driven Categorization

The Problem

A telecom industry organization needed daily intelligence on regulatory changes, M&A, and infrastructure developments across 60+ countries. A manual research team simply couldn't keep up at that scale — there's only so much a person can read before the next day's news arrives. What they actually needed was automated discovery of regulatory announcements, consistent classification into topics that mattered (SMP regulations, spectrum auctions, M&A, and so on), some notion of source credibility so official regulator sites outrank news aggregators and trade press, daily reports organized by country and topic, and a compact index that highlighted the priority sources at a glance.

So the job became: build something that autonomously discovers regulatory news, categorizes it accurately, and turns that into structured intelligence every single day.

What the Pipeline Does

At a high level, it runs through six phases. A search phase issues 200+ searches across countries and themes — SMP, spectrum, M&A, and the rest — using SerpAPI. A fetch phase downloads URLs in parallel and uses BM25 ranking to surface the most relevant pages. Alongside that, a proactive regulatory fetch injects content from known government, regulator, and operator sites, because search alone tends to bury the official sources under news coverage. Claude then extracts from the ranked results using rule-based tiering (PRIMARY/SECONDARY/LOW_RELEVANCE) plus a set of custom topic categories. Finally, report generation formats everything as markdown with a country → topic → source-tier hierarchy, and a separate index generation step builds a compact summary with emoji-highlighted sources (🏛️ regulatory, 📰 news, 🔗 other).

The whole thing is resumable. If it fails at 80% through a run, the next run skips whatever's already done and picks up from checkpoints instead of starting over.

Tech Stack

The Journey

Searching 60 countries without drowning in noise

Searching for "regulatory news" across 60 countries and 9 themes isn't something you can just throw at an API and walk away from — it needs real orchestration. Each theme (SMP, spectrum, M&A, and so on) gets its own search per country, which adds up to 500+ queries. Those run sequentially with rate limiting to stay inside SerpAPI's quotas. SerpAPI's primary engine is Google News, but if a query comes back with fewer than 5 results, the pipeline auto-switches to google_ai_mode for ranking-enhanced results instead. Everything then gets consolidated — results from multiple themes per country get merged and deduplicated.

One thing I learned early: combining too many countries or themes into a single query dilutes the results badly. Ask for "SMP regulation in India AND Pakistan AND Nepal" and you get generic hits that don't serve any of the three countries well. The fix was to just accept the query volume and run one query per country per theme.

Fetching content and ranking it with BM25

Search results give you snippets, not full articles, so the real content has to be fetched separately. A ThreadPoolExecutor with 8 workers downloads the top 5 URLs per search in parallel. HTML gets cleaned up with BeautifulSoup and a few regex passes; PDFs go through pdfplumber first, falling back to pypdf when metadata is corrupted. Then BM25 scores each page by keyword density, and only the top 3 ranked pages per URL get sent to Claude — not everything that was fetched.

That last step matters more than it sounds like it should. Without it, Claude ends up reading a lot of boilerplate and navigation noise instead of the actual regulatory content.

Making sure official sources don't get buried

Search engines favor news aggregators over regulator websites almost by default, and for this use case that's backwards — official sources are the ones that matter most. So I added a separate, targeted fetch path: a CSV registry of known regulatory bodies, government portals, and operator newsrooms, mapped to countries and topics. If someone's searching for "SMP regulations" in India, the pipeline goes and fetches the TRAI website directly instead of hoping it shows up in search results. For list-page URLs like /notices or /press-releases, it parses out the individual article links before ranking anything. Then these sources get injected as top-ranked results, ahead of whatever search returned.

The result is that official sources show up consistently now, even on days when news aggregators are dominating the raw search rankings.

Giving Claude clear rules for relevance

Going straight from raw search results to Claude extraction gets messy fast without clear rules to anchor it. So I built a three-tier relevance framework:

PRIMARY (always CONFIRMED if in date range):

• Spectrum allocation, operator licensing, regulator enforcement
• Any decision explicitly naming a mobile operator

SECONDARY (CONFIRMED only if operator/regulator named):

• Broadband policy, cybersecurity rules, mobile money regulation
• Requires credible source + named entity

LOW RELEVANCE (always MIGHT_BE_IMPORTANT):

• ISP licensing (fixed-line only), admin notices, generic background
• Still included, but flagged for stakeholder review

Claude receives this as XML, along with 25 topic categories (SMP, spectrum, M&A, taxation, devices, partnerships, and more), and outputs structured entries with a headline, body, sources, and a tier (CONFIRMED/CONTEXT/MIGHT_BE_IMPORTANT/BORDERLINE), plus country, date, a source tier (TIER1/TIER2/TIER3/BORDERLINE), and a topic category — which can be comma-separated if the entry spans multiple topics.

Turning 800 flat entries into something readable

Raw extraction output is just a flat list of 800+ items, which isn't useful to anyone trying to actually read it. Stakeholders need structure, so entries get organized three levels deep: country first (India, Bangladesh, the EU), then topic (SMP Regulations, Spectrum Auctions), then source tier — Tier A for regulatory and government sources like BTRC, TRAI, and DoT, Tier B for news sources like Daily Star, Reuters, and the Economic Times, and Tier C for trade press like Developing Telecoms and Telecompaper.

Within each tier, entries sort newest-first by date. Topics themselves are ordered by canonical importance — SMP leads, then spectrum, then the rest follow.

Emoji badges for source credibility

The index report groups entries by publisher within each topic, and I wanted source credibility to be visible at a glance rather than something you had to infer from the name. So each entry gets an emoji badge: 🏛️ for an official regulatory source, 📰 for a quality national news outlet or industry aggregator (pulled from regulatory_sources.csv), and 🔗 for everything else — trade press, specialist sites. It's a small thing, but it makes the difference between official and secondary coverage instantly obvious in the markdown.

Letting one entry belong to more than one topic

The system originally assumed one topic per entry, which broke down the moment stakeholders wanted cross-tagging. An entry about "Bharti Airtel acquiring 5G spectrum in partnership with a government agency" is genuinely both an M&A story and a government-partnership story — forcing it into one bucket loses information.

The fix was to let Claude output comma-separated topics, like "Merger and Acquisitions, Government and Telecom Operator Partnerships," and have the parser split on the comma and duplicate the entry into each bucket it belongs to. That gives proper coverage across topics without actually duplicating any data.

Making a 4-hour run resumable

Full runs take 4 to 6 hours, and a failure partway through used to mean starting over from nothing — which got old fast. So I added checkpoints at three levels. After every search, results get appended to a pickle backup, and on restart the pipeline loads that and skips whatever's already done. After fetching all URLs, the enriched SearchItem objects — complete with fetched_content — get saved too, which means Claude extraction can be re-run later without re-fetching anything. And every 5 Claude calls, processed IDs and results get saved, so a restart skips whatever already finished.

The pattern I use most: load the enriched backup to skip search and fetch entirely, then re-run just the Claude extraction step with an updated prompt. That turns a 4-hour run into about 30 minutes.

When a rename broke every old backup

The enriched backup was created back when the source tree was called pipeline/. Somewhere along the way it got renamed to src/, and every old pickle stopped loading: No module named 'pipeline'.

The fix was a small shim in backup.py that maps the old module name to the new one before unpickling:

import sys
import src.models as _src_models
for _old in ("pipeline", "pipeline.models"):
    if _old not in sys.modules:
        sys.modules[_old] = _src_models

This lets pickle find old class definitions under the new path.

The config path resolution bug

Config loading used Path("config.toml").parent to find the project root. That works fine until config.toml is a relative path, which it is in production — .parent then just returns ., the current directory, not the actual project root. The practical effect was that .env got looked up in the wrong place, which quietly broke AWS authentication.

The fix was a one-liner: Path(config_path).resolve().parent converts the relative path to absolute before taking its parent, so it always resolves correctly regardless of where the process was launched from.

Adding the topics stakeholders actually needed

Early feedback flagged three missing topics, so I added Telecom Taxation Policy and Compliance (VAT/GST changes, spectrum taxes, digital service taxes), Smartphone and Telecom Device Regulations (device approval, SAR standards, import duties), and Government and Telecom Operator Partnerships (strategic infrastructure partnerships, broadband initiatives). All 25 topics now run through the same pipeline — ordered canonically, wired into both the extraction prompt and report generation.

Deployment & Operations

The pipeline runs both on-demand and nightly. A test mode covers a single country — Bangladesh — for validating new buckets or prompt changes before they touch production. Production mode runs all 60+ countries and generates the full report plus the index report. Resumes are controlled by a use_existing_backup config flag, which skips search and fetch and re-extracts from cached data instead.

What I actually watch day to day: source fetch success rate (what percentage of regulatory_sources.csv actually returned content), extraction yield per country and topic, Claude token usage, and the deduplication ratio — how many potential duplicates got caught and removed.

What I'd Do Differently

Distributed search. Running 500+ queries sequentially takes hours. Async search with concurrent requests would speed this up considerably, as long as it still respects rate limits.

Incremental indexing. Right now the entire search runs fresh every day. Tracking the newest-seen date per source and only fetching the delta since then would cut down on wasted API calls.

Schema versioning. The extraction XML schema has evolved more than once already. A versioned schema registry would make it much easier to re-run old backups against whatever the current schema looks like.

ML-based deduplication. Dedup right now is rule-based — same URL plus same headline. A learned similarity model using semantic embeddings would catch the true duplicates that exact matching misses.

Real-time streaming. Results currently arrive in a batch. Streaming them out as Claude produces them would let stakeholders start reading mid-pipeline instead of waiting for the whole run to finish.