Real-Time Market Narrative Detection Engine
An autonomous intelligence platform that detects emerging market narratives across five data sources before they reach mainstream coverage — then ranks, dispatches, and tracks them to outcome.
The Problem
Every fund manager has the same dirty secret: they're drowning in information and starving for insight.
Our client — a quantitative investment firm — had a team of sharp analysts spending half their day scrolling through X, scanning SEC filings, sifting Reddit threads, and refreshing Bloomberg terminals. They were good at it. But "good" doesn't scale, and it definitely doesn't wake up at 4 AM to catch a narrative shift happening in Asian markets while everyone in Dubai is asleep.
The question they brought to us was deceptively simple: "Can you build something that tells us what's changing in the market right now — before everyone else notices?"
The honest answer was: sort of. No system predicts the future. But a well-designed system can notice patterns across thousands of data points simultaneously, and it can do it every single day without getting tired, distracted, or anchored to yesterday's thesis.
What We Built
The system runs autonomously every morning. No human touches it. By the time the research team opens their laptops, a ranked briefing is waiting.
Under the hood, it's a three-stage funnel: discover, converge, investigate.
Cast a Wide Net
Five independent scanners fan out in parallel, each monitoring a different corner of the information landscape. Each is powered by a different AI model, chosen for its strengths. The diversity is intentional — we didn't want five versions of the same blind spot.
Social Intelligence
What traders and analysts are actually talking about on X and social platforms, stripped of noise.
Community & Media
Reddit threads, YouTube analysis, financial blogs — the long-tail content that mainstream terminals miss.
Regulatory Filings
SEC EDGAR transcripts, earnings calls, press releases — the stuff that moves markets but nobody reads in full.
Quantitative Signals
Sector ETF flows, insider transactions, analyst consensus shifts — the numbers behind the narratives.
Digital Asset Social Data
On-chain sentiment and social intelligence across digital asset markets.
If one scanner fails, the others complete independently. In six months of daily operation, a full pipeline failure has never occurred.
Find the Signal
Five scanners produce a lot of raw material. Most of it is noise. The convergence layer is where the intelligence actually happens.
The core insight is simple: if one scanner flags a theme, it might be noise. If three scanners independently flag the same theme from different data sources, something real is probably happening.
The synthesizer compares everything against the previous day's scan. It only surfaces what's changed. This single design decision eliminated the alert fatigue that plagues most monitoring systems.
Every theme gets classified into a lifecycle stage: nascent, emerging, accelerating, consensus, or fading. Our data showed the sweet spot is the "emerging" stage — confirmed by multiple sources, but not yet crowded. By the time something reaches consensus, you're usually too late.
Go Deep (When It Matters)
Not every detected theme deserves a deep dive. Investigating everything would dilute the system's signal-to-noise ratio — the exact problem we were hired to solve.
We built hard trigger criteria. No LLM decides whether to escalate; the logic is deterministic. When a theme crosses the threshold, an in-depth analysis fires automatically — producing a conviction assessment, a counter-thesis, a catalyst timeline, and suggested trade parameters.
Each deep dive is enriched with context from prior analyses. The system remembers what it said about this theme last week, whether the thesis played out, and adjusts accordingly. Institutional memory that actually works.
The Dispatch Loop
Detection without action is just expensive journalism.
So we built an automated dispatch pipeline that ranks the day's strongest signals and feeds them directly into the firm's downstream analysis infrastructure. The top-ranked tickers — typically 5 to 10 per day — get queued for full-spectrum analysis by the client's existing systems.
The ranking isn't a black box. It weighs conviction, convergence, novelty, and risk-reward asymmetry, with adjustable weights the client can tune. Themes in the emerging stage get a bonus. Stale themes get penalized. Crowded trades get deprioritized.
Risk controls we baked in — that the client didn't ask for but definitely needed:
Regime awareness
During extreme volatility, dispatch volume automatically throttles. In severe stress, it pauses entirely. This isn't a feature you appreciate until the one day you really need it.
Outcome-aware cooldowns
If a dispatched signal resulted in a loss, the system won't re-dispatch the same thesis for two weeks. If it won, a shorter cooldown allows re-entry on pullbacks.
Concentration limits
No more than three tickers from the same sector, two from the same theme. The system enforces diversification even when every scanner is screaming about the same trade.
And the feedback loop — the part the client originally identified as their biggest gap — runs continuously. Every dispatched ticker is tracked from detection through to outcome. Win, loss, still open, closed at target, stopped out. Full attribution, from the scanner that first detected the narrative to the final P&L.
This isn't just record-keeping. It's how the system learns which types of narratives reliably produce results, and which are noise that sounds convincing.
Under the Hood
Backend
Python, FastAPI, Celery, PostgreSQL, Redis. 307 automated tests.
Frontend
Next.js 15, React 19, TypeScript, Tailwind CSS, Recharts. 58 source files.
Infrastructure
Multi-container Docker on AWS EC2, Nginx, Let's Encrypt SSL. Independent task queues per concern.
Data Model
6 core entities. Dispatch table with 30+ columns capturing full provenance from scan to trade outcome.
What Changed
A team of analysts manually monitoring dozens of sources, comparing notes in Slack, and hoping they didn't miss something important overnight.
A ranked briefing waiting every morning with the day's most actionable themes, backed by multi-source convergence data and conviction scoring — with every signal tracked to outcome.
The firm didn't replace their analysts. They gave them superpowers. The team now spends their time on the 5–10 themes that matter instead of the 500 data points they used to wade through. Research meetings got shorter. Conviction got higher. And for the first time, they have data on which types of narratives actually make money — and which ones just sound good.
The same architecture — parallel AI scanners, convergence synthesis, and outcome tracking — applies to any domain where early detection of emerging patterns drives competitive advantage: supply chain disruptions, regulatory shifts, competitive intelligence.