Informative but subsumed

Why the literature disagrees

Text-based conflict forecasting has a reproducibility problem hiding in plain sight. On one side, well-cited work reports that newspaper topics carry real predictive signal for political violence. On the other, every major forecasting competition in the space keeps reaching the same deflating conclusion: almost nothing beats “assume next month looks like this month.” Both camps publish plausible numbers. They can’t both be right at face value.

Part of the gap is evaluation hygiene. A construction you’ll find in more than one applied pipeline: label news articles by their similarity to descriptions of known events, then train a classifier to predict that label from the same text, validated on a random split. Pipelines like that produce accuracies in the 0.8-0.9 range and forecast nothing: the label is a function of the input, so the model just re-derives the labeling rule.1 I audited one such pipeline in detail as the starting point for this work; its headline number dissolved entirely under a temporal split with an independent label.

But hygiene isn’t the whole story: clean studies disagree too. So the real question became: strip away the leakage and the underpowered designs. What does news text actually contribute, measured on one benchmark, under one protocol, at a scale that can resolve small effects?

The setup: rules first, models second

Everything runs on complete unit×period grids, 994 admin-1 regions across 55 countries, monthly, on UCDP ground truth (plus an ACLED state×week track for high-resolution replication), with three frozen targets: occurrence (any violence next period), escalation (a spike above the region’s own trailing norm), and onset, violence erupting in a region calm for two years, the regime early-warning systems exist for. Four protocol rules do the heavy lifting:

• Out-of-time only. Rolling-origin backtests: train on the past, score the strictly-unseen future, refit as the window expands. Random splits are banned: they’re how time-series data lies to you.
• Beat the boring baselines or say so. Persistence and per-region climatology are always reported. Accuracy is banned outright: at base rates of 2-20%, predicting “nothing happens” scores 80-98%.
• Two model families, always. A tree ensemble and a regularized linear model must agree on the sign of an effect before it counts. This rule exists because an earlier iteration of this analysis produced a “significant” negative marginal that flipped positive under regularization, a pure artifact of 90+ features against ~140 positives.2
• Confidence intervals from released predictions. Every marginal gets a block bootstrap over test periods, recomputable by anyone from the per-row prediction files shipped with the benchmark.

What’s forecastable at all

Skill collapses by an order of magnitude at each step. Where violence already burns, next month is very learnable (0.80 AUPRC against a 0.53 persistence baseline). Whether it spikes is much harder but real (0.175 vs 0.107). And onset, eruption after two calm years, is close to unforecastable for everything tested: the best model reaches 0.025 against a 0.018 base rate, and persistence is completely blind there by construction, since every region in the risk set looks identical to it. The case that matters most is the case nobody can call.

The text ladder

Onto that scaffold goes every text signal constructible from open news data, in increasing order of representational depth: GDELT’s machine-coded tone and article volume; its thematic tags; topic models over event narratives; precipitation anomalies as a non-news control; and, because “your text signals were too shallow” is the correct objection to everything above, neural sentence embeddings of 1.37 million human-written headlines, harvested from the URL slugs of the 336 million GDELT events joined to the panel by pure geography.3

Three results, each of which survived label-sensitivity sweeps, dual-family checks, and 90% intervals:

1. Text’s real complement is tiny, but it exists. On escalation, adding news signals to event history is worth +0.3 to +0.6 points of AUPRC, significant in both model families, robust across four alternative label definitions. Roughly a 2-4% relative improvement. For calibration: that’s about two orders of magnitude smaller than what the leaky pipeline designs in this space imply.

2. Single-country studies cannot see it. Single-country confidence intervals in the high-resolution replication ran ±0.03 wide: a ±0.005 effect is invisible at that scale. Resolving it took 87,000 test region-months across 55 countries. This is, I suspect, a large piece of why the literature disagrees: the same small number, measured with instruments of wildly different power, reads as “text works” in one design and “text is useless” in another.

3. The part of news that helps isn’t the part you’d think. The embeddings, the richest representation on the ladder, and the best standalone text predictor (0.105 vs 0.085 for machine-coded counts), add nothing on top of event history. Null in both families. What carries the small complement is the count-like content of news streams: how much is being reported, where, how often. The semantic content, what the articles actually say, is precisely what a curated event record already encodes. News semantics are subsumed; news arithmetic is not.

Lessons for ML practitioners

• Your label is your result. If the target is any function of the inputs, the model will find the function, and you will find a publication-shaped artifact. Rare-event settings hide this beautifully.
• Random splits are how temporal data lies. The future must be structurally unseen, not just held out, but later. Rolling-origin evaluation should be the default, not the robustness check.
• “Significant” is a property of a pipeline, not a truth. Marginals estimated by unregularized ensembles in positives-starved regimes can flip sign under regularization. Two model families agreeing on sign is a cheap, strong filter.
• Power determines the literature you read. A field full of underpowered designs measuring a +0.005 effect will publish confident positives and confident nulls forever. Scale the evaluation to the effect size you’d need to detect, or say you can’t.