The simulator was the experiment

I wanted an RL project that was not another CartPole, so I built an environment for a problem I actually understand: you have a fraud detector, a limited number of investigations, and the only way to learn whether a case was fraud is to spend one of those investigations on it. Then the fraud population moves. I trained a policy, it beat every baseline I had, and I was ready to write that up. Running it on real data instead turned the whole thing around.

The short version: the agent was never the interesting part. My simulator was, and holding out real data is the only reason I found out.

Two in 169

Start with the measurement that made me want to build this at all. Take the Elliptic graph, 203,769 Bitcoin transactions across 49 time steps, and fit a gradient boosting detector on steps 1 to 34. Then freeze it and watch what it finds, at a budget of 2% of each step’s transactions.

(opens in a new tab)

It does not decay gracefully. It works, and then over the seven steps after the break it finds 2 of the 169 illicit transactions. Five of those steps come back at zero, and the 50% at step 46 is one transaction out of two. Not “degraded”, not “drifted”: almost nothing found, while illicit transactions were still there to find.

And the information needed to recover is sitting right there. Refit that same model on every label up to step 44, which includes the first two steps after the break, and at a 10% budget it finds 49.6% of the illicit transactions in steps 45 to 49, against 3.3% frozen and 5.0% for random picks. So the fraud is findable. You just have to look somewhere the broken detector is telling you not to.

Why this is not a classification problem

Here is the rule that makes this a decision problem. Investigating a case reveals its label. Nothing else does.

That sounds obvious and it has a nasty consequence: your detector can only ever be retrained on cases somebody chose to investigate. A policy that always spends its budget on the highest-scoring cases will keep scoring the same kind of case highly, keep being right about them, and have no way of noticing that the population moved. It does not get a warning. It gets almost nothing.

This is the selective-labels setting from the fairness literature, written as a sequential problem instead of a static one.1 It is also an instance of what the RL literature calls a Monitored MDP, where the reward exists whether or not you observe it and observing it competes for the same budget as exploiting it.

The opponent that matters

The easiest way to make an RL result look good is to pick a weak baseline. So before training anything I built the two that actually matter.

Top-k is the common baseline: score everything, investigate the highest scores, spend the budget evenly over time. It is not a strawman. Before the break it finds 95.8% of the fraud in my environment, over 30 seeds. Any policy that cannot beat it is not worth deploying.

Yield-triggered is the real bar, and I want to be honest that there is nothing clever in it. It watches its own hit rate, and when yield collapses relative to what it had been getting, it switches budget into exploring cases the detector ranks low until yield recovers. That is what a competent engineer writes in an afternoon without any machine learning, and it is the thing a learned policy has to justify itself against.

I also report an oracle ceiling everywhere, because recall under a budget is meaningless without knowing what was achievable. Over the same 30 seeds, a perfect ranker with the same budget gets 98.7% before the break (98.8% overall). Every result should be read against a ceiling like that one.

Building a simulator that lied to me

Elliptic has exactly one regime break. One break is one trajectory, and a policy fitted to one trajectory has memorized it, so I could not train on it. I wrote a procedural drift simulator instead: fraud is a mixture of modes in feature space, and at a randomly placed break some modes stop emitting and new ones start. Train on random breaks, evaluate on the real one.

The first version was useless and it took a while to see why. In it, top-k recovered a large share of the post-break fraud on its own, with no exploration at all, which is nothing like Elliptic’s near-total collapse. The reason is that my new fraud modes landed somewhere the old detector still scored middling, so top-k stumbled into a few of them every round, fed them to the refit, and quietly healed itself. Exploration had nothing left to contribute, which made the environment a poor model of the thing I built it for.

The fix was to make the break adversarial: generate candidate fraud modes, score them with a detector trained on pre-break traffic, and keep the ones it scores lowest. That reproduces the real collapse, and it is also the more honest story, because an adversary who adapts moves to where the current model is not looking.

Note what just happened, because it is the pattern for the whole project. I chose a property of the simulator in order to create the phenomenon I wanted to study. That is legitimate, and it is also exactly how you end up measuring your own assumptions.

The first run said it did not work

I trained a small PPO for 320 episodes, on one seed, and evaluated on held-out drift seeds. It got 0.596 mean recall against top-k’s 0.570 and yield-triggered’s 0.594, with a standard deviation of 0.079. In other words, it was level with the hand-written rule, and on the paired interquartile mean only a hair ahead of top-k (+0.010 [+0.001, +0.025]).

I wrote that up as a negative result, pushed it, and then noticed the obvious objection to my own work: 320 episodes is not a training run. Any reviewer kills that claim in one line. So the finding was not “RL does not help here”, it was “I did not train it”.

The second run said it did

1,600 episodes per seed, six seeds. On the same held-out episodes, with interquartile means and paired bootstrap intervals:

Paired against the hand-written rule it comes out at +0.073 [+0.053, +0.088], so it clears it, which it never did at 320 episodes. Post-break it finds 4.7 times what top-k finds. It had learned the behavior I designed the environment to reward: keep nearly all of top-k’s pre-break performance, and spend on exploration once yield collapses. (The oracle row is overall recall only. These result files do not split it before and after the break.)

At this point I had a result. A learned policy beating both top-k and a sensible hand-written rule, with intervals, on six seeds. This is the version of the story I would have published if I had stopped here.

Then I ran it on the real break

(opens in a new tab)

On my simulator, PPO has the best post-break recall of anything I tested. On the real break, every one of the six seeds finds less after the break than the hand-written rule (0.408) and the always-explore policy (0.361), and three of them find less than top-k (0.077). Five of the six seeds fall from where they were on the simulator. One rises.

Overall on Elliptic, the seeds run from 0.705 to 0.759, against top-k’s 0.730 and the rule’s 0.754. Three seeds are above top-k and three below. The rule is ahead of five of them, and the sixth is level with it. After being worth +0.076 in distribution, that is no visible advantage at all. And post-break recall across seeds runs from 0.051 to 0.286, more than a five-fold spread, so on the real break what the policy does depends heavily on which seed trained it.

There is a second reason not to lean on these numbers, and the first version of this post missed it. They depend on the last bits of the features. I first ran the evaluation on an Elliptic archive built by a different parser, and with the same checkpoints and code, that archive gives yield-triggered 0.231 post-break instead of 0.408, always-explore 0.560 instead of 0.361, and seed 0 0.440 instead of 0.286. So on this single trajectory, a difference under about 0.2 post-break, or about 0.05 overall, is not a difference between policies. That older archive is where the “3.6 times” in the first version came from.

The honest statement is narrower than either “RL works here” or “RL does not”. It is: on the simulator it trained on, the policy beats every baseline on overall recall across six seeds, and on the real break that advantage is not visible. That is an absence of evidence for transfer, not a measured loss. More training bought in-distribution skill, not transfer.

Which means my simulator does not present what the real collapse presents. I do not fully know what the difference is, and that is the most interesting open question the project produced. The drift I generate is a mode substitution in feature space; whatever happens at Elliptic’s step 43 evidently offers a different signal, or none that my observation vector exposes.

The other half: when review pays

The second environment was supposed to be the easy one. It is also where the first version of this post got its headline backwards.

Feedzai’s FiFAR records, for 30,622 synthetic fraud alerts, what each of 50 synthetic analysts would have decided. The alerts are the applications an alert model flags in the Bank Account Fraud dataset, which is itself privacy-preserving synthetic data, and the analysts come from the paper’s own generator. None of it measures real people. What makes it worth building on is that routing a case to an analyst is an exact table lookup rather than a model of a reviewer. The analysts are also fallible in uneven ways that the generator sets: false-negative rates from 0.015 to 0.312, false-positive rates from 0.015 to 0.759.

I opened the shipped scenarios expecting a sequential capacity problem and found neither sequence nor scarcity. Each test scenario is a single batch whose review capacity is 90.9% of its alerts. The training scenarios run at 100%. There is no horizon to ration across and the constraint barely binds, which is why the dataset’s own baseline solves each batch to optimality with constraint programming and does well.

Then the cost ratio, which is where I went wrong. FiFAR’s paper and DeCCaF, the dataset’s own baseline, both price a false alarm at λ_t = 0.057 of a missed fraud, about 17.5:1, and that is the regime the synthetic analysts were generated for. DeCCaF also runs about 88:1 and about 3.5:1 to test other cost structures. The first version of this post called 88:1 FiFAR’s realistic setting. It is a stress case.

At the stated regime, review pays. Sending 10% of alerts to the analysts, paced evenly and chosen nearest the model’s decision threshold, cuts expected cost by 3.2% [2.2, 4.2] across the 25 training scenarios and by 7.6% [4.0, 11.2] on held-out month-8 alerts. All 50 analysts beat blocking every alert at that ratio.

At 88:1 it flips, and the mechanism is arithmetic. There the cost-optimal threshold (0.0510) sits at the bottom of the alert scores, so the model alone blocks every alert and leaves no missed fraud for a reviewer to catch. Review can then only rescue false positives, and only 6 of the 50 analysts cost less than blocking everything:

saves = P(legit) x (1 - FPR) x fp_cost
loses = P(fraud) x FNR                      averaged over the 50 analysts

ratio              fp_cost   analytic   measured in the environment
87.7:1  stress     0.0114    -0.0121    -0.0134 [-0.0181, -0.0090]
17.5:1  stated     0.0570    +0.0155    +0.0160 [+0.0110, +0.0208]

At a 10% review budget, the 88:1 case raises total cost by 13.27% [8.86, 17.92]. That is a property of how the synthetic analysts were calibrated, for a different cost ratio, and not a finding about human review. The useful question is still when review is worth doing at all, and on this data the answer is a band rather than a line.

(opens in a new tab)

In-sample, review is significantly negative at 50:1 and above, indistinguishable from zero at 33:1, and significantly positive from 25:1 down to 3.5:1. It is not monotone either: at 2:1 the refit threshold moves to 0.42 and review’s saving falls back to +0.32% [-1.43, +2.09]. The first version of this post drew a sharp crossover at about 35:1 from a single scenario at a fixed threshold, and most of the value that chart showed at low ratios was threshold re-tuning, not review.

I have also withdrawn a paragraph. It said all this explained a published oddity in DeCCaF’s results, and that the realistic cell of that benchmark was close to degenerate for anything involving a human. I had misread their table. In DeCCaF’s own results at the 88:1 setting, every method that routes to humans, random routing included, has lower expected cost than rejecting every alert, and DeCCaF regenerates its experts for each cost setting while FiFAR’s 50 analysts are generated once. I have not trained a policy on FiFAR, so this post makes no claim about what one would do there.

What I would take from this

Five results in this project were wrong before I published. A simulator whose break was too gentle. A baseline that divided budget by batch count rather than batch size. A metric that ranked policies by how fast they burned budget, where the saving ranking turned out to be exactly the spend ranking. A decision threshold set at 0.5 on alerts that were already everything above 0.051, which manufactured a finding that routing near-threshold cases was worth ten times more per review than routing the most suspicious ones, and which evaporated when I fixed the threshold. And a negative result about RL that was really a statement about my patience.

Then the pre-release re-check of caseload found eight more, and five of them had made it into the first version of this post: the cost regime, a cost-ratio chart from an ad hoc run, an Elliptic headline spliced from two configurations with charts typed by hand, synthetic data called real, and a zero-shot table from an archive nobody else could build. The other three were reviews the triage environment counted when no analyst had made them, a routing band that picked the lowest row indices instead of the highest scores, and a stale rule in the README. That makes thirteen.

The first five were caught the same way: by comparing a number against something outside the thing that produced it. The simulator against real data. The policy against a rule I could write by hand. The arithmetic against the measurement. The eight came from doing that on purpose, claim by claim, against the code, the committed results and the cited papers, before the repository went public. The phantom-review bug only got its regression test after the re-check found it.

The specific lesson I did not expect is about simulators. I built one to make a phenomenon studyable, which is standard practice, and then trained a policy that got very good at it. The policy’s in-distribution dominance was real, reproducible across six seeds, and almost entirely uninformative about the question I cared about. A held-out real trajectory, which is one episode and carries no confidence interval at all, was worth more than 9,600 synthetic ones, even though that one trajectory turned out to be too noisy to rank policies finely.