Quincus

Engineering

The data you need is worse than you think, and that is fine

Apr 28, 20264 min read

Every optimization initiative meets the same objection in the first workshop: "our data is not ready." The instinct is to launch a data quality program, clean for 18 months, and revisit. This sequencing is almost always wrong, for a simple reason: no operating network's data is ever ready, including the networks running world-class optimization today. Missing events, duplicate records, freetext where structure should be, timestamps in local time with no zone, sensors that lie. This is the permanent condition of physical-world data, not a phase to be exited.

The correct response is architectural, not janitorial. Systems intended for production are built to metabolize imperfect data: probabilistic record matching where identifiers are unreliable, imputation with explicit uncertainty where events are missing, outlier handling that quarantines rather than crashes, and estimates that carry wider error bars where the source data is thin. A stochastic planning layer has a structural advantage here, because it already represents everything as a distribution. Bad data does not break it. Bad data widens its stated uncertainty, honestly, exactly where it should.

Deployment cleans data faster than cleaning does

There is also a second-order effect that data quality programs miss: nothing improves data like using it for decisions people care about. When a fitted dwell-time distribution drives real scheduling, the depot whose scans are missing becomes visible within a week, because its error bars are conspicuously wide and its plans conspicuously padded. Usage creates the feedback pressure that cleaning programs try to generate with governance meetings, and it creates it automatically, prioritized by commercial impact rather than by inventory of defects.

Start with the decisions, let the system quantify where data weakness actually costs money, and fix in that order. Networks that sequence it this way are optimizing in month 2 and improving data continuously. Networks that sequence it the other way are still cleaning.

Interactive

Your model is only as calm as your data.

Corrupt the inputs and watch two models diverge.

Brittle error
27.0
Robust error
19.1
Robust wins by
+7.9
Brittle
Robust
032630%10%20%30%40%

The clean-data champion and the production survivor are rarely the same model.

Journal

New essays by email.

One email when a new essay is published. Nothing else.

By subscribing you agree to our privacy policy.

If your network makes decisions under uncertainty, we should talk.

We work with a small number of operators at a time. Tell us what your network is optimizing for.

Start a conversation