The claim that AI can answer every question in financial data does not survive testing. We explain why a model that predicts downside risk but admits its direction calls are a coin flip is the more honest one.
AI products built on financial data share one weakness: they rarely say what they fail at. Yet a model's credibility is measured less by its correct predictions than by how it reports its limits. Using measurements from a model we built ourselves, this piece looks at what is and is not possible in financial prediction.
Risk is predictable; direction is not
Trained on hundreds of thousands of examples drawn from eight years of history, our model distinguishes whether a stock will fall into the worst-performing quintile over the next ninety days with an out-of-sample AUC of 0.74. A direction model built the same way sits at 0.51 — indistinguishable from a coin flip. Rather than hide that result we state it inside the product, because the product does not, and should not, say "this stock will rise".
The distinction is not accidental. Downside risk carries relatively persistent signals: leverage, deteriorating profitability, valuation stress, technical weakening. Short-term price direction, by contrast, is largely a reaction to new information and holds little structure that historical data can teach.
Calibration matters more than accuracy
A model saying "72% risk" is only meaningful if the event actually occurred in roughly 72% of comparable situations. Without calibration a probability is decoration, not a number. That is what makes an output usable for setting a decision threshold rather than merely ranking.
Leakage quietly manufactures success
The most insidious error in financial modelling is data leakage. If financial statements enter the calculation before their actual disclosure date, the model has seen the future and test results look better than they are. Likewise, because targets overlap in time, failing to place a gap the length of the horizon between training and test blocks inflates results. None of these mistakes raise an error — which is why they must be locked down by dedicated tests.
- Financial statements must enter only after their public disclosure date
- Overlapping targets require embargoed walk-forward validation
- Probabilities should be reported alongside a calibration table
- A new model should not go straight to production; it should run in shadow first
Where AI belongs: not producing the numbers, but explaining them
Our choice is that the language model never produces financial values itself. The numbers come from a deterministic engine; the AI only interprets the resulting tables and ties every claim back to its source. The metrics it may use are whitelisted. This constraint can make a product look less "clever", but it makes it auditable.
Not presenting a signal that does not work as if it does is as valuable as finding one that does.
We built Parakete on these principles; it is an analysis and monitoring tool, it does not provide investment advice, and its model outputs are statistical estimates based on historical data. If you would like to discuss building a similar decision support system on your own data, reach us via our contact page.
Let's plan this transformation together
We can bring what we covered in this article to life for your organization. Tell us your needs and we'll draw up a roadmap tailored to you.
