The most boring document in engineering

Behind every dependable machine sits a dull little document nobody except an engineer ever reads: the datasheet. It tells you exactly how the component behaves, and perhaps more importantly, how it fails. Like a fuse in your house, you only think about these things when things go wrong, but its entire purpose to keep you safe to a very specific, defined, point.

The point isn’t that the component is perfect, it’s that you know how imperfect it is and what it was built to withstand, so you can safely build around that limit. I trained and lectured in engineering a long time before any of this, and the philosophy couldn’t be more different to the world of software: you don’t ask for steel that never flexes, it doesn’t exist, you ask for steel that behaves exactly as it should under load, with some boring maths. Then you add some margins, from some boring books. Most of how we think about building AI is borrowed from engineering disciplines older than computing, and this is the core of it — you don’t build a reliable system from perfect parts, because they don’t exist. You build it from parts whose unreliability is documented.

There are of course well-designed software systems that do this too: a database transaction either completes in full or not at all, so you never find half a payment recorded. TCP guarantees your data arrives in order and intact, or that you know it didn’t. Neither promises nothing will ever go wrong. They promise that when it does go wrong, you know how.

‍

The shiny new component with no datasheet

A large language model is the most capable component most of us have ever worked with, and probably the second least specified, after the dodgy electrical item you bought from Amazon.

AI doesn’t fail predictably, of course. Torturing the engineering analogies a bit further– much like if you put too much reinforcement in concrete, it doesn’t slowly give way, it goes with a bang and without warning.

Failures in AI aren’t confined to a region you can predict or avoid. Most tools fail where you’d expect: they exceed their rating and the part gives up, so you just need to stay inside the rated zone. A language model doesn’t have a safe zone. It can work through a hard problem, like the hardest problems in mathematics, and then forget something you said just five minutes ago. Which is a problem, because the safe zone isn’t something you can draw on a graph, you can’t just give it the easy problems and hold off throwing the harder ones. When researchers scale models and fine-tuned them, they found there is no band where errors became improbable. It’s even worse, because the larger and more capable the model was, the more it produced confident, plausible, and yet still spectacularly wrong answers on questions their human supervisors then failed to catch.

Back to the dodgy electrical item you bought on Amazon – when that fails, you’ll probably know about it. The screen will stop working, or it’ll set your house on fire. A fuse goes bang. A database fails by refusing the write. A language model fails by handing you a fluent, well-formed, entirely wrong answer with exactly the composure of a right one. Commercial legal-research tools sold to lawyers as reliable were found to hallucinate 17 to 33% of the time, in such a convincing way that it’s almost impossible to spot without redoing the entire job. A component whose errors are silent can’t carry weight, because nobody downstream gets a signal telling them when to look.

‍

A larger large language model isn’t the answer

It’s easy to think of these problems as something the next generation will, but scale affects the average not the extremes in this case. A better model raises the quality of its output, in general. There’s now a sizeable amount of evidence that larger models don’t fix the problem, nor do they make the failures easier to spot; often the opposite, in fact.

For most uses, the missing datasheet doesn’t matter at all. Drafting, brainstorming, summarising, exploring an unfamiliar topic, writing a first version a human will edit — here someone’s already in the loop and hopefully might review it properly. None of this is actually intended to be an argument against language models; we use them heavily at Onteric, every single day. Most of our code is now written by AI. It’s an argument about an architectural problem that needs to be fixed.

How far this limit can be pushed isn’t agreed between researchers and academics, so you can make your own assessment. OpenAI’s team argue hallucination is not inevitable— that models guess instead of abstaining because training and evaluation reward a confident answer over “I don’t know”, and that calibration is cheaper than accuracy. But even they acknowledge that facts seen rarely in training carry an irreducible error rate, and rightly there’s an increasing fight against AI benchmarks that distort and encourage this. When Apple published evidence that frontier reasoning models collapse past a certain complexity, a rebuttal landed within days arguing the collapse was partly an artefact of the scoring. They might both be right, but that doesn’t help you decide if you want to plumb AI into your business to make things efficient.

‍

What would a datasheet even look like

Well, it’s nota better model. Or a larger model. If we stop for a minute treating “AI” as a single component, or a synonym for large language models, then in many places we can already use parts whose behaviour can be specified — and LLMs can help you do that, that’s why Claude Code and software engineering are fantastic use cases for today’s AI, you take the AI out of the production path and use it in development.

There are now model cards from the major AI players — I assume it was an intentional choice not to call these datasheets, because they contain hardly any useful information. You can’t really design a fail-safe system knowing your model scores 84.7% on OS World-Verified.

The most common thing our customers need today is reading a document and extracting the right data from it, in the right way – think bank statement, payslip, completed form. These documents are literally designed to be clear and unambiguous, the number in the box is the number in the box — and for us 99% of the time there is no language model required at all. Our neuro-symbolic approach to AI synthesises a small program that extracts and transforms only what is genuinely present in the source. That single constraint changes the failure profile. The component can still be wrong (though I should point out, this is rare!)— it could in theory pick the wrong cell or map a field incorrectly — but it cannot invent, because it has no mechanism to fabricate a value that was never actually in the document. This matters because a grounded number can be checked against the exact document it came from, and it’s easy to check if you know how it was extracted, meaning you know exactly where to look. We tell our customers that our AI can’t hallucinate – and this is what that actually means: not that it’s perfect, but that if it ever is wrong, it’s easy to spot and to test.

That’s a part with a datasheet. You can see and read what it will and won’t do, measure its error rate, and design around the margins — exactly as you would with any other critical component.

‍

The bottleneck already moved

It’d be nice to think this is all temporary and that the models get good enough and the problem dissolves. But that’s not what the evidence increasingly says is happening – the models and their capabilities aren’t the limiting factor on AI adoption anymore, and haven’t been for a while.

The largest study of enterprise AI to date found that 95% of generative-AI pilots delivered no measurable impact. The number is alarming given the amount of money involved, but more interesting is that the researchers were explicit that the cause was not model quality — it was that the tools don’t retain feedback, adapt to a workflow, or improve over time, and that organisations couldn’t bridge the gap between an impressive demo and a dependable process. One of their interviewees framed it perfectly: a model that’s excellent for first drafts but “repeats the same mistakes” and has to bere-briefed every session is not something you can build a high-stakes procession. The barrier they describe isn’t intelligence – it’s reliability, integration, and the ability to trust what it gives you.

For some reason, for most AI companies the answer is always a bigger model – enormous effort and compute is expended making the model a few percent smarter, when the thing standing between a demo and a deployment is elsewhere: consistent behaviour, predictable failures, and transparent failures. And before you can even meaningfully think about those things, you also need trustworthy and structured data to use. These are the hard problems – there’ll be more to follow in the coming weeks about how Onteric thinks about and is solving for them.

If your AI model made a loud bang and sparked when it went wrong, at least you’d know where to look.