As of the time of writing, there have been the beginnings of stirrings of malcontent among large companies. The issue is that organisations that have decided to go all-in on large language models are discovering that they’re being charged a lot of money to use the big, ‘frontier’ AI companies’ services, and that they can’t accurately work out what return-on-investment they’re getting from using AI.
Clearly, there’s not really much ROI being seen in AI deployments. We’re four years in since OpenAI made the headlines, and if there were productivity gains to be made that actually made a significant difference to on organisation’s bottom line, my but we would have heard about it.
There are two areas in which ROI has been trumpeted as having been achieved. One is the deployment of chatbots instead of human beings in call centres. This is a process that’s been going on since early language recognition and natural language processing came to be viable, something that was real around 8-9 years ago. LLMs’ acting in this space put a little spin on existing technology, but any call centre operator wanting to get rid of staff (and many did – humans are expensive) had already done so, or was well on the way.
The second area of success, although probably not actual ROI is in software coding. Individual developers can churn out much more code, especially the simpler stuff, but organisation-wide, there aren’t really savings to be made at the bottom line. After all, lots more code means lots more code review, and getting another LLM to test and scan code doesn’t go well: It’s an inherently 90% accurate code checker that’s tasked with checking code that’s inherently 90% accurate. In other words, to put out good software, the burden of work has moved up the responsibility chart – from junior engineer (now 10x more ‘productive’) to senior engineer, or the poor sod who has to try and make sense of it all before accepting the pull request.
So if there are productivity gains to be made, we should be seeing these in organisations’ bottom lines. And that doesn’t seem to be happening. What we do hear a lot about is knowledge workers (that is, workers not doing physical things like making widgets) ‘becoming more efficient’, which seems to mean summarising documents more quickly and making more PowerPoint presentations to display in meetings.
There are several issues in play here. The most obvious is to define improved efficiency among knowledge workers as a numerical process. We have a new tool, let’s call it “WIDGET”. It costs £x. By using WIDGET, a company makes extra money – more money than it would if it didn’t use WIDGET.
Either using WIDGET means the company can make and sell more product at the same cost, or make the same amount of product at a lower cost, or best of all, make and sell more product at a lower cost as well. In whichever case, let’s term the ‘extra mullah-dollar-dollar’ made is £y.
Return on investment is £y minus £x. This is very simple to understand. Complex in the detail, of course, but the principle is simple. I’ll come back to this simplicity, and likely be all rant-y.
But we have a problem. We never measured how efficient knowledge workers were in the first place. Sure, there are simplistic tells that a person is slacking off, like not responding to messages on Slack or similar. Or getting through a set number of complex reports each month and summarising them. But give anyone a target, and they’ll immediately start playing the ‘hit the target’ game, rather than actually doing things that benefit the business. In good businesses, the targets correlate with what brings value in through the door. Mostly though, targets and KPIs are drawn up by managers who don’t really know what the frontline bodies actually do, and are ignorant as to what might bring cash money in through the door anyway.
So, we have a situation in which knowledge-based workers are not accurately judged on their efficiency being given AI tools to improve their efficiency, and both the ‘before’ and ‘after’ states are either not measured, or measured in such a way that’s meaningless financially.
This brings us neatly back to this week, one of the most enjoyable three days I’ve spent professionally in the last four years, by the way.
The problems being pushed into the headlines is that the cost of using LLMs has risen sharply at the point of use. What used to cost a company, let’s say, £100,000 per month might now cost £10,000,000 per month. To use our analogy of WIDGET, £x is now a lot more than £y, so using WIDGET doesn’t make financial sense. Or does it? We can’t really say. It’s tough to say “make sure you’re careful using this new tool, and only use it to become more efficient” if there’s no meaningful measurement of efficiency. All companies have to go on is that big bills have started to arrive from the AI companies, and organisations are shitting themselves that these ‘efficiency gains’ are of less financial value than the cost of AI tools. But it’s just a feeling – after all, there are no figures to go on, are there?
Which, if one ran a business, one might want to ask the following questions:
- What were you, the employee, doing with AI tools that made your work faster and better?
- Having become faster and better, what extra cash money did that speed and efficiency bring in?
- Did we know subscription payments to AI companies were going to be temporary?
- Given the new costs – in which every interaction with an AI costs money (how much kinda depends on things outside our control) – are we getting ROI on our AI investments today?
Here are some answers a business leader may hear:
- AI allowed me to summarise text, write anodyne text or low-complexity code, create third-rate media, and introduced a level of inaccuracy in any complex process that would not be acceptable in a human.
- Faster and better you say? Well yes, by the measures which you stipulated. But no extra money.
- Did we know that AI companies couldn’t sustain their subscription-based payment systems? Yes, we could have found out, but it would have meant doing some research online, and all the AI-run searches were programmed to lie. So we just took their vapid self-promotion and promises at face value.
- Depends. If you measure ROI as doing low-level stuff badly but quicker, yes we’re in clover.
In short, dear reader, the AI bubble is starting to pop. A few months after I predicted, and I think that I’m going to be 8-12 months wrong, I grant you. But nevertheless, the end is nigh. Here’s the chain of events.
- third-rate product gets massively over-promoted, with claims of abilities it doesn’t have, and promises of improvements that can’t physically happen due to code-y things I don’t understand, and a limit on the amount of information available to steal. Sam Altman wept, for there were no more internets left to be scraped.
- third-rate product built on debt, hyperbole, and let’s be frank, lies, pushed into the headlines at every possible opportunity, usually scaremongering stories invented by the third-rate product vendors.
- third-rate product is offered to customers at a massive loss, in the hope that it will somehow improve to become a first-rate product, and everyone will find it invaluable in ways that are never defined.
- third-rate product stuffed into every piece of software so everyone has to use it, with fingers crossed aforementioned improvements will take place and users will find it addictive and irreplaceable.
- in order that vendors of third-rate product can float their companies on the stock market, they start charging customers something approaching what it actually costs. This pleases the financial types who prepare to make a few quid at the start of trading. Ker-fucking-ching.
- users of third-rate products realise that it ain’t really all that and opt to stop using it rather than pay a shit ton of money, as after all, it was never really able to do stuff that mattered.
- economies collapse, credit market tanks, economic recession on the scale of 1988 or 2008, maybe worse.
- everyone starts talking about quantum computing, which thankfully is even more complex and hard to understand than how LLMs work. Begin again at the top of the bullet points.