Fish Food: Episode 635 – Technology, functional drift, and collective blind spots

This week’s provocation: Technology, functional drift, and collective blind spots

This week I came across a news article which reported that Health New Zealand, the primary publicly funded body set up by the New Zealand government to oversee their healthcare system was using a single Excel spreadsheet to track $28 Billion of public money. An independent report by Deloitte found that the reliance on a single spreadsheet to manage their finances was a ‘major issue’. No kidding.

You might think that this kind of technological overreach is rare but I suspect that it’s far more common than we think, and it says a lot about humans, organisations and their relationship with technology – a topic worth exploring more here.

Let’s start with sunk cost fallacy and the tendency for organisations to cling on to legacy, outdated systems because of the considerable sums already spent on their implementation and maintenance. Even when newer, more efficient solutions are available. A common impulse here is to evolve software over time through bolt on solutions and end up with ‘spaghetti systems’, with large proportions of technology spend going on maintenance of core, outdated systems and hundreds of workarounds and bolt-ons, creating a highly complex house of cards.

For many years this was the challenge faced by the big banks, who relied on legacy mainframe systems and needed to spend up to 80% of their not insignificant IT budgets to undertake expensive system updates rather than investing in new, more agile technology solutions. These short-term fixes acted as a brake on innovation, giving a head-start to new and nimble fintech banks like Monzo, Revolut and Starling. As HSBC’s chief technology architect once said, these spaghetti systems can generate their own technology dependencies: ‘Everything is connected to everything else, pull on one thread and everything comes with it.’

Then there’s Gourville’s rule of thumb (see Harvard economist John Gourville’s original research paper) which states that companies tend to overvalue the benefits of their new products or technologies (by a factor of 3). Consumers, on the other hand, tend to overvalue the benefits of their existing habits and products (by a factor of 3). Substitute users in an organisation for consumers here and what this effectively means is that a new system or technology needs to be 10 times better than the last one to overcome this value gap and for it to be see as a viable replacement (I wrote more about technology acceptance models here if you’re interested).

Combine this with availability bias (familiar technology is readily available in our minds, making it seem like the best or most common option) and familiarity bias (we tend to see things that we are familiar with as being safer, and more reliable), and it’s not hard to see how people return to familiar technology (like Excel) to simplify working with complexity.

This, of course, can have serious consequences. In the autumn of 2020 at the height of the COVID-19 pandemic almost 16,000 COVID cases went unreported due to a poorly thought through use of Microsoft Excel. Public Health England (PHE) had set up an automatic process to pull data from commercial companies that were paid to analyse swab tests of the public into Excel templates. The data in these Excel files was then uploaded and made available to the NHS Test and Trace team and government COVID dashboards. Unfortunately the PHE developers had selected an old XLS file format to do this which meant that each template was limited to handling only 65,000 rows of data rather than the one million rows that Excel is actually capable of dealing with. As each test result created several rows of data this meant that each template had an upper limit of around 1400 cases that it could record and any additional cases that came in after that were simply not recorded. The mistake of using outdated software meant that there were eight days of incomplete data and thousands of cases that were not reported or passed on, with potentially very serious consequences. As the BBC noted at the time, Excel’s XLS file format goes back to 1987 and was superseded by XLSX in 2007 which would have been able to handle sixteen times the number of cases had it been used.

Excel is, of course, not an inherently bad tool (in fact quite the opposite), but the problems created by technology application creep (the growing use of comparatively basic technology for increasingly complex tasks) can be compounded by the use of the tools themselves. And spreadsheets are the perfect environment for mistakes to compound and grow unchecked.

Matt Parker’s book ‘Humble Pi: A Comedy of Maths Errors’ is brilliant on this. He notes how The European Spreadsheet Risks Interest Group, an organisation set up to look at this problem, estimates that 90 per cent of all spreadsheets contain errors. The ‘horror stories’ page of their website contains a whole series of spreadsheet challenges and errors. These include the scientific body that is in charge of standardising the names of genes (the HUGO Gene Nomenclature Committee) needing to provide guidelines to scientists for originating names for newly identified genes that avoid problems created by Excel auto formatting (for example Excel changing a gene’s alphanumeric symbol of MARCH1 into the date 1-Mar).

Then there’s the spreadsheet mistake that delayed the opening of a new £150 million Scottish hospital and led to £16 million of remedial work needing to be done to make the critical care rooms fit for use. Or how about the spreadsheet input error that lost a state fund set up in Ireland to support jobs a total of 750,000 Euros because a number was wrongly input as Euros rather than Dollars. Or the errors on a couple of spreadsheets from a County Sheriff’s Office in the US that cost that particular county almost half a million dollars. The County Sheriff is quoted as saying at the time that ‘the spreadsheets were emailed back and forth…Because of some cutting and pasting, not all the formulas were pasted correctly. It was an unintended error’. The list goes on.

I’d find it hard to believe that everyone that was working with the Health New Zealand spreadsheet thought that it was a good idea that they did things in that way. But what often happens is that it begins with solving a particular need in haste with a under-qualified system and as more and more dependency builds on that system it becomes harder and harder to unpick.

Unintended application creep becomes emergent, leading to a kind of functional drift in which technologies are applied in ways that they were never intended. As we move at pace into the wide and deep application of AI, this is something we should all be wary of.

If you do one thing this week…

‘As AI improves exponentially at climbing strategic mountains, human wisdom becomes increasingly valuable in deciding which mountains to climb in the first place.’

A big YES to Adrian Barrow’s piece on how AI is rewriting the rules of brand strategy. It’s the best thing I’ve read on what AI means for the role of the strategist in a long time.

He uses the metaphor of a chef’s relationship with recipes as they move through their career. At the start the culinary student follows existing recipes meticulously. The mid-career chef modifies their techniques and may add in additional or different ingredients to improve, depending more on taste and context. The culinary master however, breaks from the recipe entirely and crafts unique dishes that have never been created before.

Similarly, the junior strategist learns how to rigorously apply classic frameworks and models. The more experienced strategist adapts these frameworks and models according to context and specific scenarios. And the strategic maestro develops singular and distinctive solutions to respond to specific challenges.

Current AI systems, says Adrian, excel at the initial stages of this process (faithfully applying existing rules), and are starting to become useful for the second (breaking from tradition) but are a long way from effectively accomplishing the third (using intuition to transcend formal structure and create entirely new thinking). As strategy changes from optimising within known boundaries to redefining the boundaries themselves, accumulated strategic experience is a foundation for sensing opportunities, opening new paths, seeing around corners.

The opportunity that every strategist has is to not only embrace the power of AI but to transcend its limitations: ‘Technical capability without cultural intuition produces efficient meaninglessness. Cultural understanding without technological leverage creates meaningful inefficiency.’ A fantastic read.

Photo by JESHOOTS.COM on Unsplash

Links of the week

• As predicted previously here, Google are testing an AI mode in search which integrates a Gemini chatbot-like interface into a search tab. Users can use natural language questions to explore complex topics and ask follow up queries. Search is changing pretty dramatically.

• According to WARC TikTok users now average 35 hours per month, and (assuming no ban in the U.S.) global advertising spend on TikTok this year should reach $32 billion taking it to an 11% share of global social spend. Crikey.

• Researchers at Bytedance (the Chinese owners of TikTok) have developed an AI tool called Omnihuman-1 that can generate extremely realistic human avatars from a single image and a motion signal like audio or video. There are some good examples of its capabilities here

• A Chinese startup called Monica.im has revealed ‘Manus’, a service it bills as a general AI agent – you can see it performing various tasks here

• If an agent can complete your online course in minutes what does this mean for the future of online learning? A great post from Dr Philippa Hardman – ‘the agentic AI problem isn’t an integrity or security problem—it’s a learning effectiveness problem’. I liked the visual comparing the online learning era and the agentic AI era in her LinkedIn post

• ‘Our Reflective AI is designed not to provide answers but to ask questions – questions that provoke thought, spark curiosity, and encourage exploration’. An tool designed to challenge you to think – Curiouser.ai looks interesting

• Some great advice on doing your own thing from (friend of ODF) Phil Adams on his 5th anniversary of working for himself. A few years back I wrote up a few of my own thoughts on this topic if you’d like to read those.

Quote of the week

HT Deborah Hewitt

And finally…

OpenAI have trained an AI model to be good at creative writing and this week Sam Altman shared a remarkable example of its output. Writer Jeanette Winterson had some fascinating reflections on how surprisingly beautiful and moving the story actually is, noting the potential that AI has to give us new perspectives on the world. Patricia MacDonald said it really well:

‘The short story itself is intriguing; the prompt is deftly chosen to explore the tension between a powerfully human emotion and an AI trained to recognise, describe, even evoke that emotion, but never experience it. The device shortcuts any of the “uncanny valley” we might expect by framing the story as the AI groping to articulate something it can never know, despite all the language in the world at its disposal. Which of course is a theme behind many great works of literature-the impossibility of language to fully capture and convey the writer’s experience. In this case the tension is reversed: the writer is as fluent and adept with language as anyone, but has no experience to convey.’

Photo by Unseen Studio on Unsplash

Weeknotes

This week I ran the IPA AI in Advertising course in London (a course I love doing, particularly F2F), and did some writing and prepping stuff. Next week I’ll be down in the South West walking on a few beaches – so there will be no Fish Food episode next week, and normal service will resume the week after. I then have a week or two before the work travel begins again so I’m making the most of being in one (or two) place/s.

Thanks for subscribing to and reading Only Dead Fish. It means a lot. This newsletter is 100% free to read so if you liked this episode please do like, share and pass it on.

If you’d like more from me my blog is over here and my personal site is here, and do get in touch if you’d like me to give a talk to your team or talk about working together.

My favourite quote captures what I try to do every day, and it’s from renowned Creative Director Paul Arden: ‘Do not covet your ideas. Give away all you know, and more will come back to you’.

And remember – only dead fish go with the flow.