Studying AI Alignment Ep2: When Complex Systems Fail
and what it means for AI
On the morning of October 26, 1992, the London Ambulance Service (LAS) launched a Computerised Dispatch System. This was the service's second attempt to replace a manual dispatch process, following an earlier failed effort from 1987 to 1990 that cost £7.5 million but never went live.
This second attempt failed even more spectacularly. Bugs appeared during the Monday morning rush of emergency calls, and the situation worsened quickly. By Tuesday afternoon, LAS shut down most of the system and used a hybrid of computerised call-taking and manual ambulance tracking. Seven days later, the system completely locked up on November 4 at 2 a.m. Rebooting was useless, and the backup system failed. Within nine days, LAS had reverted to a fully manual dispatch system.
Although the exact death toll was unclear, the press reported that as many as 30 people may have died due to delayed ambulances. Beyond the tragic human cost, there were significant financial, social, and political repercussions.
The core problem in the LAS case was not purely technical. Unrealistic timelines and budget constraints pressured the teams, who also suffered from poor communication and a lack of trust between management and staff. Relevant data wasn't collected properly, and staff were undertrained. They conducted only piecemeal testing and never did an end-to-end integration test. Financial constraints led them to hire the lowest bidder out of 35 contractors—despite that contractor's lack of relevant experience. In hindsight, the decision to go live seems ridiculous, yet it happened.
I came across this case during an exercise in the AI alignment course I was taking. Out of a list of instances where software failures caused significant harm, I chose the LAS example because I have lived in London for several years and was curious about what happened. Well, that was not exactly reassuring! In an era of AI advancements, we might assume that implementing such a computerised dispatch system would not be difficult 30 years ago. It is sobering to realise how many things could go so wrong in this case.
Patterns of Failure
In the LAS case, at least, there was no evidence of coverup or malicious intent by the people involved. However, not all complex system failures arise from innocent mistakes. The Boeing scandal, for instance, involved years of coverups that ultimately led to disastrous consequences. Among all the cases we discussed during the exercise, similar patterns occur over and over again: Due to tight deadlines, budget pressures, greed, fear, or other incentives, people make regrettable choices that lead to harmful outcomes.
It's not unreasonable to anticipate similar problems with AI deployments. In fact, they already happened. ChatGPT ignited the new wave of AI enthusiasm, but let’s not forget that many instances of algorithmic misuse and bias existed before ChatGPT with less capable models, as described in the books Weapons of Math Destruction and AI Snake Oil. Model developers sometimes overpromise, while implementers may prioritise cost savings over robust evaluation, resulting in discriminatory or harmful effects.
The risk escalates because AI may soon become extremely capable—and thus potentially highly harmful. In his blog post "How We Could Stumble Into AI Catastrophe," Holden Karnofsky outlines how transformative AI, capable of major scientific and technological breakthroughs, could be deployed unsafely due to measurement difficulties and competitive pressures. Such misaligned AI systems might develop goals at odds with human values, leading to catastrophic outcomes.
Similarly, the Future of Life Institute's YouTube video "Artificial Escalation" depicts a fictional scenario in which two military superpowers race to deploy AI in their defence systems. Although the model was promised to involve humans, when an accident occurs, everything happens too fast for humans to react before the two superpowers' AI automatically launches nuclear bombs at each other.
These speculated long-term existential scenarios have faced criticism for being unrealistic and potentially diverting attention from existing or short-term harm and biases. While I agree that we should not focus solely on existential risks at the expense of addressing current issues, both areas can coexist without competing for attention.
As interests in AI risks grow, it could bring additional resources and expertise that can benefit both short-term and long-term concerns. Regardless of whether we are considering immediate or existential risks, a significant human flaw remains: some decision-makers may act out of fear, greed, arrogance, or other motivations, leading them to deploy AI systems too rapidly and without proper safeguards. As a result, both immediate and existential risks are plausible. After all, the Cuban missile crisis nearly brought humanity to the brink of destruction, even without the influence of advanced and opaque AI systems.
Moving Forward Safely
So, how do we avoid these pitfalls? The obvious solution is to proceed carefully, conduct thorough testing, and invest in safety research. Yet, as the LAS story shows, "obvious" precautions are not always taken. We face similar risks now with AI.
Just in recent months, there have been worrying signs of AI risks:
OpenAI and Anthropic, two prominent AI companies, have partnered with US defence contractors to collaborate on US defence and intelligence projects. While defence initiatives are not inherently harmful—indeed, much scientific and technological progress arises from defence spending—the partnership between major AI firms and the defence industry in any country warrants careful observation and regulation.
At Trump’s inauguration in the United States, tech CEOs were granted unusually prominent seating, symbolizing their close ties to the new administration. Elon Musk played a significant role in Trump’s campaign, while Jeff Bezos allegedly discouraged Washington Post editors from endorsing Kamala Harris. Meanwhile, Mark Zuckerberg introduced content moderation policies on Meta platforms that appeared more aligned with Trump’s and Musk’s ideologies. This shift suggests that the self-described “beacon of democracy” is becoming increasingly oligarchic, which poses risks not only to democratic principles but also to AI safety. Compounding these concerns, both Trump and Musk have openly opposed regulation, casting doubt on the prospects of effectively overseeing AI companies.
Recent news of DeepSeek highlighted China's advancements in AI despite US controls. Dario Amodei, the CEO of Anthropic, wrote a blog post with a central argument that "the US and other democracies must lead in AI," reflecting a cold-war mentality that likely resonates with many hawks. However, this mindset could foster distrust and arms races—precisely what we should strive to avoid for AI safety. If trust and collaboration break down, the likelihood of rushed deployments increases.
Fortunately, there are encouraging developments.
The International AI Safety Report was just published, representing international attention and effort in the area of AI safety (At least from the look of it; I haven't had a chance to read it thoroughly).
Multiple countries have been setting up their AI Safety Institutes. US and UK AI Safety Institutes have been involved in the pre-deployment evaluation of Anthropic's Upgraded Claude 3.5 Sonnet and OpenAI's o1 Model.
These are promising signs that some oversight mechanisms are starting to form. Still, as the Boeing case reminds us, self-regulation by companies is rarely sufficient. Governmental institutions are necessary, but we can (and should) do more. An essential force to involve is academia. Academic researchers need better access to data, models, and computing resources. Unlocking these resources responsibly can help identify and mitigate AI risks while advancing scientific understanding.
A Personal Reflection
As I went through the AI alignment course, I really enjoyed the materials. Still, I struggled to see how I could contribute to the field as a computational social scientist. One issue is the limited involvement of social scientists' perspectives. While social scientists can study the impact of AI on society and related policies, much less attention has been paid to examining the design and evaluation of AI models from social science perspectives. One reason for this may be limited access to data. For instance, having access to the kinds of conversations people are having with ChatGPT would be a gold mine for social science research. The findings from such research could inform various aspects, including AI development and evaluation, AI policy, and our broader understanding of human behaviour. As obtaining this kind of proprietary data could be very difficult, we might need to come up with other ways to contribute. That’s something I could explore as my next step.
Regardless of what I do, the broader lesson remains: whether it's a computerised ambulance dispatch system in 1992 or a cutting-edge AI system today, rushing development without proper safeguards can have disastrous consequences. By learning from history and engaging a diverse range of experts, including social scientists, we have a better shot at ensuring AI's benefits outweigh its risks.
P.S. I also translated this article into Chinese with the assistance of ChatGPT. Check here if you are interested.