Sometimes you come across common threads in unexpected, disparate places. I was listening to the Alignment Newsletter Podcast when I came across the idea that artificial intelligence safety is a many-to-many problem with many AI and human agents. This idea is further explored in The Age of Em.

Then I listened to the audiobook version of Deep Survival and learned about disaster siatuations in complex systems. That got me to think more about disasters - more than I already did - and to try to tie it together in some way, since disasters are in everyone’s radar these days.

What allowed me to wrap my head around disaster prevention is by reading the book Engineering a Safer World which tackles safety-critical engineering from a systems perspective. It provides a theoretical yet practical guideline on how to analyze and prevent disasters in complex systems.

Finally, I played the game Tacoma, a sci-fi exploration game which featured a disaster scenario exacerbated by AI. It seemed like the perfect case study in a time where real disaster case studies would be too depressing.

I weaved together these tangentially related threads in an op-ed for the University of Rochester student newspaper Campus Times. I am thrilled that it somehow managed to come out as a cohesive article after weeks of research, writing, and the editor’s wordsmithing.


[1] Fullbright Games. 2016. Tacoma. Retrieved from

[2] Katja Grace, John Salvatier, Allan Dafoe, Baobao Zhang, and Owain Evans. 2017. When Will AI Exceed Human Performance? Evidence from AI Experts. Arxiv (2017).

[3] Rohin Shah. [AN #118]: Risks, solutions, and prioritization in a world with many AI systems. Alignment Newsletter. Retrieved October 5, 2020 from

[4] Eliezer Yudkowsky. AI Alignment: Why It’s Hard, and Where to Start. Machine Intelligence Research Institute. Retrieved October 20, 2020 from

[5] Robert Miles, Computerphile. 2017. AI? Just Sandbox it… Youtube. Retrieved October 20, 2020 from

[6] Robin Hanson. 2016. The Age of Em. Oxford University Press. Retrieved from

[7] Nancy G Leveson. 2012. Engineering a Safer World: Systems Thinking Applied to Safety. MIT Press. Retrieved October 4, 2020 from