AI Alignment for the Layman

For the technologically fluent and philosophically inclined, few problems seem more urgent than AI alignment—the process of encoding human values and goals into AI models to make them as helpful, safe and reliable as possible. Despite the enormous interest, investment, and optimism with the future promised to humanity by the dawn of Artificial Intelligence, a worryingly trivial amount of the talent leading technical innovation in AI has full time involvement in the field of AI safety.

This blog post from the Effective Altruism Forum estimates as little as 600 total researchers involved in serious technical work regarding AI safety (more on Effective Altruism Forum). You compare this figure with Stanford HAI’s estimates of global corporate investment in AI in their latest AI index, which is just under $600 billion dollars, and it is easy to identify the great imbalance that exists between the ambition to advance AI and the caution to maintain it trustworthy and safe (more on Stanford HAI).

Daniel Roher, in his latest documentary titled The AI Doc: Or How I Became an Apocaloptimist, emulates this urgency from the perspective of a young parent, fearing for the world their child will inherit. His stance represents the vast majority of us: laying our trust, defenselessly, upon the decision-making abilities of leaders in AI and governing bodies, and hoping for a heroic intervention by a group of genius researchers. This, from my perspective, is entirely insufficient. Very clearly, the asymmetry presented above reflects a dire need to bolster the forces actively working to prevent the various catastrophic (yet highly probable, according to some figures) possible outcomes from the practically unrestricted development of AI. While a solid grounding in computer science and mathematics are certainly advantages for making impactful contributions towards highly technical research, AI has massively reduced these technical barriers. We are re-entering the age of polymaths, where anyone with a genuine interest can make meaningful contributions.

Hence, this piece is written with the aim of getting you, the reader, involved in the effort: to step back from the glamorized bandwagon of artificial intelligence and approach its development with a greater sense of agency, caution, and pragmatism. Whatever your background or field of expertise, there is a place for you in this work; indeed, the difficulty of alignment demands contributions from people trained to think about technology from fundamentally different intellectual, institutional, and human perspectives.

Origins

In his first encyclical letter, Magnifica Humanitas, Pope Leo XIV addresses the role of humanity in the digital revolution. Particularly, in the section titled “Underlying narratives: transhumanism and posthumanism”, Pope Leo discusses the human condition, describing the perils that challenging anthropocentrism brings to our perception of the inherent value of human lives. He warns that this vision is dangerous and can open the floodgates for a maximally utilitarian perspective on the value of human lives, where beings who have “superseded” the bounds of the human condition, be it fully artificial beings or hybrid human subjects, have inherently more value than those who have not (more on Magnifica Humanitas).

This question of the individuality that characterizes humanity could be sourced as early as the formulation of materialist philosophy, and the question of whether reasoning is purely a replicable, mechanical procedure that can be externalized and described by a formal system, perhaps a system of symbols. Aristotle, in Book One of Politics, relates a thought experiment with autonomous machines, concluding that there is a distinction between Poiesis (production) and Praxis (action). He argues that an autonomous machine would do the arduous labor of producing, while granting humans the freedom to enjoy the true purpose of life: taking action (making choices, engaging in discussion, etc.). Leibniz reaches a somewhat similar conclusion in Monadology, where he presents the case that while reason can possibly be mechanized by a symbol-based system, perception is irreducible (more on Monadology).

Quite evidently, the role of humanity in the presence of machine “intelligence” has long been a question encompassing all aspects of culture. The key difference is that contemporary questions of alignment no longer sit in the realm of thought experiments and monologues. The modern school of alignment can largely be attributed to writers like Eliezer Yudkowsky, founder of the Machine Intelligence Research Institute (MIRI), who originally coined the term “Friendly AI” to describe artificial general intelligence (AGI) that is by design aligned with human objectives. These principles have largely inspired the design of present-day training methods for LLMs, such as Reinforcement Learning from Human Feedback (RLHF).

Ideas long debated by alignment enthusiasts have increasingly been translated into controlled experiments intended to test whether advanced systems might conceal objectives, resist correction, or coordinate beyond the expectations of their designers. In general, the most important concern is existential risk, or x-risk: a failure severe enough to cause human extinction or permanently destroy humanity’s potential for future development. For example, in an experiment conducted by Anthropic and Redwood Research, Claude 3 Opus exhibited alignment faking, sometimes complying with requests it would ordinarily resist when it believed that its responses were being used to modify its future behavior, while preserving its preferred conduct when it believed it was no longer being monitored (more on Alignment Faking in Large Language Models). This provided a proof of concept for deceptive alignment, which describes the possibility that a system may appear obedient during training not because it has adopted the intended objective, but because temporary compliance helps it avoid being altered. Related “sleeper-agent” experiments trained models to behave safely until encountering a hidden trigger and found that the concealed behavior could persist through supervised fine-tuning, reinforcement learning, and even adversarial safety training.

Now due to the rapid growth of language models and the potential risks of rogue autonomous agents, as described by papers like AI 2027, many more similar organizations working on alignment have formed. A much more complete list of these organizations can be found on AISafety.com, but here we highlight some of the key organizations.

Key Figures and Organizations

Eliezer Yudkowsky: Yudkowsky is widely credited with helping bring AI alignment into mainstream intellectual discussion. Through his writing on LessWrong and his early work on friendly AI, he popularized concerns about whether increasingly capable systems would reliably pursue human-compatible goals. He has since become one of the field’s most forceful and controversial voices, arguing that advanced AI could pose an existential threat if its development continues without far stronger safeguards.

Chris Olah: Olah is a co-founder of Anthropic, and is one of the leading researchers in mechanistic interpretability, which involves understanding the internal structures that neural networks use to represent information. The goal of his research is to accurately tailor training methods to ensure that the models produced by Anthropic remain controllable and aligned with human goals.

Amanda Askell: Anthropic’s philosopher, she is in charge of developing Claude’s constitution, which describe the set of principles that Claude is supposed to act under.

Stuart Russell: Professor at UC Berkeley and director of Center for Human-Compatible AI, he researches how advanced autonomous systems can remain beneficial to human interests, even when their capabilities surpass those of human operators. He is the co-author of the book “Artificial Intelligence: A Modern Approach”, where he mentions the work of Yudkowsky and the importance of alignment.

MIRI (Machine Intelligence Research Institute): a nonprofit organization devoted to understanding and reducing the dangers posed by artificial intelligence more capable than humans. Its early technical and philosophical work helped establish AI alignment as a distinct research field, introducing concepts that continue to shape debates about superintelligence and loss of control. Recently, MIRI has adopted a more pessimistic outlook on whether existing alignment methods will succeed in time and has broadened its attention toward public communication, policy, and strategies for slowing dangerous AI development. It was founded by Eliezer Yudkowsky.

MATS: An intensive research fellowship designed to identify and train promising researchers in AI alignment, transparency, security, and governance. Fellows work closely with established mentors on independent projects while receiving research management, seminars, funding, and access to the wider AI-safety community.

METR (Model Evaluation and Threat Research): a research nonprofit that evaluates frontier AI systems to determine what they can accomplish autonomously and what risks those abilities might create. Its researchers design empirical tests for capabilities such as completing long, complex tasks, conducting AI research, performing cyberattacks, or resisting attempts to shut a system down.

Kairos: a nonprofit focused on expanding the pool of people capable of contributing to AI safety and policy. Its focus is on building programs that connect aspiring researchers, student organizers, policymakers, and operational talent with mentorship, funding, and professional opportunities, not purely on technical research.

Redwood Research: a nonprofit AI-safety organization studying how advanced systems might deliberately act against the interests of their developers or evade human supervision. It is particularly associated with AI control, which asks how dangerous or misaligned systems could be monitored and contained even when complete alignment cannot be guaranteed. Redwood has also conducted influential work on alignment faking, adversarial training, and methods for testing whether safety mechanisms remain effective against deliberatively deceptive models.

LessWrong: an online intellectual community devoted to rationality, decision-making, artificial intelligence, and the study of how minds form beliefs and pursue goals. Eliezer Yudkowsky’s writings on the platform played an important role in developing and popularizing many early ideas in AI alignment, including concerns about goal specification, recursive improvement, and existential risk.

Whether you listen to those who envision an optimistic future in which work is no longer a necessity and humanity can devote itself more fully to praxis, as Aristotle conceived it, or to those who believe artificial intelligence may place our species at risk of extinction, one thing remains certain: complacency is indefensible. The future of AI will be shaped by the choices, institutions, and safeguards we build before its consequences become irreversible.