Training Example: Eliezer Yudkowsky – Review the Data, Give Your Score & Compare to the Real AI Evaluation

[H2] Welcome to the website of a man who wears more than one hat.
One Eliezer Yudkowsky writes about the fine art of human rationality. Over the last few decades, science has found an increasing amount to say about sanity. Probability theory and decision theory give us the formal math; and experimental psychology, particularly the subfield of cognitive biases, has shown us how human beings think in practice. Now the challenge is to apply this knowledge to life – to see the world through that lens. That Eliezer Yudkowsky’s work can be found in the Rationality section.
The other Eliezer Yudkowsky concerns himself with Artificial Intelligence. Since the rise of Homo sapiens, human beings have been the smartest minds around. But very shortly – on a historical scale, that is – we can expect technology to break the upper bound on intelligence that has held for the last few tens of thousands of years. Artificial Intelligence is one of the technologies that potentially breaks this upper bound. The famous statistician I. J. Good coined the term “intelligence explosion” to refer to the idea that a sufficiently smart AI would be able to rewrite itself, improve itself, and so increase its own intelligence even further. This is the kind of Artificial Intelligence I work on. For more on this see the Singularity tab. (Yes, I know, the term is obsolete and no longer means what it did, I need to get around to reorganizing this website.)
My short fiction, miscellaneous essays, and various other things can be found under Other.
I am a full-time Research Fellow at the Machine Intelligence Research Institute, a small 501(c)(3) public charity supported primarily by individual donations.
A majority of all my written material is currently stored at the community blog Less Wrong, after being originally produced on the blog Overcoming Bias. (I found that I could write much faster in a blog format.) Those wishing to read through should use the indices found here.
My parents were early adopters, and I’ve been online since a rather young age. You should regard anything from 2001 or earlier as having been written by a different person who also happens to be named “Eliezer Yudkowsky”. I do not share his opinions.
Though I have friends aplenty in academia, I don’t operate within the academic system myself. (For some reason, I write extremely slowly in formal mode.) I am not a Ph.D. and should not be addressed as “Dr. Yudkowsky”.
If you’re having trouble finding something, I suggest consulting the Sitemap.
[H2] POPULAR ESSAYS:
Cognitive biases potentially affecting judgment of global risksArtificial Intelligence as a positive and negative factor in global riskAn Intuitive Explanation of Bayesian ReasoningThe Cartoon Guide to Lob’s Theorem
[H2] FEATURED ESSAY:
[H3] Twelve Virtues of Rationality
The first virtue is curiosity. A burning itch to know is higher than a solemn vow to pursue truth. To feel the burning itch of curiosity requires both that you be ignorant, and that you desire to relinquish your ignorance.

Over the last several decades, science has developed a new picture of how we succeed or fail to seek true beliefs and achieve our goals. The heuristics and biases program in cognitive psychology has exposed dozens of major flaws in human reasoning, and along with them, insight into the mental processes that produce these flaws. Bayesian probability theory and decision theory have been used in increasingly powerful statistical methods and machine learning algorithms; I believe that they also provide a lens through which to view everyday life. Evolutionary psychology and social psychology are also telling us new and disturbing things about human nature – can we, now knowing, do any better?
The vast bulk of my writings on rationality are contained on the Less Wrong community blog – follow the link to see some of my recommended posts. Hopefully that corpus will be processed into wiki pages and e-books in the near future, followed by an introductory-level physical book – to be notified when (e-)books come out, follow the Subscribe link in the right-hand column.
[H2] Essays currently available:
Twelve Virtues of RationalityThe first virtue is curiosity. A burning itch to know is higher than a solemn vow to pursue truth. To feel the burning itch of curiosity requires both that you be ignorant, and that you desire to relinquish your ignorance.Overcoming BiasCognitive Biases Potentially Affecting Judgment of Global RisksIntroduces the field of heuristics and biases (the experimental investigation of systematic human errors and what they reveal about human cognition) from the perspective of how known biases may throw off our reasoning about uncertain risks to the human species.The Simple TruthWhat is “truth”? It’s surprisingly simple.An Intuitive Explanation of Bayes’ TheoremBayes’ Theorem for the curious and bewildered; an excruciatingly gentle introduction.A Technical Explanation of Technical ExplanationMore Bayes. Many of my other writings rely on this page.(The Cartoon Guide to) Lob’s TheoremLob’s Theorem shows that Peano Arithmetic can never assert its own soundness. I prove this amazing theorem using the standard mathematical technique of cartooning.

The term “Singularity” had a much narrower meaning back when the Machine Intelligence Research Institute was founded. Since then the term has acquired all sorts of unsavory connotations. The kind of Singularity I work on has little to do with Moore’s Law. So forget the word; here’s the issue:
Since the rise of Homo sapiens , human beings have been the smartest minds around. But very shortly – on a historical scale, that is – we can expect technology to break the upper bound on intelligence that has held for the last few tens of thousands of years. Artificial Intelligence is one of the technologies that potentially breaks this upper bound.
The famous statistician I. J. Good coined the term “intelligence explosion” to refer to the idea that a sufficiently smart AI would be able to rewrite itself, improve itself, and so increase its own intelligence even further – a positive feedback cycle that would shoot upward and arrive at superintelligence, something far more capable than a human.
If you offered Gandhi a pill that made him want to kill people, he would refuse to take it, because he knows that then he would kill people, and the current Gandhi doesn’t want to kill people. This, roughly speaking, is an argument that minds sufficiently advanced to precisely modify and improve themselves, will tend to preserve the motivational framework they started in. The future of Earth-originating intelligence may be determined by the goals of the first mind smart enough to self-improve.
My mid-range long-term research program is to work out a formal theory of such matters – describe how a mind would modify itself deterministically and precisely, including modifications to the part of itself that does the modifying.
A good deal of the material I have ever produced – specifically, everything dated 2002 or earlier – I now consider completely obsolete.
Three Major Singularity SchoolsSingularity discussions seem to be splitting up into three major schools of thought: Accelerating Change, the Event Horizon, and the Intelligence Explosion.Artificial Intelligence as a Positive and Negative Factor in Global RiskHow advanced artificial intelligence relates to global risk as both a potential catastrophe and a potential solution. Contains considerable background material in cognitive sciences, and conveys much of my most recent views on intelligence, AI, and Friendly AI.The Power of IntelligenceIn our skulls we carry around 3 pounds of slimy, wet, greyish tissue, corrugated like crumpled toilet paper. You wouldn’t think, to look at the unappetizing lump, that it was some of the most powerful stuff in the known universe.Transhumanism as Simplified HumanismIf you believe professional bioethicists (people who get paid to explain ethical judgments) then the rule “Life is good, death is bad; health is good, sickness is bad” holds only until some critical age, and then flips polarity. Why should it flip? Why not just keep on with life-is-good?5-Minute Singularity IntroThis is a 5-minute spoken introduction to the Singularity I wrote for a small conference. I had to talk fast, though, so this is probably more like a 6.5 minute intro.The AI-Box ExperimentWhen we build AI, why not just keep it in sealed hardware that can’t affect the outside world in any way except through one communications channel with the original programmers? That way it couldn’t get out until we were convinced it was safe. Right?A Theory of FunHow much fun is there in the universe? What is the relation of available fun to intelligence? What kind of emotional architecture is necessary to have fun? Will eternal life be boring? Will we ever run out of fun? To answer questions like these requires Singularity Fun Theory.
[H2] Older Essays
Levels of Organization in General IntelligenceBook chapter I wrote in 2002 for an edited volume, Artificial General Intelligence, which is now supposed to come out in late 2006. I no longer consider LOGI’s theory useful for building de novo AI. However, it still stands as a decent hypothesis about the evolutionary psychology of human general intelligence.Coherent Extrapolated VolitionTakes a stab at saying what we might wish to do with a Friendly AI if we had the technical knowledge to build one.

Short fiction, assorted essays, miscellaneous things.
Yehuda YudkowskyTranshumanists are not fond of death. We would stop it if we could.FictionWhile I tend to publish most of my writing for free, I strongly believe that money is not evil. Therefore, anyone is welcome to take characters or settings from my original online fiction, such as the beisutsukai or the Baby-Eating Aliens, and use them in new commercial works of your own creation. I do ask for acknowledgment and a link or other reference to the original, but so long as the writing is your own, you may charge for access, distribute printed copies, sell the story to a magazine, etc. I don’t mind.