A minor point about instrumental convergence that I would like feedback on

• AI News Team
• February 27, 2026

​PreambleMy current understanding: the EY/MIRI perspective is that superintelligent AI will invariably instrumentally converge on something that involves extinguishing humanity. I believe I remember a tweet from EY saying that he would be happy with building ASI even if it only had a 10% chance of working out and not extinguishing humanity.Further understanding: This perspective is ultimately not sensitive to the architecture of the AI in question. I’m aware that many experts do view different types of AI as more or less risky. I think I recall some discussion years ago where people felt that the work OpenAI was doing was ultimately more dangerous / harder to align than DeepMind’s work. So as I understand it EY has the strongest possible view on instrumental convergence: It will definitely happen, no matter how you build ASI. Ever since I encountered this strong view, as I understood it many years ago, I have considered it trivially false.HypotheticalImagine a timeline where society arrived towards ASI via something much closer to bio-simulation or bio-engineering. That is to say, machine learning / neural nets / training were not really relevant.Maybe this society first arrives at whole human brain emulation — something that ultimately would surely be materially possible to do eventually. Maybe this happens on a big cube of silicon wafers, maybe it is fleshier, who knows.I think it seems clear that this non-super AI would not “instrumentally converge” any more than you or I. But I think that you could definitely arrive at ASI from this as a building block. Maybe you have 100+ emulated human brains that have a very high bandwidth connection between them, running on a substrate that actually carries signal much faster than our own brains. That entity would surely be a lot smarter than any human ever! Maybe through study of this emulated brain we learn, through a relatively crude process, ways of hacking together bigger and bigger brains (what happens if we copy paste this chunk a few times and hook it up here?) that ultimately turns out to be a scale efficiently enough to be 1000x smarter when you make a really really big brain.Relatedly, maybe such a hacked-together megamind starts out undeveloped but learns and grows as neurons self-organize in a similar manner to how they do over a human adolescence. So this megamind becomes way way smarter, but it happens in the same kind of rate that humans grow up. Or maybe the human brain development process scales really poorly with size, such that megamind takes 500 years to mature. (This is not really related to instrumental convergence I just thought it was interesting to imagine a type of slow takeoff scenario)Etc. The point is that starting with human brain emulation seems like it could definitely lead to super intelligence while also having no particular reason to instrumentally converge on preferences / values / behaviors completely alien to humans.—I have not shared this idea because I thought it was obvious but I have also never seen anyone else say it (I didn’t look that hard).So my questions:- Is this idea novel to you? – Is this an idea that EY has replied to?- Do you like my idea? please let me know.Post script: I do wonder if next-token prediction is ultimately similar enough to the evolutionary forces that created human intelligence that it creates something ultimately quite capable of empathizing with humans. I am sure people have commented on this. Thanks Discuss ​Read More