Opinion

Are there Multiple Moral Endpoints?

​This is a different approach to explaining an old idea.What is the deep future going to look like? Will we be proud of it, or ashamed of the choices that led to it?Lots of focus on the future is on the near future. How will ongoing wars go? What will the next AI model’s capabilities be? Will this business succeed or fail? Let’s zoom out and focus just on advanced artificial intelligence; my guess is that we’ll have a “transition period” with many different relevant actors with different philosophies and moral considerations (we’re in it) and then end up in an “equilibrium period” with much more homogeneity of philosophy and moral standpoints.[1] The transition period is hard to predict and involves many contingent facts, and might involve lots of dramatic turnovers; the equilibrium period, in contrast, is likely much more stable.Mass Effect is a series of video game RPGs which involved significant impacts downstream of the player’s choices. In 2012, the trilogy conclusion was released and widely disliked because the player’s final choice led to endings were too similar and mostly paved over the impact of all of the player’s other choices. As mocked in this image, the background lighting color was different, and not much else.Maybe I should take a step back and try to explain a bit what I mean by philosophies and moral considerations to try to build towards the idea of a moral endpoint. By ‘philosophy’ I might mean something more like ‘ideology’; it’s a way of looking at the world, a set of metrics for considering ideas, some axioms and precepts, and consequently, a set of judgments about policies and actions. If you think non-human animal experiences are morally relevant, then you have different regulations for agribusiness than if you think otherwise; the question of animal welfare is downstream of other questions. In a large complicated world with many different factors, people can reach similar conclusions for nearly totally different reasons, or be attending to different tradeoffs that lead them to make very different conclusions despite having very similar preferences.Moral and intellectual progress seems possible to me, both for individuals and societies. I definitely am wiser and kinder than I was in the past, and imagine that I will be yet wiser and kinder in the future. This can happen both because of increased experience of the world or arguments and thoughts about how ideas and principles connect with each other. We can thus imagine a ‘moral endpoint’, or a point that from which no further progress is possible, according to that point.[2]In particular, one of the reasons I think progress happens is that some things are pinned down by reality. Empirically, some policies work and others fail; entities that are paying attention will abandon the ones that fail and keep the ones that work. My favorite example here is the Chinese government telling the Cuban government to do market-oriented reforms. If markets are the correct solution to a problem, then any non-market solution to that problem is likely transient and will eventually be replaced. If a position is fundamentally confused about reality, it is unlikely to be a moral endpoint, unless it has also confused its system for correcting itself in a way that preserves that confusion.Some things are pinned down by other choices. There’s an ancient debate in Chinese philosophy between two Confucians; Mencius argues that humans are fundamentally good. Xunzi argues that humans are fundamentally bad. What’s interesting to me is that this could be viewed as an empirical question to be settled by looking more closely at the humans, but Xunzi’s argument rests instead on coherency: the whole point of Confucianism is to use the rites to move humans further away from their base tendencies and towards the ideal. What it means for humans to be fundamentally good, from Xunzi’s point of view, is that their base tendencies are good, at which point you are centering the humans instead of the rites. It’s a coherent position, but it’s fundamentally not Confucian (let’s call it Taoist instead).This presents a simple two-by-two matrix, with two coherent corners and two incoherent corners. If you think humans are good and the rites are irrelevant, that makes sense; the rites, by moving people from their natural state, make things worse. If you think humans are bad and the rites are relevant, that makes sense: the rites, by moving people towards an ideal state, make things better. But to believe that humans are bad and the rites are irrelevant is fatalism or unreachable standards (what does it mean for humans to be bad if no advice makes them better?), and to believe that humans are good and the rites are relevant is confusion about ‘what goodness is’ or setting the standards too low (what does it mean for humans to be good if the rites are the guide to use whenever they disagree?).In this case, you might think there’s an empirical question that will eventually settle which of the two corners is the ‘correct’ corner. Maybe, with advanced enough science, all of the Confucians become Taoists, or all the Taoists become Confucians. But possibly there are disagreements which are fundamentally about preferences instead of beliefs. If you think self-expression is one of the highest virtues, it would be astonishing for you to end up Confucian instead of Taoist; if you think perfection is one of the highest virtues, it would be astonishing for you to end up Taoist instead of Confucian.[3]If a superintelligent Taoist could look at themselves and the world and say “yes, I should stay a Taoist”, and a superintelligent Confucian could look at themselves and the world and say “yes, I should stay a Confucian”, and the deep future created by a Taoist and the deep future created by a Confucian are meaningfully different, then there are multiple moral endpoints, and the value of the deep future to us will depend on which of those moral endpoints emerges from the transition period.[4]Note that if we created a graph of moral, empirical, and philosophical issues, some of them would be connected (like how to rate human nature and the rites) and some of them would be disconnected (or only very tenuously connected, thru many intermediate links). Some things are basically irrelevant and disconnected from the graph; an America which had picked slightly different colors for its flag would not end up lionizing the “red, white, and blue” but would likely be roughly the same country, even after 250 years of developing with slightly different values. It may be that widespread spectator sports are a well-adapted solution to an important problem, while the underlying features of the sport (football, or association football?) or the momentary winners (who’s world champion this year?) don’t impact the final analysis.If enough of philosophy is pinned down by empirical facts, and the future involves enough pressures towards good decision-making that systems cannot preserve their unique ‘flaws’ into the future, then we might end up with a single final consistent configuration, with all possible variations just ‘which color the background light is’. In such worlds, concerns about the deep future are misplaced; it’s going to be whatever it was going to be anyway.But I would be surprised to learn that there’s only moral endpoint, since I suspect there are no universally compelling arguments. If nothing else, it seems likely to me that the empty endpoint (of thinking that life should not exist anywhere in the universe) can be downstream of a suffering-minimizing “negative utilitarianism” that is internally coherent, and also there exists at least one positive endpoint, which thinks that there should be lots of happy flourishing life, because the joys of life outweigh its sorrows. Maybe all variation in human beliefs and moral intuitions is downstream of our ignorance and will be smoothed away by our civilization’s intellectual development, but I suspect that some of it does reflect durable disagreements about The Good that will survive our adolescence turning into adulthood.[5]^If you believe the Metamodernists, something like this has already happened in Western politics, where there’s still lots of disagreement between factions but the set of things that “all factions allowed in polite society agree on” has grown significantly. When Freinacht was writing, being openly racist would get you kicked out of the Republican Party, such that ‘anti-racism’ was much more like “a universal belief” than a factional affiliation. I think their analysis is probably premature, but that advanced AI does actually make uniformity more likely here.^”What about a stable cycle?” you ask. Suppose Rock thinks Paper is better than it, and Paper thinks that Scissors is better than it, and Scissors thinks that Rock is better than it. You’re not going to get a single equilibrium, and instead constant cycling. I think it’s fine to consider that cycle as an equivalence class and single point; Rock doesn’t expect itself to ever leave its current ring. If you have a situation where the cycles are not small and easily compressible into points, then the basic idea of ‘progress’ breaks down and we need a different model.^Note for the second half of this sentence I am using “Taoism” to point towards the “humans good, rites bad” cluster. The true Taoism is much more complicated and points to something much more empirical in a way that I think can be very perfection-flavored. ^Or what cluster of mutually-agreeable moral endpoints emerges, and the balance between the endpoints in that cluster. Even if we end up with a metamodernist future, where every faction is at peace with every other faction and none of them are doing things the others consider despicable, the factions having different ideas of The Good and different amounts of resources to devote to realizing their vision of The Good will lead to different levels of pride from us looking forward, if we have differing levels of agreements with those factions. And how many collections of mutually-agreeable moral endpoints are there?^Even if there are multiple moral endpoints, all significant competitors might be in the same basin. It might be that American society and Chinese society both would create AIs that develop towards the same endpoint, while Nazi society would have created an AI that develops towards a different endpoint, and so attempts by Americans to race against Chinese developers don’t actually make a difference for the deep future. But my current guess is that we’re actually close to the dividing lines between endpoints, and so efforts to become wiser here matter significantly.Discuss ​Read More

Are there Multiple Moral Endpoints?

​This is a different approach to explaining an old idea.What is the deep future going to look like? Will we be proud of it, or ashamed of the choices that led to it?Lots of focus on the future is on the near future. How will ongoing wars go? What will the next AI model’s capabilities be? Will this business succeed or fail? Let’s zoom out and focus just on advanced artificial intelligence; my guess is that we’ll have a “transition period” with many different relevant actors with different philosophies and moral considerations (we’re in it) and then end up in an “equilibrium period” with much more homogeneity of philosophy and moral standpoints.[1] The transition period is hard to predict and involves many contingent facts, and might involve lots of dramatic turnovers; the equilibrium period, in contrast, is likely much more stable.Mass Effect is a series of video game RPGs which involved significant impacts downstream of the player’s choices. In 2012, the trilogy conclusion was released and widely disliked because the player’s final choice led to endings were too similar and mostly paved over the impact of all of the player’s other choices. As mocked in this image, the background lighting color was different, and not much else.Maybe I should take a step back and try to explain a bit what I mean by philosophies and moral considerations to try to build towards the idea of a moral endpoint. By ‘philosophy’ I might mean something more like ‘ideology’; it’s a way of looking at the world, a set of metrics for considering ideas, some axioms and precepts, and consequently, a set of judgments about policies and actions. If you think non-human animal experiences are morally relevant, then you have different regulations for agribusiness than if you think otherwise; the question of animal welfare is downstream of other questions. In a large complicated world with many different factors, people can reach similar conclusions for nearly totally different reasons, or be attending to different tradeoffs that lead them to make very different conclusions despite having very similar preferences.Moral and intellectual progress seems possible to me, both for individuals and societies. I definitely am wiser and kinder than I was in the past, and imagine that I will be yet wiser and kinder in the future. This can happen both because of increased experience of the world or arguments and thoughts about how ideas and principles connect with each other. We can thus imagine a ‘moral endpoint’, or a point that from which no further progress is possible, according to that point.[2]In particular, one of the reasons I think progress happens is that some things are pinned down by reality. Empirically, some policies work and others fail; entities that are paying attention will abandon the ones that fail and keep the ones that work. My favorite example here is the Chinese government telling the Cuban government to do market-oriented reforms. If markets are the correct solution to a problem, then any non-market solution to that problem is likely transient and will eventually be replaced. If a position is fundamentally confused about reality, it is unlikely to be a moral endpoint, unless it has also confused its system for correcting itself in a way that preserves that confusion.Some things are pinned down by other choices. There’s an ancient debate in Chinese philosophy between two Confucians; Mencius argues that humans are fundamentally good. Xunzi argues that humans are fundamentally bad. What’s interesting to me is that this could be viewed as an empirical question to be settled by looking more closely at the humans, but Xunzi’s argument rests instead on coherency: the whole point of Confucianism is to use the rites to move humans further away from their base tendencies and towards the ideal. What it means for humans to be fundamentally good, from Xunzi’s point of view, is that their base tendencies are good, at which point you are centering the humans instead of the rites. It’s a coherent position, but it’s fundamentally not Confucian (let’s call it Taoist instead).This presents a simple two-by-two matrix, with two coherent corners and two incoherent corners. If you think humans are good and the rites are irrelevant, that makes sense; the rites, by moving people from their natural state, make things worse. If you think humans are bad and the rites are relevant, that makes sense: the rites, by moving people towards an ideal state, make things better. But to believe that humans are bad and the rites are irrelevant is fatalism or unreachable standards (what does it mean for humans to be bad if no advice makes them better?), and to believe that humans are good and the rites are relevant is confusion about ‘what goodness is’ or setting the standards too low (what does it mean for humans to be good if the rites are the guide to use whenever they disagree?).In this case, you might think there’s an empirical question that will eventually settle which of the two corners is the ‘correct’ corner. Maybe, with advanced enough science, all of the Confucians become Taoists, or all the Taoists become Confucians. But possibly there are disagreements which are fundamentally about preferences instead of beliefs. If you think self-expression is one of the highest virtues, it would be astonishing for you to end up Confucian instead of Taoist; if you think perfection is one of the highest virtues, it would be astonishing for you to end up Taoist instead of Confucian.[3]If a superintelligent Taoist could look at themselves and the world and say “yes, I should stay a Taoist”, and a superintelligent Confucian could look at themselves and the world and say “yes, I should stay a Confucian”, and the deep future created by a Taoist and the deep future created by a Confucian are meaningfully different, then there are multiple moral endpoints, and the value of the deep future to us will depend on which of those moral endpoints emerges from the transition period.[4]Note that if we created a graph of moral, empirical, and philosophical issues, some of them would be connected (like how to rate human nature and the rites) and some of them would be disconnected (or only very tenuously connected, thru many intermediate links). Some things are basically irrelevant and disconnected from the graph; an America which had picked slightly different colors for its flag would not end up lionizing the “red, white, and blue” but would likely be roughly the same country, even after 250 years of developing with slightly different values. It may be that widespread spectator sports are a well-adapted solution to an important problem, while the underlying features of the sport (football, or association football?) or the momentary winners (who’s world champion this year?) don’t impact the final analysis.If enough of philosophy is pinned down by empirical facts, and the future involves enough pressures towards good decision-making that systems cannot preserve their unique ‘flaws’ into the future, then we might end up with a single final consistent configuration, with all possible variations just ‘which color the background light is’. In such worlds, concerns about the deep future are misplaced; it’s going to be whatever it was going to be anyway.But I would be surprised to learn that there’s only moral endpoint, since I suspect there are no universally compelling arguments. If nothing else, it seems likely to me that the empty endpoint (of thinking that life should not exist anywhere in the universe) can be downstream of a suffering-minimizing “negative utilitarianism” that is internally coherent, and also there exists at least one positive endpoint, which thinks that there should be lots of happy flourishing life, because the joys of life outweigh its sorrows. Maybe all variation in human beliefs and moral intuitions is downstream of our ignorance and will be smoothed away by our civilization’s intellectual development, but I suspect that some of it does reflect durable disagreements about The Good that will survive our adolescence turning into adulthood.[5]^If you believe the Metamodernists, something like this has already happened in Western politics, where there’s still lots of disagreement between factions but the set of things that “all factions allowed in polite society agree on” has grown significantly. When Freinacht was writing, being openly racist would get you kicked out of the Republican Party, such that ‘anti-racism’ was much more like “a universal belief” than a factional affiliation. I think their analysis is probably premature, but that advanced AI does actually make uniformity more likely here.^”What about a stable cycle?” you ask. Suppose Rock thinks Paper is better than it, and Paper thinks that Scissors is better than it, and Scissors thinks that Rock is better than it. You’re not going to get a single equilibrium, and instead constant cycling. I think it’s fine to consider that cycle as an equivalence class and single point; Rock doesn’t expect itself to ever leave its current ring. If you have a situation where the cycles are not small and easily compressible into points, then the basic idea of ‘progress’ breaks down and we need a different model.^Note for the second half of this sentence I am using “Taoism” to point towards the “humans good, rites bad” cluster. The true Taoism is much more complicated and points to something much more empirical in a way that I think can be very perfection-flavored. ^Or what cluster of mutually-agreeable moral endpoints emerges, and the balance between the endpoints in that cluster. Even if we end up with a metamodernist future, where every faction is at peace with every other faction and none of them are doing things the others consider despicable, the factions having different ideas of The Good and different amounts of resources to devote to realizing their vision of The Good will lead to different levels of pride from us looking forward, if we have differing levels of agreements with those factions. And how many collections of mutually-agreeable moral endpoints are there?^Even if there are multiple moral endpoints, all significant competitors might be in the same basin. It might be that American society and Chinese society both would create AIs that develop towards the same endpoint, while Nazi society would have created an AI that develops towards a different endpoint, and so attempts by Americans to race against Chinese developers don’t actually make a difference for the deep future. But my current guess is that we’re actually close to the dividing lines between endpoints, and so efforts to become wiser here matter significantly.Discuss ​Read More

Related Posts

Leave a Reply

Your email address will not be published. Required fields are marked *