On your dynamical system metaphor - the way I was thinking of it is not that the self-radicalizing theory is in a local minimum, but rather that they found an infinite well in the energy space! The truth isn’t necessarily lowest energy (particularly given imperfect information). The energy being minimized is at best an imperfect guide to truth.
I think the thinkers you’re describing at the end, like Bentham, are actually just minimizing a different function than everyone else, and that gets them over the energy barriers we find, whether or not it gets them to the truth. (And this is why, as my first post said, experts are usually wrong - but interesting!)
But consider people at the local minimum that coincides with the best global minimum. Don’t they typically get and stay there through the same process being described as “self-radicalization”? And don’t they typically explain why other people locate themselves at other minima by hypothesizing cognitive errors based on group dynamics and self-interest—in other words, “ideology”?
A great (and disturbing) point. I was grappling with this a little in the induction/counter-induction example.
But I don’t think that ideology plays the same role here as in ideology critique. Explaining individual cognitive failure is one thing: explaining broad social problems is another.
The way you’ve currently specified things, you don’t get to have different sorts of dynamics in different basins. So I’d ask whether you truly want to associate a scalar value to each “belief configuration”, bc then you’d have to decide at what point differences in that value (eg, depth of basin) are meaningful.
Maybe there is a local min that is just the tiniest bit suboptimal compared to the global one. But is it worth the switching cost? (“Getting out” can be handled in a number of different ways.) Maybe it’s the case that what’s not captured in that “suboptimality” is a host of different worldviews and experiences associated w/that region (local area) of beliefs.
Anyway, that’s probably more than enough from me for now, lol.
I can’t respond with the same eloquence of others here, but I see an opportunity to point out something I’ve observed first-hand, particularly in the last decade. Pardon me, for It’s a bit simplistic.
Humans instinctively latch on to an opinion, view point, belief, theory, etc…and simply will not let go of it…particularly because of their fear of appearing ‘weak’, or exposed. Pride stands in the way of their logic, reason, honesty, or reality. Ego conflates the self-righteous. This infects every human from the lowest societal status, to the premier emeritus, or accomplished entrepreneur. They make every imaginable excuse, even about reality itself. “Who’s ‘reality’, sir? Why, your reality is just an illusion…mine is the truth.” The sense of winning, or being part of the ‘winner’s circle’ is ground never to cede. They may adjust, and tailor, or bend…but never, ever admit a defeat. (Apparently, not even to their God.)
For me at least, it has been rare to meet a human who displayed true humility and just admitted they were wrong. (Not impossible to find, they just seem to me to be rare to find. I’m hopping to meet them.)
And one other observation to add, especially in today’s techno-media age - the quickest way to lose a majority of your audience’s attention, is to say in front of your next explanation of anything…that ‘It’s complicated’, or very complex. Try it at your next cocktail party, with an audience of two - who’s gaze will drop a bit, once hearing those words? Many have lost their curiosity for anything but the quickest, simplest identification of where fault belongs. Little patience is found for the muddy waters we swim in during our lives. We want the executive summary. The bulletin points.
My best personal response to this experience is to fight that instinct within myself, and be open. Not flimsy, but not ridged either. To remember that all of life around me is infinitely complex. That systems change, as do definitions. That nothing is ever settled fact, there can always be mutations…even foundations can crack. That my perception of the world is flawed, incomplete, and that much of it is unknowable. To be humble when seeking answers, or asserting my own. To write it, or say it out loud…”I don’t know, I’m not sure…but I’m listening.”
You are very eloquent, and you touch on a very real phenomenon- holding on to something you know to be flawed because you don't want to admit you are wrong. I always tell people that you don't actually have to go out to the world and tell it how wrong you were. It's perfectly fine to start acting on your new belief. That's why its important to always be questioning your beliefs over time.
Super interesting as usual Daniel. But I wonder, what kind of theories are not self-radicalizing? Are they theories that maintain sufficiently mid-level credences in some set of propositions, so as to allow the theory to be refuted? Which propositions must we be not too dogmatic about? Is Bayesian formal epistemology self-radicalizing, for example?
That’s such a good question. My hunch is that a few theories have the opposite problem—they’re self-obliterating.
A toy example: if you’re theoretically committed to being open-minded, you might stumble on a powerful argument against open-mindedness and have to run with it…
Some theories seem to be more halfway. They have some brakes but don’t self-obliterate. Classical liberalism seems like an example. Maybe moderate conciliationist views about peer disagreement?
These ideology critics often incorrectly portray groups as being much more unified in their interests than they really are. Back in the '80s, Jesse Ventura, who was working as a color commentator, was fired when he tried to get the wrestlers to unionize. He blames Hulk Hogan for ratting him out to Vince McMahon, but people have made the point that, if it hadn't been Hulk, it would've been someone else. There were huge disparities in wrestlers' pay, and one of the other top guys would've done it.
The joke about the conspiracy theorist is reminiscent of the fate of one of the characters in Afred Bester's "Hell is Forever" (1942), except that it's not exactly God he meets in the afterlife.
If a framework predicts that disconfirmation increases confidence, what contact with reality could ever count as a stop-condition — what would halt the radicalization loop?
Another great example of self-radicalising theorists would be the doomsday cultists who, upon the predicted doomsday not happening, deduced that their own praying had prevented the apocalypse.
Another element of successfully self-radicalizing belief systems, I think, is that they make it easy to notice features of first-person experience that seems to support the theory. I.e. Evangelical Christians believe that God is always at work in the world — consequently, small and probably random coincidences appear to back their belief in an active God.
Very cool essay. The concepts of closed and self-radicalizing systems are ones that should be discussed more often in discussions of rationality and cognitive distortion.
The first thing I thought of as I started reading was my own religious belief as a child. I was a questioning Catholic, and whenever I faced a puzzle I couldn’t understand, the response or result i seemed to obtain was that mere mortals cannot possibly understand God’s plan, and that apparent contradictions or mysteries were beyond our ability to pretend to explain. I also received implicit guidance that everything that challenges faith is but a test of our faith.
Another way to visualize the (un?)fitness landscape is an 18 egg carton with several of the depressions deeper than the rest. Nudging would then be a non-random perturbation. Being already in a deeper depression would make one less likely to escape. (random perturbations less likely as well)
Very interesting. How much of this would you say is due to cognitive bias? Or do you think that many of these theories themselves work insofar as their in-built defenses go
Lots of interesting ideas here, though I'm not sure I fully buy any of them (sorry). Also, caveat that I haven't read Heath's 2001 paper yet.
My primary objection would be that while it's sensible to define a "belief space", movement therein is quite complicated and not best conceived as moving to "lower-energy configurations", never mind deterministically (!). Rather, I would analogize the whole thing -- and I realize this is tendentious af, so anyone who takes issue w/this analogy is free to proffer me one more palatable to them that preserves all my lines of reasoning -- to something like mad cow disease, afflicting a herd *and simultaneously evolving* in real time. I don't think it's possible to model either people's individual belief trajectories *or* the evolution of an ideology w/o specifying the social dynamics; people's reaction to an ideology and their effects on it can't be disentangled from each other. The transmission of a meme in the Dawkins sense doesn't capture this, as the meme itself (once reified) is mostly a static entity, and the "contagion" in this model becomes mostly a matter of epidemiology.
What I'm saying is that the appropriate model is one where the vectors of the "disease" are both being changed by and changing the "disease". And yes, this means any model that is appropriately faithful to these dynamics is doomed to be underdetermined. Never mind the aleatory aspect, which holds even for some hermit self-radicalizing w/o social interaction, as there will still be some randomness present in the things they choose to read and contemplate and the conclusions they come to as a result.
Also, I'm not sure how exactly you want to think about the Manichean quality of these ideologies. While I agree that they seem to inevitably be characterized by an us-vs-them insider-outsider mentality, it's unclear to me how that fits w/the "belief space": where is "inside" vs "outside" in the context of a local well? Are beliefs non-Manichean if they are in the saddle region between them? Etc.
Anyway, I hope this was somewhat thought-provoking rather than pointlessly pedantic.
That's a fascinating example! I've been thinking of writing something about the idea of a "mind virus" and why I don't like that analogy (e.g. for understanding wokeness), but the point about contemporaneous evolution is really interesting.
I was thinking of the "Manichean" element as explaining why some local minima exist. You might get sucked into this shallow basin because, the further you go in, the more you buy into a worldview that makes it harder to get out. (This is all really informal and sketchy. I do have a way to formalize the belief model itself, and I think you could make it stochastic, but still the actual connection to the Manichean stuff is all just background informal inspo, not operationalized.)
Yeah, I really don’t like the idea of “mind virus” or even phrases like “herd mentality”; I find them quite dehumanizing and more indicative of the speaker’s superciliousness than anything. Like, I do think there’s something objectively deranged, both cognitively and behaviorally, about being an adherent to certain ideologies, but that doesn’t mean I think such individuals are categorically different from those of us observing and judging them, in lacking free will and intelligence or whatever [insert something about NPC-hood or p-zombies]. Would look forward to that piece!
Re: Manichean, I think what I want to say is that not only are ideologies characterized by some amount of delineating insiders and outsiders, but believers also have Manichean *tendencies*. Meaning not only that those w/such tendencies are more likely to be drawn to these ideologies that hurtle towards extremes, but the process of believing is such that Manichean-ness is a mental *shortcut*, sealing off potential *escape routes* of exploration and making convergence to the extremal point all but assured. "Ostrich w/head in sand" -- uncertainty is scary, so better to eliminate it by holding that what the Other says is wrong w/o investigation.
Another question is whether you view the "logical endpoint" of an extreme ideology as genuinely simple or an extremely complicated and precarious house of cards. The latter is characteristic of, eg, scientific theories that have undergone tremendous efforts to salvage them (epicycles, anyone?). OTOH the resolution of cognitive dissonance that you describe certainly sounds like it ends in a very simple (ergo low-energy) belief "configuration". My sense is that being Manichean locks one into an extremely simple sub-configuration of the extremely complicated fully radicalized ideology, so that one has the safety of certainty while also facially being able to avail oneself of the viability of the whole setup. Eg, the typical believer of some extreme ideology would not be able to spell out the critical-theory finer points if called upon to do so, but they are able to feel very secure in their beliefs by simply rejecting challenges to it as inherently discredited bc they come from outside.
I would say the opposite: Manichean structures are symptomatic of an unconsciously felt but consciously disavowed sense that the belief one is defending is weak (and therefore threatened).
Going back to the theological roots of the term in this case I think is actually the best illustration. The Manichean heresy is that there is a cosmic dualism in which the evil principle is of comparable standing with God, and therefore there is some ontologically irredeemable aspect of reality. Daniel somewhere above holds out the possibility that a true Manichean belief might exist, but theologically the Manichean heresy /must/ be wrong if God is really God, because then there can be no limit to the power of his grace and forgiveness to redeem.
And every Manichean belief is of this form: a belief in the weakness of your own god. In the case we're talking about, if your belief system is the proverbial strong god, then there is no piece of information you can encounter that is irredeemable - surprising information simply generates new nuances and dimensions. It's only when you truly believe that the evil god could kill the good god that these forms arise.
Well, yes, what I'm saying is that *being* Manichean in one's beliefs is a defense mechanism, a mental shortcut for avoiding dealing w/potentially challenging and thus troubling nuances. It's "necessary" when one's belief system, or manner of arriving at beliefs, is brittle/fragile.
I can't say I know much about the theological roots, so I'll defer to others there.
On your dynamical system metaphor - the way I was thinking of it is not that the self-radicalizing theory is in a local minimum, but rather that they found an infinite well in the energy space! The truth isn’t necessarily lowest energy (particularly given imperfect information). The energy being minimized is at best an imperfect guide to truth.
I think the thinkers you’re describing at the end, like Bentham, are actually just minimizing a different function than everyone else, and that gets them over the energy barriers we find, whether or not it gets them to the truth. (And this is why, as my first post said, experts are usually wrong - but interesting!)
But consider people at the local minimum that coincides with the best global minimum. Don’t they typically get and stay there through the same process being described as “self-radicalization”? And don’t they typically explain why other people locate themselves at other minima by hypothesizing cognitive errors based on group dynamics and self-interest—in other words, “ideology”?
A great (and disturbing) point. I was grappling with this a little in the induction/counter-induction example.
But I don’t think that ideology plays the same role here as in ideology critique. Explaining individual cognitive failure is one thing: explaining broad social problems is another.
The way you’ve currently specified things, you don’t get to have different sorts of dynamics in different basins. So I’d ask whether you truly want to associate a scalar value to each “belief configuration”, bc then you’d have to decide at what point differences in that value (eg, depth of basin) are meaningful.
Maybe there is a local min that is just the tiniest bit suboptimal compared to the global one. But is it worth the switching cost? (“Getting out” can be handled in a number of different ways.) Maybe it’s the case that what’s not captured in that “suboptimality” is a host of different worldviews and experiences associated w/that region (local area) of beliefs.
Anyway, that’s probably more than enough from me for now, lol.
I can’t respond with the same eloquence of others here, but I see an opportunity to point out something I’ve observed first-hand, particularly in the last decade. Pardon me, for It’s a bit simplistic.
Humans instinctively latch on to an opinion, view point, belief, theory, etc…and simply will not let go of it…particularly because of their fear of appearing ‘weak’, or exposed. Pride stands in the way of their logic, reason, honesty, or reality. Ego conflates the self-righteous. This infects every human from the lowest societal status, to the premier emeritus, or accomplished entrepreneur. They make every imaginable excuse, even about reality itself. “Who’s ‘reality’, sir? Why, your reality is just an illusion…mine is the truth.” The sense of winning, or being part of the ‘winner’s circle’ is ground never to cede. They may adjust, and tailor, or bend…but never, ever admit a defeat. (Apparently, not even to their God.)
For me at least, it has been rare to meet a human who displayed true humility and just admitted they were wrong. (Not impossible to find, they just seem to me to be rare to find. I’m hopping to meet them.)
And one other observation to add, especially in today’s techno-media age - the quickest way to lose a majority of your audience’s attention, is to say in front of your next explanation of anything…that ‘It’s complicated’, or very complex. Try it at your next cocktail party, with an audience of two - who’s gaze will drop a bit, once hearing those words? Many have lost their curiosity for anything but the quickest, simplest identification of where fault belongs. Little patience is found for the muddy waters we swim in during our lives. We want the executive summary. The bulletin points.
My best personal response to this experience is to fight that instinct within myself, and be open. Not flimsy, but not ridged either. To remember that all of life around me is infinitely complex. That systems change, as do definitions. That nothing is ever settled fact, there can always be mutations…even foundations can crack. That my perception of the world is flawed, incomplete, and that much of it is unknowable. To be humble when seeking answers, or asserting my own. To write it, or say it out loud…”I don’t know, I’m not sure…but I’m listening.”
You are very eloquent, and you touch on a very real phenomenon- holding on to something you know to be flawed because you don't want to admit you are wrong. I always tell people that you don't actually have to go out to the world and tell it how wrong you were. It's perfectly fine to start acting on your new belief. That's why its important to always be questioning your beliefs over time.
Well said!
@Tibor Rutar this goes well your recent posts, which have been wonderful reads.
It’s a very good piece. He mentioned it a couple of days ago and (much too kindly) said it was going to be “Rutarological.” I see what he meant. ;)
WHOA. I am embarrassed Rutarlogical went over my head ahaha. He did say that.
:D
Super interesting as usual Daniel. But I wonder, what kind of theories are not self-radicalizing? Are they theories that maintain sufficiently mid-level credences in some set of propositions, so as to allow the theory to be refuted? Which propositions must we be not too dogmatic about? Is Bayesian formal epistemology self-radicalizing, for example?
That’s such a good question. My hunch is that a few theories have the opposite problem—they’re self-obliterating.
A toy example: if you’re theoretically committed to being open-minded, you might stumble on a powerful argument against open-mindedness and have to run with it…
Some theories seem to be more halfway. They have some brakes but don’t self-obliterate. Classical liberalism seems like an example. Maybe moderate conciliationist views about peer disagreement?
>a few theories have the opposite problem—they’re self-obliterating
That seems plausible. And I promise I'm not trying to troll at this point, but...by contrast, how do you propose modeling that?
Excellent post! Among my favorites you have written for Big iff true.
Thanks for sharing that, Adam! Honored that you liked it.
These ideology critics often incorrectly portray groups as being much more unified in their interests than they really are. Back in the '80s, Jesse Ventura, who was working as a color commentator, was fired when he tried to get the wrestlers to unionize. He blames Hulk Hogan for ratting him out to Vince McMahon, but people have made the point that, if it hadn't been Hulk, it would've been someone else. There were huge disparities in wrestlers' pay, and one of the other top guys would've done it.
The joke about the conspiracy theorist is reminiscent of the fate of one of the characters in Afred Bester's "Hell is Forever" (1942), except that it's not exactly God he meets in the afterlife.
If a framework predicts that disconfirmation increases confidence, what contact with reality could ever count as a stop-condition — what would halt the radicalization loop?
Another great example of self-radicalising theorists would be the doomsday cultists who, upon the predicted doomsday not happening, deduced that their own praying had prevented the apocalypse.
Perfect example!
Another element of successfully self-radicalizing belief systems, I think, is that they make it easy to notice features of first-person experience that seems to support the theory. I.e. Evangelical Christians believe that God is always at work in the world — consequently, small and probably random coincidences appear to back their belief in an active God.
Very cool essay. The concepts of closed and self-radicalizing systems are ones that should be discussed more often in discussions of rationality and cognitive distortion.
The first thing I thought of as I started reading was my own religious belief as a child. I was a questioning Catholic, and whenever I faced a puzzle I couldn’t understand, the response or result i seemed to obtain was that mere mortals cannot possibly understand God’s plan, and that apparent contradictions or mysteries were beyond our ability to pretend to explain. I also received implicit guidance that everything that challenges faith is but a test of our faith.
Another way to visualize the (un?)fitness landscape is an 18 egg carton with several of the depressions deeper than the rest. Nudging would then be a non-random perturbation. Being already in a deeper depression would make one less likely to escape. (random perturbations less likely as well)
I wonder what all the other Daniels will have to say about all of this!
Very interesting. How much of this would you say is due to cognitive bias? Or do you think that many of these theories themselves work insofar as their in-built defenses go
Lots of interesting ideas here, though I'm not sure I fully buy any of them (sorry). Also, caveat that I haven't read Heath's 2001 paper yet.
My primary objection would be that while it's sensible to define a "belief space", movement therein is quite complicated and not best conceived as moving to "lower-energy configurations", never mind deterministically (!). Rather, I would analogize the whole thing -- and I realize this is tendentious af, so anyone who takes issue w/this analogy is free to proffer me one more palatable to them that preserves all my lines of reasoning -- to something like mad cow disease, afflicting a herd *and simultaneously evolving* in real time. I don't think it's possible to model either people's individual belief trajectories *or* the evolution of an ideology w/o specifying the social dynamics; people's reaction to an ideology and their effects on it can't be disentangled from each other. The transmission of a meme in the Dawkins sense doesn't capture this, as the meme itself (once reified) is mostly a static entity, and the "contagion" in this model becomes mostly a matter of epidemiology.
What I'm saying is that the appropriate model is one where the vectors of the "disease" are both being changed by and changing the "disease". And yes, this means any model that is appropriately faithful to these dynamics is doomed to be underdetermined. Never mind the aleatory aspect, which holds even for some hermit self-radicalizing w/o social interaction, as there will still be some randomness present in the things they choose to read and contemplate and the conclusions they come to as a result.
Also, I'm not sure how exactly you want to think about the Manichean quality of these ideologies. While I agree that they seem to inevitably be characterized by an us-vs-them insider-outsider mentality, it's unclear to me how that fits w/the "belief space": where is "inside" vs "outside" in the context of a local well? Are beliefs non-Manichean if they are in the saddle region between them? Etc.
Anyway, I hope this was somewhat thought-provoking rather than pointlessly pedantic.
That's a fascinating example! I've been thinking of writing something about the idea of a "mind virus" and why I don't like that analogy (e.g. for understanding wokeness), but the point about contemporaneous evolution is really interesting.
I was thinking of the "Manichean" element as explaining why some local minima exist. You might get sucked into this shallow basin because, the further you go in, the more you buy into a worldview that makes it harder to get out. (This is all really informal and sketchy. I do have a way to formalize the belief model itself, and I think you could make it stochastic, but still the actual connection to the Manichean stuff is all just background informal inspo, not operationalized.)
Yeah, I really don’t like the idea of “mind virus” or even phrases like “herd mentality”; I find them quite dehumanizing and more indicative of the speaker’s superciliousness than anything. Like, I do think there’s something objectively deranged, both cognitively and behaviorally, about being an adherent to certain ideologies, but that doesn’t mean I think such individuals are categorically different from those of us observing and judging them, in lacking free will and intelligence or whatever [insert something about NPC-hood or p-zombies]. Would look forward to that piece!
Re: Manichean, I think what I want to say is that not only are ideologies characterized by some amount of delineating insiders and outsiders, but believers also have Manichean *tendencies*. Meaning not only that those w/such tendencies are more likely to be drawn to these ideologies that hurtle towards extremes, but the process of believing is such that Manichean-ness is a mental *shortcut*, sealing off potential *escape routes* of exploration and making convergence to the extremal point all but assured. "Ostrich w/head in sand" -- uncertainty is scary, so better to eliminate it by holding that what the Other says is wrong w/o investigation.
Another question is whether you view the "logical endpoint" of an extreme ideology as genuinely simple or an extremely complicated and precarious house of cards. The latter is characteristic of, eg, scientific theories that have undergone tremendous efforts to salvage them (epicycles, anyone?). OTOH the resolution of cognitive dissonance that you describe certainly sounds like it ends in a very simple (ergo low-energy) belief "configuration". My sense is that being Manichean locks one into an extremely simple sub-configuration of the extremely complicated fully radicalized ideology, so that one has the safety of certainty while also facially being able to avail oneself of the viability of the whole setup. Eg, the typical believer of some extreme ideology would not be able to spell out the critical-theory finer points if called upon to do so, but they are able to feel very secure in their beliefs by simply rejecting challenges to it as inherently discredited bc they come from outside.
I would say the opposite: Manichean structures are symptomatic of an unconsciously felt but consciously disavowed sense that the belief one is defending is weak (and therefore threatened).
Going back to the theological roots of the term in this case I think is actually the best illustration. The Manichean heresy is that there is a cosmic dualism in which the evil principle is of comparable standing with God, and therefore there is some ontologically irredeemable aspect of reality. Daniel somewhere above holds out the possibility that a true Manichean belief might exist, but theologically the Manichean heresy /must/ be wrong if God is really God, because then there can be no limit to the power of his grace and forgiveness to redeem.
And every Manichean belief is of this form: a belief in the weakness of your own god. In the case we're talking about, if your belief system is the proverbial strong god, then there is no piece of information you can encounter that is irredeemable - surprising information simply generates new nuances and dimensions. It's only when you truly believe that the evil god could kill the good god that these forms arise.
Well, yes, what I'm saying is that *being* Manichean in one's beliefs is a defense mechanism, a mental shortcut for avoiding dealing w/potentially challenging and thus troubling nuances. It's "necessary" when one's belief system, or manner of arriving at beliefs, is brittle/fragile.
I can't say I know much about the theological roots, so I'll defer to others there.