[W]hy do people increase radicalism when confronted with the failure of their ideology?
Because they do not hold positions rationally, but to justify their intuitions, and their intuitions exist to advance their reproductive strategies (signals). And so they will fight to defend those signals at all costs, consistently escalating to the point of violence if necessary.
Becuase one’s self-image and social status are one’s property. So one defends them accordingly.
People change their positions only when they can both no longer see a successful use of their prior strategy, and they can now see how to use a new narrative to achieve their reproductive strategies.
You can see this not only in individuals, but the broader political phenomenon.