Dimensions of Disagreement: Divergence and Misalignment in Cognitive Science and Artificial Intelligence

Our understanding of disagreement is rooted in psychological studies of human behavior, which typically cast disagreement as divergence: two agents forming diverging evaluations of the same object. Recent work in artificial intelligence highlights how disagreement can also arise from misalignment in how agents represent that object. Here, we formally describe these two dimensions of disagreement, clarify the relationship between them, and argue that strategies for conflict resolution and collaboration are likely to be ineffective (or even backfire) if they do not consider misalignment in representations. Moreover, we identify how taking misalignment into account can enrich current research on judgment and decision making, from biased advice taking to algorithm aversion, and discuss implications for artificial intelligence research.