Parallelograms revisited: Exploring the limitations of vector space models for simple analogies

Classic psychological theories have demonstrated the power and limitations of spatial representations, providing geometric tools for reasoning about the similarity of objects and showing that human intuitions sometimes violate the constraints of geometric spaces. Recent machine learning methods for deriving vector-space embeddings of words have begun to garner attention for their surprising capacity to capture simple analogies consistently across large corpora, giving new life to a classic model of analogies as parallelograms that was first proposed and briefly explored by psychologists. We evaluate the parallelogram model of analogy as applied to modern data-driven word embeddings, providing a detailed analysis of the extent to which this approach captures human behavior in the domain of word pairs. Using a large novel benchmark dataset of human analogy completions, we show that word similarity alone surprisingly captures some aspects of human responses better than the parallelogram model. To gain a fine-grained picture of how well these models predict relational similarity, we also collect a large dataset of human relational similarity judgments and find that the parallelogram model captures some semantic relationships better than others. Finally, we provide evidence for deeper limitations of the parallelogram model of analogy based on the intrinsic geometric constraints of vector spaces, paralleling classic results for item similarity. Taken together, these results show that while modern word embeddings do an impressive job of capturing semantic similarity at scale, the parallelogram model alone is insufficient to account for how people form even the simplest analogies.