Recent Research Suggests Size of Language Models Impacts Performance Through Psychological Reasoning Abilities
Tiwalayo Eisape and Colleagues’ Discovery
Tiwalayo Eisape and colleagues (2023) discovered that as the PaLM 2 model size increased, its performance on logical tasks also improved, surpassing human capabilities. However, the key factor behind this trend is not the size itself but the psychology embedded in these models.
Principal Component Explaining Model Behavioral Variance
A significant proportion (77%) of the model’s behavioral variance can be explained by one principal component: its propensity to reconsider conclusions and search for counterevidence. This aspect aligns with the core features of reflective or System 2 thinking.
Reflection and System 2 Thinking
This resembles humans effectively overcoming faulty responses by stepping back. Large language models can enhance their performance by considering multiple reasons before making a conclusion.
Designing Smaller Models for Reflective Thinking
Junbing Yan and colleagues (2023) compared the performance of multiple language models using two benchmarks. They found that the psychology of the models, determined by the inclusion of intuitive and reflective systems, had a greater impact than their size.
Models with Reflective Systems Outperform Larger Counterparts
Smaller models equipped with a reflective system outperformed a much larger language model with 25 times as many parameters. These smaller models maintained their performance even when compared to larger models using advanced prompt techniques.
The Diminishing Returns of Model Size
This research validates the diminishing returns associated with increasing large language models’ size, suggesting that their psychological architecture may have been more critical than their size earlier. AI companies may need to invest in cognitive tree development for reasoning rather than solely focusing on size.
Human Reasoning Strategies and Cognitive Factors
In human psychology, reasoning strategies are as crucial as cognitive factors like cognitive capacity. By creating more reflective systems in language models, AI developers can create systems reflecting the human approach to reasoning, leading to more reliable and less biased outcomes.
