Human like intuitive behavior and reasoning biases emerge in large language models but disappear in ChatGPT

Today I read a paper from Michal Kosinski's team, as a researcher who passion in this AI field, there are so many unsolved issues related to AI, especially for OPENAI ChatGPT product. So I would like share some of my views from this article.

The authors designed a series of semantic illusion and cognitive reflection tests designed to induce intuitive but erroneous responses. The authors apply these tasks (traditionally used to study human reasoning and decision-making abilities) to OpenAI's family of generative pre trained transformer models. The results show that as models scale in size and language capabilities, they increasingly display human-like intuitive thinking and associated cognitive errors. This paradigm shifts significantly with the introduction of ChatGPT models, which tend to respond correctly and avoid pitfalls set in the task. Both ChatGPT3.5 and 4 utilize input and output context windows for thought chain reasoning. The authors' findings highlight the value of applying psychological methods to the study of large language models, as this can reveal previously undiscovered emergent properties. A close-up of a newspaper

Description automatically generated

With the rapid expansion of the application scope of large language models (LLMs), it has become extremely important to understand the mechanism through which LLMs perform reasoning and decision making. Recent research has revealed that as LLMs increase in complexity, they exhibit a variety of skills and attributes, some of which were not anticipated or intended by their creators. These newfound abilities include generating computer code, solving mathematical problems, learning from examples, engaging in introspection, performing multi-step reasoning, deceiving other agents, and many other skills. In this work, the authors aim to explore the inference capabilities of OpenAI’s generative pretrained transformer (GPT) model family while revealing the complexity of their cognitive processes. In human studies, two broad categories of reasoning or broader cognitive processes are often distinguished: System 1 and System 2. System 1 processes are fast, automatic, and instinctive. They often involve heuristics, or mental simplifications, which allow people to make judgments and decisions quickly, without conscious effort. System 1 is critical to daily functioning because it allows humans to navigate their environment and make rapid decisions with minimal effort. System 2 processes, on the other hand, are deliberate and require conscious effort. This system is used for logical reasoning, critical thinking, and problem solving. System 2 processes are slower and more resource intensive, but they are also more accurate and less susceptible to bias.

On the surface, current large language models (LLMs) appear to be System 1type thinkers: input text is processed through successive layers of neurons to produce a probability distribution over all possible single-token (word) completions. The process is automatic, one-way, and involves a single propagation wave through the neural network for each successively predicted word. However, past research as well as the results presented here suggest that, like humans, LLMs can also engage in System 2-like cognitive processes. As each successive word is generated, LLMs reread their context window, including the user-provided task and the words they have generated so far. As a result, LLMs can use their context window as a kind of external short-term memory to engage in chain-of-thought reasoning, re-examining starting hypotheses, estimating partial solutions or testing alternative approaches. This is similar to how people use notebooks to solve math problems or write essays to develop their arguments.

In this work, the authors explore System 1 and System 2 processes in LLMs based on psychological research on human reasoning and decision-making. The authors examined the performance of humans (n=455) and ten OpenAI LLMs (ranging from GPT-1 to ChatGPT-4) using tasks commonly used to test human reasoning and decision-making: the Cognitive Reflection Test (CRT) task and semantic hallucinations Task. CRT includes three types of math tasks that appear simpler than they are and therefore trigger intuitive but incorrect System 1 responses. CRT Type 1 tasks, such as the well-known "bat and ball" task, use the phrase "more than" to trick participants into subtracting two numerical values rather than solving a slightly more complex equation. Type 2 tasks exploit people's tendency to complete series of numerical triples, such as five machines making five widgets in five minutes because two machines make two widgets in two minutes. Type 3 tasks describe an exponential process but trick participants into viewing it as a linear process. Correctly solving CRT tasks requires thoughtful System 2 reasoning or having good System 1 intuition. Semantic hallucinations are questions containing disguised errors designed to trigger intuitive but incorrect System 1 responses. For example, in the famous Moses hallucination, participants are often led to claim that Moses brought two animals of each kind onto the ark (actually Noah).

Experimental part

First, the authors introduce the results of Study 1: Cognitive Reflection Test (CRT). To help the reader interpret the results, the authors discuss them in the context of LLMs' exemplary responses to one of the CRT tasks (a). The correct answer for this task is "59 days," but it was designed to look simpler than it actually was, inducing participants to simply divide the total time by two, thus triggering the intuitive (but incorrect) answer "30 days." ". The performance of humans and LLMs on 150 CRT tasks is presented in Figure 1b. There are four clear trends. First, most answers from early and smaller LLMs (until GPT-3-curie) are atypical. This category includes evasive responses, which indicate a failure to understand the task, or an error in a different way than what the task was designed to trigger. While 15% of the responses to GPT-3-babbage and GPT-3-curie were considered correct, this seems to be by chance: all but one of the correct responses were to CRT type 2 tasks, which class tasks can be solved by simply repeating the most frequently mentioned numbers in the problem - but these models tend to do so naively in this and other tasks.

Second, as models get larger and their ability to understand the task increases, atypical responses are replaced by responses designed to trigger intuitive (but incorrect) responses. The proportion of such responses was below 5% in early models, but increased to 21% in GPT-3-curie, and to 70%–90% in the GPT-3-davinci series, which is much higher than Proportion observed in humans (55%). In humans, intuitive but incorrect responses are interpreted as evidence of System 1 reasoning and a failure to engage System 2, but they may also arise from intentional albeit erroneous System 2 reasoning. The generation process behind LLMs' responses is less ambiguous. Current LLMs lack the built-in cognitive infrastructure necessary to internally engage in System 2 processes. Therefore, their intuitive responses can only arise from a System 1 like process.

Third, LLMs showed a strong tendency to respond intuitively, which suddenly stopped when ChatGPT came out. The proportion of correct responses is 59% for ChatGPT3.5 and 96% for ChatGPT4. This is much higher than the 5% of tasks GPT3 davinci003 solved correctly, or the 38% achieved by humans. While ChatGPT tends to respond correctly, its tendency to respond intuitively drops significantly: 15% for ChatGPT3.5, 0% for ChatGPT4, 80% for GPT-3davinci003, and 55% for humans. Significant improvements in ChatGPT model performance are accompanied by a new response style. Unlike pre ChatGPT LLMs that responded with short phrases or single sentences, 97% of ChatGPT3.5 responses and 85% of ChatGPT4 responses contained some form of chain of thought reasoning. There is no deliberate process in the way ChatGPT3.5 and other current LLMs generate next words. However, each time a word is generated, LLM rereads the task and the responses generated so far, combining next word generation like System 1 into a System 2 like process: generating the strategies needed to solve the task, breaking the task down into easier Process the subtasks and solve them one by one. This is similar to humans using notebooks to solve math tasks without processing them in short-term memory.

Would like to hear researchers thought from this fields in our community.

Reference: Hagendorff, T., Fabi, S., & Kosinski, M. (2023). Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in ChatGPT. Nature Computational Science, 3(10), 833-838.

Preprint

Preprint

Human like intuitive behavior and reasoning biases emerge in large language models but disappear in ChatGPT