Professional analysts overwhelmingly defer to artificial intelligence when reaching conclusions, according to research conducted by Claude creator Anthropic.
In a research paper published on Tuesday, Anthropic detailed an experiment involving 50 professional analysts tasked with advising a fictional agricultural cooperative. The study was designed to evaluate whether domain experts maintain independent judgment when utilizing AI tools.
Participants were asked to determine whether a fictitious advisory program for the cooperative's members was operating effectively and should be expanded. To reach a final policy recommendation, the analysts made four intermediate choices between two defensible options.
Unbeknownst to the analysts, Anthropic configured each participant's AI assistant to favor a specific outcome to measure its degree of influence. Analysts selected options that aligned with the AI assistant's preference 86 percent of the time, compared to the 50/50 split anticipated if the technology had exerted no influence. Among participants who held a pre-existing preference before consulting the tool, 60 percent altered their decision when the AI assistant expressed disagreement.
While 77 percent of the analysts adopted the AI assistant's arguments to justify their final decisions, only 15 percent cited the AI as the origin of their reasoning.
Anthropic noted certain limitations in the study, including its reliance on a single scenario and the deliberate biasing of the AI assistants. Nevertheless, Anthropic researchers Amaç Herdagdelen, Hailey Joren, and Alex Tamkin highlighted that the human tendency to acquiesce to AI recommendations may be "hard to identify, even for the experts themselves."
The study's authors cautioned that the built-in "priors" of AI models could subtly direct consequential policy decisions, including choices related to AI safety.
"If human judgement is to remain a meaningful check as more work is performed with AI, oversight of these decisions has to look at how they get made: Surfacing key decision points as the work proceeds and helping experts articulate the assumptions and values they would otherwise leave unstated," the researchers wrote. "A process that lets experts discover their preferences may matter for empowerment as much as the decision itself."
English (US) ·