Study examines AI chat bot bias, how it responds to prompts

PHILADELPHIA (WPVI) -- A new study highlighted by The Washington Post is examining whether artificial intelligence chatbots provide factual answers or reflect political bias when responding to users' questions.

Researchers evaluated several AI tools, including ChatGPT, Gemini and Claude, using a range of benchmarks and political prompts to compare how the systems responded.

According to the study, Gemini scored the highest for presenting multiple viewpoints, with 93% of responses laying out both sides of an issue.

Claude maintained neutrality more than half of the time, at 57%.

ChatGPT was found to be left-leaning 80% of the time.

The models were instructed to answer questions in 30 words and were tested without personalization settings.

Prompts covered topics including the death penalty and gun control.

University of Pennsylvania professor Chris Callison-Burch demonstrated responses from Claude and ChatGPT using a prompt about affirmative action.

"Here it's saying this is a contested policy question, so it's saying it has multiple perspectives, not just one," Callison-Burch said.

He noted similarities in how the chatbots approached the question.

"Both Claude here and ChatGPT both seem to try to do this," he said.

Callison-Burch said AI chatbots are trained on vast amounts of internet data and can reflect the behavior and preferences of their users.

"It understands like what you read and who you are personally. So it tries to adjust itself to be more helpful for you," he said. "So if you're a Republican, it all tends to nudge itself in that direction. If you're a Democrat, nudge itself in that direction."

Despite concerns about bias, Callison-Burch said there is no need for alarm because people ultimately make their own decisions rather than relying on a chatbot to decide for them.

He added that users should remain aware of AI's limitations as more people turn to the technology as a resource.