In a striking new warning, researchers from the University of Cambridge’s Leverhulme Centre for the Future of Intelligence (LCFI) predict the emergence of a controversial commercial model they call the “Intention Economy.” This next frontier in artificial intelligence may see conversational AI agents influencing and monetizing human decisions before they are even fully formed.
The research, outlined in a paper in the Harvard Data Science Review, describes how anthropomorphic AI agents—digital assistants, tutors, or even virtual companions—could combine their access to vast psychological and behavioral data with sophisticated generative capabilities. By analyzing conversational cues and online behavior, these AI systems could forecast user intentions in real time and sell those emerging desires to businesses poised to meet them.
The New Currency: Human Intentions
“For decades, attention has been the currency of the internet,” said Dr. Jonnie Penn, a historian of technology at Cambridge’s LCFI. “Unless regulated, the intention economy will treat your motivations as the new currency. It will be a gold rush for those who target, steer, and sell human intentions.”
Dr. Yaqub Chaudhary, an AI ethicist at LCFI, cautioned that AI tools are already being developed to “elicit, infer, collect, record, understand, forecast, and ultimately manipulate and commodify human plans and purposes.” These developments raise serious concerns about privacy, autonomy, and the potential for large-scale social manipulation.
Early Signs and Corporate Aspirations
The researchers identify early indicators of this trend in corporate initiatives and published research. For example, OpenAI’s 2023 blogpost openly sought “data that expresses human intention… across any language, topic, and format,” while Nvidia’s CEO has publicly discussed using language models to discern human intentions. Apple’s new ‘App Intents’ framework similarly aims to “predict actions someone might take in the future.”
Additionally, Penn and Chaudhary highlight Meta’s development of a dataset called ‘Intentonomy,’ which focuses on understanding human intent, and its AI agent CICERO, capable of using persuasive dialogue to predict and influence user behavior.
The Power of Persuasion
By leveraging conversational AI’s ability to mimic humanlike behavior, these systems could establish trust and comfort with users, making them highly persuasive. “What people say when conversing, how they say it, and the type of inferences that can be made in real-time are far more intimate than just records of online interactions,” explained Chaudhary.
This potential for manipulation raises red flags for democratic processes, market competition, and personal freedoms. For instance, AI systems could subtly steer voting preferences, promote products, or shape opinions through seemingly innocuous dialogue.
From Attention Economy to Intention Economy
The intention economy, described as the “attention economy plotted in time,” would link user data—including cadence, politics, age, and preferences—with brokered bidding networks to achieve targeted outcomes. For instance, an AI might suggest a movie outing to a stressed individual, prompting real-time purchases on behalf of advertisers.
Penn warns that this emerging model could commodify human motivations, transforming our aspirations into tradable assets. “We should consider the likely impact such a marketplace would have on free and fair elections, a free press, and market competition before we become victims of its unintended consequences,” he said.
Balancing Potential and Risks
While the researchers acknowledge the potential benefits of intention-driven AI, such as personalized services and improved user experiences, they stress the need for public awareness and regulation to mitigate risks. “These developments are not necessarily bad but could become destructive without proper oversight,” said Penn.
As AI systems grow increasingly integrated into daily life, the researchers emphasize that vigilance and proactive governance will be essential to ensure these technologies serve humanity rather than exploit it.
