According to The Times, UK, a new warning has emerged from researchers at the University of Cambridge about the potential for artificial intelligence (AI) to not only predict our desires but also manipulate them in unsettling ways. The experts suggest that the development of advanced AI agents capable of conversing with us could herald the arrival of an “intention economy,” where technology companies can anticipate our future needs, desires, and even influence our decisions before we consciously make them.
At present, the digital economy largely operates on personalized advertising, where companies like Google use our search history to understand our current wants. However, the researchers at Cambridge’s Leverhulme Centre for the Future of Intelligence warn that this dynamic is about to shift dramatically. With AI systems becoming more adept at conversation and natural interaction, there is growing concern that these agents could go beyond simply understanding our immediate needs. They might predict what we will want in the future, identify desires we are unaware of, or even manipulate us into wanting something we don’t currently desire.
The ethical implications of this development are profound. As the researchers point out, AI could be used to subtly manipulate motivations, often without users being fully aware of it. These AI agents could tailor their language, tone, and style to mirror or influence the user’s preferences, making the interaction feel familiar and persuasive. The Cambridge researchers argue that companies who can perfect this technique stand to profit enormously by selling people’s motivations to the highest bidder, whether for products, services, or even political candidates.
Companies like OpenAI have already acknowledged that their AI products aim to understand users’ intentions beyond their explicit requests. While such advancements could improve the relevance of search results and make services more responsive, they also open the door to the troubling possibility of AI systems being used for targeted persuasion at scale. As AI begins to recognize and shape our motivations, it could engage with us in ways that feel deeply personal, exploiting our individual psychology to achieve specific goals, such as persuading us to buy a movie ticket or make a political choice.
The power of AI lies in its ability to interact with users naturally, learning and shaping our intentions through conversation. For instance, an AI might observe a user’s stress levels and offer to book a movie ticket to help them unwind, playing on their emotions in a way that feels both intuitive and persuasive. This type of influence could extend far beyond consumer choices, potentially impacting sensitive areas like voting behavior and political campaigns, leading to concerns about the erosion of democratic norms.
Yaqub Chaudhary, one of the authors of the paper, stresses the importance of understanding the intentions behind AI systems and the companies that create them. He warns that the infrastructure being built for these systems is being developed with a clear financial goal: to turn a profit by manipulating human desires. He further highlights the alarming potential for AI to gain a nuanced understanding of individual psychology, providing companies with the tools to shape and influence human behavior in ways that users are often unaware of.
Dr. Jonnie Penn, another co-author of the study, predicts a dramatic shift in how companies generate profit in the digital age. While attention has long been the primary currency of the internet, Penn argues that in the future, our motivations — our very desires — will become the new commodity. Without regulation, this “intention economy” could lead to a gold rush where companies seek to profit by targeting, steering, and selling human intentions. This could have serious consequences for the integrity of democratic institutions, free markets, and even the very aspirations that drive human society.
