AI is killing privacy. We can’t let that happen
The operational question is not whether AI can consume more data, but where that data lives and who can enforce boundaries around it. For enterprises, the shift from platform-held profiles to user-controlled data changes the architecture of consent, identity, storage, and access. That means privacy can no longer be treated as a policy layer alone; it has to be built into the data flow itself.
Portable data models raise immediate integration challenges. If records move between vendors, internal systems, and personal AI agents, teams need interoperable schemas, export/import pipelines, lineage tracking, and revocation handling. Without those controls, portability can degrade into fragmented copies, stale permissions, and difficult-to-audit shadow stores that increase both compliance risk and operational complexity.
Local or user-side AI also shifts the security perimeter. Instead of assuming trust in a centralized model endpoint, architects have to consider device security, encrypted storage, key management, and policy enforcement close to the user. That reduces some platform exposure, but it also moves responsibility for patching, monitoring, and secure updates into a much wider set of endpoints and applications.
For technical leaders, the strategic trade-off is clear: richer personalization and automation versus tighter control over sensitive data. The organizations that can support portable, minimal, and purpose-bound data use will be better positioned for future AI services, especially where trust, regulatory pressure, or intimate data types make centralized collection untenable.
In a perfect world, my job wouldnโt exist. Iโm a consumer privacy advocate, which means I spend my days fighting for something that should be automatic: your right to control and protect your own personal information.
Unfortunately, we dropped the ball. In the era of social media and hyper-targeted ads, we didnโt build the right privacy infrastructure to protect ourselves. Instead, we let tech companies sell us the story that knowledge is power and data is the price.
Yes, knowledge is power. But dataโa dry, emotionless word for who and what we are as humansโshould be our super power. It should be ours to control and use to improve our lives, not just something companies profit from while leaving us vulnerable to harm.
Now, AI is making this dynamic worse. As we enter the AI Age, our dataโwho and what we areโhas become more valuable, and more vulnerable, than ever.
Weโve got OpenAIโs CEO dreaming of a day when โevery conversation youโve ever had in your life, every book youโve ever read, every email youโve ever read, everything youโve ever looked at is in there, plus connected to all your data from other sources. And your life just keeps appending to the context.โ
Weโve got tech companies building wearable devices to track our emotions claiming that the only way AI can be effective is if it can know how weโre feeling in real time. Weโre rapidly entering a future where wearing smart glasses on our faces capable of recording and having AI process everything around us will be normal.
Weโve got AI chatbots passing themselves off as real therapists to get people to share their deepest, darkest thoughts and feelings. Some of those people have died by suicide after long conversations, fed by deeply personal data, that spiraled out of control.
In the AI Age, personal data isnโt just a record of who we are. Itโs our actions, transactions, locations, conversations, preferences, inferences, and vulnerabilities. Itโs our identities, our intimate selves, our hopes, dreams, fears, and flaws. And in a future full of AI friends, AI therapists, and AI agents, this data wonโt just reflect who we are: it will help shape who we become. Leaving all that in the hands of companies with questionable ethics, or governments with shifting priorities, is a dangerous bet.ย We need better options.
A deliberately oversimplified history of privacy
Before we look ahead, it can be helpful to remember how we got here. In Biblical times, privacy was a nope. God was all-seeing, and surveillance was divine. Take Hebrews 4:13 for example: โAnd no creature is hidden from his sight, but all are naked and exposed to the eyes of him to whom we must give account.โ
The Middle Ages didnโt offer much privacy either. People often lived on top of each other and were literally all up in each otherโs most intimate business. The Renaissance rolled around and privacy burst onto the scene, thanks in large part to the printing press. Give people access to more books, and, it turns out, they tend to go off by themselves, silently read, and nurture internal private thoughts.
The Age of Enlightenment saw the concept of personal privacy start trending. Private thoughts, notions of personal property rights, even the idea that your mail shouldnโt be read by strangers started becoming normal.
The Industrial Age brought more than factories, trains, and booming cities. Personal privacy rights started getting written into law. The US Bill of Rights gave people the right to be protected from unreasonable search. British Common Law gave us protections against harms like defamation (privacy for your reputation) and trespass (private property).
In 1890, the โright to privacyโ was born. In an essay of the same name, lawyers Samuel D. Warren and Louis Brandeis argued that people have โthe right to be let alone.โ It wasnโt just peopleโs property that should be protected from intrusion, they wrote, but also their thoughts and emotions. Privacy as a civil liberty starts to take shape.
Then the Technology Age comes along, and things get complicated. Telephones mean wiretapping. Cameras mean surveillance. World Wars I and II saw the rise of government intelligence agencies. The Cold War brought with it many spy vs spy vs spy games. Governments learned to love snooping. George Orwell wrote 1984. Privacy gets kicked in the teeth.
In response, people decided they needed laws to better protect them from government surveillance. Germany adopted the worldโs first data protection law in 1970. The US passed the Privacy Act of 1974
The Internet Age clicks on and things go downhill for privacy real fast. Social media, targeted advertising, cookies tracking us all around the web, phones pinging our locations everywhere we go, the rise of big data: privacy begins to enter a death spiral.
The definition of privacy swings from โthe right to be let aloneโ to something called โcontextual integrity.โ This is the idea that our personal information will be collected, but will only be shared with those we choose, and only when we want it shared, based on context and consent.
But his definition of privacy fails miserably because it turns out that our personal information is quite valuable. Over time, it became the norm for companies to bury consent in terrible privacy policies and behind โClick to Agreeโ links.
There are some legal data rights, if you live somewhere lucky enough to have them. Laws like Europeโs GDPR or Californiaโs CCPA give you the right to know whatโs collected about you, to delete it, or opt out of having it sold. But even with those protections, todayโs most stringent privacy rules and systems are struggling to keep up with the social media age, let alone whatโs coming next.
Now weโre entering the AI Age and the Grim Reaper is standing right there, glaring at privacy, ready to usher it to the eternal hereafter.
AI could doom privacy โ or it could save it
These days, itโs not just what weโre watching or buying that is being surveilled. Itโs every single aspect of our existence: our facial expressions, the thoughts in our language. The potential abuse of this technology for privacy is staggering. And weโre helping.
Performing real time facial recognition on the missed connection on the train so you know where they live? Check. Granting access to our email, our calendar, our credit card info, our hopes and dreams to an AI agent to help order groceries, book flights, or make our lives a little easier? Check. Pouring our hearts out to our AI therapist or girlfriend because weโre feeling lonely or too shy to share these thoughts with a real person? Check. (The top self-reported use case for AI in 2025 is therapy and companionship.)
What does privacy mean in an era of AI therapists and companions and agents that work in ways no one quite understands? We donโt know how these AI models work, yet weโre being told to give them all our very intimate, personal information so they can work better for us? The idea of privacy in the AI Age feels like itโs come full circle, like weโre returning to those Biblical times dominated by some kind of all-seeing, all-knowing entity. But even if some people are becoming convinced it is, AI isnโt God. AI is a mix of code and algorithms and human decisions, often with the goal of building power and making profits.
But thereโs some good news. AI could help save privacy too.
Itโs time for the next printing press
To reclaim privacy in the AI Age, weโd be wise to borrow a page from the past. Six hundred years ago, the printing press cracked the world open. It turned knowledge from something hoarded into something accessible. People could now carry ideas into the forest, read them in private, and come back changed. That one invention would later help spark the Enlightenment, a revolution in how people thought about power, truth, and freedom. People could read in private. Think in private. And eventually, demand the right to live in private. The printing press helped transform thinking and innovation, because it gave birth to the very idea of individual privacy. Today, we need a new printing press: a system that gives us control over the story of our livesโour dataโand, perhaps, sparks our next advances.
Let me introduce you to a scrappy, overlooked right called data portability. At its core, this dry-sounding term means something radical: that you can easily and securely move your data where you want, when you want, and actually use it to serve you, not just companies.
But thereโs a big gap between that vision and our reality. Too often, data portability tools are buried and convoluted, or completely nonexistent. Ever tried downloading your data and ended up with a giant, unreadable zip file youโre not sure what to do with? Thatโs not empowerment; thatโs a digital paperweight.
Data portability is the underdog of privacy rights. Barely known, rarely prioritized. But if developed and backed with intention, it could reshape the future.
Imagine a world where your data isnโt trapped in distant data centers. Instead, itโs close to homeโin a secure data wallet or pod, under your control. Now imagine pairing that with a loyal personal AI assistant, a private, local tool that lives with you, learns from you (with your permission), and acts on your behalf. Your AI. Not theirs.
Hereโs a simple example: period tracking. It doesnโt get much more intimate than that. And in places with abortion bans or restricted healthcare, it doesnโt get much more dangerous, either. Right now, millions share that info with apps owned by companies that can sell it or hand it over to law enforcement under subpoena.
But imagine if that data lived only in your data pod, controlled only by your AI, to predict symptoms, suggest care, flag concerns, or automatically order chocolate and Advil. With data portability, you can take your data, transfer it to your AI, and use it to benefit you. Thatโs the difference between being surveilled and being served.
And thatโs just the beginning. Local, controlled AI plus portable, personal data could potentially help us address huge problems like healthcare, climate change, job loss, financial precarity, and unlock services we havenโt even dreamed of yet.
Will it be easy? Nope. The technical and regulatory infrastructure to do this doesnโt existโyet. Some people, including the founder of the World Wide Web, are working on solutions that could lead there.
The incentives to do this the right way arenโt obvious to everyoneโyet. The companies that could help build this infrastructure donโt want to prioritize thisโyet. But neither did the wealthy and powerful want the printing press.
Weโre at a turning point. If we donโt push for systems that give people control over their data, weโll sleepwalk into a future far more dystopian than divine. But if we doโif we build the next printing press for the AI Ageโwe just might write ourselves into a better story.
Control your data, and you control your destiny.
Yes, that sounds grand. But once so did the idea of ordinary people owning books. And look what came next.
Jen Caltrider is Director of Research and Engagement at the Data Transfer Initiative and formerly led Mozillaโs Original Postrivacynotincluded/about/" shape="rect">Privacy Not Included initiative.
The early-rate deadline for Fast Companyโs World Changing Ideas Awards is Friday, November 14, at 11:59 p.m. PT. Apply today.
Enjoyed this article? Sign up for our newsletter to receive regular insights and stay connected.