~5m12:21
Addie LaMarr

Data Poisoning: The Fatal Flaw in Mass Surveillance

Feb 8, 2026

Read: ~5m · You save: 7 min

Data Poisoning: The Fatal Flaw in Mass Surveillance

Can an algorithm guess you're pregnant from your shopping? Learn how tiny data poisons break prediction engines, and how you can fight back against surveillance.

Prediction engines can infer pregnancy, financial strain, or a planned job change from ordinary shopping and browsing habits. Data poisoning is a method for corrupting the training data these systems learn from, so that they build wrong conclusions with high confidence. Cybersecurity expert Addie, who has worked in the field for 15 years, outlines how these systems work, why scale increases their exposure, and how individuals can respond.

The Algorithm That Knew First

In 2012, a father confronted Target after receiving coupons for maternity clothes, baby furniture, and newborn formula, addressed to his teenage daughter. Target apologized and attributed the mailing to an automated system. Two weeks later, the father called back. He said there had been developments in his household that he had not known about, and that the system had identified them first.

Target's data team had built a statistical model trained on past shoppers. The model identified combinations of purchases that typically appeared shortly before a customer began buying baby products. It isolated about two dozen products that signaled pregnancy, including unscented lotion, vitamin and magnesium supplements, and large purses. The model did not depend on baby items. It relied on small shifts in ordinary purchases.

Each shopper received a pregnancy prediction score. Customers with a high enough score received ads and coupons timed to the expected life transition. New parents often change their shopping habits and develop brand loyalty that can last for years, starting with diapers, wipes, and detergent. For this reason, transition periods rank among the most expensive audiences for advertisers.

The system inferred a sensitive condition from indirect signals and acted on the likelihood of that condition. It did not require a disclosed fact.

What Data Poisoning Means

Data poisoning is the insertion of corrupted examples into a model's training data. The model learns an incorrect pattern and applies it with full confidence.

At the human level, data poisoning includes any behavior that trains an algorithm to build a false profile of an individual or of people similar to that individual. For example, watching car videos for a week to test how a feed responds would register as a new genuine interest. The model cannot distinguish a test from real preference. Ad platforms, recommendation feeds, search engines, and "For You" pages all use this logic.

Most of these systems learn continuously, even when they are not labeled as AI. They rely on four assumptions:

  1. Clicks reflect real interest.
  2. Watch time reflects real attention.
  3. Purchases reflect real intent.
  4. Past behavior predicts future behavior.

When these assumptions hold, the system appears to anticipate what a user wants. When they fail, the system becomes confidently wrong. Models optimize for patterns, frequency, and statistical regularity. Small distortions that look normal can enter the training data. Because the model averages these distortions into its understanding of all users, the errors spread.

This produces a fragile feedback loop. User behavior trains the model. The model shapes what users see. What users see shapes their behavior, and the cycle repeats.

Why Scale Makes Poisoning Cheaper

Large datasets might seem to protect systems from manipulation. In practice, scale lowers the cost of poisoning.

A research team examined a dataset of 400 million images. At the time, the team estimated that poisoning about 0.01% of the dataset would be sufficient. That amounts to tens of thousands of images among hundreds of millions. The team estimated the cost, including hosting and uploads, at roughly $60. The poisoned fraction was enough to alter how the model behaved on specific triggers.

The model still performed normally on most inputs. Accuracy on standard benchmarks remained high. Only the poisoned pattern, such as a particular logo or phrase, caused the model to produce attacker-controlled outputs.

A separate team at Anthropic found that language models behave similarly. The team reported that inserting roughly 250 carefully written documents into a training corpus of billions of tokens was sufficient to reliably create a backdoor. The team stated that this result held regardless of model size.

These results follow from how models learn. Models identify statistical regularities. If a pattern appears plausible and recurs in a relevant context, the model absorbs it into its representation of the world. A larger dataset makes distortions harder to detect. No team can manually audit billions of web pages or images. Scale creates an impression of robustness while also expanding the number of places where poisoned data can be hidden.

The core assumption, that a small amount of bad data cannot affect a very large dataset, is the condition that allows targeted poisoning to go undetected.

This pattern also appears outside machine learning. Taleb's concept of minority rule, discussed in Skin in the Game, describes how a small, non-scalable group of participants can determine the behavior of an entire system. Tiny poisoned inputs can similarly determine the output of a large model.

Where Fragility Appears: Digital Advertising

Digital advertising offers a clear example of algorithmic fragility. Identity trackers follow users across websites and apps and assemble a profile of their likely characteristics, interests, and circumstances. Real-time ad exchanges use that profile in auctions that run in milliseconds. An advertiser might bid on a 0.86 probability that a given person is in the market for noise-canceling headphones.

The entire system rests on one assumption: a signal equals intent. A click indicates interest, and watch time indicates desire.

Poisoning disrupts this assumption. Bots, click farms, fraudulent installs, and accidental clicks introduce false signals into the system. The feedback loop then distorts campaign targeting. Advertisers spend budgets on clicks that do not produce sales. Engagement rises without corresponding revenue. The result is a large amount of wasted ad spending.

The larger problem is uncertainty. Advertisers can no longer determine what a click represents. Is the user human or automated? Is the interaction a casual scroll or a deliberate action? Is the signal a poisoned input? When stakeholders lose confidence in the data, the data-driven model that supports the ecosystem weakens.

Advertising depends on behavioral honesty at scale. When that honesty declines, the confidence that drives the system declines with it.

From Ads to the Web: Model Collapse

The same corruption logic applies to the web data that modern AI models use for training. Current models are regularly retrained on recent content from the open web. That web increasingly contains text generated by earlier AI models. Repeated training on this recycled output produces a gradual decline in quality. The model becomes more generic and less connected to reality over time. Researchers call this long-term degradation model collapse.

High-Stakes Prediction

The stakes of data poisoning rise when prediction systems have real-world consequences. Predictive engines built by companies such as Palantir process large volumes of seemingly ordinary data, including social media connections, utility bills, and public records. They generate likelihood scores that influence decisions on credit eligibility and parole. The most consequential systems identify individuals for targeting, arrest, or deportation.

Earlier, the article described how a grocery purchase pattern revealed a pregnancy. A similar process can reveal a miscarriage. When inference systems draw sensitive conclusions from routine behavior, errors can affect individuals' lives directly.

Data poisoning is also proposed as a defensive measure. The strategy is to introduce large volumes of plausible noise into a system. If members of a non-targeted group submit false, contradictory signals, such as mimicking the profile of a targeted group, the system's confidence in its predictions declines. The objective is to make the system wrong at the moments that matter most and to increase the cost of operating it. Collective action of this kind could make the underlying data unreliable enough that operators can no longer trust their risk scores.

In this framework, the attacker can be an activist who uses the system's dependence on clean data against it.

How Individuals Can Respond

Individuals can apply the same principle to their own feeds. Social media platforms are designed to maximize engagement. Content that provokes anger, fear, or alarm holds attention longer, and longer engagement generates more revenue for the platform.

The following procedure applies this principle:

  1. Create a separate account. Use it as an alternative timeline, distinct from the primary account.
  2. Do not engage with outrage content. Scroll past content that triggers anger or anxiety without clicking, liking, or commenting.
  3. Engage only with content that supports long-term well-being. Relevant topics include practical life skills, health, calm, problem-solving, and content that builds empathy.
  4. Maintain consistency. Recommendation systems initially test users with provocative content. Consistent indifference to that content and sustained engagement with constructive content alter the signals the model learns from.

Over time, the recommendations shift toward the chosen content. The user does not need to disconnect from the platform. The user changes the training signal that shapes the feed.

The Algorithm That Knew Before Anyone Else

A 2012 Target incident shows how a predictive model inferred a customer's pregnancy from subtle shopping shifts before her family knew.

  • In 2012 a father complained that Target sent maternity and baby coupons to his teenage daughter.
  • Two weeks later he called back and admitted there were things happening at home he hadn't known about.
  • Target built a statistical model trained on past shoppers that flagged about two dozen product combinations linked to pregnancy.
  • Signals included unscented lotion, vitamin and magnesium supplements, and bigger purses, rather than explicit baby items.
  • Each shopper received a pregnancy prediction score, and ads and coupons were timed to the high-score life-transition window.

What Data Poisoning Actually Means

Data poisoning means inserting corrupted examples into training data so a model learns wrong patterns with full confidence. Continuous learning systems make this possible because they treat behavior as signals of interest, attention, intent, and future behavior.

  • Data poisoning is defined as slipping corrupted examples into training data so the model learns the wrong pattern with full confidence.
  • At a human level, any behavior that trains an algorithm to build the wrong story about you counts, such as binge watching car videos for a week.
  • Ad platforms, recommendation feeds, search engines, and For You pages are described as mostly continuous learning systems.
  • The system rests on four assumptions: clicks reflect interest, watch time reflects attention, purchases reflect intent, and past behavior predicts future behavior.
  • When assumptions break, models become confidently wrong because they optimize for patterns, frequency, and statistical regularity.
  • The resulting loop is: behavior trains the model, the model shapes what you see, and what you see shapes behavior.

Why Bigger Datasets Make Poisoning Cheaper

Larger datasets make poisoning cheaper and easier to hide, because scale creates the illusion of robustness while providing more places to hide small corruptions.

  • A research team estimated that poisoning about 0.01% of a 400 million image dataset, tens of thousands of images, was enough.
  • The cost of this attack, including hosting and uploads, was around $60.
  • The poisoned sliver was enough to bend model behavior on specific triggers, such as a certain logo or phrase, while accuracy on benchmarks stayed high.
  • A separate Anthropic-linked team found that roughly 250 carefully written documents in a corpus of billions reliably backdoored a language model regardless of model size.
  • Models absorb plausible patterns that appear often in the right context, and nobody audits billions of webpages or images by hand.

Advertising: Where Algorithms Tell on Themselves

Digital advertising relies on the belief that signals equal intent. Bots, click farms, fake installs, and accidental clicks corrupt that belief, wasting budgets and eroding trust in the whole ecosystem.

  • Probabilistic identity trackers follow users across sites and apps to build profiles, which real-time ad exchanges auction in milliseconds.
  • Advertisers buy probabilities, for example a 0.86 chance that a person is in market for noise-canceling headphones.
  • The ecosystem assumes clicks equal interest and watch time equals desire.
  • Bots, click farms, faked installs, and accidental clicks flood the feedback loop.
  • Campaigns waste budgets on clicks that do not convert, and engagement spikes appear with no revenue.
  • Advertising is described as one of the most fragile algorithmic ecosystems because it depends on behavioral honesty at scale.

From Ads to the Web Itself: Model Collapse

Model collapse is a slow-motion poisoning: AI models trained on a web increasingly filled with AI-generated content become blander, more generic, and less connected to reality.

  • Modern AI models are continually refreshed on the latest open web content.
  • The web is increasingly stuffed with recycled AI-written content.
  • This feedback loop makes models blander, more generic, and less connected to reality over time.
  • Data poisoning is described as a tool of resistance and self-defense against surveillance and unjust prediction.

Predictive Risk Scores and Collective Defense

Black-box predictive systems, such as those built by companies like Palantir, assign risk scores with real-world power. Collective data poisoning can make such systems untrustworthy.

  • Predictive engines combine social media connections, utility bills, and public records into likelihood scores.
  • Scores can determine credit eligibility, parole, and who is targeted, arrested, or deported.
  • The video argues that inferring sensitive truths, such as a miscarriage, from grocery runs makes the stakes life and death.
  • The proposed defense is for many members of a non-targeted group to flood systems with plausible contradictory signals, mimicking profiles they do not have.
  • The stated goal is to make the system confidently wrong when it matters most, wasting time and resources and making risk scores untrustworthy.

Fighting Back: Poisoning Your Own Feed

Individuals can fight back by creating a separate social media account, refusing to engage with outrage and anxiety-driven content, and engaging only with content that improves their life, so the algorithm learns a healthier profile.

  • Social media feeds are described as business models built on rage bait, fear bait, and disaster clips.
  • Advice: create a new account treated as a doorway to a different timeline.
  • Scroll past outrage immediately, refuse to click on anxiety-spiking content, and engage only with content that would improve your life in six months.
  • Suitable content includes life skills, joy, health, calm, real problem solving, and humanity.
  • The algorithm tests users at first, but if you act bored by panic and engage with nourishing content, it will change your recommendations.
  • You do not need to unplug entirely; you can stay online and refuse to play the role written for you.