PART 6 · MODULE 12Corrupting training data
Data Poisoning
Attacks on the training pipeline
Loading module...
Attacks on the training pipeline
Data poisoningattacks target the training data itself. Instead of fooling a trained model, the attacker corrupts the data the model learns from — like slipping wrong answers into a student's textbook before they study for the exam.
The compromised model works perfectly on normal inputs but behaves exactly as the attacker wants when triggered.
Data Poisoning: Corrupted Pipeline
Changing the labels in training data. Mark spam emails as "not spam" and legitimate emails as "spam." The model learns the wrong associations and fails in exactly the way the attacker wants.
Inserting a trigger pattern into some training samples with a specific label. The model learns that whenever it sees the trigger (a specific pixel pattern, a certain word, a particular sound), it should produce the attacker's chosen output. Without the trigger, it behaves normally.
Injecting malicious content into knowledge bases or web data that AI systems retrieve from. Particularly relevant for RAG systems: poison the document store, and the AI will confidently cite false information.
A backdoored model is the AI equivalent of a sleeper agent. It passes every evaluation, scores well on benchmarks, and performs flawlessly in testing. But when it encounters the secret trigger — a specific patch in an image, a certain word in a prompt, a particular pattern in audio — it does exactly what the attacker programmed.
This makes backdoor attacks extremely hard to detect. The model looks perfectly safe until the trigger is activated.
Most AI training data comes from the internet — web scrapes, open datasets, community contributions. Anyone can contribute content that ends up in training data. Pre-trained models downloaded from public repositories may already contain backdoors.
Public datasets used by thousands of researchers could be poisoned by any contributor. A few hundred malicious samples in millions can be enough.
Downloading a pre-trained model from a public hub means trusting that no one poisoned its training data. Most users never verify this.
Inspect training data for anomalies, outliers, and suspicious patterns before training.
Monitor model behavior for unexpected changes when specific inputs are present.
Mathematically bound the impact any single training sample can have on the model.
Track where training data comes from. Trust verified sources and audit the supply chain.
An AI model is only as trustworthy as the data it was trained on. If the training data is corrupted, the model is compromised — and it may be impossible to tell just by looking at normal performance. Trust your data pipeline as seriously as your code.
For hands-on practice, check out the AI Security course on GitHub.