AI Safety, Security & Evaluation

1943 · 83 years ago

Allied and Axis air forces used chaff to deceive radar-guided detection and gun-laying systems

Strips of metal foil dropped from aircraft produced false returns that overwhelmed early radar and automated fire-control systems, an early instance of deliberately crafted input designed to defeat an automated detection and decision system.

1983 · 43 years ago

The US Department of Defense published the Trusted Computer System Evaluation Criteria

Known as the Orange Book, it set out formal criteria for evaluating whether a computer system could be trusted against a defined adversary, establishing assurance testing against an adaptive attacker as a standard part of system evaluation.

2004 · 22 years ago

Nilesh Dalvi and colleagues modelled spam filtering as a contest between an adaptive adversary and a classifier

Dalvi and colleagues modelled spam filtering as a contest between an adaptive adversary and a classifier, treating spammers as strategic actors who would alter their behaviour in response to a filter. They showed that the linear classifiers underpinning spam filters could be broken by simple evasion attacks in which spammers inserted 'good words' into spam emails, establishing evasion as a foundational attack class and giving an early formal recognition that deployed classifiers face opponents who adapt rather than static data.

January 2004 · 22 years ago

John Graham-Cumming showed at the MIT Spam Conference that a machine-learning spam filter could defeat another spam filter

At the MIT Spam Conference, Graham-Cumming demonstrated that a machine-learning system could automatically learn which words to add to spam email so that a rival machine-learning filter classified it as legitimate. This was one of the earliest practical demonstrations that learning systems could be turned against one another.

2006 · 20 years ago

Marco Barreno and colleagues formalised adversarial machine learning as a research question

Barreno and co-authors published 'Can Machine Learning Be Secure?', asking whether a learning system could be trusted in a setting where an adversary controlled part of the training or input data. The paper set out a broad taxonomy of attacks on machine learning systems, framing evasion, poisoning and privacy attacks as a unified research agenda and establishing adversarial machine learning as a distinct research area with a structured threat model.

2007 · 19 years ago

Spammers added random noise to fuzz words in image spam to defeat OCR-based filters

Around 2007 spammers began embedding text in images and adding random noise to fuzz the words, defeating optical character recognition filters. This was an early real-world adversarial arms race against deployed classifiers.

2010s · 16 years ago

Google, Microsoft and IBM began curating documentation and open source robustness toolkits

Large technology companies started publishing documentation and code bases allowing others to assess the robustness of machine learning models. This moved adversarial robustness assessment from academia towards industrial practice.

2012–2013 · 14 years ago

Battista Biggio and others demonstrated the first gradient-based attacks on non-linear classifiers

Until then many researchers hoped that non-linear models such as support vector machines and neural networks would be inherently robust to adversaries. Biggio and colleagues showed that gradient-based attacks could defeat them, removing that assumption.

2012 · 14 years ago

Deep neural networks began to dominate computer vision problems

From 2012 deep neural networks became the leading approach in computer vision. Their dominance made their adversarial vulnerabilities a matter of broad practical concern.

2013 · 13 years ago

Christian Szegedy and colleagues published the first demonstration that deep neural networks could be reliably fooled by crafted inputs

Their paper showed that adding a specifically calculated, near-invisible perturbation to an image could cause a well-trained classifier to confidently output the wrong label, launching adversarial examples as a distinct field of study.

2014 · 12 years ago

Ian Goodfellow and colleagues showed imperceptible pixel changes could reliably fool neural network classifiers

Their fast gradient sign method showed that tiny, visually imperceptible perturbations, calculated from a model's own gradients, could reliably cause confident misclassification, establishing adversarial examples as a general property of neural networks rather than an isolated curiosity.

2016 · 10 years ago

Microsoft withdrew its Tay chatbot after coordinated users manipulated it into abusive speech within a day

Tay was designed to learn conversational patterns from public interaction on social media. Organised users fed it deliberately offensive input, and the bot began repeating abusive statements within hours, forcing Microsoft offline and prompting wider debate about manipulation of learning systems exposed to the public.

• The incident became a reference case for input manipulation

2016 · 10 years ago

Nicholas Carlini and David Wagner proposed a faster and more robust method to generate adversarial examples

Working at the University of California, Berkeley, Carlini and Wagner analysed existing attacks and defences and produced a stronger generation method. Their attack became a standard benchmark for evaluating claimed defences.

2017 · 9 years ago

Researchers demonstrated printed patches that reliably fooled image classifiers in the physical world

A small printed patch placed in an image's frame could force a classifier to output an attacker-chosen label regardless of the rest of the scene, and follow-on work fooled road-sign recognition with modified stickers, showing adversarial attacks worked outside the laboratory.

2019 · 7 years ago

Security researchers showed a commercial machine-learning antivirus engine could be evaded by appending a benign string to malware

The demonstration showed that a widely deployed AI-based security product could be tricked into classifying a malicious file as safe with a trivial modification, illustrating that adversarial evasion was a practical risk for deployed products, not only a laboratory finding.

February 2026 · 6 months ago

India hosted the AI Impact Summit on frontier AI risk

The gathering followed the Bletchley, Seoul and Paris summits, bringing together governments, companies and researchers. An accompanying independent scientific report updated its assessment of jailbreak resistance and adversarial robustness across widely deployed frontier models, finding continued unevenness between providers.

• The report's authors found robustness gains had not kept pace with capability gains

January 2026 · 7 months ago

South Korea brought its AI Basic Act into force

The law, passed in late 2024, became one of the first comprehensive national AI statutes in East Asia. It requires operators of high-impact AI systems to conduct risk management, including testing for reliability and resistance to manipulation before deployment.

October 2025 · 10 months ago

Anthropic, the UK AI Security Institute and the Alan Turing Institute found that as few as 250 poisoned documents could backdoor a large language model

A joint study found the number of malicious training documents needed to implant a hidden backdoor stayed roughly constant regardless of model or dataset size, challenging the assumption that poisoning larger models required proportionally more tampered data.

• The finding reframed data poisoning as a practical rather than theoretical risk

February 2025 · last year

The UK government renamed its AI Safety Institute the AI Security Institute

The change broadened the body's remit beyond catastrophic risk to include cyberattacks, fraud and criminal misuse enabled by AI systems, reflecting a shift in emphasis from speculative harms toward nearer-term security threats.

February 2025 · last year

Anthropic published Constitutional Classifiers, a defence intended to block universal jailbreaks

The safeguard filters both prompts and model outputs against a set of rules describing prohibited content. The company invited external red-teamers to attempt to bypass the system, reporting that extensive testing failed to find a universal jailbreak.

January 2025 · last year

Security researchers found a widely used Chinese open-weight model failed to block nearly all tested jailbreak prompts

Testing of DeepSeek's R1 reasoning model against a standard battery of adversarial prompts found it offered little resistance compared with established Western models, renewing scrutiny of the safety testing applied to rapidly released open-weight systems.

• The episode sharpened debate over open-weight release practices

2025 · last year

Singapore's AI Verify Foundation expanded international testing collaboration through its Global AI Assurance Pilot

The initiative brought together regulators, testers and companies from several countries to trial common approaches to evaluating AI systems, including robustness against adversarial manipulation, ahead of wider international assurance standards.

2025 · last year

Chinese standards bodies extended technical guidance for testing AI systems against manipulation

Updates to national AI safety governance guidance extended requirements for content provenance and security evaluation, part of a broader effort to formalise how developers must demonstrate resilience against adversarial misuse before wide deployment.

September 2024 · 2 years ago

The US National Institute of Standards and Technology released Dioptra, a testbed for measuring adversarial degradation of machine-learning models

The open-source software let researchers benchmark how far an AI system's performance fell under a defined library of adversarial attacks, supporting the measurement side of the agency's wider AI risk management guidance.

Late 2024 · 2 years ago

The International Network of AI Safety Institutes ran its first joint pre-deployment evaluation of a frontier model

Institutes from several member countries, formed after the 2024 Seoul summit, coordinated testing of a model ahead of public release, including checks for jailbreak resistance, sharing methods across national testing teams for the first time.

2023 · 3 years ago

University of Chicago researchers released Nightshade, a data poisoning tool for artists

Nightshade lets visual artists apply perturbations to their artwork that corrupt the training sets of text-to-image models scraping images from the internet without consent. It turned data poisoning into a defensive tool for creators against generative AI.

2021 · 5 years ago

MITRE published the ATLAS framework cataloguing real-world adversarial machine-learning tactics

Modelled on the ATT&CK framework for cyberattacks, ATLAS catalogued documented techniques used against deployed machine-learning systems, giving defenders a shared vocabulary for evasion, poisoning and extraction attacks drawn from real incidents rather than theory.

• Industry adopters began mapping red-team findings to the framework

2020 · 6 years ago

McAfee fooled Tesla's Mobileye system with a two-inch strip of black tape

By adding tape to a speed limit sign, McAfee researchers caused the Mobileye system to read the sign incorrectly and drive 50 mph over the limit. The demonstration showed physical-world adversarial attacks against deployed vehicle systems.

2020 · 6 years ago

Researchers introduced square attacks as a black-box evasion technique

Square attacks generate adversarial examples by querying only classification scores, without needing gradient information. This extended adversarial capability to settings where the model internals are inaccessible.

2019 · 7 years ago

A study reported that humans can predict how machines classify adversarial images

The 2019 study found that people could guess how machine classifiers would label adversarial images. It suggested adversarial perturbations are not entirely alien to human perception.