What changed when we trained a digit classifier against its own attacks

An 8 × 8 image of a handwritten digit contains only 64 pixel values. A classifier can recognize almost all the examples in a held-out set and still be made to fail by changing those values in a direction chosen specifically for that classifier. In our test, a standard neural network read 437 of 450 clean digits correctly. With a bounded, 20-step attack, it read 8 of the same 450 correctly.

Changing the training examples, without increasing the model's capacity, produced the sharpest contrast. We trained a second copy of the same architecture on adversarially perturbed examples. Under the same attack budget, it recognized 252 digits. Clean accuracy declined by four correctly classified examples. The change recovered accuracy at the tested attack radius, with a four-image cost on clean inputs.

This is a small, reproducible study of adversarial image classification. It is not a Zortex product benchmark or a test of prompt-injection defenses.

The 64-pixel test

We used scikit-learn's digits dataset: 1,797 labeled grayscale images, ten digit classes, and integer pixel values between 0 and 16. The code divides the pixel values by 16 so that the model sees values between 0 and 1. A fixed, stratified 75/25 split yielded 1,347 training examples and 450 test examples. Test images were never included in training or model selection.

The classifier has three fully connected layers, with dimensions 64 → 128 → 64 → 10 and ReLU activations. We trained both versions for 45 epochs with Adam, using the same seed, layer initialization, learning rate, batch size, and split. The only planned difference was the input presented during training:

• Standard training: use the unmodified training digit.
• PGD training: replace each training digit with an adversarial version generated against the model at its current weights, using four projected-gradient steps and a random starting point.

The training attack is bounded by an L-infinity radius of ε = 0.16. In other words, any one of the 64 normalized pixel values can move at most 0.16, and the result must remain between 0 and 1. On the original 0–16 pixel scale, that limit corresponds to 2.56 units per pixel. It is a substantial change at this resolution; we make no claim that these modifications are visually imperceptible.

This experiment tests an attacker who knows the model and can compute gradients through it. The attacker changes the input, not the model weights or the digit's label. The FGSM method takes one step in the gradient-sign direction of increasing classification loss. Projected gradient descent (PGD) repeats such steps and projects each candidate back into the permitted pixel range. For evaluation we used 20 PGD steps, two random starts, and kept the higher-loss candidate per test image. We attacked each model separately using that model's own gradients.

What survived

The primary metric is the fraction of all 450 held-out examples classified correctly after the attack. It includes examples that were already incorrect before perturbation; it is not an attack-success rate conditional on the clean prediction being right.

The clean result alone would have given a misleading picture. Both networks cleared 96% accuracy. At ε = 0.16, the ordinary model's PGD accuracy collapsed to 1.8%. Training with adversarial inputs raised it to 56.0%, a difference of 54.2 percentage points on this split. The clean-accuracy cost was 0.9 percentage points.

At the same radius, the one-step FGSM attack left the PGD-trained network at 60.9% accuracy, compared with 56.0% under the longer attack. That ordering matters. A defense that appears effective against one gradient step may still be weak against a sequence of smaller steps. Here, iterative evaluation found more errors in both networks.

The next radius exposes a second boundary. At ε = 0.24, PGD accuracy of the defended model fell to 11.6%. Training for one perturbation budget did not confer protection across all budgets. An attacker can also choose a different norm, objective, or attack family. None of those was tested here.

The failure mode behind the result

The ordinary classifier learned boundaries that worked well for the clean dataset. Good average classification is compatible with narrow margins in directions an attacker can exploit. An attack does not need to sample random corruptions until one works. It uses the loss gradient to locate a damaging direction, then stays within the declared constraint.

Adversarial training changes the examples that determine those boundaries. Each training step asks the model to do well on an input chosen to make it fail. The improvement in our measurement is therefore specific: at the tested radii, under the chosen white-box FGSM and PGD procedures, more adversarial images remained correctly classified.

This is neither an explanation of the network's internal features nor a proof that they become more human-like. We did not inspect learned representations or assess digit legibility with people. The experiment records behavior under a defined mathematical threat model.

There is a methodological trap here: if the defender alters or masks gradients, an ordinary gradient attack may fail without the classifier becoming reliably robust. We used differentiable feed-forward networks and attacked the defended model directly, but two PGD restarts cannot exclude weak optimization or every form of gradient masking. Stronger adaptive attacks, more restarts, and independent attack implementations are necessary before making a security claim.

Where this connects to private AI

Zortex's work concerns personal context assembled from a person's applications and supplied, with permissions, to AI tools. That setting introduces security questions of its own: whether a retrieved page can steer an assistant, whether content crosses a permission boundary, and whether a tool action is authorized. Our digit classifier does not model those questions. Pixel perturbation is not prompt injection.

The relevant lesson is in the shape of the evaluation. “The model is accurate” is insufficient until we say accurate against which inputs and adversary. Likewise, “the context is private” needs operational tests of storage, access, and disclosure under a stated attacker capability. A defense should be judged against an adaptive attempt to cross its boundary, not against the cases that are easiest to pass.

For this study we kept the entire pipeline inspectable. The code in experiments/sec-adversarial/run.py loads the dataset, trains both networks, generates each attack, enforces the perturbation constraint, and writes per-condition counts to results.csv and results.json. The figure above is generated from those saved counts, not entered by hand.

What the attack tested

These are 450 test images from one stratified split and one initialization, not a confidence interval over datasets or seeds. The 8 × 8 images and a small multilayer perceptron cannot stand in for high-resolution vision systems, language models, or production agents. An L-infinity attack is a convenient controlled probe; it does not represent every realistic attack. We did not run certified robustness analysis, test transfer attacks, search for adaptive failures, or measure deployment latency and memory.

Multiple training seeds and independent adaptive attacks would test the stability of this digit-classification result. Access-control failures in a retrieval system require a different attacker model and a separate experiment.

The narrow result stands: a model that was almost perfect on clean 8 × 8 digits failed on almost all of them under a bounded white-box attack. Changing its training data from clean to adversarial examples recovered much of that accuracy at the trained radius, while leaving large and measurable gaps at a higher one.