We made most of a digit network binary. It did not get faster.

A network can store a weight as a single bit instead of a 32-bit floating-point number. That suggests a smaller model, less memory traffic, and perhaps faster inference. The last part is not automatic. When we trained a partial binary neural network on handwritten digits, its accuracy was almost unchanged, and its calculated packed tensor payload was much smaller. But our actual CPU inference was slower than the full-precision model.

A one-bit matrix helps only when the runtime can use its representation. Storing or multiplying shadow weights as floating-point values can erase the expected advantage.

A deliberately small comparison

We used the same publicly packaged scikit-learn digits dataset as our adversarial-robustness experiment: 1,797 grayscale images, each 8 × 8, belonging to ten classes. Pixel values were divided by 16. We made a fixed stratified split into 1,347 training images and 450 held-out images using seed 2609. This is a classification exercise, not a representation of personal user data.

Both networks have layer widths 64 → 256 → 256 → 10 and batch normalization between the linear layers. They train for 80 epochs with Adam at learning rate 0.0015, batch size 128, and the same seeded mini-batch order. The baseline uses ordinary floating-point linear layers and ReLU activations.

The binary variant is deliberately partial, not an end-to-end one-bit network. It retains a floating-point input projection, a floating-point output classifier, biases, and normalization parameters. Its middle 256 × 256 weight matrix is binarized to −1 or +1 in the forward pass. The input activations to that middle matrix are also binarized to −1 or +1. Before the binary activation, the first normalization output remains signed; applying sign after a ReLU would collapse every nonnegative input to +1 and destroy information.

Training uses a straight-through estimator: the forward pass uses the sign function, while the backward pass passes the gradient through values whose magnitude is at most one. The underlying weights remain floating-point parameters for optimization. This follows the practical idea explored in Binarized Neural Networks, without reproducing the full architecture or custom bitwise execution kernels from that work.

We chose this middle matrix because it accounts for 65,536 of 86,026 trainable parameters. Binarizing a tiny final classifier would say little about size. Keeping the two outer layers in full precision, however, also means we cannot claim a fully binary inference graph.

Three measurements, three different answers

Accuracy differed by just one held-out image. That is encouraging for this network and dataset, but the sample is too small to infer a general accuracy advantage or a stable 0.2-percentage-point loss. A different split, depth, or task could give a different outcome. We ran one seed, without a variance estimate.

The storage figure needs more care. We totaled the bytes in every parameter and buffer in the PyTorch state dictionaries. For the binary network's calculated packed payload, we replaced the 262,144 bytes of its floating-point middle matrix with 8,192 bytes: one bit for each of its 65,536 entries. We left every other tensor unchanged. The resulting estimate fell from 348,216 bytes to 94,264 bytes, a 72.9% reduction in this narrowly defined payload.

We did not create or benchmark that packed model file. There is no custom serializer, bit-packed runtime, alignment calculation, or model loader in the experiment. The float shadow weights still have to be stored in the training checkpoint. As a result, both actual torch.save files measured 353,331 bytes, effectively the same size. Calling the 92.1 KiB figure an achieved download size would be false.

The CPU result is the important counterexample

The binary forward pass uses torch.sign-equivalent values produced by the straight-through function, followed by an ordinary floating-point matrix multiplication. It does not dispatch an XNOR or population-count kernel. The extra operations therefore cost time while failing to exploit the principal computational advantage of binary arithmetic.

We timed forward inference on CPU using PyTorch 2.10.0 with two threads. Each number is the median of ten groups of 200 consecutive forwards, after 40 warm-up forwards. Inputs were already in memory. We did not include loading, image preprocessing, disk reads, or user-interface work.

For a single digit, the partial binary implementation took about 4.0 times as long as full precision: 0.1599 versus 0.0395 milliseconds. At a batch size of 128, it took about 3.2 times as long: 0.8381 versus 0.2621 milliseconds. These are measurements of this particular CPU, kernel path, and short workload. They are not portable speed ratios, and each is small enough that host scheduling and Python overhead matter. The result is still sufficient to refute a claim that binarizing the weights alone made this implementation faster.

The original XNOR-Net research distinguished binarized representation from execution using binary operations. Our code only tests the representation and a training path that can learn with it. To test the promised runtime benefit, we would need to build an inference backend that packs both weights and compatible activations, implements binary dot products with suitable bitwise operations, and measures the end-to-end path against a well-optimized floating-point or int8 baseline.

What this means for private computation

Keeping computation on a laptop or desktop can reduce the need to send raw personal context to a remote system. But on-device models compete with other work for RAM, storage, memory bandwidth, battery, and thermal headroom. For that reason, compact representations are worth testing, particularly in always-available components that perform classification or retrieval alongside a larger AI tool.

This test says that a small network can tolerate one binary internal layer without sacrificing much accuracy on easy digits. It does not say that Zortex uses this network, that binary weights improve the confidentiality of a model, or that the resulting model would be appropriate for a context engine. Privacy depends on data placement, permissions, and disclosure controls. Precision is a separate engineering choice that may help a local design fit its resource budget.

There is another constraint in the numbers: 92.1 KiB is attractive only because the middle matrix is relatively large compared with the full-precision boundaries. A model with most of its parameters in unquantized embeddings or output heads would see less benefit. The count of trainable parameters alone cannot reveal how much space a deployable mixed-precision file will occupy.

What packed inference requires at runtime

Deployment performance depends on real bit-packing and a specialized inference kernel, evaluated against both int8 and float32 baselines. It would record the exported file size, peak resident memory, wall-clock latency across batch sizes, power if measurable, and accuracy on a task representative of the application. Repeating the training with several seeds would separate quantization effects from chance variation in the split and optimizer.

The script in experiments/sec-binary/run.py trains both networks, counts correct test predictions, measures state-dictionary payloads and checkpoint bytes, and times inference. results.json preserves the measured outputs, timing method, seed, and software versions. The displayed chart reads those results directly.

The three outcomes are distinct: 98.2% accuracy, a 72.9% calculated packed-payload reduction, and slower measured CPU inference. A storage estimate cannot stand in for a compact checkpoint or a faster executed model. The next decision is whether the saved memory would be large enough to justify building and maintaining an optimized binary runtime.