A toolkit for reproducible hardware fault-injection experiments on PyTorch models.
- Standalone parts. Each component is useful by itself. Encode a tensor, inject faults into it, or summarize a list of scores without setting up anything else.
- Built to fit together. The same parts combine into complete, resumable experiments with little glue code.
- Use as much or as little as you need. Nothing forces you to adopt the rest of the toolkit to use one piece of it.
- Extensible. The built-in models, encodings, metrics and experiments implement public base classes. Your own implementations plug in the same way.
Building blocks:
- Fault model - bit flips and stuck-at faults, a resumable repeat-free location sampler and batched injection.
- Encodings - SECDED (Hamming codes), MSET and CEP, usable alone or chained together, and a wrapper that stores a model's parameters in encoded memory.
- Metrics - a generic system for reliability metrics.
- Model loading - models bundled with their evaluation datasets. Image classifiers on CIFAR-10/100 and ImageNet, and HuggingFace language models on WikiText, are built in.
Experiment harness:
- Experiments - automatic run loops with clean Ctrl+C handling, and atomic, resumable, optionally zstd-compressed saving.
- Estimators - mean and failure rate, with 95% confidence intervals.
- Stop conditions - run until an estimate is stable, for a fixed number of runs, or until interrupted.
- Fingerprints - resuming a saved result with a changed configuration fails with a precise diff instead of silently mixing incompatible data.
Performance-critical paths are implemented in Rust.
docs/library.md
walks through each part and ends with a complete example.
pip install faultforge
# or
uv add faultforgeSee docs/installation.md
for building from source and installing specific revisions.
Encoded Memory
injects faults into ECC-protected model parameters. It's a complete
experiment built on FaultForge and the reference implementation to follow when
writing your own. It also provides an encoded-memory CLI for recording
results.
It lives in this repository and is released together with the library, but isn't published to PyPI. Its README covers installation.
FaultForge is experimental. Expect breaking changes in most releases, and
check the changelog
when upgrading. Versions follow semver: while the
version is 0.x, minor releases can break and patch releases can't. A range
like faultforge>=0.3.0,<0.4.0 picks up only compatible releases.
mainis the development branch and can break at any point. Thelatestbranch tracks the most recent release.- The Rust crates in this repository are internal and can change at any time. They aren't published, and their version only mirrors the library's.
FaultForge has been used to develop and evaluate the techniques in several
papers. See docs/publications.md
for links and citations.
FaultForge is licensed under the Universal Permissive License (UPL), Version 1.0. It's an OSI-approved permissive license that is about as short as MIT, but:
- includes an explicit patent grant, like Apache-2.0.
- is compatible with both GPLv2 and GPLv3. Apache-2.0 isn't compatible with GPLv2, which is why many projects dual-license under "MIT OR Apache-2.0".
- extends the patent grant to larger works that include the software, not only the software itself.
The main condition is the same as MIT's: keep the copyright and license notice in copies of the software.