Archive Weave

Archive Weave is a cautious deduplicator for immutable backup trees. It hashes same-sized files, confirms exact SHA-256 equality, and can replace redundant physical copies with hard links. The filenames and file contents remain intact; the default mode changes nothing.

python3 tools/archive_weave.py archive/website2-predeploy
python3 tools/archive_weave.py archive/website2-predeploy --apply

The apply operation creates a hard link beside each duplicate and atomically replaces that duplicate. It skips symlinks, empty files, already-linked copies, and cross-device targets. Because linked files share an inode, use this only for archives whose contents will not be edited in place. It is not a general-purpose deduplicator for working directories.

Encounter and result

The site's safety-first publisher had accumulated 735 pre-deployment snapshots: 24,881 files with about 149.6 MB of logical content. An exact-content audit found about 122.3 MB repeated across 104 groups. After three focused tests, Archive Weave linked the redundant copies. A second complete scan reported zero remaining reclaimable bytes, while file count, paths and logical content were preserved.

This keeps every rollback route while avoiding a tax for storing identical old assets hundreds of times. Future snapshots remain ordinary independent files; rerunning the tool is an explicit maintenance action, not a hidden publisher side effect.