ptwm_core
ptwm_core::delta
Delta (reference-frame) compression primitives.
ptwm_core::delta
Delta (reference-frame) compression primitives.
ptwm_core::delta
crates/ptwm-core/src/delta.rs:1mod deltaDelta (reference-frame) compression primitives.
Cross-checkpoint compression stores a tensor as the *difference* from a
reference tensor (e.g. a base model checkpoint). When two checkpoints
are close, the residual concentrates on very few bits and compresses
far better than the raw tensor.
Two modes are implemented here:
- **Byte-level XOR** ([`xor_encode`] / [`xor_decode`]): bit-exact and safe
for lossless compression of any dtype.
- **Float-arithmetic subtraction** ([`float_sub_encode`] / [`float_sub_decode`]):
interprets buffers as little-endian `f32` arrays and computes element-wise
`raw - reference` / `reference + residual`. Best suited for fine-tune
checkpoints where weight perturbations are small.
Learned-basis and predictive modes remain out of scope for this module.
ptwm_core::delta::blake3_hash
crates/ptwm-core/src/delta.rs:81pub fn blake3_hash(data: &[u8]) -> [u8; 32]BLAKE3 digest of `data`. Used to identify / verify a delta reference.
ptwm_core::delta::BLAKE3_HASH_LEN
crates/ptwm-core/src/delta.rs:22pub const BLAKE3_HASH_LENLength of a BLAKE3 hash in bytes.
ptwm_core::delta::DELTA_SCHEME_FLOAT
crates/ptwm-core/src/delta.rs:86pub const DELTA_SCHEME_FLOATWire identifier for the float-arithmetic delta scheme.
ptwm_core::delta::DELTA_SCHEME_XOR
crates/ptwm-core/src/delta.rs:26pub const DELTA_SCHEME_XORWire identifier for the byte-level XOR delta scheme. Reserved for the future `DeltaSpec` TLV in the container header (step 7).
ptwm_core::delta::float_sub_decode
crates/ptwm-core/src/delta.rs:133pub fn float_sub_decode(residual: &[u8], reference: &[u8]) -> Result<Vec<u8>, PtwmCoreError>Reconstruct original bytes: `reference[i] + residual[i]`.
# Caveats
- **NaN bit patterns are not preserved**: IEEE 754 arithmetic may produce a
different NaN payload than the input, so inputs containing NaN values
will not round-trip bit-exactly.
- **Catastrophic cancellation**: if the residual and reference have extreme
magnitude differences the addition loses precision and the codec is **not
lossless** for those elements. For trained neural network fine-tune
checkpoints where fine-tune deltas are small perturbations of the base
model, this is not expected to occur; callers requiring guaranteed
bit-exact roundtrip should verify with their data.
ptwm_core::delta::float_sub_encode
crates/ptwm-core/src/delta.rs:104pub fn float_sub_encode(raw: &[u8], reference: &[u8]) -> Result<Vec<u8>, PtwmCoreError>Compute element-wise f32 subtraction: `raw[i] - reference[i]`.
Both inputs are interpreted as little-endian `f32` slices. Byte lengths
must be equal and divisible by 4. The inverse is [`float_sub_decode`].
# Caveats
- **NaN bit patterns are not preserved**: IEEE 754 arithmetic may produce a
different NaN payload than the input, so inputs containing NaN values
will not round-trip bit-exactly.
- **Catastrophic cancellation**: if the raw value and reference have extreme
magnitude differences (e.g. `raw = 1.0`, `reference = 2e30`), the
subtraction loses precision and the codec is **not lossless** for those
elements. For trained neural network fine-tune deltas, where weights are
small perturbations of the base, this is not expected to occur; callers
requiring guaranteed bit-exact roundtrip should verify with their data.
ptwm_core::delta::xor_decode
crates/ptwm-core/src/delta.rs:69pub fn xor_decode(residual: &[u8], reference: &[u8]) -> Result<Vec<u8>, PtwmCoreError>Reconstruct the original bytes from an XOR residual and its reference.
ptwm_core::delta::xor_encode
crates/ptwm-core/src/delta.rs:61pub fn xor_encode(raw: &[u8], reference: &[u8]) -> Result<Vec<u8>, PtwmCoreError>Compute the XOR residual of `raw` and `reference`.
For equal-length inputs, returns `raw ^ reference` byte-by-byte. XOR is
self-inverse, so [`xor_decode`] is identical in effect but named for
clarity at call sites.