PTWM

Rust API

Reference for the ptwm-core Rust crate.

Rust API

Reference for the ptwm-core Rust crate.

129 modules documented.

  • ptwm_core — Pure-Rust core for lossless neural-network weight compression. (5 symbols)
  • ptwm_core::chain — PPG chain construction, wire encoding, validation, runtime. (0 symbols)
  • ptwm_core::chain::dispatch — Op dispatch: construct a Box<dyn Op> from an OpId and its raw params. (1 symbol)
  • ptwm_core::chain::graph — Chain DAG types — Chain, ChainNode, ChainEdge, TerminalRef. (4 symbols)
  • ptwm_core::chain::runtime — PPG DAG runtime — forward executor, inverse executor, fusion picker. (8 symbols)
  • ptwm_core::chain::validate — Static validator for PPG chain DAGs. (1 symbol)
  • ptwm_core::chain::wire — Wire encoding / decoding for Chain. (4 symbols)
  • ptwm_core::codec (6 symbols)
  • ptwm_core::codec_tagged — Codec-agnostic per-plane encode / decode. (8 symbols)
  • ptwm_core::codecs (0 symbols)
  • ptwm_core::codecs::arithmetic — Arithmetic-coding plane codecs: three peer models over the shared range coder. Each frames the true plane length into its payload (like rans.rs), so decode is self-delimiting and ignores the container's decoded_len (which is source-derived and unreliable for byte-split planes). (3 symbols)
  • ptwm_core::codecs::context_mixing — Opt-in context-mixing plane codec. Stateless; self-delimiting (the coder frames the plane length, so decode ignores the container's decoded_len). NOT in the default dispatch menu — selected only when named in codec_menu. (1 symbol)
  • ptwm_core::codecs::fpc — FPC-style leading-zero plane codec. (1 symbol)
  • ptwm_core::codecs::huff_llm — huff_llm_5bit — field-separated Huffman codec for 16-bit float planes. (1 symbol)
  • ptwm_core::codecs::huffman (1 symbol)
  • ptwm_core::codecs::identity (1 symbol)
  • ptwm_core::codecs::neural_predictor — Opt-in NNCP-style neural-predictor plane codec. Stateless; self-delimiting (the coder frames the plane length, so decode ignores decoded_len). NOT in the default dispatch menu — selected only when named in codec_menu. (1 symbol)
  • ptwm_core::codecs::order1_arithmetic — Order1Arithmetic: static order-1 (lag-1) arithmetic coder over the exponent byte plane — a learned dynamic-length float coder. Fits a conditional table once, serializes it into codec state (the static table is what makes a future GPU decode kernel feasible), and range-codes against the frozen model. Self-delimiting: it frames the true plane length and ignores the container's source-derived decoded_len. (1 symbol)
  • ptwm_core::codecs::order1_scale_ac — Order1ScaleAC: per-row lag-1 conditional arithmetic coder over the SCALE plane of MXFP4 / NVFP4 tensors. (6 symbols)
  • ptwm_core::codecs::per_group_codebook — PerGroupCodebook codec. (15 symbols)
  • ptwm_core::codecs::rans — rANS plane codec. (1 symbol)
  • ptwm_core::codecs::tans — tANS (tabled-ANS / Finite State Entropy) plane codec. (1 symbol)
  • ptwm_core::codecs::zstd — Zstd plane codec. Wraps the existing crate::codec_tagged::zstd_{encode,decode} helpers so the container can use it as a generic meta-compressor without new dependencies. (1 symbol)
  • ptwm_core::codecs::zstd_dict — Zstd plane codec with dictionary support. (1 symbol)
  • ptwm_core::compressor — PPG compressor entry point. (4 symbols)
  • ptwm_core::container — Container writer and reader orchestration. (3 symbols)
  • ptwm_core::delta — Delta (reference-frame) compression primitives. (8 symbols)
  • ptwm_core::discovery — Filesystem discovery + index cache for installed extensions. (14 symbols)
  • ptwm_core::dispatch — PPG codec dispatch. Codecs declare capabilities via PlaneCodec::accepts and PlaneCodec::priority_for; this module filters and ranks them for a given PlaneDescriptor. (1 symbol)
  • ptwm_core::dtype — First-class dtype enum for the compression pipeline. (2 symbols)
  • ptwm_core::entropy — Entropy: measurement and coding. (2 symbols)
  • ptwm_core::entropy::arithmetic — Continuous-precision arithmetic coding over plane bytes. (0 symbols)
  • ptwm_core::entropy::arithmetic::model — Byte models for the arithmetic plane codecs. (8 symbols)
  • ptwm_core::entropy::context_mixing — Opt-in context-mixing plane codec (cmix/lpaq lineage). Bit-level prediction from byte-history models, context-gated logistic mixing, binary range coding. Fully integer/deterministic; stateless on the wire. (0 symbols)
  • ptwm_core::entropy::context_mixing::coder — Bit-level encode/decode driver. Ties the context models + gated mixer to the shared range coder, coding each byte MSB-first. The payload is self-delimiting: a u32 LE length prefix precedes the coded bits, so decode never depends on the container's decoded_len. (3 symbols)
  • ptwm_core::entropy::context_mixing::mixer — Context-gated logistic mixer. One weight vector per gating context (the partial-byte register c0, bounded to 256). Mixes the models' stretched logits, squashes to a probability, and learns the weights online by logistic gradient. Integer/deterministic. (1 symbol)
  • ptwm_core::entropy::context_mixing::model — Four byte-history context models (orders 0–3). Each maps a context — a hash of recent whole bytes combined with the partial-byte register c0 — to an adaptive 12-bit probability counter in a fixed hash table. (3 symbols)
  • ptwm_core::entropy::context_mixing::squash — lpaq squash (logistic) and stretch (logit) over 12-bit probabilities. squash(d) maps a stretched logit d to a probability in 0..=4095; stretch(p) is its inverse. Both are integer and deterministic. (2 symbols)
  • ptwm_core::entropy::huff_llm — Field-separated Huffman coding of 16-bit float planes (Huff-LLM). (3 symbols)
  • ptwm_core::entropy::huffman — Pure-Rust canonical Huffman codec. (3 symbols)
  • ptwm_core::entropy::neural_predictor — Opt-in NNCP-style neural-predictor plane codec. A tiny online-learning fixed-point MLP predicts each bit; a binary range coder writes the residual. Fully integer/deterministic; stateless on the wire (the model trains from scratch identically on both sides). (0 symbols)
  • ptwm_core::entropy::neural_predictor::coder — Bit-level encode/decode driver tying Net to the shared range coder, coding each byte MSB-first. Self-delimiting: a u32 LE length prefix precedes the coded bits, so decode never depends on the container's decoded_len. (3 symbols)
  • ptwm_core::entropy::neural_predictor::net — Tiny online-learning neural predictor: a trainable per-context embedding feeds a ReLU hidden layer and a linear output. Pure integer/fixed-point and fully deterministic, so encode and decode evolve in lockstep — losslessness does NOT depend on the net compressing well. No weights cross the wire. (1 symbol)
  • ptwm_core::entropy::outcome — Typed return for the [huffman]crate::entropy::huffman and [rans]crate::entropy::rans compressors. (1 symbol)
  • ptwm_core::entropy::rans — rANS (range Asymmetric Numeral Systems) entropy codec. (2 symbols)
  • ptwm_core::entropy::stream_frame — Shared framing helpers for the 4-stream interleaved entropy codecs (huffman::compress / rans::compress). Both codecs split their input into 4 roughly-equal segments, encode each segment as an independent stream, and record per-stream compressed lengths in a fixed-size jump table so the decoder can locate each stream without scanning. (5 symbols)
  • ptwm_core::entropy::tans — Tabled ANS (tANS / Finite State Entropy) entropy codec. (2 symbols)
  • ptwm_core::error (2 symbols)
  • ptwm_core::extension — PTWM v1 extension system: contributions, manifests, capabilities, lifecycle, and the per-file Extension Table that ties them to the on-disk wire format. (2 symbols)
  • ptwm_core::extension::builder — Helpers for building an ExtensionTable while serializing chains. (1 symbol)
  • ptwm_core::extension::builtins — Materialize ExtensionTableEntry values for in-tree built-ins. (3 symbols)
  • ptwm_core::extension::capability — Sealed-but-extensible capability vocabulary. (5 symbols)
  • ptwm_core::extension::dispatch — Static dispatch table for in-tree built-in ops and codecs. (2 symbols)
  • ptwm_core::extension::error — Extension-system error type. (1 symbol)
  • ptwm_core::extension::id — Canonical contribution identity = blake3(pubkey || name || version). (2 symbols)
  • ptwm_core::extension::kind — Closed-but-versioned taxonomy of contribution kinds. (1 symbol)
  • ptwm_core::extension::lifecycle — Lifecycle declares the instance-allocation policy the host should use. (1 symbol)
  • ptwm_core::extension::manifest — TOML manifest schema for an extension bundle (manifest.toml). (3 symbols)
  • ptwm_core::extension::table — On-disk Extension Table format. (8 symbols)
  • ptwm_core::fit — Shared codec state fitters. Each codec contributes its own fitter module; the top-level fit_shared aggregates their outputs into a flat SharedStates::entries list (codec-agnostic prelude allocation). (3 symbols)
  • ptwm_core::flavor — PTWM flavor system: WASM (Wasmtime), native cdylib (dlopen), and host-language entry points (Python). Each flavor materializes a Box from a disk path + manifest entry. (0 symbols)
  • ptwm_core::flavor::abi — Per-kind ABI: Rust traits the host dispatcher invokes. (30 symbols)
  • ptwm_core::flavor::host — Host-language (Python) extension descriptors. (1 symbol)
  • ptwm_core::flavor::native — Native cdylib loader. Loads a .so / .dylib / .dll via libloading, probes for the per-kind ABI symbols, and exposes a thin handle the dispatcher uses to invoke each method. (6 symbols)
  • ptwm_core::flavor::router — PlaneCodecRouter: canonical-id-keyed dispatch for both built-in and third-party plane codecs. (6 symbols)
  • ptwm_core::flavor::third_party — Bridge adapter that exposes a third-party DispatchedPlaneCodec (native or WASM, dispatched by canonical id) as if it were an in-tree crate::codec::PlaneCodec. The container decode loop can then treat every plane uniformly through a single Box<dyn PlaneCodec>. (1 symbol)
  • ptwm_core::flavor::wasm — Wasmtime integration. Host calls into a WASM module via a byte- buffer ABI (no Component Model — plain (ptr, len) pairs through linear memory). (5 symbols)
  • ptwm_core::header — v1 PTWM wire-format header. (5 symbols)
  • ptwm_core::index — EOF sentinel and tensor index. (6 symbols)
  • ptwm_core::install — Source-format dispatch for ptwm ext install. (4 symbols)
  • ptwm_core::install::git — Git source: git+https://...#<commit-sha>. Only commit-SHA pins are accepted so that re-installs are reproducible. (2 symbols)
  • ptwm_core::install::https — HTTPS install source. Fetches a .tar.zst from the given URL, unpacks into a staging directory, returns the TempDir. (1 symbol)
  • ptwm_core::install::local — Local install sources: directory and .tar.zst archive. (2 symbols)
  • ptwm_core::install::oci — OCI artifact install source: oci://<registry>/<path>:<tag>. (1 symbol)
  • ptwm_core::install::pip — Pip-package install source. Two paths: (3 symbols)
  • ptwm_core::layout — Per-plane physical structure passed to codecs at encode/decode time. (1 symbol)
  • ptwm_core::metadata — Tensor metadata convention for the .ptwm container. (2 symbols)
  • ptwm_core::plane_record — Plane record serialization. (8 symbols)
  • ptwm_core::policy — PTWM policy system: capability checks, native-dep verification, policy-file TOML schema, and the resolver that computes the encode-time candidate pool. (0 symbols)
  • ptwm_core::policy::capability_check — Capability verification against a HostPolicy. (4 symbols)
  • ptwm_core::policy::file — Policy TOML schema. (4 symbols)
  • ptwm_core::policy::native_deps — Verify declared native-library dependencies against the OS package manager. Linux: dpkg + rpm. macOS: codesign. Other platforms: fall through to Unverified-PlatformUnsupported. (12 symbols)
  • ptwm_core::policy::resolve — Compose a PolicyFile + trust state into a ResolvedPolicy. (2 symbols)
  • ptwm_core::prelude — Shared-state prelude. (5 symbols)
  • ptwm_core::quantize — Shared PMF quantization + validation primitives. Used by PerGroupCodebook (PGC, alphabet=16, total=256) and Order1ScaleAC (alphabet=256, total=4096); each codec passes its own total constant. (4 symbols)
  • ptwm_core::range_coder — Minimal, correct 32-bit range coder for PerGroupCodebook / Order1ScaleAC. (2 symbols)
  • ptwm_core::select — Per-plane trial-encode: try every menu item in parallel, pick smallest. (3 symbols)
  • ptwm_core::split — Byte/bit preprocessing for neural network weight compression. (0 symbols)
  • ptwm_core::split::dtype16 — BF16/FP16 split/combine. (5 symbols)
  • ptwm_core::split::dtype32 — FP32 split/combine. (5 symbols)
  • ptwm_core::split::dtype64 — 64-bit integer split/combine. (3 symbols)
  • ptwm_core::split::dtype8 — FP8 split and combine — supports passthrough, per-byte bit reorder, and nibble split. (3 symbols)
  • ptwm_core::tensor_record — Tensor record serialization. (9 symbols)
  • ptwm_core::transcode — PTWM container ⇄ canonical PTWM-JSON manifest transcoder. (4 symbols)
  • ptwm_core::transcode::manifest — Serde data model for the canonical PTWM-JSON manifest (Layer 1). (18 symbols)
  • ptwm_core::transforms — PPG op catalogue. The trait surface and per-op modules live here. (0 symbols)
  • ptwm_core::transforms::alpha_stable_normalize — AlphaStableNormalize — per-row robust affine normalization (v − δ)/γ with an exact XOR residual. δ = median(row), γ = median(|row − δ|) (MAD). Lossless by construction: the fit is encode-only, γ, δ are stored per row, and the residual repairs all round-off — so reconstruction is bit-exact regardless of the (heavy-tailed) distribution. (2 symbols)
  • ptwm_core::transforms::bit_reorder_fp8 — BitReorderFp8E4M3 and BitReorderFp8E5M2 ops — per-byte bit shuffle for FP8 formats that moves the exponent into the high nibble for better entropy coding. (2 symbols)
  • ptwm_core::transforms::bit_reorder_ieee — BitReorderIeee16 and BitReorderIeee32 ops — rearrange IEEE-754 float bits so the exponent occupies the most-significant position, improving entropy coding. (2 symbols)
  • ptwm_core::transforms::block_microscaling_repack — BlockMicroscalingRepack op — re-express a 16-bit float plane as a per-block shared scale (E8M0) plus a scale-relative element plane. (1 symbol)
  • ptwm_core::transforms::burrows_wheeler — BurrowsWheeler op — block-wise Burrows–Wheeler Transform (BWT). (2 symbols)
  • ptwm_core::transforms::byte_passthrough — BytePassthrough op — identity transform that passes one plane unchanged. (1 symbol)
  • ptwm_core::transforms::byte_split — ByteSplit op — round-robin deinterleave of a flat byte plane into N planes. (1 symbol)
  • ptwm_core::transforms::concat — Concat op — concatenates N input planes into a single byte plane. (2 symbols)
  • ptwm_core::transforms::delta — XorDelta and FloatDelta ops — cross-tensor residual encoding. (3 symbols)
  • ptwm_core::transforms::entropy — EntropyEstimate op — computes per-byte Shannon entropy and stores it in the output plane's vendor_bytes as a Q8.24 fixed-point value. (1 symbol)
  • ptwm_core::transforms::fusion — Fused transform implementations for hot pairs (4 symbols)
  • ptwm_core::transforms::index_pack — IndexBitwidthPack op — packs u8 indices into a tight MSB-first bit stream. (2 symbols)
  • ptwm_core::transforms::int_delta — IntDelta op — intra-plane integer-subtraction delta between consecutive elements (ndzip / FPC-family preprocessing). (1 symbol)
  • ptwm_core::transforms::mantissa_zero — MantissaZeroStrip op — strip a uniform run of trailing zero bits from every byte of a quantized-weight mantissa plane. (1 symbol)
  • ptwm_core::transforms::move_to_front — MoveToFront op — the move-to-front (MTF) byte transform. (1 symbol)
  • ptwm_core::transforms::mxfp4_deinterleave — MxFp4Deinterleave op — split OCP MXFP4 block-packed data into a nibble VALUE plane and a byte SCALE plane. (1 symbol)
  • ptwm_core::transforms::nibble_split — NibbleSplit op — split each input byte into high-nibble (exponent) and low-nibble (sign+mantissa) planes. (1 symbol)
  • ptwm_core::transforms::op — Op trait + OpId enum. The op catalogue's wire form lives here; per-op behaviour lives in sibling modules under transforms/. (3 symbols)
  • ptwm_core::transforms::predictor_xor — PredictorXor — FPC-style intra-plane predictor + XOR residual. (1 symbol)
  • ptwm_core::transforms::reshape — Reshape op — re-tags a plane's layout without modifying its bytes. (2 symbols)
  • ptwm_core::transforms::source — Source op — graph entry node that describes the raw input tensor. (3 symbols)
  • ptwm_core::transforms::spherical_normalize — SphericalNormalize — per-row (radius, direction) reparameterization with an exact XOR residual. Lossless by construction (the inverse reconstructs the prediction from the stored radius/direction and XORs the residual back, so float-op inaccuracy only affects residual size, never correctness). (2 symbols)
  • ptwm_core::transforms::terminal — Terminal op — graph leaf node that consumes one plane and feeds a codec. (2 symbols)
  • ptwm_core::trust — PTWM trust system: Ed25519 signatures, bundled-then-pinned keyring, org-trust-manifest delegation, and the Verifier that decides whether an ExtensionTableEntry is trusted. (0 symbols)
  • ptwm_core::trust::bundled — Bundled-then-pinned trust state evaluator. (14 symbols)
  • ptwm_core::trust::keyring — User-facing trust keyring. (2 symbols)
  • ptwm_core::trust::org — Org trust manifests — signed TOML files vouching for member author keys. (6 symbols)
  • ptwm_core::trust::signature — Ed25519 sign/verify wrappers. (2 symbols)
  • ptwm_core::trust::verifier — Decides whether an ExtensionTableEntry is trusted given the active keyring and the bundled-key status. Built-ins (whose canonical_id was derived from BUILTIN_PUBKEY) are trusted by construction. (2 symbols)
  • ptwm_core::types — PPG type system: plane descriptors and supporting types. (0 symbols)
  • ptwm_core::types::descriptor — Plane descriptors and supporting type-level values. (5 symbols)
  • ptwm_core::types::plane_role — On-disk plane role tag (single byte stored in each plane record). (1 symbol)
  • ptwm_core::types::role — Role tagged union and supporting format enums. Wire encoding uses tag bytes (5 symbols)

utils

Utility functions for PTWM.

ptwm_core

Pure-Rust core for lossless neural-network weight compression.