ptwm
ptwm.sharding
Sharded .ptwm writers/readers.
ptwm.sharding
Sharded .ptwm writers/readers.
ptwm.sharding
python/ptwm/sharding/__init__.py:1module ptwm.shardingSharded ``.ptwm`` writers/readers.
ptwm.sharding.compress_ptwm_blob
python/ptwm/sharding/_writer.py:30def compress_ptwm_blob(tensors: Sequence[tuple[str, bytes, Any, tuple[int, ...]]], chains_per_tensor: list[list[bytes]] | None = None, method_hint: int = 3, emit_payload_hash: bool = True, emit_plane_crc: bool = True, streaming_chunk: int | None = None) -> bytesCompress ``tensors`` into a single in-memory ``.ptwm`` container blob.
``tensors`` is a sequence of ``(name, raw_bytes, dtype, shape)``.
``chains_per_tensor`` — one list of chain wire-byte blobs per tensor;
each blob is produced by ``Chain.to_bytes()``. When ``None``, chains are
auto-derived from each tensor's dtype via the STANDARD production table.
``method_hint``: 1=HUFFMAN, 2=ZSTD, 3=MICROSCALE, 4=RANS, 5=IDENTITY.
ptwm.sharding.plan_shards
python/ptwm/sharding/_sharder.py:18def plan_shards(sized_tensors: Sequence[tuple[str, int]], max_shard_size: int) -> list[list[tuple[str, int]]]Greedy pack tensors in input order, respecting ``max_shard_size``.
A single tensor larger than ``max_shard_size`` gets its own shard.
ptwm.sharding.PtwmIndex
python/ptwm/sharding/_index.py:12class PtwmIndex:ptwm.sharding.PtwmIndex.__init__
python/ptwm/sharding/_index.py:1def __init__(self, total_size: int, compressed_total_size: int, weight_map: dict[str, str] = dict()) -> Noneptwm.sharding.PtwmIndex.compressed_total_size
python/ptwm/sharding/_index.py:15compressed_total_size: intptwm.sharding.PtwmIndex.read
python/ptwm/sharding/_index.py:30def read(cls, path: str | Path) -> PtwmIndexptwm.sharding.PtwmIndex.total_size
python/ptwm/sharding/_index.py:14total_size: intptwm.sharding.PtwmIndex.weight_map
python/ptwm/sharding/_index.py:16weight_map: dict[str, str] = field(default_factory=dict)ptwm.sharding.PtwmIndex.write
python/ptwm/sharding/_index.py:18def write(self, path: str | Path) -> Noneptwm.sharding.read_sharded_ptwm
python/ptwm/sharding/_reader.py:13def read_sharded_ptwm(model_dir: str | Path) -> dict[str, bytes]Open every shard listed in ``model.ptwm.index.json``, return the union.
If no index file is present, falls back to a single ``model.ptwm``
in the directory.
| Name | Type | Description |
|---|---|---|
| model_dir | str | Path | Directory containing ``.ptwm`` shard files and optionally a ``model.ptwm.index.json`` index. |
Returns
dict[str, bytes] — Flat mapping of tensor name → raw decompressed bytes, merged
across all shards. Raises
FileNotFoundError— If neither ``model.ptwm.index.json`` nor ``model.ptwm`` exists.ValueError— If the same tensor name appears in more than one shard.
ptwm.sharding.shard_filename
python/ptwm/sharding/_sharder.py:10def shard_filename(idx_one_based: int, total: int, suffix: str) -> strReturn ``model-NNNNN-of-MMMMM.<suffix>``.
ptwm.sharding.write_ptwm_shard
python/ptwm/sharding/_writer.py:78def write_ptwm_shard(out_path: str | Path, tensors: Sequence[tuple[str, bytes, Any, tuple[int, ...]]], chains_per_tensor: list[list[bytes]] | None = None, method_hint: int = 3, emit_payload_hash: bool = True, emit_plane_crc: bool = True, streaming_chunk: int | None = None) -> NoneWrite a single ``.ptwm`` container holding ``tensors``.
See :func:`compress_ptwm_blob` for the parameter semantics.