PTWM
ptwm

ptwm.integrations

Integrations for third-party libraries.

ptwm.integrations

Integrations for third-party libraries.

pymodule

ptwm.integrations

python/ptwm/integrations/__init__.py:1
module ptwm.integrations

Integrations for third-party libraries.

pyfunction

ptwm.integrations.compress_safetensors_file

python/ptwm/integrations/_compress_safetensors.py:169
def compress_safetensors_file(src_path: str | Path, out_dir: str | Path, mode: Literal['a', 'b'] = 'a', classifier: TensorClassifier | ClassifierChain | None = None, max_shard_size: int = _DEFAULT_MAX_SHARD_SIZE, method_hint: int = 3, explore_options: ExploreOptions | None = None, use_user_cache: bool = True, fast: bool = False, fast_chunk: int = 512 * 1024) -> AuditLog

Compress a safetensors file into a sharded ``.ptwm`` directory.

NameTypeDescription
src_pathstr | PathPath to the source ``.safetensors`` file.
out_dirstr | PathOutput directory. Created if it does not exist.
mode = 'a'Literal['a', 'b']``"a"`` — native ``.ptwm`` shards (default). ``"b"`` — ``.safetensors`` shell containing a ``.ptwm`` blob (requires ``write_mode_b_shell``).
classifier = NoneTensorClassifier | ClassifierChain | NoneOptional classifier or chain. ``None`` → empty chain → every tensor classified ``STANDARD``.
max_shard_size = _DEFAULT_MAX_SHARD_SIZEintMaximum uncompressed byte budget per shard (default 5 GiB).
method_hint = 3intMethod hint forwarded to ``write_ptwm_shard`` (1=HUFFMAN, 2=ZSTD, 3=MICROSCALE, 4=RANS, 5=IDENTITY).
Returns
AuditLogPer-tensor classification audit log.
pyfunction

ptwm.integrations.discover_ptwm_shards

python/ptwm/integrations/_hf.py:164
def discover_ptwm_shards(model_dir: str | os.PathLike[str]) -> list[Path]

Return the list of ``.ptwm`` shards in ``model_dir`` (or empty list).

Uses ``model.ptwm.index.json`` if present; otherwise looks for a single ``model.ptwm``.
NameTypeDescription
model_dirstr | os.PathLike[str]Directory to search for ``.ptwm`` shards.
Returns
list of :class:`pathlib.Path`Shard paths in sorted order, or an empty list if none are found.
pyfunction

ptwm.integrations.patch_safetensors

python/ptwm/integrations/_safetensors.py:162
def patch_safetensors() -> None
pyfunction

ptwm.integrations.patch_transformers

python/ptwm/integrations/_hf.py:240
def patch_transformers() -> None

Install the ``.ptwm``-aware ``load_state_dict`` replacement.

Patches :func:`transformers.modeling_utils.load_state_dict` so that any transformers path funnelling through it — including :meth:`PreTrainedModel.from_pretrained` with an explicit local ``.ptwm`` path — decompresses transparently. Also installs a best-effort patch on :func:`transformers.modeling_utils._get_resolved_checkpoint_files` so ``from_pretrained`` discovers ``model.ptwm.index.json`` + ``.ptwm`` shards in local directories. If the patch target is absent in the installed transformers version, emits a warning and skips the patch (:func:`materialize_hf_cache` still works as a fallback). For HF cache snapshots that store ``*.safetensors.ptwm`` instead of ``*.safetensors``, run :func:`materialize_hf_cache` on the snapshot directory once; transformers' own file discovery in ``from_pretrained`` then finds the decompressed files by their native names with no further patching.
pyclass

ptwm.integrations.SafeOpen

python/ptwm/integrations/_safetensors.py:41
class SafeOpen:

``safetensors.safe_open`` wrapper recognising .ptwm + ptwm-shell.

Lazy decoding: the compressed blob stays in memory; individual tensors decode only on :meth:`get_tensor`. This preserves safetensors' lazy-load contract for large multi-shard checkpoints.
pymethod

ptwm.integrations.SafeOpen.__enter__

python/ptwm/integrations/_safetensors.py:109
def __enter__(self) -> Self
pymethod

ptwm.integrations.SafeOpen.__exit__

python/ptwm/integrations/_safetensors.py:112
def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None
pymethod

ptwm.integrations.SafeOpen.__init__

python/ptwm/integrations/_safetensors.py:49
def __init__(self, filename: str, framework: str, device: str = 'cpu') -> None
pyattribute

ptwm.integrations.SafeOpen.compressed_tensors_metadata

python/ptwm/integrations/_safetensors.py:61
compressed_tensors_metadata: dict[str, Any] = {}
pymethod

ptwm.integrations.SafeOpen.get_slice

python/ptwm/integrations/_safetensors.py:101
def get_slice(self, name: str) -> Any
pymethod

ptwm.integrations.SafeOpen.get_tensor

python/ptwm/integrations/_safetensors.py:94
def get_tensor(self, name: str) -> torch.Tensor
pymethod

ptwm.integrations.SafeOpen.keys

python/ptwm/integrations/_safetensors.py:84
def keys(self) -> list[str]
pymethod

ptwm.integrations.SafeOpen.metadata

python/ptwm/integrations/_safetensors.py:89
def metadata(self) -> dict[str, str] | None