ptwm
ptwm.integrations
Integrations for third-party libraries.
ptwm.integrations
Integrations for third-party libraries.
ptwm.integrations
python/ptwm/integrations/__init__.py:1module ptwm.integrationsIntegrations for third-party libraries.
ptwm.integrations.compress_safetensors_file
python/ptwm/integrations/_compress_safetensors.py:169def compress_safetensors_file(src_path: str | Path, out_dir: str | Path, mode: Literal['a', 'b'] = 'a', classifier: TensorClassifier | ClassifierChain | None = None, max_shard_size: int = _DEFAULT_MAX_SHARD_SIZE, method_hint: int = 3, explore_options: ExploreOptions | None = None, use_user_cache: bool = True, fast: bool = False, fast_chunk: int = 512 * 1024) -> AuditLogCompress a safetensors file into a sharded ``.ptwm`` directory.
| Name | Type | Description |
|---|---|---|
| src_path | str | Path | Path to the source ``.safetensors`` file. |
| out_dir | str | Path | Output directory. Created if it does not exist. |
| mode = 'a' | Literal['a', 'b'] | ``"a"`` — native ``.ptwm`` shards (default). ``"b"`` — ``.safetensors`` shell containing a ``.ptwm`` blob (requires ``write_mode_b_shell``). |
| classifier = None | TensorClassifier | ClassifierChain | None | Optional classifier or chain. ``None`` → empty chain → every tensor classified ``STANDARD``. |
| max_shard_size = _DEFAULT_MAX_SHARD_SIZE | int | Maximum uncompressed byte budget per shard (default 5 GiB). |
| method_hint = 3 | int | Method hint forwarded to ``write_ptwm_shard`` (1=HUFFMAN, 2=ZSTD, 3=MICROSCALE, 4=RANS, 5=IDENTITY). |
Returns
AuditLog — Per-tensor classification audit log.ptwm.integrations.discover_ptwm_shards
python/ptwm/integrations/_hf.py:164def discover_ptwm_shards(model_dir: str | os.PathLike[str]) -> list[Path]Return the list of ``.ptwm`` shards in ``model_dir`` (or empty list).
Uses ``model.ptwm.index.json`` if present; otherwise looks for a single
``model.ptwm``.
| Name | Type | Description |
|---|---|---|
| model_dir | str | os.PathLike[str] | Directory to search for ``.ptwm`` shards. |
Returns
list of :class:`pathlib.Path` — Shard paths in sorted order, or an empty list if none are found.ptwm.integrations.patch_safetensors
python/ptwm/integrations/_safetensors.py:162def patch_safetensors() -> Noneptwm.integrations.patch_transformers
python/ptwm/integrations/_hf.py:240def patch_transformers() -> NoneInstall the ``.ptwm``-aware ``load_state_dict`` replacement.
Patches :func:`transformers.modeling_utils.load_state_dict` so that any
transformers path funnelling through it — including
:meth:`PreTrainedModel.from_pretrained` with an explicit local ``.ptwm``
path — decompresses transparently.
Also installs a best-effort patch on
:func:`transformers.modeling_utils._get_resolved_checkpoint_files` so
``from_pretrained`` discovers ``model.ptwm.index.json`` + ``.ptwm`` shards
in local directories. If the patch target is absent in the installed
transformers version, emits a warning and skips the patch
(:func:`materialize_hf_cache` still works as a fallback).
For HF cache snapshots that store ``*.safetensors.ptwm`` instead of
``*.safetensors``, run :func:`materialize_hf_cache` on the snapshot
directory once; transformers' own file discovery in ``from_pretrained``
then finds the decompressed files by their native names with no further
patching.
ptwm.integrations.SafeOpen
python/ptwm/integrations/_safetensors.py:41class SafeOpen:``safetensors.safe_open`` wrapper recognising .ptwm + ptwm-shell.
Lazy decoding: the compressed blob stays in memory; individual tensors
decode only on :meth:`get_tensor`. This preserves safetensors' lazy-load
contract for large multi-shard checkpoints.
ptwm.integrations.SafeOpen.__enter__
python/ptwm/integrations/_safetensors.py:109def __enter__(self) -> Selfptwm.integrations.SafeOpen.__exit__
python/ptwm/integrations/_safetensors.py:112def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> Noneptwm.integrations.SafeOpen.__init__
python/ptwm/integrations/_safetensors.py:49def __init__(self, filename: str, framework: str, device: str = 'cpu') -> Noneptwm.integrations.SafeOpen.compressed_tensors_metadata
python/ptwm/integrations/_safetensors.py:61compressed_tensors_metadata: dict[str, Any] = {}ptwm.integrations.SafeOpen.get_slice
python/ptwm/integrations/_safetensors.py:101def get_slice(self, name: str) -> Anyptwm.integrations.SafeOpen.get_tensor
python/ptwm/integrations/_safetensors.py:94def get_tensor(self, name: str) -> torch.Tensorptwm.integrations.SafeOpen.keys
python/ptwm/integrations/_safetensors.py:84def keys(self) -> list[str]ptwm.integrations.SafeOpen.metadata
python/ptwm/integrations/_safetensors.py:89def metadata(self) -> dict[str, str] | None