Low effort scraping Python's pickle format in Rust. It is to complete pickle parsing as BeautifulSoup was to complete HTML parsing.



Because it is, isn't it?

This is a Rust crate for dealing with the Python pickle format.

It also has support for opening PyTorch files (.pth, .pt) which are basically ZIP files with a pickle hidden inside.


The Python pickle format. There are people who get paid way more than me who spent massive amounts of effort working on copy protection schemes that were less effective at preventing interoperation than the Python pickle format.

Why would anyone actually make such a thing? My theory is that the originator has a deep and abiding hate for their fellow humans. The only thing that gives them any pleasure is to cause pain to as many people as possible. To that end: the Python pickle format.

It's basically impossible to handle reliably unless you're Python. In an ideal world, you'd just leave the pickles alone but sadly that is not the world we find ourselves living in. Sometimes you need to get some data out of a pickle file and you don't want to embed a whole Python interpreter into your application or something crazy like that.

That's where repugnant-pickle comes in. It does what it can and gives you what it came up with.

The name is a riff on BeautifulSoup, a HTML scraper library. It didn't (doesn't?) try to parse HTML completely (which is insanely hard), it just gives you a relatively simple way to to to extract the data you need. Most of the time that's good enough.


Here's the kind of output you could expect to get from parsing a PyTorch file:

    Raw(GLOBAL("collections", "OrderedDict"), [
      Seq(Tuple, []),
      Seq(Tuple, [
        Seq(Dict, [
          Global(Raw(GLOBAL("torch._utils", "_rebuild_tensor_v2")), [
            Seq(Tuple, [
              PersId(Seq(Tuple, [
                Raw(GLOBAL("torch", "BFloat16Storage")),
              Seq(Tuple, [Int(50277), Int(1024)]),
              Seq(Tuple, [Int(1024), Int(1)]),
              Global(Raw(GLOBAL("collections", "OrderedDict")), [
                Seq(Tuple, [])
        Seq(Dict, [
          Global(Raw(GLOBAL("torch._utils", "_rebuild_tensor_v2")), [
            // Etc

What about serde-pickle?

Use it if you can. It absolutely will be a much nicer experience than this crate. However, there are things it can't handle such as persistant IDs. PyTorch files use persistant IDs, so you don't really have a choice in that case.

There are also some odd choices like forcing the Rust toolchain to version 1.41.


You can enable support for attempting to deal with PyTorch files with the torch feature. If your Torch file has weird stuff, you may have to deal with it manually. Look in src/torch.rs for an example of where to start.

Otherwise, you can use repugnant_pickle::torch::RepugnantTorchTensors::new_from_file to load the tensor metadata. For example, if it works, you'll get something like this:

       RepugnantTorchTensor {
           name: "emb.weight",
           device: "cuda:0",
           tensor_type: BFloat16,
           storage: "archive/data/0",
           storage_len: 430348288,
           storage_offset: 327378944,
           absolute_offset: 327445248,
           shape: [50277, 1024],
           stride: [1024, 1],
           requires_grad: false,
       RepugnantTorchTensor {
           name: "blocks.0.ln1.weight",
           device: "cuda:0",
           tensor_type: BFloat16,
           storage: "archive/data/0",
           storage_len: 430348288,
           storage_offset: 13639680,
           absolute_offset: 13705984,
           shape: [1024],
           stride: [1],
           requires_grad: false,

You'll have to calculate the length yourself from the type and shape. For example, the first tensor has shape [1024, 50277] and type BFloat16. So the length in bytes would be (1024 * 50277) * 2 and the data would start at offset 327445248 in the Torch file, or if you access the specific file archive/data/0 in the Torch ZIP, it would start at 327378944.

It worked on the .pth and .pt LLM models I have.

Source: Trust me bro.


Look at the examples in examples:

  • dump_raw.rs — Dumps the raw Pickle opcodes from a file.
  • dump.rs — Dumps the Values from a file.
  • dump_torch.rs — Dumps the tensor metadata for a PyTorch file.

Note that for the last one you'll need to have the torch feature enabled. You can also run the example like:

cargo run --features torch --example dump_raw -- /path/to/torchfile.pth

Warning: Don't try to run dump or dump_raw on a Torch file. It'll try to load the entire massive file as a pickle.


This isn't very well tested. It probably won't delete all your cat pictures when it fails, but it could fail.

Also, be careful of recursive Pickle files. It won't crash, but it'll loop until it hits the recursion limit (about 250) and you'll get nested data up to that limit.

If you need to handle large Pickle files, ideally don't use this at all, but if you must then you'd probably be better off using the single Pickle op Nom parser in src/parsers.rs.

You might also like...
Rust library for Self Organising Maps (SOM).
Rust library for Self Organising Maps (SOM).

RusticSOM Rust library for Self Organising Maps (SOM). Using this Crate Add rusticsom as a dependency in Cargo.toml [dependencies] rusticsom = "1.1.0"

Rust language bindings for TensorFlow
Rust language bindings for TensorFlow

TensorFlow Rust provides idiomatic Rust language bindings for TensorFlow. Notice: This project is still under active development and not guaranteed to

Machine learning crate for Rust

rustlearn A machine learning package for Rust. For full usage details, see the API documentation. Introduction This crate contains reasonably effectiv

Rust bindings for the C++ api of PyTorch.

tch-rs Rust bindings for the C++ api of PyTorch. The goal of the tch crate is to provide some thin wrappers around the C++ PyTorch api (a.k.a. libtorc

个人的 rust 学习资料

🔝 通知: 项目文档迁移到: https://github.com/higker/learn-rust learning-rust-zh 个人的 rust 学习资料 学习目录 目录 源代码地址 相关解析 第一个rust程序 https://github.com/higker/learning-ru

Distributed compute platform implemented in Rust, and powered by Apache Arrow.
Distributed compute platform implemented in Rust, and powered by Apache Arrow.

Ballista: Distributed Compute Platform Overview Ballista is a distributed compute platform primarily implemented in Rust, powered by Apache Arrow. It

Tensors and differentiable operations (like TensorFlow) in Rust

autograd Differentiable operations and tensors backed by ndarray. Motivation Machine learning is one of the field where Rust lagging behind other lang

Rust numeric library with R, MATLAB & Python syntax

Peroxide Rust numeric library contains linear algebra, numerical analysis, statistics and machine learning tools with R, MATLAB, Python like macros. W

A fast, safe and easy to use reinforcement learning framework in Rust.
A fast, safe and easy to use reinforcement learning framework in Rust.

RSRL (api) Reinforcement learning should be fast, safe and easy to use. Overview rsrl provides generic constructs for reinforcement learning (RL) expe

I like strawberry nicecream.
A pure, low-level tensor program representation enabling tensor program optimization via program rewriting

Glenside is a pure, low-level tensor program representation which enables tensor program optimization via program rewriting, using rewriting frameworks such as the egg equality saturation library.

Gus Smith 45 Dec 28, 2022
Serenade: Low-Latency Session-Based Recommendations

Serenade: Low-Latency Session-Based Recommendations This repository contains the official code for session-based recommender system Serenade, which em

bol.com 61 Dec 16, 2022
The CBNF neural network header format.

cbnf The CBNF neural network header format. What is CBNF? CBNF is a neural network header format for use with efficiently updatable neural networks fo

Cosmo Bobak 2 Jul 11, 2023
PyQIR is a set of APIs for generating, parsing, and evaluating Quantum Intermediate Representation (QIR).

PyQIR PyQIR is a set of APIs for generating, parsing, and evaluating Quantum Intermediate Representation (QIR). It consists of the following component

QIR Alliance 37 Dec 31, 2022
Msgpack serialization/deserialization library for Python, written in Rust using PyO3, and rust-msgpack. Reboot of orjson. msgpack.org[Python]

ormsgpack ormsgpack is a fast msgpack library for Python. It is a fork/reboot of orjson It serializes faster than msgpack-python and deserializes a bi

Aviram Hassan 139 Dec 30, 2022
Practice repo for learning Rust. Currently going through "Rust for JavaScript Developers" course.

rust-practice ?? Practice repo for learning Rust. Directories /rust-for-js-dev Files directed towards "Rust for JavaScript Developers" course. Thank y

Sammy Samkough 0 Dec 25, 2021
A Rust library with homemade machine learning models to classify the MNIST dataset. Built in an attempt to get familiar with advanced Rust concepts.

mnist-classifier Ideas UPDATED: Finish CLI Flags Parallelize conputationally intensive functions Class-based naive bayes README Image parsing Confusio

Neil Kaushikkar 0 Sep 2, 2021
🦀Rust Turkiye - Rust Dersleri

Rust Turkiye - Rust Dersleri CURIOSITY - Featuring Richard Feynman Bu repo Rust Turkiye tarafindan duzenlenen Rust Dersleri egitiminin alistirma ve ko

Theo M. Bulut 12 Jan 14, 2023
A Rust machine learning framework.

Linfa linfa (Italian) / sap (English): The vital circulating fluid of a plant. linfa aims to provide a comprehensive toolkit to build Machine Learning

Rust-ML 2.2k Jan 2, 2023
Machine Learning library for Rust

rusty-machine This library is no longer actively maintained. The crate is currently on version 0.5.4. Read the API Documentation to learn more. And he

James Lucas 1.2k Dec 31, 2022