Pyo3 - Rust bindings for the Python interpreter

Last update: Jun 26, 2022

PyO3

actions status benchmark codecov crates.io minimum rustc 1.48 dev chat contributing notes

Rust bindings for Python, including tools for creating native Python extension modules. Running and interacting with Python code from a Rust binary is also supported.

Usage

PyO3 supports the following software versions:

  • Python 3.7 and up (CPython and PyPy)
  • Rust 1.48 and up

You can use PyO3 to write a native Python module in Rust, or to embed Python in a Rust binary. The following sections explain each of these in turn.

Using Rust from Python

PyO3 can be used to generate a native Python module. The easiest way to try this out for the first time is to use maturin. maturin is a tool for building and publishing Rust-based Python packages with minimal configuration. The following steps install maturin, use it to generate and build a new Python package, and then launch Python to import and execute a function from the package.

First, follow the commands below to create a new directory containing a new Python virtualenv, and install maturin into the virtualenv using Python's package manager, pip:

# (replace string_sum with the desired package name)
$ mkdir string_sum
$ cd string_sum
$ python -m venv .env
$ source .env/bin/activate
$ pip install maturin

Still inside this string_sum directory, now run maturin init. This will generate the new package source. When given the choice of bindings to use, select pyo3 bindings:

$ maturin init
โœ” ๐Ÿคท What kind of bindings to use? ยท pyo3
  โœจ Done! New project created string_sum

The most important files generated by this command are Cargo.toml and lib.rs, which will look roughly like the following:

Cargo.toml

[package]
name = "string_sum"
version = "0.1.0"
edition = "2018"

[lib]
name = "string_sum"
# "cdylib" is necessary to produce a shared library for Python to import from.
#
# Downstream Rust code (including code in `bin/`, `examples/`, and `tests/`) will not be able
# to `use string_sum;` unless the "rlib" or "lib" crate type is also included, e.g.:
# crate-type = ["cdylib", "rlib"]
crate-type = ["cdylib"]

[dependencies]
pyo3 = { version = "0.15.1", features = ["extension-module"] }

src/lib.rs

use pyo3::prelude::*;

/// Formats the sum of two numbers as string.
#[pyfunction]
fn sum_as_string(a: usize, b: usize) -> PyResult<String> {
    Ok((a + b).to_string())
}

/// A Python module implemented in Rust. The name of this function must match
/// the `lib.name` setting in the `Cargo.toml`, else Python will not be able to
/// import the module.
#[pymodule]
fn string_sum(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(sum_as_string, m)?)?;
    Ok(())
}

Finally, run maturin develop. This will build the package and install it into the Python virtualenv previously created and activated. The package is then ready to be used from python:

$ maturin develop
# lots of progress output as maturin runs the compilation...
$ python
>>> import string_sum
>>> string_sum.sum_as_string(5, 20)
'25'

To make changes to the package, just edit the Rust source code and then re-run maturin develop to recompile.

To run this all as a single copy-and-paste, use the bash script below (replace string_sum in the first command with the desired package name):

mkdir string_sum && cd "$_"
python -m venv .env
source .env/bin/activate
pip install maturin
maturin init --bindings pyo3
maturin develop

As well as with maturin, it is possible to build using setuptools-rust or manually. Both offer more flexibility than maturin but require more configuration to get started.

Using Python from Rust

To embed Python into a Rust binary, you need to ensure that your Python installation contains a shared library. The following steps demonstrate how to ensure this (for Ubuntu), and then give some example code which runs an embedded Python interpreter.

To install the Python shared library on Ubuntu:

sudo apt install python3-dev

Start a new project with cargo new and add pyo3 to the Cargo.toml like this:

[dependencies.pyo3]
version = "0.15.1"
features = ["auto-initialize"]

Example program displaying the value of sys.version and the current user name:

use pyo3::prelude::*;
use pyo3::types::IntoPyDict;

fn main() -> PyResult<()> {
    Python::with_gil(|py| {
        let sys = py.import("sys")?;
        let version: String = sys.getattr("version")?.extract()?;

        let locals = [("os", py.import("os")?)].into_py_dict(py);
        let code = "os.getenv('USER') or os.getenv('USERNAME') or 'Unknown'";
        let user: String = py.eval(code, None, Some(&locals))?.extract()?;

        println!("Hello {}, I'm Python {}", user, version);
        Ok(())
    })
}

The guide has a section with lots of examples about this topic.

Tools and libraries

  • maturin Zero configuration build tool for Rust-made Python extensions.
  • setuptools-rust Setuptools plugin for Rust support.
  • pyo3-built Simple macro to expose metadata obtained with the built crate as a PyDict
  • rust-numpy Rust binding of NumPy C-API
  • dict-derive Derive FromPyObject to automatically transform Python dicts into Rust structs
  • pyo3-log Bridge from Rust to Python logging
  • pythonize Serde serializer for converting Rust objects to JSON-compatible Python objects
  • pyo3-asyncio Utilities for working with Python's Asyncio library and async functions

Examples

  • hyperjson A hyper-fast Python module for reading/writing JSON data using Rust's serde-json
  • html-py-ever Using html5ever through kuchiki to speed up html parsing and css-selecting.
  • point-process High level API for pointprocesses as a Python library
  • autopy A simple, cross-platform GUI automation library for Python and Rust.
    • Contains an example of building wheels on TravisCI and appveyor using cibuildwheel
  • orjson Fast Python JSON library
  • inline-python Inline Python code directly in your Rust code
  • Rogue-Gym Customizable rogue-like game for AI experiments
    • Contains an example of building wheels on Azure Pipelines
  • fastuuid Python bindings to Rust's UUID library
  • wasmer-python Python library to run WebAssembly binaries
  • mocpy Astronomical Python library offering data structures for describing any arbitrary coverage regions on the unit sphere
  • tokenizers Python bindings to the Hugging Face tokenizers (NLP) written in Rust
  • pyre Fast Python HTTP server written in Rust
  • jsonschema-rs Fast JSON Schema validation library
  • css-inline CSS inlining for Python implemented in Rust
  • cryptography Python cryptography library with some functionality in Rust
  • polaroid Hyper Fast and safe image manipulation library for Python written in Rust
  • ormsgpack Fast Python msgpack library
  • bed-reader Read and write the PLINK BED format, simply and efficiently
    • Shows Rayon/ndarray::parallel (including capturing errors, controlling thread num), Python types to Rust generics, Github Actions

Articles and other media

Contributing

Everyone is welcomed to contribute to PyO3! There are many ways to support the project, such as:

  • help PyO3 users with issues on GitHub and Gitter
  • improve documentation
  • write features and bugfixes
  • publish blogs and examples of how to use PyO3

Our contributing notes and architecture guide have more resources if you wish to volunteer time for PyO3 and are searching where to start.

If you don't have time to contribute yourself but still wish to support the project's future success, some of our maintainers have GitHub sponsorship pages:

License

PyO3 is licensed under the Apache-2.0 license. Python is licensed under the Python License.

GitHub

https://github.com/PyO3/pyo3
Comments
  • 1. Added Rust initialisation of Python-allocated bytes

    Adds new_with<F: Fn(&mut [u8])>(py: Python<'_>, len: usize, init: F) -> &PyBytes as suggested by @davidhewitt in #617. This allows initialising new PyBytes in Rust like such:

    let py_bytes = PyBytes::new_with(py, 10, |b: &mut [u8]| {
        b.copy_from_slice(b"Hello Rust");
    });
    

    Currently, it follows the semantics of PyBytes::new in that it panics if a memory error occurs in Python. Maybe my implementation of that could be improved.

    Reviewed by MomoLangenstein at 2020-08-01 10:00
  • 2. FromPyObject derivation for structs and enums

    This is a draft addressing a bit of both topics in #301 and #1055.

    I have not written proc-macro code before, so this might be quite ugly. This is mostly proof-of-concept, but I think it'd be a great quality-of-life improvement if something like this was available.

    This produces the following behaviour:

    >>> union.foo(["Test"])
    ["Test"]
    >>> union.foo("Test")
    Test
    >>> union.foo(1)
    1
    >>> union.foo(None)
    TypeError: Can't convert NoneType to Union[Str, Int, StringList]
    

    This implementation covers both the cheap downcast variant and extraction of proper Rust types. Although I'm not sure whether relying on the variant holding a reference is the proper heuristic to apply here.


    Todos are marked in the code, a proper implementation should in my opinion:

    • make the reported names of the variants in the error configurable through an attribute on the variant, i.e.:
      #[union]
      pub enum MyUnion {
          #[rename(str)]
          SomeVariant(String)
      } 
      
    • support named fields & variants with multiple fields
    Reviewed by sebpuetz at 2020-07-23 14:18
  • 3. Add catch_unwind! macro to prevent panics crossing ffi boundaries

    Fixes #492

    This is a first attempt at updating all the wrapping code to use catch_unwind to prevent unsoundness of panics inside Rust callback functions.

    I implemented this using a pyo3::catch_unwind macro, which takes a Python token and a body of code which should evaluate to a PyResult.

    I ran into complications around lifetimes and the callback converters so I ended up simplifying a lot of the callback conversion code. I think a lot of those macros are now in slightly better shape but there's probably more refactoring that could be done.

    Please give this a thorough review, I'm prepared to rework this as many times as needed to get it right.

    TODO:

    • [X] Custom Exception type from BaseException
      • [ ] Mechanism for users to import this Exception (will do later after #805)
      • [x] Resume unwind when pyO3 gets this Exception in a PyErr
    • [ ] Panic section for the book
    Reviewed by davidhewitt at 2020-03-08 21:22
  • 4. Get rid of specialization

    ~The last step for #5 is getting rid of specialization. I've remove most uses in 7c0379b1 and subsequent commits, except for the following ~17~ ~10~ ~9~ ~7~ 4:~ https://github.com/rust-lang/rust/pull/64564 changed the rules so there are again many default fn.

    src/class/descr.rs
    76:    default fn tp_descr_get() -> Option<ffi::descrgetfunc> {
    97:    default fn tp_descr_set() -> Option<ffi::descrsetfunc> {
    130:    default fn tp_as_descr(_type_object: &mut ffi::PyTypeObject) {}
    
    src/class/buffer.rs
    46:    default fn tp_as_buffer() -> Option<ffi::PyBufferProcs> {
    74:    default fn cb_bf_getbuffer() -> Option<ffi::getbufferproc> {
    110:    default fn cb_bf_releasebuffer() -> Option<ffi::releasebufferproc> {
    
    src/class/gc.rs
    27:    default fn update_type_object(_type_object: &mut ffi::PyTypeObject) {}
    72:    default fn tp_traverse() -> Option<ffi::traverseproc> {
    124:    default fn tp_clear() -> Option<ffi::inquiry> {
    
    src/class/iter.rs
    49:    default fn tp_as_iter(_typeob: &mut ffi::PyTypeObject) {}
    71:    default fn tp_iter() -> Option<ffi::getiterfunc> {
    94:    default fn tp_iternext() -> Option<ffi::iternextfunc> {
    
    src/class/basic.rs
    152:    default fn tp_as_object(_type_object: &mut ffi::PyTypeObject) {}
    153:    default fn nb_bool_fn() -> Option<ffi::inquiry> {
    183:    default fn tp_getattro() -> Option<ffi::binaryfunc> {
    252:        default fn set_attr() -> Option<ffi::setattrofunc> {
    274:        default fn del_attr() -> Option<ffi::setattrofunc> {
    296:        default fn set_del_attr() -> Option<ffi::setattrofunc> {
    324:    default fn tp_str() -> Option<ffi::unaryfunc> {
    344:    default fn tp_repr() -> Option<ffi::unaryfunc> {
    364:    default fn tp_hash() -> Option<ffi::hashfunc> {
    389:    default fn nb_bool() -> Option<ffi::inquiry> {
    409:    default fn tp_richcompare() -> Option<ffi::richcmpfunc> {
    
    src/class/mapping.rs
    83:    default fn tp_as_mapping() -> Option<ffi::PyMappingMethods> {
    116:    default fn mp_length() -> Option<ffi::lenfunc> {
    139:    default fn mp_subscript() -> Option<ffi::binaryfunc> {
    162:    default fn mp_ass_subscript() -> Option<ffi::objobjargproc> {
    187:    default fn mp_del_subscript() -> Option<ffi::objobjargproc> {
    201:    default fn det_set_dispatch() -> Option<ffi::objobjargproc> {
    
    src/class/pyasync.rs
    94:    default fn tp_as_async() -> Option<ffi::PyAsyncMethods> {
    121:    default fn am_await() -> Option<ffi::unaryfunc> {
    144:    default fn am_aiter() -> Option<ffi::unaryfunc> {
    167:    default fn am_anext() -> Option<ffi::unaryfunc> {
    
    src/conversion.rs
    107:    default fn with_borrowed_ptr<F, R>(&self, py: Python, f: F) -> R
    
    src/class/sequence.rs
    135:    default fn tp_as_sequence() -> Option<ffi::PySequenceMethods> {
    168:    default fn sq_length() -> Option<ffi::lenfunc> {
    190:    default fn sq_item() -> Option<ffi::ssizeargfunc> {
    239:        default fn set_item() -> Option<ffi::ssizeobjargproc> {
    285:        default fn del_item() -> Option<ffi::ssizeobjargproc> {
    328:        default fn del_set_item() -> Option<ffi::ssizeobjargproc> {
    372:    default fn sq_contains() -> Option<ffi::objobjproc> {
    394:    default fn sq_concat() -> Option<ffi::binaryfunc> {
    416:    default fn sq_repeat() -> Option<ffi::ssizeargfunc> {
    438:    default fn sq_inplace_concat() -> Option<ffi::binaryfunc> {
    465:    default fn sq_inplace_repeat() -> Option<ffi::ssizeargfunc> {
    
    src/class/number.rs
    625:    default fn tp_as_number() -> Option<ffi::PyNumberMethods> {
    692:    default fn nb_add() -> Option<ffi::binaryfunc> {
    714:    default fn nb_subtract() -> Option<ffi::binaryfunc> {
    736:    default fn nb_multiply() -> Option<ffi::binaryfunc> {
    758:    default fn nb_matrix_multiply() -> Option<ffi::binaryfunc> {
    780:    default fn nb_true_divide() -> Option<ffi::binaryfunc> {
    802:    default fn nb_floor_divide() -> Option<ffi::binaryfunc> {
    824:    default fn nb_remainder() -> Option<ffi::binaryfunc> {
    846:    default fn nb_divmod() -> Option<ffi::binaryfunc> {
    868:    default fn nb_power() -> Option<ffi::ternaryfunc> {
    890:    default fn nb_lshift() -> Option<ffi::binaryfunc> {
    912:    default fn nb_rshift() -> Option<ffi::binaryfunc> {
    934:    default fn nb_and() -> Option<ffi::binaryfunc> {
    956:    default fn nb_xor() -> Option<ffi::binaryfunc> {
    978:    default fn nb_or() -> Option<ffi::binaryfunc> {
    1000:    default fn nb_inplace_add() -> Option<ffi::binaryfunc> {
    1022:    default fn nb_inplace_subtract() -> Option<ffi::binaryfunc> {
    1044:    default fn nb_inplace_multiply() -> Option<ffi::binaryfunc> {
    1066:    default fn nb_inplace_matrix_multiply() -> Option<ffi::binaryfunc> {
    1088:    default fn nb_inplace_true_divide() -> Option<ffi::binaryfunc> {
    1110:    default fn nb_inplace_floor_divide() -> Option<ffi::binaryfunc> {
    1132:    default fn nb_inplace_remainder() -> Option<ffi::binaryfunc> {
    1154:    default fn nb_inplace_power() -> Option<ffi::ternaryfunc> {
    1176:    default fn nb_inplace_lshift() -> Option<ffi::binaryfunc> {
    1198:    default fn nb_inplace_rshift() -> Option<ffi::binaryfunc> {
    1220:    default fn nb_inplace_and() -> Option<ffi::binaryfunc> {
    1242:    default fn nb_inplace_xor() -> Option<ffi::binaryfunc> {
    1264:    default fn nb_inplace_or() -> Option<ffi::binaryfunc> {
    1287:    default fn nb_add_fallback() -> Option<ffi::binaryfunc> {
    1309:    default fn nb_sub_fallback() -> Option<ffi::binaryfunc> {
    1331:    default fn nb_mul_fallback() -> Option<ffi::binaryfunc> {
    1353:    default fn nb_matmul_fallback() -> Option<ffi::binaryfunc> {
    1375:    default fn nb_truediv_fallback() -> Option<ffi::binaryfunc> {
    1397:    default fn nb_floordiv_fallback() -> Option<ffi::binaryfunc> {
    1419:    default fn nb_mod_fallback() -> Option<ffi::binaryfunc> {
    1441:    default fn nb_divmod_fallback() -> Option<ffi::binaryfunc> {
    1463:    default fn nb_pow_fallback() -> Option<ffi::ternaryfunc> {
    1485:    default fn nb_lshift_fallback() -> Option<ffi::binaryfunc> {
    1507:    default fn nb_rshift_fallback() -> Option<ffi::binaryfunc> {
    1529:    default fn nb_and_fallback() -> Option<ffi::binaryfunc> {
    1551:    default fn nb_xor_fallback() -> Option<ffi::binaryfunc> {
    1573:    default fn nb_or_fallback() -> Option<ffi::binaryfunc> {
    1595:    default fn nb_negative() -> Option<ffi::unaryfunc> {
    1618:    default fn nb_positive() -> Option<ffi::unaryfunc> {
    1640:    default fn nb_absolute() -> Option<ffi::unaryfunc> {
    1662:    default fn nb_invert() -> Option<ffi::unaryfunc> {
    1684:    default fn nb_int() -> Option<ffi::unaryfunc> {
    1706:    default fn nb_float() -> Option<ffi::unaryfunc> {
    1728:    default fn nb_index() -> Option<ffi::unaryfunc> {
    
    src/instance.rs
    387:    default fn to_managed_py_ref<'p>(&self, py: Python<'p>) -> ManagedPyRef<'p, Self> {
    396:    default fn drop_impl(borrowed: &mut ManagedPyRef<Self>) {
    
    src/types/sequence.rs
    268:                default fn extract(obj: &'a PyAny) -> PyResult<Self> {
    307:    default fn extract(obj: &'a PyAny) -> PyResult<Self> {
    

    ~Once those are removed, we pyo3 can be stable on rust 1.30! I'm happy to take pull requests tackling one or more of those default fn uses~

    Reviewed by konstin at 2018-08-25 18:52
  • 5. Questions about multithreading and asyncio

    Hey, I've got a rather complicated program that involves mixing Rust async/await with Python asyncio. I know this problem isn't sorted out yet in pyo3 (although I think it's being worked on), but I figured I could get around those limitations by running two event loops, one for Rust and one for Python on its own dedicated thread.

    So here's what I'm doing:

    • Spin up an async Rust application with async-std runtime
    • Spawn a thread to act as the main thread for the asyncio event loop and call asyncio.set_event_loop(loop) and loop.run_forever() on that thread
    • Host a websocket server with async-tungstenite that occasionally interacts with the Python event loop via pyo3.
    • Interact with the Python event loop via call_soon_threadsafe and asyncio.run_coroutine_threadsafe to run my application
    • When the application exits, I call loop.call_soon_threadsafe(loop.stop) and join my thread

    My tests run exactly as expected with pyo3. The only problem is that when the test exits, I get one of three errors:

    If I'm lucky, my test exits successfully this warning:

    Exception ignored in: <module 'threading' from '/usr/lib/python3.8/threading.py'>
    Traceback (most recent call last):
      File "/usr/lib/python3.8/threading.py", line 1373, in _shutdown
        assert tlock.locked()
    AssertionError: 
    

    If I'm unlucky I either get this error:

    Fatal Python error: This thread state must be current when releasing
    
    Thread 0x00007f7dd1dd8b40 (most recent call first):
    
    Current thread 0x00007f7dcece9700 (most recent call first):
      File "/usr/lib/python3.6/asyncio/selector_events.py", line 148 in _write_to_self
      File "/usr/lib/python3.6/asyncio/base_events.py", line 643 in call_soon_threadsafe
    
    Thread 0x00007f7dcc4a0700 (most recent call first):
      File "/usr/lib/python3.6/selectors.py", line 445 in select
      File "/usr/lib/python3.6/asyncio/base_events.py", line 1415 in _run_once
      File "/usr/lib/python3.6/asyncio/base_events.py", line 438 in run_forever
    
    Thread 0x00007f7dce8e7700 (most recent call first):
    error: test failed
    

    or this one:

    FATAL: exception not rethrown
    error: test failed
    

    Things I've tried:

    • Spawning native rust threads with std::thread to run the event loop
    • Spawning python threads with the threading module to run my event loop
    • Calling pyo3::prepare_freethreaded_python() before the first call to Python::with_gil

    Sorry the problem description is so vague. I've tried simplifying my program to narrow the problem down, but it's frustratingly hard to replicate in a smaller codebase. Before I spend too much time on that, I just wanted to check with someone more experienced with embedding Python to see if I'm setting things up properly.

    Questions:

    • Am I setting up my threads correctly?
    • Do I need to call pyo3::prepare_freethreaded_python() before the first call to Python::with_gil in this situation?
    • Do I need to use a thread from the Python threading module or is it ok to use Rust's std::thread
    • Do I need to use some of the FFI calls to ensure that my Python thread state is correct?
    • Are there any other multi-threading limitations that I should know about with pyo3?

    ๐ŸŒ Environment

    • OS: Ubuntu 20.04
    • Python version: 3.8.5 (installed with distro)
    • Rust version (rustc --version): rustc 1.47.0-nightly (6c8927b0c 2020-07-26)
    • PyO3 version: "0.12.3"
    • I have not tried pyo3 master yet (I think the problem is with my code)
    Reviewed by awestlake87 at 2020-11-12 19:15
  • 6. Complete abi3 support

    For #1125. Continuation of #1132. What I did so far are

    • Prepared the proper feature flag. I named it all-apis but feel free to propose alternative.
      • EDITED: I renamed it unstable-api adviced by konstin.
    • Restructured method tables with cfg.

    TODO:

    • [x] Add CI setting for abi3
    • [x] Refactor tp_dict initialization codes (=classattr). It should be changed a lot.
    • [x] __text__signature__.
    • [x] Raise compile errors if some features (e.g., Buffer Protocol) is used without unstable-api feature.
    • [ ] Support additional feature flags with the minimum required Python version. See #1195 about the discussion.
    • [x] Documentation
      • Document about abi3 in the README and the guide.

    ... and so on.

    Sorry @alex for the conflict. I'm going to work on tp_dict problem for now and contributions for other areas are welcome.

    Reviewed by kngwyu at 2020-09-05 16:38
  • 7. Support rust extensions for PyPy via cpyext

    Hi, Following the discussion on https://github.com/PyO3/pyo3/pull/370, I've taken some time to dust off my work on support for PyPy via cpyext.

    It runs nicely on my machine (osx) for the several test cases I've been using (raising errors, moving objects back and forth and so on).

    I've managed to run the word_count example successfully, altough for this usecase pypy beats rust in terms of performance (overhead is greater than benefit).

    ~~Unfortunely, it seems the rustapi_module example is still crashing, because of 2 outstanding issues:~~ EDIT: Both solved!

    ~~1. The datetime API wrapper.~~ ~~2. The way initialize_type behaves (causes SIGSEGV on attempt to register a new class) My guess is that there is still some outstanding difference in the implementation of the PyTypeObject struct that causes it to crash~~

    I'd like to outline some changes I've done, and why I think there is definitely some more work needed on this before it's ready for prime time.

    1. cpyext does not offer complete support for the entirety of the cpython API. So I've had (mostly by dumping symbols out of the pypy interpreter) manually add the parts of it that it implements with annotations like this: #[cfg_attr(PyPy, link_name = "PyPyErr_SetInterrupt")]

    I couldn't find any automatic way to set this up, since it's not for all the symbols, so it's quite verbose (I've done this with a regexes), but it makes it very easy to spot changes when the cpyext api layer changes and add them.

    1. I've never succeeded in getting this to work with python2, and since I've read in the PR that it's likely to be discontinued, I didn't invest more time with it.

    2. I've had to manually shim some functions that pypy expects to find when compiling against their version of Python.h, like PyUnicodeDecodeError_Create, these are mostly utility functions which they never bothered implementing, so it looks something like this: image

    3. I've modified the build.rs script originally in my implementation, since I needed it to detect and correctly emit configuration variables that are suitable for PyPy, it's a slight drift from the current model (single build.rs file), into a more structured internal crate that I've used. I didn't want to spend time porting all the changes back into the big build.rs file without hearing your thoughts on it first.

    4. As the title suggests, this only supports python -> rust, and not embedding pypy (due to pypy not behaving nicely with embedding), though I don't think embedding pypy is a significant use case, as it's mostly used to squeeze performance out of existing python codebases.

    EDIT: Functionallity that is still missing from cpyext that is used by PyO3 internally: These functions will need to be implemented in rust.

    • complex numbers _Py_c_* functions
    • PyDict_MergeFromSeq2
    • _PyLong_FromByteArray, _PyLong_AsByteArray
    • PySequence_Count
    • Py_CompileStringExFlags (EDIT: PyPy exports Py_CompileStringFlags, so this is actually OK)

    I'm still certain there are some semantics that I've missed, but i'd love to get another pair of eyes looking over some of this stuff.

    Would love to hear your thoughts :) Cheers ๐Ÿ

    P.S I'm away for next week, but I will be able to address suggestions the week after.

    Reviewed by omerbenamram at 2019-03-09 12:58
  • 8. #1064: Comparisons with __eq__ should not raise TypeError

    RichComparisonToSelf expects to compare itself with objects of the same type, but then we shouldn't raise TypeError but return NotImplemented.

    Fixes #1064

    Reviewed by mvaled at 2020-07-28 16:19
  • 9. enum WIP

    Thank you for contributing to pyo3!

    Here are some things you should check for submitting your pull request:

    • Run cargo fmt (This is checked by travis ci)
    • Run cargo clippy and check there are no hard errors (There are a bunch of existing warnings; This is also checked by travis)
    • If applicable, add an entry in the changelog.
    • If applicable, add documentation to all new items and extend the guide.
    • If applicable, add tests for all new or fixed functions
    • If you changed any python code, run black .. You can install black with pip install black)

    You might want to run tox (pip install tox) locally to check compatibility with all supported python versions. If you're using linux or mac you might find the Makefile helpful for testing.

    Reviewed by stillinbeta at 2020-07-17 20:34
  • 10. Compile with stable rust

    PyO3 requires following unstable

    • [x] const_fn rust-lang/rust#24111
    • [x] specialization rust-lang/rust#31844
    • [x] use_extern_macros rust-lang/rust#35896
    • [x] proc_macro (proc_macro_attribute) rust-lang/rust#35896
    • [x] associated_consts
    • [x] ~~Optional, for wrap_function!: concat_idents (rust-lang/rust#29599)~~ Solved with mashup
    Reviewed by fafhrd91 at 2017-05-19 15:05
  • 11. Use PyType_Spec for creating new types in Rust

    This is a proof of concept -- implements just enough of it to prove the idea works! There's still a lot of missing elements of this (look at all the commented out code!)

    Before I went too much further and implemented all the different function pointers, I wanted to check in and see if this approach looked ok.

    Reviewed by alex at 2020-09-01 00:37
  • 12. implicit conversions

    Thanks for the great work you do on PyO3.

    Pybind11 has a feature called 'implicit conversions' https://pybind11.readthedocs.io/en/stable/advanced/classes.html#implicit-conversions

    This can be quite handy, let me give an example:

    #[pyclass]
    #[derive(Clone, Copy)]
    pub struct Vector3D {
        pub x: f64
        pub y: f64
        pub z: f64
    }
    
    #[pymethods]
    impl Vector3D {
        #new
        pub fn __new__(v: [f64, 3]) -> Self {
            Self {x: v[0], y: v[1], z: v[2]}
        }
    }
    
    
    #[pyfunction]
    fn do_something(vec: Vector3D) -> f64 {
        vec.x * 2
    }
    
    >>> from py_rust_module import *
    
    >>> do_something([1,2,3])
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: argument 'vec': 'list' object cannot be converted to 'Vector3D'
    
    >>> do_something(Vector3D([1,2,3]))
    2
    

    In Pybind11 Vector3D could be declared as implicitly convertible from list and do_something([1,2,3]) would be valid and implicitly casting the list to a Vector3D object.

    Is this something that could be done in pyo3? It is quite handy and can save a lot of verbosity...

    Reviewed by hiaselhans at 2022-06-26 09:13
  • 13. security: fix use-after-free in PyCapsule implementation

    Thanks to @saethlin for alerting me to this issue.

    The current implementation of PyCapsule::new does not require a specific lifetime on the name: &CStr argument. This is a fundamental flaw; the argument is converted to a pointer and handed over to CPython, which simply stores the pointer.

    This means that as soon as PyCapsule::new is complete, current implementations can deallocate the name and any subsequent calls to PyCapsule_GetName will read from freed memory (including our safe PyCapsule::name wrapper).

    A snippet as simple as this is enough to achieve this:

    let capsule: Py<PyCapsule> = Python::with_gil(|py| {
        let name = CString::new("foo").unwrap();
        PyCapsule::new(py, 5usize, &name).unwrap().into()
    });
    
    Python::with_gil(|py| {
        let name = capsule.as_ref(py).name();  // <-- Reading from freed memory previous to this PR's fix
    });
    

    The fix without any API change is to take a copy of the name as a CString and store that where we place the rest of the capsule data. We can then pass CPython the pointer from there, which guarantees that the name will outlive the capsule.


    Given this is a reasonably clear-cut soundness error, I think we should file a RustSec advisory and also yank versions 0.16.0 -> 0.16.5 (after publishing this as a 0.16.6).

    Reviewed by davidhewitt at 2022-06-26 06:44
  • 14. Support `__del__`

    I'd like to be able to use a __del__ method in #[pymethods], however it's not being called.

    There are two other related issue #402 and #774. #774 seems most relevant but the OP get's told they shouldn't be using __del__. AFAIK issuing a ResourceWarning is a legitimate use of __del__.

    I could print a warning in Drop but that won't be a python warning and will confuse people, without support for __del__ I don't see another way around this.

    Related PR https://github.com/samuelcolvin/watchfiles/pull/164

    Reviewed by samuelcolvin at 2022-06-25 11:14
  • 15. Update PyTryFrom for PyMapping to more accurately check for mapping types

    This is intended to fix #2072 - see further discussion in that thread.

    The implementation is such that:

    • For Python 3.10 and not Py_LIMITED_API
      • Check ffi::PyType_HasFeature(ty, fft::Py_TPFLAGS_MAPPING) != 0
    • For Python < 3.10 or Py_LIMITED_API
      • Call into python to check isinstance(<obj>, collections.abc.Mapping)

    One quirk I ran into is that using #[pyclass(mapping)] to define a new mapping class in Rust will fail this new check. This is a breaking change and caused tests/test_mapping.rs::mapping_is_not_sequence to fail. The proper solution is probably to subclass collections.abc.Mapping (see #991 for context on why this is not simple). For now to pass that test I first called collections.abc.Mapping.register(<RustMappingPyClass>).

    TODO

    • [ ] an entry in CHANGELOG.md
    • [ ] docs to all new functions and / or detail in the guide
    • [x] tests for all new or changed functions
    Reviewed by aganders3 at 2022-06-24 15:56
  • 16. Reduce `gh-pages` branch size?

    It gets annoying git pull receives several hundred megabytes due to gh-pages branch updates.

    โฏ git pull
    remote: Enumerating objects: 23466, done.
    remote: Counting objects: 100% (23466/23466), done.
    remote: Compressing objects: 100% (2336/2336), done.
    remote: Total 23466 (delta 21105), reused 23257 (delta 20905), pack-reused 0
    Receiving objects: 100% (23466/23466), 214.82 MiB | 5.80 MiB/s, done.
    Resolving deltas: 100% (21105/21105), completed with 2622 local objects.
    From github.com:PyO3/pyo3
       53b83cccbf..ef308ffaad  main       -> upstream/main
       74f4ccd485..2d9e575e3d  gh-pages   -> upstream/gh-pages
    Updating 53b83cccbf..ef308ffaad
    

    How about only keep the latest commit in gh-pages branch?

    https://github.com/peaceiris/actions-gh-pages#%EF%B8%8F-force-orphan-force_orphan

    Reviewed by messense at 2022-06-23 06:21
RustPython - A Python Interpreter written in Rust
RustPython - A Python Interpreter written in Rust

RustPython A Python-3 (CPython >= 3.9.0) Interpreter written in Rust ?? ?? ?? . Usage Check out our online demo running on WebAssembly. RustPython req

Jun 27, 2022
Create a Python project automatically with rust (like create-react-app but for python)

create-python-project Create a Python project automatically with rust (like create-react-app but for python) Installation cargo install create-python-

Mar 12, 2022
Create, open, manage your Python projects with ease, a project aimed to make python development experience a little better
Create, open, manage your Python projects with ease, a project aimed to make python development experience a little better

Create, open, manage your Python projects with ease, a project aimed to make python development experience a little better

Apr 26, 2022
A simple Pascal interpreter written in rust.

rascal A simple Pascal interpreter written in rust. Usage Download the latest rascal executable from the release page. Run the executable. rascal.exe

Nov 19, 2021
A memory safe Lua interpreter

Hematita Da Lua Hematita Da Lua is an interpreter for the scripting language Lua, written entirely in 100% safe Rust. Hematita is the portugese word f

Jun 24, 2022
An interpreter for the esoteric programming language, Brainf*ck.

Brainf*ck Interpreter This is just a normal Brainf*ck interpreter written in Rust. If you don't know what Brainf*ck is, you can check out the wikipedi

Dec 23, 2021
WebAssembly (Wasm) interpreter.

Continuous Integration Test Coverage Documentation Crates.io wasmi- WebAssembly (Wasm) Interpreter wasmi was conceived as a component of parity-ethere

Jun 18, 2022
Lisp interpreter that might be fast someday maybe?

ehlisp Pronunciation I'm not really sure. Maybe like an incorrect pronunciation of "ellipse", like "ellisp"? Also maybe like "a lisp". I named it this

May 9, 2022
Rust <-> Python bindings

rust-cpython Rust bindings for the python interpreter. Documentation Cargo package: cpython Copyright (c) 2015-2020 Daniel Grunwald. Rust-cpython is l

Jun 17, 2022
Implementation of Monte Carlo PI approximation algorithm in Rust Python bindings

rusty_pi Implementation of Monte Carlo PI approximation algorithm in Rust Python bindings. Time of 100M iterations approximation on Core i7 10th gen:

Jun 7, 2022
Very experimental Python bindings for the Rust biscuit-auth library

Overview This is a very experimental take on Python bindings for the biscuit_auth Rust library. It is very much a work in progress (limited testing, m

Apr 20, 2022
Python bindings for heck, the Rust case conversion library

pyheck PyHeck is a case conversion library (for converting strings to snake_case, camelCase etc). It is a thin wrapper around the Rust library heck. R

Jun 15, 2022
lavalink-rs bindings for Python

lavasnek_rs Dev Docs: Main Site | Fallback: GitHub Pages GitHub repo GitLab repo Using the library The library is available on PyPi, and you can insta

Jun 26, 2022
The polyglot bindings generator for your library (C#, C, Python, โ€ฆ) ๐Ÿ™

Interoptopus ?? The polyglot bindings generator for your library. Interoptopus allows you to deliver high-quality system libraries to your users, and

Jun 20, 2022
A script language like Python or Lua written in Rust, with exactly the same syntax as Go's.

A script language like Python or Lua written in Rust, with exactly the same syntax as Go's.

Jun 17, 2022
Rust Python modules for interacting with Metaplex's NFT standard.

Simple Metaplex Metadata Decoder Install the correct Python wheel for your Python version with pip: pip install metaplex_decoder-0.1.0-cp39-cp39-manyl

Mar 31, 2022
A simple library to allow for easy use of python from rust.

Rustpy A simple library to allow for easy use of python from rust. Status Currently this library has not received much love (pull requests welcome for

Jun 20, 2022
Robust and Fast tokenizations alignment library for Rust and Python
Robust and Fast tokenizations alignment library for Rust and Python

Robust and Fast tokenizations alignment library for Rust and Python Demo: demo Rust document: docs.rs Blog post: How to calculate the alignment betwee

Jun 20, 2022
Arrowdantic is a small Python library backed by a mature Rust implementation of Apache Arrow

Welcome to arrowdantic Arrowdantic is a small Python library backed by a mature Rust implementation of Apache Arrow that can interoperate with Parquet

Jun 13, 2022