Rust <-> Python bindings

Daniel Grunwald

Last update: Dec 29, 2022

Related tags

Foreign Function Interface rust-cpython

Overview

rust-cpython

Rust bindings for the python interpreter.

Documentation
Cargo package: cpython

Supported Python versions:

Python 2.7
Python 3.3 to 3.9

Requires Rust 1.41.1 or later.

Usage

To use cpython, add this to your Cargo.toml:

[dependencies]
cpython = "0.5"

Example program displaying the value of `sys.version`:

use cpython::{Python, PyDict, PyResult};

fn main() {
    let gil = Python::acquire_gil();
    hello(gil.python()).unwrap();
}

fn hello(py: Python) -> PyResult<()> {
    let sys = py.import("sys")?;
    let version: String = sys.get(py, "version")?.extract(py)?;

    let locals = PyDict::new(py);
    locals.set_item(py, "os", py.import("os")?)?;
    let user: String = py.eval("os.getenv('USER') or os.getenv('USERNAME')", None, Some(&locals))?.extract(py)?;

    println!("Hello {}, I'm Python {}", user, version);
    Ok(())
}

Example library with python bindings:

The following two files will build with cargo build, and will generate a python-compatible library. On Mac OS, you will need to rename the output from *.dylib to *.so. On Windows, you will need to rename the output from *.dll to *.pyd.

Note:

At build time python3-sys/build.rs will look for interpreters in:

PYTHON_SYS_EXECUTABLE
python
python3

picking the first one that works and is compatible with the configured expected version (by default, any Python 3.X interpreter will do). If a specific interpreter is desired, the PYTHON_SYS_EXECUTABLE environment variable should point to it.

Cargo.toml:

[lib]
name = "rust2py"
crate-type = ["cdylib"]

[dependencies.cpython]
version = "0.5"
features = ["extension-module"]

src/lib.rs

use cpython::{PyResult, Python, py_module_initializer, py_fn};

// add bindings to the generated python module
// N.B: names: "rust2py" must be the name of the `.so` or `.pyd` file
py_module_initializer!(rust2py, |py, m| {
    m.add(py, "__doc__", "This module is implemented in Rust.")?;
    m.add(py, "sum_as_string", py_fn!(py, sum_as_string_py(a: i64, b:i64)))?;
    Ok(())
});

// logic implemented as a normal rust function
fn sum_as_string(a:i64, b:i64) -> String {
    format!("{}", a + b).to_string()
}

// rust-cpython aware function. All of our python interface could be
// declared in a separate module.
// Note that the py_fn!() macro automatically converts the arguments from
// Python objects to Rust values; and the Rust return value back into a Python object.
fn sum_as_string_py(_: Python, a:i64, b:i64) -> PyResult<String> {
    let out = sum_as_string(a, b);
    Ok(out)
}

On windows and linux, you can build normally with cargo build --release. On Mac Os, you need to set additional linker arguments. The simplest solution is to create a .cargo/config with the following content:

[target.x86_64-apple-darwin]
rustflags = [
  "-C", "link-arg=-undefined",
  "-C", "link-arg=dynamic_lookup",
]

For setup.py integration, see https://github.com/PyO3/setuptools-rust

Comments

Windows support

I actually started writing rust-cpython on Windows, but had to give up due to strange issues when linking against the official python binaries. Now that the x86_64-pc-windows-msvc target exists, we should give it another try.
enhancement help wanted

opened by dgrunwald 24
How to Best Thread Work
Which approach will provide the most performance / make most sense when embedding Python in Rust?

Create multiple Rust threads where each thread calls a Python function.

Use a single Rust thread that calls a Python function which then distributes work to multiple Python owned threads.

Just getting started here and it's not obvious (unless its 1 :) which approach to choose, hopefully someone can point me in the right direction.
question
opened by liveresume 22
Modernize the codebase

Putting this up as an RFC.

This updates to more modern code, including edition 2018. It now requires rustc 1.36 so that we can use things like AssumeUninit (without this, all the std::mem::uninitialized() calls show up as warnings).

Rust 1.36 was released over 6 months ago. However, it looks like Debian stable is pinned at 1.34, so if we want to keep targetting that as a minumum, we won't be able to take all of this. In practice I would expect users of Rust that are keeping their dependent crates up-to-date would also be using rustup to keep the compiler up-to-date, too.

Any comments?

opened by markbt 14
allow all visibility keywords on classes and functions

Teach the py_class macros about allowing arbitrary visibility keywords (e.g., pub(crate)) on class definitions and the method definitions within that class definition. This PR also makes the create_instance method have the same visibility as the class.

The vis capture type for visibility in macro_rules has been available since Rust v1.30.0. Given that this crate requires v1.32 or higher, this PR does not break the crate's compatibility guarantee.

Note: The ultimate motivation for this PR is to allow me to split a very complicated use of this crate into multiple modules without having to make many types pub.

opened by tdyas 13

Support array.array('f', ...) -> Vec

rust-cpython can convert Python lists to Rust vectors, but it can't yet convert arrays created by the array module. It also doesn't support numpy arrays, which is probably a bigger project, but also valuable goal.

For example, I have an example where I am writing a dot product function:

mod dot {
pub fn dot(a : &Vec<f32>, b : &Vec<f32>) -> f32 {                               
    let mut out = 0.;                                                           
    for (x, y) in a.iter().zip(b.iter()) {                                      
        out += x*y;                                                             
    }                                                                           
    out                                                                         
}                                                                               

} // mod 
fn dot<'p>(py: Python<'p>, args: &PyTuple<'p>) -> PyResult<'p, f32> {           
    if args.len() < 2 {
// TODO: write a macro for making PyErr so it's less painful.                                                         
        let msg = "dot takes two arrays of float";                              
        let pyerr = PyErr::new_lazy_init(py.get_type::<exc::ValueError>(), Some(msg.to_py_object(py).into_object()));
        return Err(pyerr);                                                      
    }                                                                           
    let arg0 = match args.get_item(0).extract::<Vec<f32>>() {                   
        Ok(x) => x,                                                             
        Err(_) => {                                                             
            let msg = "Could not convert argument 0 to array of float";         
            let pyerr = PyErr::new_lazy_init(py.get_type::<exc::ValueError>(), Some(msg.to_py_object(py).into_object()));
            return Err(pyerr);                                                  
        }                                                                       
    };                                                                          

    let arg1 = match args.get_item(1).extract::<Vec<f32>>() {                   
        Ok(x) => x,                                                             
        Err(_) => {                                                             
            let msg = "Could not convert argument 0 to array of float";         
            let pyerr = PyErr::new_lazy_init(py.get_type::<exc::ValueError>(), Some(msg.to_py_object(py).into_object()));
            return Err(pyerr);                                                  
        }                                                                       
    };                                                                          

    if arg0.len() != arg1.len() {                                               
            let msg = "arg0 and arg1 must be the same length";                  
            let pyerr = PyErr::new_lazy_init(py.get_type::<exc::ValueError>(), Some(msg.to_py_object(py).into_object()));
            return Err(pyerr);                                                  
    }                                                                           

    Ok(dot::dot(&arg0, &arg1))                                                  
}  

py_module_initializer!(librust_python_example, |_py, m| {                       
    try!(m.add("__doc__", "Dot product"));                                                     
    try!(m.add("dot", py_fn!(dot)));                                            
    Ok(())                                                                      
});

It works pretty well for a toy example:

In [1]: def dot(a, b):
    return sum(map(lambda x: x[0]*x[1], zip(a,b)))
   ...: 

In [2]: import numpy as np

In [3]: import librust_python_example as r

In [4]: timeit dot([0,2,3], [1,2,3])
100000 loops, best of 3: 1.86 µs per loop

In [5]: timeit np.dot([0,2,3], [1,2,3])
100000 loops, best of 3: 3.89 µs per loop

In [6]: timeit r.dot([0,2,3], [1,2,3])
1000000 loops, best of 3: 920 ns per loop

However, it won't convert numpy arrays or array.arrays:

In [8]: r.dot(np.arange(3), np.arange(3))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-8-0e9a2e3217bb> in <module>()
----> 1 r.dot(np.arange(3), np.arange(3))

ValueError: Could not convert argument 0 to array of float

In [9]: import array
In [10]: r.dot(array.array('f', [0,2,3]), array.array('f', [1,2,3]))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-10-3eab6b6f9184> in <module>()
----> 1 r.dot(array.array('f', [0,2,3]), array.array('f', [1,2,3]))

ValueError: Could not convert argument 0 to array of float

opened by ehiggs 13

PyObject<'p> and Send

When defining a python extension type in rust, it's possible for python code to retain a reference to instances of that type, and later use them on another (python) thread. Thus, python extension types require that their contents are Send. (and also 'static, but we could potentially allow 'p if we make the 'p lifetime invariant and use a higher-order lifetime when acquiring the GIL)

The problem: PyObject itself is not Send, so it is not possible to create python extension objects that contain references to other python objects :(

So, why is PyObject not Send? The python runtime requires that python objects are only used while the global interpreter lock (GIL) is held. But it's no problem to hold a reference to a python object without having the GIL acquired -- as long as you don't access the python object until re-acquiring the GIL. Currently, a PyObject<'p> is a *mut ffi::PyObject with two invariants: a) The PyObject owns one reference to the python object (will call Py_DECREF() in the Drop impl) b) The GIL is held for at least the lifetime 'p. (i.e. PyObject<'p> contains a Python<'p>)

Python<'p> represents "the current thread holds the GIL for lifetime 'p", and thus is fundamentally not Send. We could attempt to remove the Python<'p> from PyObject<'p>. This would require the user to explicitly pass in the Python<'p> for every operation on the PyObject. But on the other hand, it means we don't need a lifetime on PyObject (and transitively, PyResult etc.), so overall it should be an ergonomics win. It would open up some possibilities for a safe API for temporarily releasing the GIL during a long pure-rust computation (Py_BEGIN_ALLOW_THREADS).

But there's a big issue with this approach: the Drop impl. Calling Py_DECREF() requires holding the GIL, so we wouldn't be able to do that in Drop -- we'd have to provide an explicit fn release(self, Python), and any call to Drop would leak a reference count. Forgetting to call release() would be a serious footgun.

This becomes less of an issue with static analysis to find missing release() calls: humpty_dumpty might be able to help; and hopefully Rust will get linear types in the future.

So, do we adopt this change? When writing python extension in rust, I'd say yes, it's worth it. But for embedding python in a rust program, trading a few lifetime annotations for a serious footgun is not exactly a good trade-off.

Maybe we could make Drop (re-)acquire the GIL? Locking the GIL recursively is allowed. That would reduce the footgun from a ref leak to a performance problem: instead of a simple decrement (which is getting inlined into the rust code), Drop would involve two calls into python library, both of which read from TLS and then do some cheap arithmetic+branching. Nothing too expensive, so this might be a workable approach (considering you can prevent the performance loss by explicitly calling release()).

opened by dgrunwald 13

Python class TypeError

I defined a Python class and invoked, it triggered TypeError, below is my Python code and Rust code :

# test_hello.py
import hello


class TestMe:
    age = 20


t = TestMe()
print(hello.print_py_object(t))

// lib.rs
#[macro_use]
extern crate cpython;

use cpython::{PyObject, PyResult, Python};

py_class!(class TestMe |py| {
    data age: i32;
});

fn print_py_object(py: Python, obj: PyObject) -> PyResult<String> {
    let v = obj.cast_as::<TestMe>(py)?;
    let out = format!("{}", v.age(py));
    Ok(out)
}

py_module_initializer!(hello, inithello, PyInit_hello, |py, m| {
    m.add(
        py,
        "print_py_object",
        py_fn!(py, print_py_object(val: PyObject)),
    )?;

    Ok(())
});

I don't know what's wrong, all I want to do is pass Python class to a Rust function. I want to get fields in that Python class, please help me. Thanks.

opened by josephok 12

Inadequate CI
Travis is pretty inadequate for a project of this type where what's really required is a selection of build slaves for the supported platforms and python versions.

It seems like currently travis builds on linux x64 only, with python 2.7.

Raising this issue in case anyone knows of a free hosted CI platform that supports other platform types (32 bit, OS X, Windows), has a cunning free solution to the problem, or is willing to donate an instance to the problem.

OS:

[x] Linux

[x] OS X

[x] Windows

Platform:

[X] x64

[ ] x86

help wanted
opened by novocaine 12
rust-cpython fails to build on Mac

Compiling python27-sys v0.0.6 failed to run custom build command for python27-sys v0.0.6 Process didn't exit successfully: target/debug/build/python27-sys-c6f0449b2a9ec4bf/build-script-build (exit code: 101) --- stderr thread '
' panicked at 'called Result::unwrap() on an Err value: "\"pkg-config\" \"--libs\" \"--cflags\" \"python-2.7\" did not exit successfully: exit code: 1\n--- stderr\nPackage python-2.7 was not found in the pkg-config search path.\nPerhaps you should add the directory containing `python-2.7.pc'\nto the PKG_CONFIG_PATH environment variable\nNo package 'python-2.7' found\n"', /Users/rustbuild/src/rust-buildbot/slave/stable-dist-rustc-mac/build/src/libcore/result.rs:729

opened by traff 11
Add utility to share rust reference across Python objects
This was originally developed for the Mercurial's Rust extension. It allows us to implement a Python iterator over a Rust iterator relatively safely.

The problem is that a Rust iterator typically has a reference to the collection behind, and thus cannot be a data member of a Python object. We get around this problem by casting &'a to &'static and add runtime validation instead. The basic idea is described in the following blog post.

https://raphaelgomes.dev/blog/articles/2019-07-01-sharing-references-between-python-and-rust.html

In order to make the reference-sharing business safe, we have the following structs:

PySharedRefCell defines a storage holding a value to be shared

PySharedRef is a lifetime-bound reference to the value above, which provides a safe interface on top of the PySharedRefCell and its owner PyObject

UnsafePyLeaked is the reference not bound to the real lifetime, and its validity is checked at runtime

PyLeakedRef/PyLeakedRefMut are borrowed references from UnsafePyLeaked

In order to create PySharedRef in safe manner, we plan to add py_class!() macro like this:

py_class!(class List |py| shared_data vec: Vec<i32>; // the storage type is PySharedRefCell<Vec<i32>>, but its accessor // returns PySharedRef<'a, Vec<i32>>. It makes sure that the right // owner PyObject and its data member are paired. });

The macro extension is not included in this PR since we'll probably need some bikeshedding.

There are a couple of unsafe business in this module, and some public functions are unfortunately unsafe. Please refer to the inline comments why some of them can be safe and some can't.

Thanks to @gracinet for the basic ideas and experiments, @Alphare for the core implementation, @markbt for the generation-counter idea, and Mercurial developers for reviewing the original patches.
opened by yuja 10
Release 0.5.0 and some doc clarifications

Tried to make some things more explicit in the documentation of properties, and then proceed to the 0.5.0 release.

I'll merge this myself, so that I can upload to crates.io and put a tag right away.

opened by gracinet 9
fixed linking python when venv is active
By checking for base_prefix instead of exec_prefix, linking with pythonXY.lib works even when a virtual environment is active.

Changes:

printing sys.base_prefix instead of sys.exec_prefix

This PR references issue #213
opened by nemjit001 1
add support for handling posix-like libary style on windows

ld.exe: cannot find -lpython310: No such file or directory collect2.exe: error: ld returned 1 exit status

The python core library is libpython3.10.dll

opened by ArchGuyWu 0
$error: process didn't exit successfully: `target\debug\cpythontest3.exe` (exit code: 0xc0000135, STATUS_DLL_NOT_FOUND)$

error: process didn't exit successfully: `target\debug\cpythontest3.exe` (exit code: 0xc0000135, STATUS_DLL_NOT_FOUND)

This must be a dumb doubt for sure Im asking here atlast after I coudnt find solution anywhere... I'm intermediate in python and completed until chapter 5 in rust Im trying to use python in rust while using the example i couldnt find out how to solve this... Maybe its this because I didnt know how to do this step i checked out py03 example which suggests just installing python in ubuntu but nothing about in windows and while i just tested installing python using wsl it quite didnt work thats why

opened by ninetynin 4
Missing autocomplete

Hello, I'm running VSC 1.62.3 with Visual Studio IntelliCode v1.2.14, but seem to lack autocomplete results. I can import mypylib properly (the .dll was renamed to .pyd and moved to .), but I don't get any autocomplete for it's members (which should show 2 built-in functions). I've confirmed that those functions do exist and work properly, it's purely lack of autocomplete. Is that normal or is something wrong with my IDE settings?

opened by MiniaczQ 1
How to set or modify tp_flags?

In an extension type written with py_class!(), is there a way to set or fix the tp_flags of the resulting PyTypeObject?

For this specific use-case I want to add Py_TPFLAGS_BASETYPE so the extension class can be sub-classed in Python. Is this a quote-unquote "difficult" thing to do while still using the py_class! macro?

Any example or code appreciated.

opened by fsh 0