A setuptools/wheel/cffi extension to embed a binary data in wheels

Overview

Milksnake

Milksnake is an extension for setuptools that allows you to distribute dynamic linked libraries in Python wheels in the most portable way imaginable.

It gives you a hook to invoke your own build process and to then take the resulting dynamic linked library.

Why?

There are already other projects that make Python and native libraries play along but this one is different. Unlike other projects that build Python extension modules the goal of this project is to build regular native libraries that are then loaded with CFFI at runtime. Why not just use CFFI? Because CFFI's setuptools support alone does not properly work with such wheels (it does not provide a way to build and properly tag wheels for shared libraries) and it does not provide a good way to invoke an external build process (like a makefile, cargo to build rust binaries etc.)

In particular you will most likely only need two wheels for Linux, one for macs and soon one for Windows independently of how many Python interpreters you want to target.

What is supported?

  • Platforms: Linux, Mac, Windows
  • setuptools commands: bdist_wheel, build, build_ext, develop
  • pip install --editable .
  • Universal wheels (PACKAGE-py2.py3-none-PLATFORM.whl); this can be disabled with milksnake_universal=False in setup() in case the package also contains stuff that does link against libpython.

How?

This example shows how to build a rust project with it:

This is what a setup.py file looks like:

from setuptools import setup

def build_native(spec):
    # build an example rust library
    build = spec.add_external_build(
        cmd=['cargo', 'build', '--release'],
        path='./rust'
    )

    spec.add_cffi_module(
        module_path='example._native',
        dylib=lambda: build.find_dylib('example', in_path='target/release'),
        header_filename=lambda: build.find_header('example.h', in_path='target'),
        rtld_flags=['NOW', 'NODELETE']
    )

setup(
    name='example',
    version='0.0.1',
    packages=['example'],
    zip_safe=False,
    platforms='any',
    setup_requires=['milksnake'],
    install_requires=['milksnake'],
    milksnake_tasks=[
        build_native
    ]
)

You then need a rust/ folder that has a Rust library (with a crate type of cdylib) and a example/ python package.

Example example/__init__.py file:

from example._native import ffi, lib


def test():
    return lib.a_function_from_rust()

And a rust/src/lib.rs:

#[no_mangle]
pub unsafe extern "C" fn a_function_from_rust() -> i32 {
    42
}

And the rust/Cargo.toml:

[package]
name = "example"
version = "0.1.0"
build = "build.rs"

[lib]
name = "example"
crate-type = ["cdylib"]

[build-dependencies]
cbindgen = "0.4"

And finally the build.rs file:

extern crate cbindgen;

use std::env;

fn main() {
    let crate_dir = env::var("CARGO_MANIFEST_DIR").unwrap();
    let mut config: cbindgen::Config = Default::default();
    config.language = cbindgen::Language::C;
    cbindgen::generate_with_config(&crate_dir, config)
      .unwrap()
      .write_to_file("target/example.h");
}
Comments
  • "milksnake" should not be listed in install_requires

    In the current README.md, the milksnake package is listed in the example setup.py's install_requires, besides the setup_requires.

    I understand it being a build requirement, but why does milksnake also need to be listed as an installation requirement, ie. required at runtime by the extension module that used milksnake in its building process?

    Note that I haven't used this package yet -- i'm just passing by and considering to use in my projects and too lazy to try and see what happens.

    Thanks anyway!

    enhancement 
    opened by anthrotype 15
  • Windows support?

    Windows support?

    I'd like to use this tool for a project, but the project requires Windows support. I'm curious how far out Windows support is, and what work needs to be done to support Windows.

    opened by kazimuth 14
  • Clarify what's wrong with cffi's setuptools support

    Clarify what's wrong with cffi's setuptools support

    In the readme it say "CFFI's setuptools support alone does not properly work with wheels" - but what does that exactly mean? Some examples and clarifications would help a lot.

    opened by ionelmc 6
  • Python 3 uses abi tagged dylibs

    Python 3 uses abi tagged dylibs

    I noticed that, when using milksnake under python3 (tested with both 3.6.3 on Linux and Windows), the fake/empty extension module that is built to trick distutils/setuptools to generate a binary wheel is still present in the generated wheel package along with the shared library built with the external build tool (e.g. cargo).

    When running on python2.7, only the shared lib is present.

    Is this normal or intended behaviour? I thought this empty extension module is only a temporary file that is not needed in the final distribution package. In fact, I can delete the extension module file and I am still able to import the generated cffi module.

    opened by anthrotype 5
  • OSError: error 0xc1 unable to load .pyd file on windows

    OSError: error 0xc1 unable to load .pyd file on windows

    Hi, I keep getting the following error:

    error 0xc1  cannot load library 'C:\Users\a11123434\projects\venv\Lib\site-packages\xlsbconvert-0.0.1-py3.6-win32.egg\xlsbconvert\_native__lib.cp36-win32.pyd': error 0xc1
    

    my setup.py:

    from setuptools import setup
    
    def build_native(spec):
        # build an example rust library
        build = spec.add_external_build(
            cmd=['cargo', 'build', '--release'],
            path='.'
        )
    
        spec.add_cffi_module(
            module_path='xlsb_csv._native',
            dylib=lambda: build.find_dylib('xlsb_csv', in_path='target/release'),
            header_filename=lambda: build.find_header('xlsb_csv.h', in_path='target'),
            rtld_flags=['NOW', 'NODELETE']
        )
    
    setup(
        name='xlsb_csv',
        version='0.0.1',
        packages=['xlsb_csv'],
        zip_safe=False,
        platforms='any',
        setup_requires=['milksnake'],
        install_requires=['milksnake'],
        milksnake_tasks=[
            build_native
        ]
    )
    

    my cargo.toml:

    [package]
    name = "xlsb_csv"
    version = "0.1.0"
    authors = ["a random person [email protected]"]
    edition = "2018"
    build = "build.rs"
    
    [lib]
    name = "xlsb_csv"
    path = "src/lib.rs"
    crate-type=["cdylib"]
    
    [dependencies]
    calamine = "0.15.5"
    csv = "1.1.1"
    
    [build-dependencies]
    cbindgen="0.9.0"
    

    lib.rs:

    #[no_mangle]
    pub unsafe extern "C" fn a_test()->String{
        return String::from("abc")
    }
    

    xlsb_csv/init.py:

    from xlsb_csv._native import ffi, lib
    
    def test():
        return lib.a_test()
    

    build.rs:

    extern crate cbindgen;
    
    use std::env;
    
    fn main() {
        let crate_dir = env::var("CARGO_MANIFEST_DIR").unwrap();
        let mut config: cbindgen::Config = Default::default();
        config.language = cbindgen::Language::C;
        cbindgen::generate_with_config(&crate_dir, config)
          .unwrap()
          .write_to_file("target/xlsb_csv.h");
    }
    

    folder structure:

    • xlsb_csv\ __init__.py
    • src\lib.rs
    • cargo.toml
    • build.rs
    • setup.py
    opened by hpca01 3
  • Improve Windows compatibility.

    Improve Windows compatibility.

    1. The MSVC linker can't cope with undefined exported functions, compiling an extension using setuptools always assumes that the initialisation function is defined so they need to be added to empty.c
    2. Add Windows DLL style in the lookup function
    3. Skip rtld flag definitions (fail because they are undefined and would be ignored anyways on Windows)
    opened by filmor 3
  • milksnake long description markdown is not properly rendered on pypi

    milksnake long description markdown is not properly rendered on pypi

    seen here: https://pypi.org/project/milksnake/#description

    image

    per the internet:

    As of March 16, 2018, PyPI.org aka Warehouse (finally) supports Markdown in long descriptions. Warehouse replaced the old legacy PyPI implementation in April 2018.
    
    You need to:
    
     - Make sure setuptools is upgraded to version 38.6.0 or newer
     - Make sure twine is upgraded to version 1.11.0 or newer
     - Make sure wheel is upgraded to version 0.31.0 or newer
     - Add a new field named long_description_content_type to your setup() call, and set it to 'text/markdown':
    
    setup(
        long_description="""# Markdown supported!\n\n* Cheer\n* Celebrate\n""",
        long_description_content_type='text/markdown',
        # ....
    )
    
    - Use twine to upload your distribution to PyPI.
    
    opened by kwlzn 2
  • When no wheel available, all output of build  process is swallowed, making it appear as if pip froze

    When no wheel available, all output of build process is swallowed, making it appear as if pip froze

    This has also been an issue with snaek. When you install a package without wheels, installation might take a very long time (particularly when rustc is involved). The user isn't notified about that and might get the impression that the installation froze.

    I attempted a patch to forward all output from the build functions (i.e. cargo's output), but it seems like this requires a lot of workarounds.

    opened by untitaker 2
  • Fix bundled library and generated cffi module paths for nested packages

    Fix bundled library and generated cffi module paths for nested packages

    This is fix for #15 - first change is properly determine module base without loosing name parts and second is determine file name for generated cffi module using only last name component without bringing extraneous parts in case of deeply nested packages.

    opened by daa 1
  • Document how to do callbacks calling a (Python function from Rust)

    Document how to do callbacks calling a (Python function from Rust)

    I have a hard time getting a callback (Python function that is registered to Rust code) to work.

    Between the CFFI documentation and milksnake I find it very hard to see what I should be doing here..

    https://cffi.readthedocs.io/en/latest/using.html#extern-python-new-style-callbacks lets met to think that I should actually include a extern "Python" in the cbindgen generated header or something...

    In my case the rust side already exposes a signature, but of course it is not marked extern "Python"

    opened by hmvp 1
  • Pass header_strip_directives to make_ffi

    Pass header_strip_directives to make_ffi

    I am getting this kind of errors because strip_directives hardcoded to True and it removed #define JIEBA_API extern

    cffi.error.CDefError: cannot parse "JIEBA_API void jieba_free(jieba_t);"
    

    Pass header_strip_directives to make_ffi so it can be disabled. (I think self. header_strip_directives was intended for this but left unused?)

    opened by messense 1
  • Global inline flags bug since Python 3.11

    Global inline flags bug since Python 3.11

    Hello, The regular expression hereis no longer tolerated by the version of Python 3.11 "Global inline flags (e.g. (?i)) can now only be used at the start of the regular expressions"

    Small suggested fix: _directive_re = re.compile(r'(?m)^\s*#.*?$')

    opened by NieWil 0
  • Examples using Python-specific data structures?

    Examples using Python-specific data structures?

    The README (is there more documentation somewhere?) only has a very trivial example of how to use this, and it's not immediately clear how to do anything very complicated. For example, how would I:

    1. Create a function that returns a list?
    2. Define a class in Rust?
    3. Write a function that takes arguments?
    4. Write a function that manipulates a Python dictionary?

    It is possible that these are "out of scope" for this library (which I gather is very low level), but I would find it difficult to write an interesting Python extension without doing any of these things. If it is out of scope for this library to document the idiomatic use in writing Python backends, maybe you could have a section linking to a few open source projects actually using milksnake so a new user can try to pick up the idioms?

    opened by pganssle 3
  • Fix logic in find_dylib() and find_header()

    Fix logic in find_dylib() and find_header()

    Milksnake does not support searching for dylib/header in absolute directory.

    It appends path to the in_path variable (if given): https://github.com/getsentry/milksnake/blob/ef0723e/milksnake/setuptools_ext.py#L146

    This prevent setting CARGO_TARGET_DIR to an absolute path different of the rust/ subdirectory.

    This PR changes the logic so that appending happens only if in_path is not an absolute path. Also, this in_path is normalized as described in #24.

    Closes #24.

    opened by nbigaouette 2
  • Non portable paths issue

    Non portable paths issue

    I had an issue with milksnake when setuping a CI pipeline for Linux, Mac and Windows.

    For Windows I SET the CARGO_TARGET_DIR to a folder I can cache between runs. To prevent issue with Windows backslash being wrongly escaped (or not), I use a path similar to C:/cargo_cache/project_name/target.

    After some debugging, I've realized that milksnake assumes / is used as paths separators when looking for files. See for example https://github.com/getsentry/milksnake/blob/ef0723e41df23d8f6357570c69c1e69cb31f9e9e/milksnake/setuptools_ext.py#L146

    and https://github.com/getsentry/milksnake/blob/ef0723e41df23d8f6357570c69c1e69cb31f9e9e/milksnake/setuptools_ext.py#L165

    Those two lines will join() the search path with the in_path argument which is split with forward slashes.

    I thus end up with paths that look like C:cargo_cache/project_name/target and milksnake can't find the expected files.

    It seems those lines attempt to replace the forward slashes with the platform's separator, but it's not doing a proper job.

    If that is the expected behaviour, I would suggest doing something like that instead:

    path = os.path.join(os.path.normalize(path), *os.path.normalize(in_path).split(os.sep))
    

    or even just

    path = os.path.join(os.path.normalize(path), os.path.normalize(in_path))
    

    or maybe even simpler

    path = os.path.normalize(os.path.join(path, in_path))
    
    opened by nbigaouette-eai 0
Owner
Sentry
Real-time crash reporting for your web apps, mobile apps, and games.
Sentry
A safe Rust FFI binding for the NVIDIA® Tools Extension SDK (NVTX).

NVIDIA® Tools Extension SDK (NVTX) is a C-based Application Programming Interface (API) for annotating events, code ranges, and resources in your applications. Official documentation for NVIDIA®'s NVTX can be found here.

Spencer Imbleau 78 Jan 2, 2023
Mod_wasm - an extension module for the Apache HTTP Server (httpd) that enables the usage of WebAssembly (Wasm).

mod_wasm is an extension module for the Apache HTTP Server (httpd) that enables the usage of WebAssembly (Wasm). This module will allow to execute certain tasks in the backend in a very efficient and secure way.

VMware  Labs 67 Dec 21, 2022
Turns running Rust code into a serializable data structure.

WasmBox WasmBox turns running Rust code into a serializable data structure. It does this by compiling it to WebAssembly and running it in a sandbox. T

drifting in space 12 Dec 7, 2022
Displaying data for the Soroban Futurenet.

Soroban Fiddle https://leighmcculloch.github.io/soroban-fiddle Web frontend-only application that displays data on the Soroban Futurenet network. Feat

Leigh McCulloch 2 Nov 17, 2022
Build a python wheel from a dynamic library

build_wheel Small utility to create a Python wheel given a pre-built dynamic library (.so, .dylib, .dll). If you are just trying to produce a wheel fr

Tangram 1 Dec 2, 2021
A timer based on a multi-time wheel structure

wheel-timer2 A timer based on a multi-time wheel structure This library uses a multi-layered time wheel structure. When a task is added to the wheel,

orange soeur 1 Jan 25, 2022
Easily embed and manage assets for your web application to build standalone-executables. Offers filename hashing, templating and more.

reinda: easily embed and manage assets This library helps you manage your assets (external files) and is mostly intended to be used in web application

Lukas Kalbertodt 23 Jul 15, 2022
Image proxy and embed generator.

January Description Image proxy and metadata scraper. Features: Can scrape metadata from websites, e.g. OpenGraph Can scrape embeds from websites, e.g

Revolt 26 Dec 26, 2022
PyOxidizer is a utility for producing binaries that embed Python

PyOxidizer is a utility for producing binaries that embed Python. The over-arching goal of PyOxidizer is to make complex packaging and distribution problems simple so application maintainers can focus on building applications instead of toiling with build systems and packaging tools.

Gregory Szorc 4.5k Jan 4, 2023
Safe API to embed an ECMAScript engine.

Kopi Kopi is a small abstraction to easily and safely embed an ECMAScript runtime inside a Rust based application. It uses the V8 execution engine to

Nils Hasenbanck 3 Dec 20, 2022
Binary coverage tool without binary modification for Windows

Summary Mesos is a tool to gather binary code coverage on all user-land Windows targets without need for source or recompilation. It also provides an

null 384 Dec 30, 2022
Binary coverage tool without binary modification for Windows

Summary Mesos is a tool to gather binary code coverage on all user-land Windows targets without need for source or recompilation. It also provides an

null 381 Dec 22, 2022
postgres-ical - a PostgreSQL extension that adds features related to parsing RFC-5545 « iCalendar » data from within a PostgreSQL database

postgres-ical - a PostgreSQL extension that adds features related to parsing RFC-5545 « iCalendar » data from within a PostgreSQL database

Edgar Onghena 1 Feb 23, 2022
A guide for Mozilla's developers and data scientists to analyze and interpret the data gathered by our data collection systems.

Mozilla Data Documentation This documentation was written to help Mozillians analyze and interpret data collected by our products, such as Firefox and

Mozilla 75 Dec 1, 2022
Scalable and fast data store optimised for time series data such as financial data, events, metrics for real time analysis

OnTimeDB Scalable and fast data store optimised for time series data such as financial data, events, metrics for real time analysis OnTimeDB is a time

Stuart 2 Apr 5, 2022
I/O and binary data encoding for Rust

nue A collection of tools for working with binary data and POD structs in Rust. pod is an approach at building a safe interface for transmuting POD st

null 38 Nov 9, 2022
A Rust library for zero-allocation parsing of binary data.

Zero A Rust library for zero-allocation parsing of binary data. Requires Rust version 1.6 or later (requires stable libcore for no_std). See docs for

Nick Cameron 45 Nov 27, 2022
A performant binary encoding for geographic data based on flatbuffers

FlatGeobuf A performant binary encoding for geographic data based on flatbuffers that can hold a collection of Simple Features including circular inte

FlatGeobuf 477 Jan 5, 2023
TestDrive automatically scrapes input/output data from BOJ(Baekjoon Online Judge) and runs tests for your executable binary file!

?? TestDrive What does it do? TestDrive automatically scrapes input/output data from BOJ(Baekjoon Online Judge) and runs tests for your executable bin

Hyeonseok Jung 3 Mar 5, 2022
binocle is a graphical tool to visualize binary data

a graphical tool to visualize binary data

David Peter 773 Dec 30, 2022