Python module implemented in Rust for counting the number of one bits in a numpy array.

Andrew MacDonald

Last update: Jul 9, 2022

Related tags

Foreign Function Interface bit-counter

Overview

bit-counter

Package for counting the number of one bits in a numpy array of uint8 values. Implemented as a Python module using Rust, providing high performance counting.

Building

To build this package an installation of Rust and Python with the maturin package is required. The Maturin documentation on maturin local development is a useful reference for more details.

This project is configured by default to target CPUs with the POPCNT instruction. The builds available on PyPI have been built with this configuration. If you require a version for an older CPU without popcnt support, build with the RUSTFLAGS environment variable to exclude the popcnt target-feature. i.e. RUSTFLAGS='-C target-feature=-popcnt' maturin build -r.

Example usage

import numpy as np
from bit_counter import count_ones

arr = np.packbits(np.random.choice([True, False], 1000000))

count_of_true_values = count_ones(arr)

Performance

When built to target a CPU with popcnt support the count_ones method provided is substantially faster than a naive np.unpackbits(arr).sum(). The count_ones method also doesn't require unpacking the bit packed numpy array, so doesn't require any addtional memory to do the calculation.

For example with the following test code for 100 million True/False values that are packed:

import numpy as np
from bit_counter import count_ones

arr = np.packbits(np.random.choice([True, False], 100000000))

Using this package on a mid-2013 MacBook Air (1.7 GHz Intel Core i7) yields:

%timeit count_ones(arr)

45.3 ms ± 2.08 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

Compared to the simple unpack and sum case:

%timeit np.unpackbits(arr).sum()

108 ms ± 6.89 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

Comments

rustflags in cargo config?
I wonder if it would be worth putting something like the following in .cargo/config.toml:

[build] rustflags = ["-C", "target-feature=+popcnt"]

That way builds would automatically use of popcnt by default and you could simplify the documentation and CI a bit. If you did go down this path you could add a note about the inverse: If someone wants to build for a CPU that doesn't support popcnt then they should build with RUSTFLAGS="-C target-feature=-popcnt"
opened by wezm 1

Python module implemented in Rust for counting the number of one bits in a numpy array.

Related tags

Overview

bit-counter

Building

Example usage

Performance

You might also like...

Implementation of Monte Carlo PI approximation algorithm in Rust Python bindings

A simple library to allow for easy use of python from rust.

Pyo3 - Rust bindings for the Python interpreter

RustPython - A Python Interpreter written in Rust

Robust and Fast tokenizations alignment library for Rust and Python

Very experimental Python bindings for the Rust biscuit-auth library

Python bindings for heck, the Rust case conversion library

Arrowdantic is a small Python library backed by a mature Rust implementation of Apache Arrow

lavalink-rs bindings for Python

Comments

rustflags in cargo config?

Owner

Andrew MacDonald

Create, open, manage your Python projects with ease, a project aimed to make python development experience a little better

Easy way to write Node.js module using Rust

Another cursed Garry's Mod module. This time, it adds the C preprocessor to Lua scripts

Mod_wasm - an extension module for the Apache HTTP Server (httpd) that enables the usage of WebAssembly (Wasm).

Slitter is a C- and Rust-callable slab allocator implemented primarily in Rust, with some C for performance or to avoid unstable Rust features.

lens implemented in rust

Rust <-> Python bindings

Rust bindings for the Python interpreter

A script language like Python or Lua written in Rust, with exactly the same syntax as Go's.

Rust Python modules for interacting with Metaplex's NFT standard.