Unofficial python bindings for the rust llm library. 🐍❤️🦀

Lukas Kreussel

Last update: May 20, 2023

Related tags

Overview

llm-rs-python: Python Bindings for Rust's llm Library

Welcome to llm-rs, an unofficial Python interface for the Rust-based llm library, made possible through PyO3. Our package combines the convenience of Python with the performance of Rust to offer an efficient tool for your machine learning projects. 🐍 ❤️ 🦀

With llm-rs, you can operate a variety of Large Language Models (LLMs) including LLama and GPT-NeoX directly on your CPU.

For a detailed overview of all the supported architectures, visit the llm project page.

Installation

Simply install it via pip: pip install llm-rs

Usage

The package is type-hinted for easy usage.

A Llama model can be run like this:

from llm_rs import Llama

#load the model
model = Llama("path/to/model.bin")

#generate
print(model.generate("The meaning of life is"))

Documentation

For in-depth information on customizing the loading and generation processes, refer to our detailed documentation.

Rust bindings for libjuice. Look at datachannel-rs if you need more batteries.

3 Sep 25, 2022

GGML bindings that aim to be idiomatic Rust rather than directly corresponding to the C/C++ interface

rusty-ggml GGML bindings that aim to be idiomatic Rust rather than directly corresponding to the C/C++ interface. GG-what? See: https://github.com/gge

6 May 16, 2023

Rust bindings to llama.cpp, using metal on macOS

llama-rs Rust bindings to llama.cpp, for macOS, with metal support, for testing and evaluating whether it would be worthwhile to run an Llama model lo

7 Aug 31, 2023

High-level, optionally asynchronous Rust bindings to llama.cpp

llama_cpp-rs Safe, high-level Rust bindings to the C++ project of the same name, meant to be as user-friendly as possible. Run GGUF-based large langua

4 Nov 21, 2023

A highly modular Bitcoin Lightning library written in Rust. Its Rust-Lightning, not Rusty's Lightning!

Rust-Lightning is a Bitcoin Lightning library written in Rust. The main crate, lightning, does not handle networking, persistence, or any other I/O. Thus, it is runtime-agnostic, but users must implement basic networking logic, chain interactions, and disk storage. More information is available in the About section.

850 Jan 3, 2023

Rust library that can be reset if you think it's slow

GoodbyeKT Rust library that can be reset if you think it's slow

39 Jun 16, 2022

Notion Offical API client library for rust

Notion API client library for rust.

65 Dec 26, 2022

Rust library for program synthesis of string transformations from input-output examples 🔮

Synox implements program synthesis of string transformations from input-output examples. Perhaps the most well-known use of string program synthesis in end-user programs is the Flash Fill feature in Excel. These string transformations are learned from input-output examples.

21 Apr 27, 2022

SE3 Rust library for Robotics

Algebraic Robots A small Rust Library for SE3 Supported: Twist Screw SE3 Group se3 algebra Adjoint SE3 Twist Chains Wrenches Future plans: Jacobians V

4 Jul 18, 2021

Releases(0.2.1)

0.2.1(May 19, 2023)

The ability to quantize models is now available for every architecture via quantize.
Source code(tar.gz)
Source code(zip)
0.2.0(May 19, 2023)
Added support for Mosaic ML's MPT models.

Added support for LoRA adapters for all architectures.

⚠️Caution⚠️ Due to changes in the ggml format old quantized models are not supported anymore!
Source code(tar.gz)
Source code(zip)
0.1.1(May 8, 2023)

Added the tokenize and decode functions to each model, to enable access to the internal tokenizer.

The generation of tokens is now GIL free, meaning other background threads can run at the same time.
Source code(tar.gz)
Source code(zip)
0.1.0(May 3, 2023)
Since llama-rs was renamed to llm and now supports multiple model architectures, this wrapper was also expanded to support the new trait system and library structure.

Supported architectures for now:

Llama

GPT2

GPTJ

GPT-NeoX

Bloom

The loader was also reworked and now supports the mmap-able ggjt. To support this the SessionConfig was expandend with the prefer_mmap field.
Source code(tar.gz)
Source code(zip)
0.0.2(Apr 19, 2023)

Source code(tar.gz)
Source code(zip)
0.0.1(Apr 18, 2023)

Source code(tar.gz)
Source code(zip)