🚧 WIP 🚧 Vector database plugin for Postgres, written in Rust, specifically designed for LLM.

TensorChord

Last update: Apr 26, 2023

Related tags

Machine learning vector nearest-neighbor-search gpt postgress faiss vector-database llm chatgpt

Overview

pgvecto.rs

pgvecto.rs is a Postgres extension that provides vector similarity search functions. It is written in Rust and based on pgrx.

Features

cosine_distance: cosine distance
dot_product_distance: dot product distance
square_euclidean_distance: square Euclidean distance

Build from source

cargo install cargo-pgrx
cargo pgrx init
cargo pgrx run

Getting Started

-- install the extension
DROP EXTENSION IF EXISTS vectors;
CREATE EXTENSION vectors;
-- check the extension related functions
\df+

-- call the distance function through operators

-- square Euclidean distance
SELECT array[1, 2, 3] <-> array[3, 2, 1];
-- dot product distance
SELECT array[1, 2, 3] <#> array[3, 2, 1];
-- cosine distance
SELECT array[1, 2, 3] <=> array[3, 2, 1];

-- create table
CREATE TABLE items (id bigserial PRIMARY KEY, emb numeric[]);
-- insert values
INSERT INTO items (emb) VALUES (ARRAY[1,2,3]), (ARRAY[4,5,6]);
-- query the similar embeddings
SELECT * FROM items ORDER BY emb <-> ARRAY[3,2,1] LIMIT 5;
-- query the neighbors within a certain distance
SELECT * FROM items WHERE emb <-> ARRAY[3,2,1] < 5;

Roadmap

See the open issues

Contributors ✨

Thanks goes to these wonderful people (emoji key):

_{Ce Gao} 💼 🖋 📖	_Keming 🐛 💻 📖 🤔 🚇	_{Jinjing Zhou} 🎨 🤔 📆	_{Alex Chi} 💻
Add your contributions

This project follows the all-contributors specification. Contributions of any kind welcome!

Acknowledgements

Thanks to the following projects:

pgrx - Postgres extension framework in Rust
pgvector - Postgres extension for vector similarity search written in C

A tiny embedding database in pure Rust.

tinyvector - a tiny embedding database in pure Rust ✨ Features Tiny: It's in the name. It's literally just an axum server. Extremely easy to customize

210 Jul 12, 2023

A plugin for Jupyter Notebooks that shows you its energy use.

Jupyter Energy Jupyter Notebooks are a data science tool mostly used for statistics and machine learning, some of the most energy-intensive computing

3 Jun 29, 2022

Msgpack serialization/deserialization library for Python, written in Rust using PyO3, and rust-msgpack. Reboot of orjson. msgpack.org[Python]

ormsgpack ormsgpack is a fast msgpack library for Python. It is a fork/reboot of orjson It serializes faster than msgpack-python and deserializes a bi

139 Dec 30, 2022

Fwumious Wabbit, fast on-line machine learning toolkit written in Rust

Comments

feat: Roadmap
Feature:

Vectorize distance operations

Custom Index using IVF

Custom index using HNSW

Custom index using ScaNN

Support Vec8, Vec16

Tests:

Add benchmarks

Add unit tests

Reference

https://github.com/pgvector/pgvector

https://github.com/postgres/postgres/tree/master/contrib/bloom
opened by VoVAllen 1