An AI-native lightweight, reliable, and high performance open-source vector database.

Related tags

Learning Resources open-source vector-database
Overview

Oasys

License Contributor Covenant Discord

What is OasysDB?

OasysDB is a vector database that can be used to store and query high-dimensional vectors. Our goal is to make OasysDB fast and easy to use. We are also working on making it easy to deploy and scale.

Features

  • HTTP-based API: All operations are exposed via a RESTful API. This makes it easy to integrate with other systems without having to install any client libraries.

  • Persistent storage: Embeddings and graphs data are stored on disk and are persisted across restarts.

  • HNSW indexing: OasysDB uses the HNSW algorithm to build graphs to index embeddings. This allows for fast and accurate nearest neighbor search.

  • Multi-graph support (WIP): OasysDB supports multiple HNSW graphs. This allows you to version and customize your graphs to suit different use cases. For example, optimizing speed for a specific query type or optimizing accuracy for a specific dataset.

Getting Started

Installation

The easiest way to get started is to use Docker. You can pull the latest image from GitHub Container Registry:

docker pull ghcr.io/oasysai/oasysdb:latest

This will pull the latest version of the server from the GitHub Container Registry. You can then run the server with the following command:

docker run \
    --publish 3141:3141 \
    --env OASYSDB_DIMENSION=512 \
    --env OASYSDB_TOKEN=token \
    ghcr.io/oasysai/oasysdb:latest

  • OASYSDB_DIMENSION: An integer representing the dimension of your embedding. Different embedding model will have different dimension. For example, OpenAI Ada 2 has a dimension of 1536.

  • OASYSDB_TOKEN: A string that you will use to authenticate with the server. You need to add x-oasysdb-token header to your request with the value of this environment variable.

This will start OasysDB that is accessible on port 3141. You can change this by changing the port number in the --publish flag and setting the OASYSDB_PORT environment variable to the port number that you want to use.

Testing the server

You can test the server by calling GET / using your favorite HTTP client. For example, you can use curl:

curl http://localhost:3141

You can replace localhost with the IP address of the server if you are running the server on a remote machine.

Quickstart

To put it simply, these are the primary steps to get started with OasysDB:

  1. Set a value.
  2. Create a graph.
  3. Query the graph.

Set a value

POST /values/<key>

{
  "embedding": [1.0, 2.0, 3.0],
  "data": { "text": "OasysDB is awesome!" }
}

This endpoint sets a value for a given key. The value is an embedding and an optional data object. The embedding is a list of floating-point numbers. The data object is a JSON object of string keys and values.

Create a graph

POST /graphs

Optional request body:

{
  "name": "my-graph",
  "ef_construction": 10,
  "ef_search": 10
}

This endpoint creates a graph. The graph is used to query for nearest neighbors. If there is no data provided, the server will create a default graph with the name default and the default ef_construction and ef_search values of 100 for both.

Query the graph

POST /graphs/<name>/query

{
  "embedding": [1.0, 2.0, 3.0],
  "k": 10
}

This endpoint queries the graph for the nearest neighbors of the given embedding. The k parameter is the number of nearest neighbors to return.

Note

  • All embedding dimensions must match the dimension configured in the server using the OASYSDB_DIMENSION environment variable.
  • Requests to /graphs and /values endpoints must include the x-oasysdb-token header with the value of the OASYSDB_TOKEN environment variable.
  • Currently, OasysDB doesn't support auto graph building. This means whenever you add or remove a value, you need to rebuild the graph.

More resources

For more information, please see the OasysDB documentation.

Disclaimer

This project is still in the early stages of development. We are actively working on it and we expect the API and functionality to change. We do not recommend using this in production yet.

We also don't have a benchmark yet. We are working on it and we will publish the results once we have them.

Contributing

We welcome contributions from the community. Please see contributing.md for more information.

We are also looking for advisors to help guide the project direction and roadmap. If you are interested, please contact us via Discord or alternatively, you can email me at [email protected].

Code of Conduct

We are committed to creating a welcoming community. Any participant in our project is expected to act respectfully and to follow the Code of Conduct.

Comments
  • FEAT: database persistence to disk

    FEAT: database persistence to disk

    Use case

    Currently, the database data, key-value pairs and index, is stored only in memory. This means that every time the database is restarted, we need to re-add the key-value and rebuild the index.

    This feature should allow the database data to be persisted to disk so that when there is crash or shutdown, the data and index of previous session can still be used.

    Proposed solution

    The database should store a snapshot of the key-value pairs periodically. Every time, the database is started, the database will lookup the most recent snapshot and prepopulate the key-value store and rebuild the index based on the existing key-value pairs.

    Additional Context

    • https://crates.io/crates/bincode
    enhancement help wanted accepted 
    opened by edwinkys 1
  • fix!: rewrite architecture

    fix!: rewrite architecture

    Purpose

    This PR rewrites the entire server architecture of OasysDB. Initially, OasysDB use a custom built HTTP server from scratch. This is not optimal and hard to maintain. So this PR introduce the usage of Rocket framework as the server.

    Approach

    This PR rewrites the server architecture using Rocket web framework and separate the server architecture from the database architecture.

    Testing

    • [x] I have tested this PR locally.
    • [x] I added tests to cover my changes, if not applicable, I have added a reason why.

    I have tested the PR locally and added a whole new set of test suite compared to the last one. I have also tested the builds either by using Cargo or Docker.

    Chore checklist

    • [x] I have updated the documentation accordingly.
    • [x] I have added comments to most of my code, particularly in hard-to-understand areas.
    opened by edwinkys 0
  • feat: database persistence

    feat: database persistence

    Purpose

    This PR will replace OasysDB HashMap key-value store that doesn't persist with sled embeddable database that persist. There might be a trade-off in the performance but I haven't test anything related with this yet.

    Sled is an embeddable key-value database that is ACID compliant making it a suitable option to persist data of OasysDB. For more information about Sled, check out their GitHub repo or Rust docs.

    Approach

    Although it's not the best option to add more dependencies, Sled is a pretty decent option that we have to achieve great persistance with minimal time investment.

    I have tried creating a append-only log from scratch using commitlog crate but I encounter a roadblock and unable to resolve it even after reading the docs.

    Testing

    • [x] I have tested this PR locally.
    • [x] I added tests to cover my changes, if not applicable, I have added a reason why.

    This PR introduce a new functionality in the test suite, creating different data storage when running the test and deleting them once the test is run. This is needed because the main data storage /data can't be shared across different server task runner.

    Beyond that, to test the persistance, I test it manually using my HTTP client to create a value, restart the server, and fetch the same value which result in the same value being returned.

    Chore checklist

    • [x] I have updated the documentation accordingly.
    • [x] I have added comments to most of my code, particularly in hard-to-understand areas.
    enhancement 
    opened by edwinkys 0
  • fix: post values remove response body

    fix: post values remove response body

    Purpose

    This PR will remove returning the value which contains embedding and data when creating a new key-value pair because it's redundant. For high-dimensional embeddings, this will cause extra overhead to write response back to the client.

    Approach

    Delete the part of code that constructs and returns the body for the response. Instead, we return None for the body with code 201 which is the same as previously.

    Testing

    • [x] I have tested this PR locally.
    • [x] I added tests to cover my changes, if not applicable, I have added a reason why.

    I have tested this PR manually using Insomnia to make a request to POST /values which after this change will return an empty object.

    Chore checklist

    • [x] I have updated the documentation accordingly.
    • [x] I have added comments to most of my code, particularly in hard-to-understand areas.
    enhancement 
    opened by edwinkys 0
  • fix: increase read buffer

    fix: increase read buffer

    Purpose

    This PR introduce a flexible configuration to configure read buffer for the TCP stream. It was initially set on 1024 which is not enough to read full length body of high-dimensional embeddings (tested using 512 and 1536 dimensions embedding).

    Approach

    Added an environment variable OASYSDB_BUFFER_SIZE that will define the read buffer size for when reading data from stream. By default, this is set to 32768 (enough to read 1536 dimension embedding created by Python random.random() function).

    This environment variable allows users who use lower dimensional embedding to set the buffer size lower to optimize the performance.

    Testing

    • [x] I have tested this PR locally.
    • [x] I added tests to cover my changes, if not applicable, I have added a reason why.

    Tested this PR using a Python script to generate the embedding and calling the POST /values endpoint to create a new value. I didn't add anything to the test suite as making a custom body for the request is time consuming.

    Chore checklist

    • [x] I have updated the documentation accordingly.
    • [x] I have added comments to most of my code, particularly in hard-to-understand areas.
    bug 
    opened by edwinkys 0
  • fix: fix server multithreading

    fix: fix server multithreading

    Purpose

    This PR solves the issue where the server doesn't acccept multiple connections/requests at once. It will throw connection refused error or even sometimes crash the server.

    Approach

    1. I modified the server in a way to prevent direct mutation to the server by using Arc and Mutex for shared values.
    2. I fixed the tokio::spawn code which is the primary source of this problem.
    3. I remove the Server.serve() function and replace it with a standalone serve function that will handle the server and thread creation. In this function, the server is wrapped by Arc to allow threads to refer to the same server.

    Testing

    • [x] I have tested this PR locally.
    • [x] I added tests to cover my changes, if not applicable, I have added a reason why.

    I have modified the test suite to accomodate the changes that I made to the server functionality.

    I tested this PR by using a custom Python script to create a multi-threaded request to the server while the server is running on localhost port 3141. Without this changes, the server will throw an error when encountering multi-thread requests.

    Chore checklist

    • [x] I have updated the documentation accordingly.
    • [x] I have added comments to most of my code, particularly in hard-to-understand areas.
    bug enhancement 
    opened by edwinkys 0
  • feat: token-based authentication

    feat: token-based authentication

    Purpose

    This PR adds a layer of HTTP authentication before performing actions to the database via the endpoints. This authentication is applied only for /index and /values routes.

    Approach

    This PR add a feature that will get x-oasysdb-token header from the request and match it against the OASYSDB_TOKEN environment variable used to run the server. If the token doesn't match, the server will return a 401 unauthorized response.

    Testing

    • [x] I have tested this PR locally.
    • [x] I added tests to cover my changes, if not applicable, I have added a reason why.

    I tested this PR by running cargo test and manually checking functionality by calling the endpoints directly from Insomnia. There is no new tests being added to the test suite. But, I added functionality to the existing tests to guard for unauthorized request without or invalid x-oasysdb-token.

    Chore checklist

    • [x] I have updated the documentation accordingly.
    • [x] I have added comments to most of my code, particularly in hard-to-understand areas.
    enhancement 
    opened by edwinkys 0
  • feat: graph build allow data filter

    feat: graph build allow data filter

    Purpose

    This PR introduce a feature that allows users to filter the data while building a graph. This allows building a graph with a specific set of values. This is useful to support multiple custom graph in the same database.

    For example, if we have data with type image or text, we can have one graph for image type only and another one for text type only.

    Approach

    We simply iterate over the filters which now is included in the data to build the graph, the graph configuration data. Then, we match if the values that we want to use to build a graph match any of the filter. If it is, we include it in the values used to build graph.

    Testing

    • [x] I have tested this PR locally.
    • [x] I added tests to cover my changes, if not applicable, I have added a reason why.

    I have tested the functionality locally with an HTTP client to create multiple values with 2 different IDs and used this functionality to create 2 different graph for each ID.

    Chore checklist

    • [ ] I have updated the documentation accordingly.
    • [x] I have added comments to most of my code, particularly in hard-to-understand areas.
    enhancement accepted 
    opened by edwinkys 0
Releases(v0.1.0-beta.1)

  • v0.1.0-beta.1(Dec 23, 2023)

    What's changed?

    • The 2 main endpoints are changed to /graphs which represents a queriable HNSW graph to query for nearest neighbors and /values which is the endpoint that handle operations on the key-value store.
    • Removes in-memory support of both the key-values and the graphs. But, the key-values and the graphs are now persisted to disk using Sled, an embeddable database built with Rust.
    • Rewriting the entire server architecture using Rocket web framework instead of a custom TCP algorithm that accepts HTTP.
      • Fix the concurrency issue of the previous release.
      • Remove returning the value back when setting a value to remove unnecessary HTTP overhead.
      • Fix read buffer issue of the custom TCP algorithm that prevents a large embedding size to be stored.
    • Fix GH actions for deployment and quality check.
    Source code(tar.gz)
    Source code(zip)

  • v0.1.0-alpha.2(Dec 15, 2023)

    What's changed?

    • Change routing name to be more semantical /kvs to /values.
    • Improvement of documentation both readme.md and contributing.md.
    • Add HTTP authentication method for the server. Users are now required to add OASYSDB_TOKEN as environment variable when setting up OasysDB server. When calling /index or /values endpoint, users need to include x-oasysdb-token header in the request.
    • Fix log not showing up when starting the server.
    Source code(tar.gz)
    Source code(zip)

  • v0.1.0-alpha.1(Dec 9, 2023)

    This is the first alpha release of OasysDB. This release will introduce the core functionality of a simple key value vector database.

    Features

    • OasysDB accepts HTTP protocols from clients to perform the core functionality.
    • The key-value functionality to get, set, and delete key-value pairs.
    • The vector functionality to build HNSW graph index and search nearest neighbors given the embedding.

    What's not included?

    • Key-value store persistance.
    • Resetting/flushing the key-value store.
    • HNSW index persistance and load.
    • Database authentication.
    Source code(tar.gz)
    Source code(zip)
Owner
Oasys
Accelerating your LLM app development with our LLM-focused BaaS platform.
Oasys
GitHub https://www.oasysai.com
Linked Atomic Random Insert Vector: a thread-safe, self-memory-managed vector with no guaranteed sequential insert.

Linked Atomic Random Insert Vector Lariv is a thread-safe, self-memory-managed vector with no guaranteed sequential insert. It internally uses a linke

Guillem Jara 8 Feb 1, 2023
Vemcache is an in-memory vector database.

Vemcache Vemcache is an in-memory vector database. Vemcache can be thought of as the Redis equivalent for vector databases. Getting Started Prerequisi

Faizaan Chishtie 8 May 21, 2023
An inquiry into nondogmatic software development. An experiment showing double performance of the code running on JVM comparing to equivalent native C code.

java-2-times-faster-than-c An experiment showing double performance of the code running on JVM comparing to equivalent native C code ⚠️ The title of t

xemantic 49 Aug 14, 2022
Cogo is a high-performance library for programming stackful coroutines with which you can easily develop and maintain massive concurrent programs.

Cogo is a high-performance library for programming stackful coroutines with which you can easily develop and maintain massive concurrent programs.

co-rs 47 Nov 17, 2022
Nyah is a programming language runtime built for high performance and comes with a scripting language.

?? Nyah ( Unfinished ) Nyah is a programming language runtime built for high performance and comes with a scripting language. ??️ Status Nyah is not c

Stacker 3 Mar 6, 2022
Support SIMD low-memory overhead and high-performance adaptive radix tree.

Artful Artful is an adaptive radix tree library for Rust. At a high-level, it's like a BTreeMap. It is based on the implementation of paper, see The A

future 3 Sep 7, 2022
Diosic is an open source web-based music collection server and streamer

Diosic is an open source web-based music collection server and streamer. Mainly suitable for users who need to deploy on servers with low hardware specifications.

Jinker 45 Jan 28, 2023
Powerful math lib for Vector, Matrix and Quaternion operations

An opinionated, powerful math lib for Vector2, Vector3, Matrix and Quaternion operations Vector2 Add, Sub, Div, Mul, Eq Distance Move towards target a

O(ʒ) 4 Mar 28, 2022
Powerful math lib for Vector, Matrix and Quaternion operations

An opinionated, powerful math lib for Vector2, Vector3, Matrix and Quaternion operations Vector2 Add, Sub, Div, Mul, Eq Distance Move towards target a

O(ʒ) 4 Mar 28, 2022
Compact, clone-on-write vector and string.

ecow Compact, clone-on-write vector and string. Types An EcoVec is a reference-counted clone-on-write vector. It takes up two words of space (= 2 usiz

Typst 78 Apr 18, 2023
Open-source Autonomy Software in Rust-lang with gRPC for the Roomba series robot vacuum cleaners

CleanIt Open-source Autonomy Software in Rust-lang with gRPC for the Roomba series robot vacuum cleaners Motivation Motivation is to build a complete

Kristoffer Rakstad Solberg 216 Dec 13, 2022
Maai_core is an open source AI that is written in rust

maai_core is an open source AI that is written in rust. It is more focused on being lightweight and speedy.

Max Beier 2 Aug 1, 2022
An open source WCH-Link library/command line tool written in Rust.

wlink - WCH-Link command line tool NOTE: This tool is still in development and not ready for production use. Known Issue: Only support binary firmware

WCH MCU for Rust 22 Mar 7, 2023
Open source bible study program on Linux!

Open Witness Library Open Witness Library is a program to read articles that carries God's true name. Warning This branch is outdated, please refer to

Wellington Júnior 5 Jun 29, 2023
An open-source UCIe implementation developed at UC Berkeley.

UCIe An open-source UCIe implementation developed at UC Berkeley. This project is very early-stage. If you are interested in contributing, see below.

null 3 Aug 18, 2023
A HashMap/Vector hybrid: efficient, ordered key-value data storage in Rust.

hashvec A HashVec is a hash map / dictionary whose key-value pairs are stored (and can be iterated over) in a fixed order, by default the order in whi

Skye Terran 2 May 16, 2022
Type erased vector. All elements have the same type.

Type erased vector. All elements have the same type. Designed to be type-erased as far as possible - most of the operations does not know about concre

null 7 Dec 3, 2022
The Fast Vector Similarity Library is designed to provide efficient computation of various similarity measures between vectors.

Fast Vector Similarity Library Introduction The Fast Vector Similarity Library is designed to provide efficient computation of various similarity meas

Jeff Emanuel 243 Sep 6, 2023
The first compute-centric vector graphic video game

??️ Vong This repository contains source code for the first native use of a compute-centric vector graphics video game, inspired by Pong. ✍️ Authors @

Spencer C. Imbleau 29 Nov 24, 2023