mvSQLite
Distributed, MVCC SQLite that runs on top of FoundationDB.
Features
- Full feature-set from SQLite: mvSQLite integrates with SQLite as a custom VFS layer. It is a layer "below" SQLite, and all of SQLite's features are available.
- Time travel: Checkout the snapshot of your database at any point of time in the past.
- Lock-free, scalable reads and writes: Optimistic fine-grained concurrency with BEGIN CONCURRENT-like semantics. mvSQLite inherits FoundationDB's lock-free property - not a single distributed lock is acquired during data plane operation.
- Get the nice properties from FoundationDB, without its limits: Correctness, really fast and scalable distributed transactions, synchronous and asynchronous replication, integrated backup and restore. Meanwhile, there's no five-second transaction limit any more, and a SQLite transaction can be ~39x larger than FDB's native transaction.
- Drop-in addition: Use
LD_PRELOADor a patchedlibsqlite3.soto plug mvSQLite into your existing apps. Read the docs
Releases
Grab the latest binaries from the Releases page. You can also build your own binaries to run on a platform other than x86-64.
Quick reference
Check the single-page mvSQLite Quick Reference for common operations with mvSQLite.
Try it
Install FoundationDB:
wget https://github.com/apple/foundationdb/releases/download/7.1.15/foundationdb-clients_7.1.15-1_amd64.deb
sudo dpkg -i foundationdb-clients_7.1.15-1_amd64.deb
wget https://github.com/apple/foundationdb/releases/download/7.1.15/foundationdb-server_7.1.15-1_amd64.deb
sudo dpkg -i foundationdb-server_7.1.15-1_amd64.deb
Download the binaries:
curl -L -o ./libmvsqlite_preload.so https://github.com/losfair/mvsqlite/releases/download/v0.2.1/libmvsqlite_preload.so
curl -L -o ./mvstore https://github.com/losfair/mvsqlite/releases/download/v0.2.1/mvstore
chmod +x ./mvstore
Run mvstore, the server-side half that should be colocated with the FoundationDB cluster in production:
RUST_LOG=info ./mvstore \
--data-plane 127.0.0.1:7000 \
--admin-api 127.0.0.1:7001 \
--metadata-prefix mvstore \
--raw-data-prefix m
Create a namespace with the admin API:
curl http://localhost:7001/api/create_namespace -i -d '{"key":"test"}'
Build libsqlite3 and the sqlite3 CLI: (note that a custom build is only needed here because the sqlite3 binary shipped on most systems are statically linked to libsqlite3 and LD_PRELOAD don't work)
wget https://www.sqlite.org/2023/sqlite-amalgamation-3410000.zip
unzip sqlite-amalgamation-3410000.zip
cd sqlite-amalgamation-3410000
gcc -O2 -fPIC --shared -o libsqlite3.so ./sqlite3.c -lpthread -ldl -lm
gcc -O2 -o sqlite3 ./shell.c -L. -lsqlite3
Set environment variables, and run the shell:
export RUST_LOG=info MVSQLITE_DATA_PLANE="http://localhost:7000"
# "test" is the key of the namespace we created earlier
LD_PRELOAD=../libmvsqlite_preload.so LD_LIBRARY_PATH=. ./sqlite3 test
You should see the sqlite shell now :) Try creating a table and play with it.
Contributing
mvsqlite can be built with the standard Rust toolchain:
cargo build --release -p mvstore
cargo build --release -p mvsqlite
make -C mvsqlite-preload
Internals are documented in the wiki.
16 Jan 2, 2023
516 Jan 2, 2023
8.3k Dec 30, 2022
7.6k Dec 31, 2022
1.9k Jan 5, 2023
952 Jan 5, 2023
27 Nov 22, 2022
309 Dec 20, 2022
20 Jan 2, 2023
45 Nov 16, 2022
14 Dec 5, 2022
10 Nov 13, 2022
139 Dec 28, 2022
36 Dec 7, 2022
5 Sep 23, 2022
11 Sep 3, 2022
33 Dec 16, 2022