SQLite compiled to WASM with pluggable data storage

Related tags

Database sqlite wasm vfs
Overview

wasm-sqlite

SQLite compiled to WASM with pluggable data storage. Useful to save SQLite in e.g. Cloudflare Durable Objects (example: https://github.com/rkusa/do-sqlite).

Status: This is very experimental. Don't use it for real applications yet! See the conclusion of my blog post for reasons for why not to use it.

npm install -S @rkusa/wasm-sqlite

Example

const sqlite = await Sqlite.instantiate(
  // get page
  async (ix: number) => {
    return await storage.get(ix) ?? new Uint8Array(4096);
  },

  // put page
  async (ix: number, page: Uint8Array) => {
    await storage.put(ix, page);
  }
);

await sqlite.execute("...", []);
const query: T = await sqlite.query("...", []);

Build

Execute the following once:

brew install cmake ninja binaryen
./build-wasi-sdk.sh

After that, build it via:

npm run build
You might also like...
Simple and handy btrfs snapshoting tool. Supports unattended snapshots, tracking, restoring, automatic cleanup and more. Backed with SQLite.
Simple and handy btrfs snapshoting tool. Supports unattended snapshots, tracking, restoring, automatic cleanup and more. Backed with SQLite.

Description Simple and handy btrfs snapshoting tool. Supports unattended snapshots, tracking, restoring, automatic cleanup and more. Backed with SQLit

Some bunch of test scripts to generate a SQLite DB with 1B rows in fastest possible way

To find out the fastest way to create an SQLite DB with 1B random rows.

A tool for automated migrations for PostgreSQL, SQLite and MySQL.

Models Models is an implementation for a SQL migration management tool. It supports PostgreSQL, MySQL, and SQLite. Quick Start install the CLI by runn

Provides a Rust-based SQLite extension for using Hypercore as the VFS for your databases.

SQLite and Hypercore A Rust library providing SQLite with an virtual file system to enable Hypercore as a means of storage. Contributing The primary r

ChiselStore is an embeddable, distributed SQLite for Rust, powered by Little Raft.

ChiselStore ChiselStore is an embeddable, distributed SQLite for Rust, powered by Little Raft. SQLite is a fast and compact relational database manage

cogo rust coroutine database driver (Mysql,Postgres,Sqlite)

cdbc Coroutine Database driver Connectivity.based on cogo High concurrency,based on coroutine No Future'q,Output=*,No async fn, No .await , no Poll*

Interface to SQLite

SQLite The package provides an interface to SQLite. Example Open a connection, create a table, and insert some rows: let connection = sqlite::open(":m

Rusqlite is an ergonomic wrapper for using SQLite from Rust

Rusqlite Rusqlite is an ergonomic wrapper for using SQLite from Rust. It attempts to expose an interface similar to rust-postgres. use rusqlite::{para

Build SQLite virtual file systems (VFS) by implementing a simple Rust trait.

sqlite-vfs Build SQLite virtual file systems (VFS) by implementing a simple Rust trait. Documentation | Example This library is build for my own use-c

Comments
  • database disk image is malformed

    database disk image is malformed

    Hello !

    Nice project, I would like to use wasm-sqlite to mimic D1 on Deno Deploy or any other V8 "serverless" workers. I tried to edit your project for a basic test using Deno and localStorage but got the following error:

    error: Uncaught (in promise) Error: failed to get next row: database disk image is malformed
          throw new Error(err);
                ^
        at SqliteConnection.throwLastError (file:///Users/sqlite-deploy/vfs.ts:217:13)
        at async SqliteConnection.queryRaw (file:///Users/sqlite-deploy/vfs.ts:267:7)
        at async file:///Users/sqlite-deploy/index.ts:32:20
    

    The code:

    import { Sqlite } from "./vfs.ts";
    
    const sqlite = await Sqlite.instantiate({
      pageCount(): number {
        return self.pageCount;
      },
      async getPage(ix: number): Promise<Uint8Array> {
        const page =
          new Uint8Array(JSON.parse(await localStorage.getItem(ix))) ??
          new Uint8Array(4096);
        return page;
      },
      async putPage(ix: number, page: Uint8Array): Promise<void> {
        await localStorage.setItem(ix, JSON.stringify(Array.from(page)));
        self.pageCount = Math.max(self.pageCount, ix + 1);
      },
      async delPage(ix: number): Promise<void> {
        await localStorage.removeItem(ix);
        if (ix + 1 >= self.pageCount) {
          self.pageCount = ix;
        }
      },
    });
    
    localStorage.clear();
    const conn = await sqlite.connect();
    const create: T = await conn.queryRaw(`CREATE TABLE td(a INT) STRICT;`);
    const insert: T = await conn.queryRaw(`INSERT INTO td VALUES('000123');`);
    console.log(create, insert);
    

    And the vfs.ts diff:

    1c1,2
    < import * as Asyncify from "https://unpkg.com/asyncify-wasm?module";
    ---
    > import * as Asyncify from "asyncify-wasm";
    > import module from "./wasm_sqlite.wasm";
    23,24c24
    <     const { instance } = await Asyncify.instantiateStreaming(
    <         fetch(new URL("./wasm_sqlite.wasm", import.meta.url)), {
    ---
    >     const instance = await Asyncify.instantiate(module, {
    

    Do you have any clue why the database got corrupted ?

    Best,

    opened by nestarz 2
Owner
Markus Ast
Self-employed web engineer. #Rust #Wasm #Go #TypeScript #React #REST
Markus Ast
Mycelite is a SQLite extension that allows you to synchronize changes from one instance of SQLite to another.

Mycelite What is Mycelite? Mycelite is a SQLite extension that allows you to synchronize changes from one instance of SQLite to another. Currently, it

Mycelial 16 Jan 2, 2023
🐸Slippi DB ingests Slippi replays and puts the data into a SQLite database for easier parsing.

The primary goal of this project is to make it easier to analyze large amounts of Slippi data. Its end goal is to create something similar to Ballchasing.com but for Melee.

Max Timkovich 20 Jan 2, 2023
Command-line tool to convert Apple HealthKit data to a SQLite database.

healthkit-to-sqlite Command-line tool to convert Apple HealthKit data to a SQLite database. Getting Started Open the Health app on your iOS device. Cl

Justin Shrake 3 Jan 7, 2023
A Key-Value data storage system. - dorea db

Dorea DB ?? Dorea is a key-value data storage system. It is based on the Bitcask storage model Documentation | Crates.io | API Doucment 简体中文 | English

ZhuoEr Liu 112 Dec 2, 2022
Zenith substitutes PostgreSQL storage layer and redistributes data across a cluster of nodes

Zenith substitutes PostgreSQL storage layer and redistributes data across a cluster of nodes

null 5.7k Jan 6, 2023
OBKV Table Client is Rust Library that can be used to access table data from OceanBase storage layer.

OBKV Table Client is Rust Library that can be used to access table data from OceanBase storage layer. Its access method is different from JDBC, it skips the SQL parsing layer, so it has significant performance advantage.

OceanBase 4 Nov 14, 2022
Scalable and fast data store optimised for time series data such as financial data, events, metrics for real time analysis

OnTimeDB Scalable and fast data store optimised for time series data such as financial data, events, metrics for real time analysis OnTimeDB is a time

Stuart 2 Apr 5, 2022
🧰 The Rust SQL Toolkit. An async, pure Rust SQL crate featuring compile-time checked queries without a DSL. Supports PostgreSQL, MySQL, SQLite, and MSSQL.

SQLx ?? The Rust SQL Toolkit Install | Usage | Docs Built with ❤️ by The LaunchBadge team SQLx is an async, pure Rust† SQL crate featuring compile-tim

launchbadge 7.6k Dec 31, 2022
Ergonomic bindings to SQLite for Rust

Rusqlite Rusqlite is an ergonomic wrapper for using SQLite from Rust. It attempts to expose an interface similar to rust-postgres. use rusqlite::{para

Rusqlite 1.9k Jan 5, 2023
SQLite clone from scratch in Rust

Rust-SQLite (SQLRite) Rust-SQLite, aka SQLRite , is a simple embedded database modeled off SQLite, but developed with Rust. The goal is get a better u

João Henrique Machado Silva 952 Jan 5, 2023