ReadySet is a lightweight SQL caching engine written in Rust that helps developers enhance the performance and scalability of existing applications.

Overview


ReadySet is a SQL caching engine designed to help developers enhance the performance and scalability of their existing database-backed applications.

What is ReadySet?

ReadySet is a lightweight SQL caching engine that precomputes frequently-accessed query results and automatically keeps these results up-to-date over time as the underlying data in your database changes. ReadySet is wire-compatible with MySQL and Postgres and can be adopted without code changes.



ReadySet acts as both a SQL cache and a proxy– when you first connect ReadySet to your application, it defaults to proxying all of your queries to your backing database, so it doesn't change the behavior of your application.

From there, you can monitor the performance of your queries using the ReadySet dashboard, and cache a subset of them in ReadySet. ReadySet automatically keeps cached results up-to-date over time as the underlying data in your database changes, so there's no need to write cache maintenance logic.

Getting Started

Deploy ReadySet

The ReadySet orchestrator is a command line tool that uses Docker Compose to spin up a ReadySet instance on your local machine. The following command downloads and runs the ReadySet orchestrator:

bash -c "$(curl -sSL https://launch.readyset.io)"

The orchestrator will prompt you to select a wire protocol, and to connect ReadySet to an existing database (or create a new one).

Connect to ReadySet

  • To connect to ReadySet via the MySQL or Postgres command line, run the command that the orchestrator prints after successfully deploying a ReadySet instance.
  • To connect ReadySet to your application, use the ReadySet connection string (printed by the orchestrator after successfully deploying ReadySet) in the same way you would if it were a MySQL or Postgres connection string. Check out our docs for more information.

Once you connect to ReadySet, you can start issuing queries as you would with a traditional database.

Monitor Query Performance

By default, all queries sent to ReadySet are proxied to the backing database. You can see the current list of proxied queries and their performance profiles via the ReadySet dashboard which is accessible on http://localhost:4000.

Cache Queries

ReadySet supports caching both prepared statements (i.e. parameterized queries) and one-off queries. To cache a query in ReadySet, you’ll need either the query ID (which can be found in the dashboard, or by running SHOW CACHES) or the full query text (i.e. the SELECT statement).

From there, you can run CREATE CACHE FROM <query ID> or CREATE CACHE FROM <select statement> via the MySQL or Postgres client.

Write Handling

You can either send writes to ReadySet or directly to your backing database. If you send ReadySet a write, it will be proxied to your backing database. ReadySet waits to receive updates from your database's bin logs before updating the cached state to reflect those writes.

Note: The ReadySet Orchestrator is updated frequently with releases built off our internal ReadySet repository. Because of this, the orchestrator may have features not yet in our public repo. We're working on making our live codebase available on Github.

FAQs

Q: How does ReadySet work under the hood?

A: The heart of ReadySet is a query engine based on partially-stateful, streaming data flow. To learn more about how it works, see our docs.

Q: How does ReadySet keep cached state up to date?

A: ReadySet receives updates about data changes from your backing database via binlog replication and uses those updates to automatically update its internal state.

Q: Do I have to send all of my database traffic to ReadySet?

A: You can if you want to, but it’s not required. You can manually route a subset of your traffic to ReadySet (as you would with a traditional database read replica), or you can send all of it to ReadySet. It's important to note that not all of the queries sent to ReadySet need to be cached– you have fine-grained control over what is cached vs. what is proxied.

Documentation

For more information, check out our full docs site here.

Join the Community

For questions or support, join us on the ReadySet Community Slack. You can also post questions on our Github forum. Everyone is welcome!

Development

For instructions on how to build ReadySet from source, see our developer quick start guide.

Note: This repository contains a snapshot of the current state of the ReadySet internal repository. Our team is currently working on getting our live codebase, including all development history, ready to be hosted on GitHub, at which point we'll force push to this repository.

License

ReadySet is licensed under the BSL 1.1 license, converting to the open-source Apache 2.0 license after 4 years. The ReadySet team is hard at work getting the codebase ready to be hosted on Github.

ReadySet is also available as a paid cloud service that can be deployed to your accounts or hosted entirely by ReadySet. You can get early access to our cloud product by signing up here.

Comments
  • Bug: Error with enum type and geography type at Postgres

    Bug: Error with enum type and geography type at Postgres

    So, i just install ReadySet at my local with my Postgres 14 existing.

    Then, i just want to select my "Table" with specific type like enum or geography. When i try to query got this error.

    ERROR:  internal error: could not retrieve expected column index 3 from row while parsing psql result: error deserializing column 3: cannot convert between the Rust type `noria_data::DataType` and the Postgres type `geography`
    

    this is bug? or i just wrong install

    btw, great work for doing this. Keep it up.

    opened by bungambohlah 4
  • Type conversion error when selecting any `character varying(n)[]` column

    Type conversion error when selecting any `character varying(n)[]` column

    Similar to #6:

    Any query through Readyset that attempts to select a character varying(n)[] column seems to result in the following type of error:

    ERROR:  internal error: could not retrieve expected column index 1 from row while parsing psql result: error deserializing column 1: cannot convert between the Rust type `noria_data::DataType` and the Postgres type `_varchar`
    

    Ie, for a table that looks like:

    CREATE TABLE some_table (
        id uuid DEFAULT uuid_generate_v4() PRIMARY KEY,
        some_text_array character varying(5)[],
    );
    
    select id, some_text_array from some_table -- query will result in the above error
    
    select id from some_table -- query will succeed
    
    opened by zi-nt 4
  • Webhooks support to persist data to third party end-points

    Webhooks support to persist data to third party end-points

    Is your feature request related to a problem? Please describe. See a potential to have a feature to move 3rd party data integrations closer to the DB. For example, based on some condition or CRUD operations on any db table - to be able to post data to 3rd party end-points via https POST

    Describe the solution you'd like Maybe add an easy to use configurable user interface to build such Integrations and along with a feature to view all the request / response logs for traceability.

    Describe alternatives you've considered

    Additional context Since keeping integrations in the application layer can be tricky some cases leading to data inconsistency - it is always fail safe to handle data integrations closer to the DB right before a CRUD operation.

    A usecase would be to send data generated in a legacy system to a new system

    enhancement 
    opened by pythonpen 3
  • Type conversion error when selecting any UUID column

    Type conversion error when selecting any UUID column

    Using PG13:

    Any query through Readyset that attempts to select a UUID column seems to result in the following type of error:

    ERROR:  internal error: could not retrieve expected column index 0 from row while parsing psql result: error deserializing column 0: cannot convert between the Rust type `noria_data::DataType` and the Postgres type `uuid`
    

    Ie, for a table that looks like:

    CREATE TABLE books (
        id uuid DEFAULT uuid_generate_v4() PRIMARY KEY,
        title text NOT NULL
    )
    
    select id, title from books -- query will result in the above error
    
    select title from books -- query will succeed
    
    bug 
    opened by zi-nt 2
  • deployment error

    deployment error

    Downloading ReadySet orchestrator Welcome to the ReadySet orchestrator.

    I found an existing deployment named crmeb. We can continue with this deployment, or create a new one.

    ✔ Would you like to continue with the existing deployment? · no Before proceeding we need to tear down all other deployments. ✔ Tear down other deployments now? · yes Error: missing field advanced_settings at line 1 column 262

    opened by zhangkg 0
  • SET String type

    SET String type

    Is your feature request related to a problem? Please describe. A lot of popular CMS/Ecommerce applications are utilizing SET sql_mode="" in SELECT queries

    Describe the solution you'd like Add SET String type to readyset application

    Describe alternatives you've considered So far, ProxySQL can rewrite each query, but that is not a permanent solution.

    Additional context Could expand application popularity and use cases

    enhancement 
    opened by nemke82 2
  • Clickhouse Support

    Clickhouse Support

    Is your feature request related to a problem? Please describe.

    Support for Clickhouse DB

    Additional context

    With Clickhouse being primarily a large data OLAP DB, integration with it could lead to a wider adoption of ReadySet

    enhancement 
    opened by tomtom215 2
  • Bounded Consistency

    Bounded Consistency

    I couldn't find anything related to this in the docs, so please correct me if I'm wrong.

    Is your feature request related to a problem? Please describe.

    Systems like FlightTracker/TAO at Facebook or Zanzibar at Google use tokens (tickets, zookies respectively) to put a minimum bound on the required consistency for a request. This enables read-after-write consistency guarantees for only the requests that need it, rather than having to wait for eventual consistency to asynchronously propagate.

    Describe the solution you'd like

    Some form of bounded consistency to address this issue or read a view as of a specific "time" (some SQL databases have an "as of system time" predicate).

    Describe alternatives you've considered

    N/A

    Additional context

    N/A

    enhancement 
    opened by jzelinskie 1
  • What to do if the query is out of the cache range?

    What to do if the query is out of the cache range?

    Is your feature request related to a problem? Please describe. What to do if the query is out of the cache range? readiest is a very interesting project, as I understand it, Readyset is caching the data from the backend database(mysql, Postgres), but I have a question:

    1. Is the cache all the data of the database? (I understand that some data should be cached)
    2. if the user's query exceeds the range of cached data, how to deal with it? Forward the user's SQL to the back-end database(mysql, Postgres) for processing?
    enhancement 
    opened by xnge 3
  • Surrealdb support

    Surrealdb support

    Surrealdb is a NewSQL database. I couldn't find any cache layer for it. Surrealdb SQL is very similar to regular SQL. it's still in beta but works fine. it would be great if Readyset supports it. thanks.

    opened by ehsanonline 1
Releases(beta-2022-12-15)
Rust client for Timeplus Proton, a fast and lightweight streaming SQL engine

Rust Client for Timeplus Proton Rust client for Timeplus Proton. Proton is a streaming SQL engine, a fast and lightweight alternative to Apache Flink,

Timeplus 4 Feb 27, 2024
🧰 The Rust SQL Toolkit. An async, pure Rust SQL crate featuring compile-time checked queries without a DSL. Supports PostgreSQL, MySQL, SQLite, and MSSQL.

SQLx ?? The Rust SQL Toolkit Install | Usage | Docs Built with ❤️ by The LaunchBadge team SQLx is an async, pure Rust† SQL crate featuring compile-tim

launchbadge 7.6k Dec 31, 2022
X-Engine: A SQL Engine built from scratch in Rust.

XNGIN (pronounced "X Engine") This is a personal project to build a SQL engine from scratch. The project name is inspired by Nginx, which is a very po

Jiang Zhe 111 Dec 15, 2022
A Rust SQL query builder with a pleasant fluent API closely imitating actual SQL

Scooby An SQL query builder with a pleasant fluent API closely imitating actual SQL. Meant to comfortably build dynamic queries with a little bit of s

Aleksei Voronov 100 Nov 11, 2022
Gh-sql - Query GitHub Projects (beta) with SQL

gh-sql: Query GitHub Projects (beta) with SQL Installation gh extension install KOBA789/gh-sql Features SELECT items DELETE items UPDATE item fields

Hidekazu Kobayashi 108 Dec 7, 2022
SQL validator tool for BigQuery standard SQL.

bqvalid What bqvalid does bqvalid is the SQL validator tool for BigQuery standard SQL. bqvalid fails with error message if there's the expression that

null 10 Dec 25, 2022
Scalable and encrypted embedded database with 3-tier caching

Infinitree is a versioned, embedded database that uses uniform, encrypted blobs to store data.

Symmetree Research Labs 116 Dec 27, 2022
SQL/JSON path engine in Rust.

sql-json-path SQL/JSON Path implementation in Rust. ?? Under development ?? Features Compatible with SQL/JSON Path standard and PostgreSQL implementat

RisingWave Labs 3 Nov 22, 2023
A Toy Query Engine & SQL interface

Naive Query Engine (Toy for Learning) ?? This is a Query Engine which support SQL interface. And it is only a Toy for learn query engine only. You can

谭巍 45 Dec 21, 2022
ReefDB is a minimalistic, in-memory and on-disk database management system written in Rust, implementing basic SQL query capabilities and full-text search.

ReefDB ReefDB is a minimalistic, in-memory and on-disk database management system written in Rust, implementing basic SQL query capabilities and full-

Sacha Arbonel 75 Jun 12, 2023
FeOphant - A SQL database server written in Rust and inspired by PostreSQL.

A PostgreSQL inspired SQL database written in Rust.

Christopher Hotchkiss 27 Dec 7, 2022
GlueSQL is a SQL database library written in Rust

GlueSQL is a SQL database library written in Rust. It provides a parser (sqlparser-rs), execution layer, and optional storage (sled) packaged into a single library.

GlueSQL 2.1k Jan 8, 2023
Distributed SQL database in Rust, written as a learning project

toyDB Distributed SQL database in Rust, written as a learning project. Most components are built from scratch, including: Raft-based distributed conse

Erik Grinaker 4.6k Jan 8, 2023
A high-performance storage engine for modern hardware and platforms.

PhotonDB A high-performance storage engine for modern hardware and platforms. PhotonDB is designed from scratch to leverage the power of modern multi-

PhotonDB 466 Jun 22, 2023
Rust library to parse, deparse and normalize SQL queries using the PostgreSQL query parser

This Rust library uses the actual PostgreSQL server source to parse SQL queries and return the internal PostgreSQL parse tree.

pganalyze 37 Dec 18, 2022
Query LDAP and AD with SQL

SQLDAP Ever wanted to query AD or LDAP with SQL like queries ? I'm going to answer this question myself: yes ! Why ? Because I never could remember al

null 9 Nov 15, 2022
Distributed, version controlled, SQL database with cryptographically verifiable storage, queries and results. Think git for postgres.

SDB - SignatureDB Distributed, version controlled, SQL database with cryptographically verifiable storage, queries and results. Think git for postgres

Fremantle Industries 5 Apr 26, 2022
PRQL is a modern language for transforming data — a simpler and more powerful SQL

PRQL Pipelined Relational Query Language, pronounced "Prequel". PRQL is a modern language for transforming data — a simpler and more powerful SQL. Lik

PRQL 6.5k Jan 5, 2023
SQL database to read and write "discord"

GlueSQL Discord Storage After discussing how CI testing will be managed, we plan to move it upstream. Precautions for use discord ToS https://discord.

Jiseok CHOI 9 Feb 28, 2023