A highly scalable MySQL Proxy framework written in Rust

Overview

mysql-proxy-rs

CircleCI Version Docs

An implementation of a MySQL proxy server built on top of tokio-core.

Overview

This crate provides a MySQL proxy server that you can extend with your own custom logic. Here are some examples of use cases for a proxy:

  • Capture a log of SQL queries issued by an application
  • Profiling of query execution time
  • Monitor query patterns e.g. threat detection
  • Record SQL traffic for later playback for automated testing

Usage

The proxy defines the following trait for defining a packet handler for handling request and response packets:

pub trait PacketHandler {
    fn handle_request(&self, p: &Packet) -> Action;
    fn handle_response(&self, p: &Packet) -> Action;
}

/// Handlers return a variant of this enum to indicate how the proxy should handle the packet.
pub enum Action {
    /// drop the packet
    Drop,
    /// forward the packet unmodified
    Forward,
    /// forward a mutated packet
    Mutate(Packet),
    /// respond to the packet without forwarding
    Respond(Vec<Packet>),
    /// respond with an error packet
    Error { code: u16, state: [u8; 5], msg: String },
}

Example

The example proxy passes all queries to MySQL except for queries containing the word 'avocado'. Use the following command to run the example.

Note that we have tested the proxy with rustc 1.13.0-nightly (cbe4de78e 2016-09-05).

$ cargo run --example proxy

Then in a separate window you can test out the proxy. The proxy binds to port 3307 and assumes that a MySQL server is running on port 3306.

$ mysql -u root -p -h 127.0.0.1 -P 3307
mysql> select 'banana';
+--------+
| banana |
+--------+
| banana |
+--------+
1 row in set (0.00 sec)

mysql> select 'avocado';
ERROR 1064 (12345): Proxy rejecting any avocado-related queries

License

mysql-proxy-rs is distributed under the terms of the Apache License (Version 2.0).

See LICENSE-APACHE for details.

Comments
  • TPC-C Load fails via proxy

    TPC-C Load fails via proxy

    The load works with mysql-router but will not work with mysql-proxy-rs and here is the error from our TPCC:

    java.lang.RuntimeException: Error loading into table 'item' with SQL: INSERT IGNORE INTO `item` (i_id,i_im_id,i_name,i_price,i_data) VALUES (1,2816,'A6orZi0q5MGNmqNcMO',76.14,'k5LFMuM05JbY3ztNZa4Y5r8L0rU0VrpwVQpMPlP0g76MEVBspS'),(2,896,'Q78ZjbjJa3DJuUgWhZw',36.62,'aXQNsiRyGQZHJlNjgpLlWm2CpoSJR'),(3,9431,'7JZrAOnIHSVoYPwp1sLi',77.29,'Qo3aiqTrLLYutXg7NXb8YBo9cKSPyHHoOrJjWIcpsCjCRAIzmP')….( a lot more inserts here)
        at com.codefutures.tpcc.load.JdbcStatementLoader.executeBulkInsert(JdbcStatementLoader.java:72)
        at com.codefutures.tpcc.load.JdbcStatementLoader.load(JdbcStatementLoader.java:59)
        at com.codefutures.tpcc.Load.loadItems(Load.java:83)
        at com.codefutures.tpcc.TpccLoad.runLoad(TpccLoad.java:263)
        at com.codefutures.tpcc.TpccLoad.main(TpccLoad.java:334)
    Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
    
    The last packet successfully received from the server was 4 milliseconds ago.  The last packet sent successfully to the server was 2 milliseconds ago.
    

    all the proxy says is this:

    Failed to spawn future: Error { repr: Custom(Custom { kind: Other, error: StringError("connection closed") }) }
    
    opened by andygrove 3
  • Panic when using mutate

    Panic when using mutate

    I'm completely new to rust so this could well be user error.

    I've been playing around with the proxy example to mutate CREATE TABLE queries so that they will always have ROW_FORMAT=DYNAMIC appended if a row format is not already defined. The modified proxy example has been changed to

    if sql.to_lowercase().starts_with("create table ") && ! sql.to_lowercase().contains(" row_format") {
        let new_sql = sql + " ROW_FORMAT=DYNAMIC";
    
        let mut payload: Vec<u8> = vec![];
        payload.extend_from_slice(&p.bytes[0..5]);
        payload.extend_from_slice(new_sql.as_bytes());
    
        let new_packet = Packet { bytes: payload };
    
        print_packet_bytes(&new_packet.bytes);
    
        Action::Mutate(new_packet)
    }
    

    When running a create statement such as CREATE TABLE test1 (id int); the printed bytes are correct (i.e. containing the extra appended ROW_FORMAT=DYNAMIC) but the application crashes with

    thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { repr: Os { code: 107, message: "Transport endpoint is not connected" } }', ../src/libcore/result.rs:799
    note: Run with `RUST_BACKTRACE=1` for a backtrace.
    error: Process didn't exit successfully: `target/debug/examples/proxy` (exit code: 101)
    

    I can provide a back trace if required. Additionally the table does get created but without the ROW_FORMAT being set.

    Is this something in the application or am I doing something wrong?

    opened by alexwhitman 2
  • Non UTF-8 ?

    Non UTF-8 ?

    Hi so I am using your code to provide a mysql facade for postgres for a particular commercial product. Code works great thanks.

    However, sometimes where I get the byte stream form said product I get

    error: Utf8Error let sql = String::from_utf8(slice.to_vec()).expect("Invalid UTF-8");

    How can I deal with this ? I'm not sure how non utf-8 can be sent.

    Thanks

    opened by glennpierce 0
  • expected a key but found an empty string

    expected a key but found an empty string

    Caused by: failed to parse manifest at/home/$USER/.cargo/registry/src/github.com-88ac128001ac3a9a/openssl-sys-0.7.17/Cargo.toml`

    Caused by: could not parse input as TOML /home/$USER/.cargo/registry/src/github.com-88ac128001ac3a9a/openssl-sys-0.7.17/Cargo.toml:45:9 expected a key but found an empty string /home/$USER/.cargo/registry/src/github.com-88ac128001ac3a9a/openssl-sys-0.7.17/Cargo.toml:45:9-45:10 expected ., but found '

    `

    opened by rosehgal 0
  • Cannot send error packet to terminate result set

    Cannot send error packet to terminate result set

    If we hit an error during processing packets for a result set and then send an error packet to the client and drop the remainder of the result packets, the client terminates with a "Lost connection to MySQL" error.

    The protocol documentation says that we should be able to do this. This needs further investigation, probably involving reading through the MySQL client source code.

    opened by andygrove 0
  • Investigate use of tokio-proto and/or tokio-middleware

    Investigate use of tokio-proto and/or tokio-middleware

    We’re currently working with the base tokio-core crate, but would most likely benefit from moving up the stack to tokio-proto and potentially tokio-middleware.

    opened by andygrove 0
  • Packet handler should be able to override which MySQL to connect to

    Packet handler should be able to override which MySQL to connect to

    Currently, the proxy maintains a one to one mapping of a client connection to a server connection. For more flexibility, the API should be extended so that the user-provided proxy logic can decide which MySQL instance to connect to. This would facilitate basic query routing and sharding.

    opened by andygrove 0
Owner
AgilData
AgilData
🧰 The Rust SQL Toolkit. An async, pure Rust SQL crate featuring compile-time checked queries without a DSL. Supports PostgreSQL, MySQL, SQLite, and MSSQL.

SQLx ?? The Rust SQL Toolkit Install | Usage | Docs Built with ❤️ by The LaunchBadge team SQLx is an async, pure Rust† SQL crate featuring compile-tim

launchbadge 7.6k Dec 31, 2022
Asyncronous Rust Mysql driver based on Tokio.

mysql-async Tokio based asynchronous MySql client library for rust programming language. Installation Library hosted on crates.io. [dependencies] mysq

Anatoly I 292 Dec 30, 2022
Mysql client library implemented in rust.

mysql This crate offers: MySql database driver in pure rust; connection pool. Features: macOS, Windows and Linux support; TLS support via nativetls cr

Anatoly I 548 Dec 31, 2022
cogo rust coroutine database driver (Mysql,Postgres,Sqlite)

cdbc Coroutine Database driver Connectivity.based on cogo High concurrency,based on coroutine No Future<'q,Output=*>,No async fn, No .await , no Poll*

co-rs 10 Nov 13, 2022
rust-mysql-simple support library for the r2d2 connection pool

r2d2-mysql rust-mysql-simple support library for the r2d2 connection pool.

outersky 44 Nov 1, 2022
This project provides a Rust-based solution for migrating MSSQL databases to MySQL.

MSSQL to MySQL Database Migration A Rust project to migrate MSSQL databases to MySQL, including table structures, column data types, and table data ro

Bitalizer 2 Jul 10, 2023
A tool for automated migrations for PostgreSQL, SQLite and MySQL.

Models Models is an implementation for a SQL migration management tool. It supports PostgreSQL, MySQL, and SQLite. Quick Start install the CLI by runn

null 45 Nov 16, 2022
CRUD example with Rocket, Toql and MySQL

Todo (Rocket, Toql, MySQL) This is a little REST server built with Rocket, Toql and MySQL. The server allows to call CRUD functions on a Todo item. It

Artos 7 Jan 5, 2022
Gibbs MySQL Spyglass

Gibbs MySQL Spyglass Ahoy Matey! The Gibbs MySQL Spyglass is a application used to capture application traffic into a MySQL database. It is designed t

AgilData 82 Nov 14, 2021
fast & easy CLI and vscode extension specialized to format MySQL INSERT queries.

insertfmt fast & easy CLI specialized to format MySQL INSERT queries. format queries so that they look like a table. NOTE: If you wanna use the VSCode

canalun 7 May 2, 2023
tectonicdb is a fast, highly compressed standalone database and streaming protocol for order book ticks.

tectonicdb crate docs.rs crate.io tectonicdb tdb-core tdb-server-core tdb-cli tectonicdb is a fast, highly compressed standalone database and streamin

Ricky Han 525 Dec 23, 2022
RedisLess is a fast, lightweight, embedded and scalable in-memory Key/Value store library compatible with the Redis API.

RedisLess is a fast, lightweight, embedded and scalable in-memory Key/Value store library compatible with the Redis API.

Qovery 145 Nov 23, 2022
Scalable and encrypted embedded database with 3-tier caching

Infinitree is a versioned, embedded database that uses uniform, encrypted blobs to store data.

Symmetree Research Labs 116 Dec 27, 2022
A scalable, distributed, collaborative, document-graph database, for the realtime web

is the ultimate cloud database for tomorrow's applications Develop easier. Build faster. Scale quicker. What is SurrealDB? SurrealDB is an end-to-end

SurrealDB 16.9k Jan 8, 2023
Scalable and fast data store optimised for time series data such as financial data, events, metrics for real time analysis

OnTimeDB Scalable and fast data store optimised for time series data such as financial data, events, metrics for real time analysis OnTimeDB is a time

Stuart 2 Apr 5, 2022
The most efficient, scalable, and fast production-ready serverless REST API backend which provides CRUD operations for a MongoDB collection

Optimal CRUD Mongo Goals of This Project This is meant to be the most efficient, scalable, and fast production-ready serverless REST API backend which

Evaluates2 1 Feb 22, 2022
Redis compatible server framework for Rust

Redis compatible server framework for Rust Features Create a fast custom Redis compatible server in Rust Simple API. Support for pipelining and telnet

Josh Baker 61 Nov 12, 2022
A raft framework, for regular people

RmqttRaft - A raft framework, for regular people This is an attempt to create a layer on top of tikv/raft-rs, that is easier to use and implement. Thi

null 16 Dec 21, 2022
A yaml-based SQL planner test framework

SQLPlannerTest A yaml-based SQL planner test framework. SQLPlannerTest is a regression test framework. It will read a special yaml file for describing

The RisingLight Project 23 Sep 21, 2022