Noria: data-flow for high-performance web applications

Overview

Noria: data-flow for high-performance web applications

noria on crates.io noria on docs.rs noria-server on crates.io noria-server on docs.rs Azure Status

Noria is a new streaming data-flow system designed to act as a fast storage backend for read-heavy web applications based on Jon Gjengset's Phd Thesis, as well as this paper from OSDI'18. It acts like a database, but precomputes and caches relational query results so that reads are blazingly fast. Noria automatically keeps cached results up-to-date as the underlying data, stored in persistent base tables, change. Noria uses partially-stateful data-flow to reduce memory overhead, and supports dynamic, runtime data-flow and query change.

Noria comes with a MySQL adapter that implements the binary MySQL protocol. This lets any application that currently talks to MySQL or MariaDB switch to Noria with minimal effort. For example, running a Lobsters-like workload that issues the equivalent SQL queries to the real Lobsters website, Noria improves throughput supported by 5x:

Noria speeds up Lobsters queries by 5x

At a high level, Noria takes a set of parameterized SQL queries (think prepared statements), and produces a data-flow program that maintains materialized views for the output of those queries. Reads now become fast lookups directly into these materialized views, as if the value had been directly cached in memcached. The views are then kept up-to-date incrementally through the data-flow, which yields high write throughput.

Running Noria

Like most databases, Noria follows a server-client model where many clients connect to a (potentially distributed) server. The server in this case is the noria-server binary, and must be started before clients can connect. Noria also uses Apache ZooKeeper to announce the location of its servers, so ZooKeeper must be running.

You (currently) need nightly Rust to build noria-server. This will be arranged for automatically if you're using rustup.rs. To build noria-server, run

$ cargo build --release --bin noria-server

You may need to install some dependencies for the above to work:

  • clang
  • libclang-dev
  • libssl-dev
  • liblz4-dev
  • build-essential

To start a long-running noria-server instance, ensure that ZooKeeper is running, and then run:

$ cargo r --release --bin noria-server -- --deployment myapp --no-reuse --address 172.16.0.19 --shards 0

myapp here is a deployment. Many noria-server instances can operate in a single deployment at the same time, and will share the workload between them. Workers in the same deployment automatically elect a leader and discovery each other via ZooKeeper.

Interacting with Noria

There are two primary ways to interact with Noria: through the Rust bindings or through the MySQL adapter. They both automatically locate the running worker through ZooKeeper (use -z if ZooKeeper is not running on localhost:2181).

Rust bindings

The noria crate provides native Rust bindings to interact with noria-server. See the noria documentation for detailed instructions on how to use the library. You can also take a look at the example Noria program using Noria's client API. You can also see a self-contained version that embeds noria-server (and doesn't require ZooKeeper) in this example.

MySQL adapter

We have built a MySQL adapter for Noria that accepts standard MySQL queries and speaks the MySQL protocol to make it easy to try Noria out for existing applications. Once the adapter is running (see its README), you should be able to point your application at localhost:3306 to send queries to Noria. If your application crashes, this is a bug, and we would appreciate it if you open an issue. You may also want to try to disable automatic re-use (with --no-reuse) or sharding (with --shards 0) in case those are misbehaving.

CLI and Web UI

You can manually inspect the data stored in Noria using any MySQL client (e.g., the mysql CLI), or use Noria's own web interface.

Noria development

Noria is a large piece of software that spans many sub-crates and external tools (see links in the text above). Each sub-crate is responsible for a component of Noria's architecture, such as external API (noria), mapping SQL to data-flow (server/mir), and executing data-flow operators (server/dataflow). The code in server/src/ is the glue that ties these pieces together by establishing materializations, scheduling data-flow work, orchestrating Noria program changes, handling failovers, etc.

server/src/lib.rs has a pretty extensive comment at the top of it that goes through how the Noria internals fit together at an implementation level. While it occasionally lags behind, especially following larger changes, it should serve to get you familiarized with the basic building blocks relatively quickly.

The sub-crates each serve a distinct role:

  • noria/: everything that an external program communicating with Noria needs. This includes types used in RPCs as arguments/return types, as well as code for discovering Noria workers through ZooKeeper, establishing a connection to Noria through ZooKeeper, and invoking the various RPC exposed by the Noria controller (server/src/controller.rs). The noria sub-crate also contains a number of internal data-structures that must be shared between the client and the server like DataType (Noria's "value" type). These are annotated with #[doc(hidden)], and should be easy to spot in noria/src/lib.rs.

  • applications/: a collection of various Noria benchmarks. The most frequently used one is vote, which runs the vote benchmark from §8.2 of the OSDI paper. You can run it in a bunch of different ways (--help should be useful), and with many different backends. The localsoup backend is the one that's easiest to get up and running with.

  • server/src/: the Noria server, including high-level components such as RPC handling, domain scheduling, connection management, and all the controller operations (listening for heartbeats, handling failed workers, etc.). It contains two notable sub-crates:

    • dataflow/: the code that implements the internals of the data-flow graph. This includes implementations of the different operators (ops/), "special" operators like leaf views and sharders (node/special/), implementations of view storage (state/), and the code that coordinates execution of control, data, and backfill messages within a thread domain (domain/).
    • mir/: the code that implements Noria's SQL-to-dataflow mapping. This includes resolving columns and keys, creating dataflow operators, and detecting reuse opportunities, and triggering migrations to make changes after new SQL queries have been added. @ms705 is the primary author of this particular subcrate, and it builds largely upon nom-sql.
    • common/: data-structures that are shared between the various server sub-crates.

To run the test suite, use:

$ cargo test

Build and open the documentation with:

$ cargo doc --open

Once noria-server is running, its API is available on port 6033 at the specified listen address.

Alternatively, you can discover Noria's REST API listen address and port through ZooKeeper via this command:

$ cargo run --bin noria-zk -- \
    --show --deployment myapp
    | grep external | cut -d' ' -f4

A basic graphical UI runs at http://IP:PORT/graph.html and shows the running data-flow graph. You can also deploy Noria's more advanced web UI that serves the REST API endpoints in a human-digestible form and includes the graph visualization.

License

Licensed under either of

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Comments
  • Recovery

    Recovery

    This adds a Blender.recover() method that goes through and replays the log entries it finds for its base nodes.

    • The log flushes are now separated by a newline, so that each JSON array can be parsed individually.
    • persistence::Parameters now has an optional log_prefix that'll be used to replace soup in e.g. soup-log_0_0-0.json. Without this parallel tests would sometimes log to the same file - at the same time.
    • Packets are re-created from the log entries and passed on to Domain.handle. Transactional packets are first passed through checktable.apply_batch to retrieve a timestamp. I'm not sure if the latter is completely correct though - cc @fintelia.

    I've added a couple of tests, but let me know if there's anything else I should cover.

    I also changed the basic example so that it calls .recover() and votes with a unique timestamp (so that the count increments each time it's run). Let me know if I should move it to another example or something though.

    opened by ekmartin 17
  • AUTO_INCREMENT support

    AUTO_INCREMENT support

    This implements support for auto incrementing IDs in base nodes.

    • 0 values are replaced with the next auto increment value.
    • Incoming non-zero values for auto increment columns are used directly, and the next auto increment value for that column will continue from there (MySQL behavior).

    To make this work with shards, all auto increment columns get their values held in a DataType::ID(shard, value) type. value here is always an i64. An alternative would be to encode the shard info into the numeric value itself, i.e. through use of the first bits.

    The shard index is passed through the base with on_input. Some things to consider:

    • Can a base node "move" to a different shard while Soup is running? This could probably lead to collisions if that's the case. For recovery purposes I think it should still work, as long as the sharding is completely deterministic.

    • How do you shard incoming rows if the key is auto incremented, and always 0? Right now this would just cause all rows to land in the first shard (see this test as an example).

    • How should the user signify that they want an auto increment value for a column? Currently it's done by passing in a 0 value, but it might be better to let the value be omitted altogether (this might conflict with the default value stuff though).

    opened by ekmartin 12
  • `cargo doc` fails

    `cargo doc` fails

    Trying to build the docs from the root of this repository fails on macOS 10.15.2 with rustc 1.42.0-nightly (698fcd38f 2020-01-26)

    I receive the following error:

    error[E0277]: the trait bound `(): std::future::Future` is not satisfied
      --> noria/src/controller.rs:65:5
       |
    65 |     type Future = impl Future<Output = Result<Self::Response, Self::Error>> + Send;
       |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `std::future::Future` is not implemented for `()`
       |
       = note: the return type of a function must have a statically known size
    
    error[E0277]: the trait bound `(): std::future::Future` is not satisfied
      --> noria/src/view.rs:38:5
       |
    38 | /     type Future = impl Future<
    39 | |         Output = Result<multiplex::MultiplexTransport<Transport, Tagger>, tokio::io::Error>,
    40 | |     >;
       | |______^ the trait `std::future::Future` is not implemented for `()`
       |
       = note: the return type of a function must have a statically known size
    
    error[E0277]: the trait bound `(): std::future::Future` is not satisfied
      --> noria/src/table.rs:42:5
       |
    42 | /     type Future = impl Future<
    43 | |         Output = Result<multiplex::MultiplexTransport<Transport, Tagger>, tokio::io::Error>,
    44 | |     >;
       | |______^ the trait `std::future::Future` is not implemented for `()`
       |
       = note: the return type of a function must have a statically known size
    
    error[E0277]: the trait bound `(): std::future::Future` is not satisfied
       --> noria/src/controller.rs:179:1
        |
    179 | type RpcFuture<A, R> = impl Future<Output = Result<R, failure::Error>>;
        | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `std::future::Future` is not implemented for `()`
        |
        = note: the return type of a function must have a statically known size
    
    error[E0277]: the trait bound `(): std::future::Future` is not satisfied
       --> noria/src/table.rs:235:5
        |
    235 |     type Future = impl Future<Output = Result<Tagged<()>, TableError>> + Send;
        |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `std::future::Future` is not implemented for `()`
        |
        = note: the return type of a function must have a statically known size
    
    error[E0277]: the trait bound `(): std::future::Future` is not satisfied
       --> noria/src/view.rs:201:5
        |
    201 |     type Future = impl Future<Output = Result<Vec<Datas>, ViewError>> + Send;
        |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `std::future::Future` is not implemented for `()`
        |
        = note: the return type of a function must have a statically known size
    
    error: aborting due to 6 previous errors
    
    For more information about this error, try `rustc --explain E0277`.
    error: Could not document `noria`.
    
    
    opened by goodSyntax808 11
  • Trouble Embedding noria_server

    Trouble Embedding noria_server

    I am currently trying to embed noria_server into a testing project, and I can't seem to figure out what needs to go into Cargo.toml to pull in the noria_server code. Attempting to only include the noria crate in Cargo.toml throws an error stating that the noria_server crate cannot be found, while trying to pull it from GitHub directly throws a 404 error, presumably because noria_server lives in a subfolder of the noria repo. Advise on this would be much appreciated.

    opened by zack-emmert 10
  • Run rustfmt on Travis

    Run rustfmt on Travis

    This runs rustfmt with --write-mode=diff on Travis, so that the build fails if there are formatting errors. It installs rustfmt from git, with the SHA 6e41100725267974fa6dcc61134d4377b676ad01.

    Also shortened a few lines that were above 100 characters, to make the build pass.

    opened by ekmartin 10
  • Arithmetic Expressions in Projections

    Arithmetic Expressions in Projections

    This adds support for e.g. SELECT 2 * price FROM ... and SELECT price * MAX(column) FROM ... by handling nom-sql's ArithmeticExpression.

    Arithmetic between different numeric data types is converted implicitly:

    • BigInt + Int = BigInt
    • Real + Int = Real
    • Not supported: BigInt + Real (since Real's integer part is represented using an i32)

    Similar to how nom-sql handles arithmetic this currently only deals with simple, non-nested, expressions, but it should definitely be possible to add support for nested expressions in the future.

    I'm not completely sure if the way I've handled ArithmeticColumns in the query graph is correct (especially when it comes to re-use), but it seems to work for the tests I've written so far. The output formatting of arithmetic nodes could probably also need some tuning.

    opened by ekmartin 10
  • Windows build failed

    Windows build failed

    error[E0412]: cannot find type `clockid_t` in module `libc`
      --> C:\Users\xxxku\.cargo\registry\src\github.com-1ecc6299db9ec823\timekeeper-0.3.0\src\source.rs:26:31
       |
    26 | fn clock_gettime(clock: libc::clockid_t) -> Result<libc::timespec, ()> {
       |                               ^^^^^^^^^ help: a type alias with a similar name exists: `clock_t`
    
    error[E0425]: cannot find function `clock_gettime` in module `libc`
      --> C:\Users\xxxku\.cargo\registry\src\github.com-1ecc6299db9ec823\timekeeper-0.3.0\src\source.rs:33:15
       |
    33 |         libc::clock_gettime(clock, &mut tp)
       |               ^^^^^^^^^^^^^ not found in `libc`
    help: possible candidate is found in another module, you can import it into scope
       |
    2  | use source::clock_gettime;
       |
    
    error[E0425]: cannot find value `CLOCK_PROCESS_CPUTIME_ID` in module `libc`
      --> C:\Users\xxxku\.cargo\registry\src\github.com-1ecc6299db9ec823\timekeeper-0.3.0\src\source.rs:47:40
       |
    47 |         let time = clock_gettime(libc::CLOCK_PROCESS_CPUTIME_ID).unwrap();
       |                                        ^^^^^^^^^^^^^^^^^^^^^^^^ not found in `libc`
    
    error[E0425]: cannot find value `CLOCK_THREAD_CPUTIME_ID` in module `libc`
      --> C:\Users\xxxku\.cargo\registry\src\github.com-1ecc6299db9ec823\timekeeper-0.3.0\src\source.rs:56:40
       |
    56 |         let time = clock_gettime(libc::CLOCK_THREAD_CPUTIME_ID).unwrap();
       |                                        ^^^^^^^^^^^^^^^^^^^^^^^ not found in `libc`
    
    error: aborting due to 4 previous errors
    
    Some errors occurred: E0412, E0425.
    For more information about an error, try `rustc --explain E0412`.
    error: Could not compile `timekeeper`.
    warning: build failed, waiting for other jobs to finish...
    error: build failed
    
    opened by xNxExOx 9
  • Decide on terminology

    Decide on terminology

    As we move to distributed fault tolerant Soup, the terms we're using have become increasingly ambiguous and overloaded. I don't love all these names, but we should have terminology for all of the following concepts:

    • domain: the collection of all shards and replicas that do computation for the same nodes in the graph.
    • replica set: the part of a domain that manages a single shard.
    • replica: single running instantiation of the nodes in a domain.

    And for these higher level components...

    • controller: component that coordinates control plane operations. Runs in the same address space as one worker, but typically interacts with more over the network.
    • controller stub: component that participates in leader election. Becomes a controller if it wins.
    • worker: component manages data plane operations
    • worker thread pool: the collection of threads on a worker that do actual processing
    • worker thread: single thread in the worker thread pool.

    We should also probably have distinct DomainIndex, ReplicaSetIndex, and ReplicaIndex types as well as WorkerIndexs to identify individual processes. Depending on how they're implemented, worker indexes may have to be randomly generated to avoid collisions.

    opened by fintelia 9
  • Produce even better DOT graphs

    Produce even better DOT graphs

    Output DOT graphs without using petgraph's stock formatter so that we can output each node's configuration using a dense, human-readable, relational-algebra-like format.

    Here's what the main/web graph looks like in this new world:

    dot-example

    opened by benesch 8
  • insert_all method not found in `noria::table::Table`

    insert_all method not found in `noria::table::Table`

    I am probably doing something dumb, but I am trying to use the insert_all function, but it doesn't seem to exist.

    // schema not needed for mre //
    let mut votes = db.table("votes").await.unwrap();
    let noria_recs: Vec<Vec<DataType>> = ...
    votes.insert_all(noria_recs).await.unwrap();
    

    gives: no method named insert_all found for struct noria::table::Table in the current scope: method not found in noria::table::Table

    The definition for insert_all is

    pub fn insert_all<I, V>(&mut self, i: I) -> Result<(), TableError> 
    where    
    I: IntoIterator<Item = V>,   
    V: Into<Vec<DataType>>, 
    

    which I believe a Vec impls IntoIterator.

    I think a code snippet in the docs would be helpful

    -- versions -- My Cargo.toml has noria = "0.5.0" rustc 1.45.0-nightly (47c3158c3 2020-06-04)

    opened by anandijain 7
  • SQL support

    SQL support

    Sorry if I missed an obvious doc link somewhere, but I wasn't able to find comprehensive documentation on the extend of SQL support in either noria and or noria-mysql.

    I'm looking to replace some part of a 1bn+ row (postgres-based) query system with Noria, since a lot of it is written in Rust and queries are already using mat views in most cases, so here goes my feature list:

    • recursive and non-recursive CTE, where side-effect-free non-recursive CTE are not meant to be optimization fences (Postgres 12)
    • subqueries
    • UPSERT-type queries with a lot of rows (1-15 million)
    • all kinds of JOINs, but SQL-default JOINs would be fine for now
    opened by turbo 7
  • Incoming nightly breakage

    Incoming nightly breakage

    While your crate does nothing wrong per se, the incoming breakage may be somewhat annoying, and you can get ahead of the breakage.

    The PR that will break this crate: https://github.com/rust-lang/rust/pull/94081

    The breakage: https://crater-reports.s3.amazonaws.com/pr-94081/try%23e251c42c969d6f8dbaae00e1028489c3a29f8789/reg/noria-common-0.6.0/log.txt

    The fix is to move https://github.com/mit-pdos/noria/blob/465184ee4b57a2791d41b0f12fb8f71c45bae6b6/noria/src/controller.rs#L358 and https://github.com/mit-pdos/noria/blob/465184ee4b57a2791d41b0f12fb8f71c45bae6b6/noria/src/controller.rs#L179 into their own module, so that the type alias doesn't try to get inferred at the same time as the impl Trait usage in the return type.

    On a positive note: https://github.com/rust-lang/rust/issues/65442 is now very much on my radar, with the PR that breaks your code this becomes actually fixable without a complex hack!

    opened by oli-obk 5
  • Project fix

    Project fix

    This fixes what I think is a bug in the implementation of query_through in the project operator. The implementation assumed that columns in emit were monotonically increasing, but I did not read about this as an explicit invariant. As a result the implementation would not permute columns if that was requested in emit. The following pseudo-rust illustrates the inconsistency.

    let p = Project::new(emit=[0,2,1]); // <- permutation requested
    
    let v = [1,2,3];
    p.on_input([v]) // returns [1,3,2] <- correctly permuted
    ancestor_state.insert(v);
    p.query_through(..., states={key=1, states={..., ancestor_state }); // returns [1,2,3] <- not permuted
    

    The new version should be as efficient as before but correctly permutes in query_through though I have not benchmarked it.

    I added safety assertions for debug mode and added a test case.

    The new implementation is less pretty and maybe less simple? So I added some comments explaining what it does.

    Lmk if there are any questions or issues with the PR.

    opened by JustusAdam 4
  • dose is support minus?

    dose is support minus?

    Although mysql does not support minus, however some subqueries may turn to minus, can these subqueries also be used to build incremental materialized views?

    opened by nnsgmsone 1
  • Build Errors when interfacing with the evmap crate

    Build Errors when interfacing with the evmap crate

    Hi,

    I'm trying to build Noria on the following system:

    OS and distribution: Lubuntu 20.10, Linux kernel 5.8.0-25-generic Rust: rustc 1.53.0-nightly (c051c5ddd 2021-04-06) Noria commit: cd639f57e8fa3fa1cdc1ca3a4c7ef3cedbe45c8c. This is the master's HEAD at the time of writing this issue.

    I get the following compile errors. It seems all have to do with interfacing with the evmap crate

       Compiling noria-dataflow v0.7.0 (/home/ppf/Documents/noria/noria/server/dataflow)
    error[E0603]: struct `ReadHandle` is private
       --> server/dataflow/src/backlog/multir.rs:7:19
        |
    7   |     Single(evmap::ReadHandle<DataType, Vec<DataType>, i64, RandomState>),
        |                   ^^^^^^^^^^ private struct
        |
    note: the struct `ReadHandle` is defined here
    
    error[E0603]: struct `ReadHandle` is private
       --> server/dataflow/src/backlog/multir.rs:8:19
        |
    8   |     Double(evmap::ReadHandle<(DataType, DataType), Vec<DataType>, i64, RandomState>),
        |                   ^^^^^^^^^^ private struct
        |
    note: the struct `ReadHandle` is defined here
    
    error[E0603]: struct `ReadHandle` is private
       --> server/dataflow/src/backlog/multir.rs:9:17
        |
    9   |     Many(evmap::ReadHandle<Vec<DataType>, Vec<DataType>, i64, RandomState>),
        |                 ^^^^^^^^^^ private struct
        |
    note: the struct `ReadHandle` is defined here
    
    error[E0603]: struct `WriteHandle` is private
       --> server/dataflow/src/backlog/multiw.rs:7:19
        |
    7   |     Single(evmap::WriteHandle<DataType, Vec<DataType>, i64, RandomState>),
        |                   ^^^^^^^^^^^ private struct
        |
    note: the struct `WriteHandle` is defined here
    
    error[E0603]: struct `WriteHandle` is private
       --> server/dataflow/src/backlog/multiw.rs:8:19
        |
    8   |     Double(evmap::WriteHandle<(DataType, DataType), Vec<DataType>, i64, RandomState>),
        |                   ^^^^^^^^^^^ private struct
        |
    note: the struct `WriteHandle` is defined here
    
    error[E0603]: struct `WriteHandle` is private
       --> server/dataflow/src/backlog/multiw.rs:9:17
        |
    9   |     Many(evmap::WriteHandle<Vec<DataType>, Vec<DataType>, i64, RandomState>),
        |                 ^^^^^^^^^^^ private struct
        |
    note: the struct `WriteHandle` is defined here
    
    warning: unnecessary trailing semicolon
       --> server/dataflow/src/ops/topk.rs:224:10
        |
    224 |         };
        |          ^ help: remove this semicolon
        |
        = note: `#[warn(redundant_semicolons)]` on by default
    
    error[E0308]: mismatched types
      --> server/dataflow/src/backlog/mod.rs:55:39
       |
    55 |             (multir::Handle::$variant(r), multiw::Handle::$variant(w))
       |                                       ^ expected struct `ReadHandle`, found struct `evmap::handles::WriteHandle`
    ...
    61 |         1 => make!(Single),
       |              ------------- in this macro invocation
       |
       = note: expected struct `ReadHandle<noria_common::DataType, Vec<noria_common::DataType>, i64, ahash::RandomState>`
                  found struct `evmap::handles::WriteHandle<_, _, {integer}, ahash::RandomState>`
       = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)
    
    error[E0308]: mismatched types
      --> server/dataflow/src/backlog/mod.rs:55:68
       |
    55 |             (multir::Handle::$variant(r), multiw::Handle::$variant(w))
       |                                                                    ^ expected struct `evmap::handles::WriteHandle`, found struct `ReadHandle`
    ...
    61 |         1 => make!(Single),
       |              ------------- in this macro invocation
       |
       = note: expected struct `evmap::handles::WriteHandle<noria_common::DataType, Vec<noria_common::DataType>, i64, ahash::RandomState>`
                  found struct `ReadHandle<_, _, {integer}, ahash::RandomState>`
       = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)
    
    error[E0308]: mismatched types
      --> server/dataflow/src/backlog/mod.rs:55:39
       |
    55 |             (multir::Handle::$variant(r), multiw::Handle::$variant(w))
       |                                       ^ expected struct `ReadHandle`, found struct `evmap::handles::WriteHandle`
    ...
    62 |         2 => make!(Double),
       |              ------------- in this macro invocation
       |
       = note: expected struct `ReadHandle<(noria_common::DataType, noria_common::DataType), Vec<noria_common::DataType>, i64, ahash::RandomState>`
                  found struct `evmap::handles::WriteHandle<_, _, {integer}, ahash::RandomState>`
       = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)
    
    error[E0308]: mismatched types
      --> server/dataflow/src/backlog/mod.rs:55:68
       |
    55 |             (multir::Handle::$variant(r), multiw::Handle::$variant(w))
       |                                                                    ^ expected struct `evmap::handles::WriteHandle`, found struct `ReadHandle`
    ...
    62 |         2 => make!(Double),
       |              ------------- in this macro invocation
       |
       = note: expected struct `evmap::handles::WriteHandle<(noria_common::DataType, noria_common::DataType), Vec<noria_common::DataType>, i64, ahash::RandomState>`
                  found struct `ReadHandle<_, _, {integer}, ahash::RandomState>`
       = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)
    
    error[E0308]: mismatched types
      --> server/dataflow/src/backlog/mod.rs:55:39
       |
    55 |             (multir::Handle::$variant(r), multiw::Handle::$variant(w))
       |                                       ^ expected struct `ReadHandle`, found struct `evmap::handles::WriteHandle`
    ...
    63 |         _ => make!(Many),
       |              ----------- in this macro invocation
       |
       = note: expected struct `ReadHandle<Vec<noria_common::DataType>, Vec<noria_common::DataType>, i64, ahash::RandomState>`
                  found struct `evmap::handles::WriteHandle<_, _, {integer}, ahash::RandomState>`
       = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)
    
    error[E0308]: mismatched types
      --> server/dataflow/src/backlog/mod.rs:55:68
       |
    55 |             (multir::Handle::$variant(r), multiw::Handle::$variant(w))
       |                                                                    ^ expected struct `evmap::handles::WriteHandle`, found struct `ReadHandle`
    ...
    63 |         _ => make!(Many),
       |              ----------- in this macro invocation
       |
       = note: expected struct `evmap::handles::WriteHandle<Vec<noria_common::DataType>, Vec<noria_common::DataType>, i64, ahash::RandomState>`
                  found struct `ReadHandle<_, _, {integer}, ahash::RandomState>`
       = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)
    
    error[E0599]: no method named `read` found for reference `&ReadHandle<noria_common::DataType, Vec<noria_common::DataType>, i64, ahash::RandomState>` in the current scope
      --> server/dataflow/src/backlog/multir.rs:28:29
       |
    28 |                 let map = h.read()?;
       |                             ^^^^ method not found in `&ReadHandle<noria_common::DataType, Vec<noria_common::DataType>, i64, ahash::RandomState>`
    
    error[E0599]: no method named `read` found for reference `&ReadHandle<(noria_common::DataType, noria_common::DataType), Vec<noria_common::DataType>, i64, ahash::RandomState>` in the current scope
      --> server/dataflow/src/backlog/multir.rs:57:33
       |
    57 |                     let map = h.read()?;
       |                                 ^^^^ method not found in `&ReadHandle<(noria_common::DataType, noria_common::DataType), Vec<noria_common::DataType>, i64, ahash::RandomState>`
    
    error[E0599]: no method named `read` found for reference `&ReadHandle<Vec<noria_common::DataType>, Vec<noria_common::DataType>, i64, ahash::RandomState>` in the current scope
      --> server/dataflow/src/backlog/multir.rs:64:29
       |
    64 |                 let map = h.read()?;
       |                             ^^^^ method not found in `&ReadHandle<Vec<noria_common::DataType>, Vec<noria_common::DataType>, i64, ahash::RandomState>`
    
    warning: use of deprecated associated function `evmap::handles::WriteHandle::<K, V, M, S>::empty`: Renamed to remove_entry
      --> server/dataflow/src/backlog/multiw.rs:38:19
       |
    38 |                 h.empty(key_to_single(k).into_owned());
       |                   ^^^^^
       |
       = note: `#[warn(deprecated)]` on by default
    
    warning: use of deprecated associated function `evmap::handles::WriteHandle::<K, V, M, S>::empty`: Renamed to remove_entry
      --> server/dataflow/src/backlog/multiw.rs:41:19
       |
    41 |                 h.empty(key_to_double(k).into_owned());
       |                   ^^^^^
    
    warning: use of deprecated associated function `evmap::handles::WriteHandle::<K, V, M, S>::empty`: Renamed to remove_entry
      --> server/dataflow/src/backlog/multiw.rs:44:19
       |
    44 |                 h.empty(k.into_owned());
       |                   ^^^^^
    
    error[E0599]: no method named `refresh` found for mutable reference `&mut evmap::handles::WriteHandle<noria_common::DataType, Vec<noria_common::DataType>, i64, ahash::RandomState>` in the current scope
      --> server/dataflow/src/backlog/multiw.rs:67:19
       |
    67 |                 h.refresh();
       |                   ^^^^^^^ method not found in `&mut evmap::handles::WriteHandle<noria_common::DataType, Vec<noria_common::DataType>, i64, ahash::RandomState>`
    
    error[E0599]: no method named `refresh` found for mutable reference `&mut evmap::handles::WriteHandle<(noria_common::DataType, noria_common::DataType), Vec<noria_common::DataType>, i64, ahash::RandomState>` in the current scope
      --> server/dataflow/src/backlog/multiw.rs:70:19
       |
    70 |                 h.refresh();
       |                   ^^^^^^^ method not found in `&mut evmap::handles::WriteHandle<(noria_common::DataType, noria_common::DataType), Vec<noria_common::DataType>, i64, ahash::RandomState>`
    
    error[E0599]: no method named `refresh` found for mutable reference `&mut evmap::handles::WriteHandle<Vec<noria_common::DataType>, Vec<noria_common::DataType>, i64, ahash::RandomState>` in the current scope
      --> server/dataflow/src/backlog/multiw.rs:73:19
       |
    73 |                 h.refresh();
       |                   ^^^^^^^ method not found in `&mut evmap::handles::WriteHandle<Vec<noria_common::DataType>, Vec<noria_common::DataType>, i64, ahash::RandomState>`
    
    error[E0599]: no method named `read` found for reference `&evmap::handles::WriteHandle<noria_common::DataType, Vec<noria_common::DataType>, i64, ahash::RandomState>` in the current scope
      --> server/dataflow/src/backlog/multiw.rs:85:29
       |
    85 |                 let map = h.read()?;
       |                             ^^^^ method not found in `&evmap::handles::WriteHandle<noria_common::DataType, Vec<noria_common::DataType>, i64, ahash::RandomState>`
    
    error[E0599]: no method named `read` found for reference `&evmap::handles::WriteHandle<(noria_common::DataType, noria_common::DataType), Vec<noria_common::DataType>, i64, ahash::RandomState>` in the current scope
       --> server/dataflow/src/backlog/multiw.rs:114:33
        |
    114 |                     let map = h.read()?;
        |                                 ^^^^ method not found in `&evmap::handles::WriteHandle<(noria_common::DataType, noria_common::DataType), Vec<noria_common::DataType>, i64, ahash::RandomState>`
    
    error[E0599]: no method named `read` found for reference `&evmap::handles::WriteHandle<Vec<noria_common::DataType>, Vec<noria_common::DataType>, i64, ahash::RandomState>` in the current scope
       --> server/dataflow/src/backlog/multiw.rs:121:29
        |
    121 |                 let map = h.read()?;
        |                             ^^^^ method not found in `&evmap::handles::WriteHandle<Vec<noria_common::DataType>, Vec<noria_common::DataType>, i64, ahash::RandomState>`
    
    opened by pratikfegade 2
  • Question - io-uring, Glommio and Upmem peformance benefits for Noria?

    Question - io-uring, Glommio and Upmem peformance benefits for Noria?

    Hi,

    I have a question as to whether Noria's performance would benefit from:

    1. io-uring ?

    2. from the rust crate Glommio (a thread per core crate - https://www.datadoghq.com/blog/engineering/introducing-glommio/). I think the author wrote another blog about API performance and io_uring (https://itnext.io/modern-storage-is-plenty-fast-it-is-the-apis-that-are-bad-6a68319fbc1a)

    3. As well as from Upmem (https://www.upmem.com/) which seems to have the most benefit (index searches) potential and smallest code change required to enable (index searches) it according to their site.

    I understand the answer might be 'if you need that much performance then you should be able to afford to buy more cores rather than creating technical complexity' but some of the items above might only benefit incoming API calls from source and not Noria itself, so just want to understand at a simplistic level / answer if these would or wouldnt benefit Noria'S performance.

    Thanks in advance!

    opened by unibum 1
Owner
MIT PDOS
Parallel and Distributed Operating Systems group at MIT CSAIL
MIT PDOS
Super Fast & High Performance minimalist web framework for rust

Super Fast & High Performance minimalist web framework for rust

null 6 Oct 12, 2022
🌱🦀🌱 Trillium is a composable toolkit for building web applications with async rust 🌱🦀🌱

?????? Trillium is a composable toolkit for building web applications with async rust ??????

Trillium 243 Jan 2, 2023
Layers, extractors and template engine wrappers for axum based Web MVC applications

axum-template Layers, extractors and template engine wrappers for axum based Web MVC applications Getting started Cargo.toml [dependencies] axum-templ

Altair Bueno 11 Dec 15, 2022
Sauron is an html web framework for building web-apps. It is heavily inspired by elm.

sauron Guide Sauron is an web framework for creating fast and interactive client side web application, as well as server-side rendering for back-end w

Jovansonlee Cesar 1.7k Dec 26, 2022
Actix Web is a powerful, pragmatic, and extremely fast web framework for Rust.

Actix Web is a powerful, pragmatic, and extremely fast web framework for Rust.

Actix 16.2k Jan 2, 2023
Hot reload static web server for deploying mutiple static web site with version control.

SPA-SERVER It is to provide a static web http server with cache and hot reload. 中文 README Feature Built with Hyper and Warp, fast and small! SSL with

null 7 Dec 18, 2022
Code template for a production Web Application using Axum: The AwesomeApp Blueprint for Professional Web Development.

AwesomeApp rust-web-app More info at: https://awesomeapp.dev/rust-web-app/ rust-web-app YouTube episodes: Episode 01 - Rust Web App - Course to Produc

null 45 Sep 6, 2023
A highly customizable, full scale web backend for web-rwkv, built on axum with websocket protocol.

web-rwkv-axum A axum web backend for web-rwkv, built on websocket. Supports BNF-constrained grammar, CFG sampling, etc., all streamed over network. St

Li Junyu 12 Sep 25, 2023
A Rust library to extract useful data from HTML documents, suitable for web scraping.

select.rs A library to extract useful data from HTML documents, suitable for web scraping. NOTE: The following example only works in the upcoming rele

Utkarsh Kukreti 829 Dec 28, 2022
A Google-like web search engine that provides the user with the most relevant websites in accordance to his/her query, using crawled and indexed textual data and PageRank.

Mini Google Course project for the Architecture of Computer Systems course. Overview: Architecture: We are working on multiple components of the web c

Max 11 Aug 10, 2022
📝 Web-based, reactive Datalog notebooks for data analysis and visualization

Percival is a declarative data query and visualization language. It provides a reactive, web-based notebook environment for exploring complex datasets, producing interactive graphics, and sharing results.

Eric Zhang 486 Dec 28, 2022
Bundle Shiny Applications for serving with WebR.

WebR Bundle A tool for bundling Shiny Apps with WebAssembly. R Package Installation Install the package from GitHub: # install.packages("pak") pak::pa

Andrés Felipe Quintero Moreano 12 Oct 5, 2023
Bundle Shiny Applications for serving with WebR.

WebR Bundle A tool for bundling Shiny Apps with WebAssembly. R Package Installation Install the package from GitHub: # install.packages("pak") pak::pa

Appsilon 14 Oct 29, 2023
Starlight is a JS engine in Rust which focuses on performance rather than ensuring 100% safety of JS runtime.

starlight Starlight is a JS engine in Rust which focuses on performance rather than ensuring 100% safety of JS runtime. Features Bytecode interpreter

null 453 Dec 31, 2022
A Rust web framework

cargonauts - a Rust web framework Documentation cargonauts is a Rust web framework intended for building maintainable, well-factored web apps. This pr

null 179 Dec 25, 2022
A rust web framework with safety and speed in mind.

darpi A web api framework with speed and safety in mind. One of the big goals is to catch all errors at compile time, if possible. The framework uses

null 32 Apr 11, 2022
A web framework for Rust.

Rocket Rocket is an async web framework for Rust with a focus on usability, security, extensibility, and speed. #[macro_use] extern crate rocket; #[g

Sergio Benitez 19.4k Jan 4, 2023
Rust / Wasm framework for building client web apps

Yew Rust / Wasm client web app framework Documentation (stable) | Documentation (latest) | Examples | Changelog | Roadmap | 简体中文文档 | 繁體中文文檔 | ドキュメント A

Yew Stack 25.8k Jan 2, 2023
A super-easy, composable, web server framework for warp speeds.

warp A super-easy, composable, web server framework for warp speeds. The fundamental building block of warp is the Filter: they can be combined and co

Sean McArthur 7.5k Jan 2, 2023