πŸ“ Web-based, reactive Datalog notebooks for data analysis and visualization

Overview

Percival

Percival is a declarative data query and visualization language. It provides a reactive, web-based notebook environment for exploring complex datasets, producing interactive graphics, and sharing results.

Percival combines the flexibility of Datalog as a query language for relational data with the beauty of exploratory visualization grammars. These declarative components interact through a reactive dataflow system. Because Percival uses web technologies (including Web Workers for multithreaded, sandboxed execution), fully-interactive notebooks can be shared with anyone on the Internet, making data analyses more tangible to others.

At the core of Percival is a custom Datalog compiler, built with Rust and WebAssembly, which integrates with its notebook runtime. This compiles the query language to JavaScript through a staged evaluation process that also allows users to embed their own JavaScript code. The interface aims to be lightweight, friendly, and accessible, and there is no hidden workspace state.

This is an early-stage research project, and we welcome your feedback, so please feel free to say hello at our discussions page!

Getting Started

Building Percival from scratch requires Node v16+, NPM v8+, Rust 1.56+, Cargo, and Wasm-Pack installed on your machine. To build the Rust/WebAssembly portion of the project, use the command:

wasm-pack build --target web crates/percival-wasm

Next, run npm install to install JavaScript dependencies, then run the following command to start the development server:

npm run dev

This should open a Percival notebook in your browser.

Development

To build, lint, and format the Svelte project, use the corresponding scripts:

npm run build
npm run check
npm run format

For the Rust crates, you can run unit tests for the core functionality with:

cargo test

You can also run tests for the WebAssembly component using a headless Chrome or Firefox browser:

wasm-pack test --chrome --headless crates/percival-wasm

Since Percival uses a Rust-based compiler but outputs JavaScript, the easiest way to test code generation functionality is within the browser. We use Mocha and Puppeteer for this, and tests can be run with:

npm test

Acknowledgement

Created by Eric Zhang (@ekzhang1). Licensed under the MIT license.

Comments
  • Adjust Divider UX

    Adjust Divider UX

    Hi, nice project! Datalog/prolog-like facts+rules is a great way to work with data, web-based notebooks are an ideal experience for exploring and sharing it, and I've been interested in trying out svelte + tailwindcss, so your project is surprisingly synchronous for me. Thank you for sharing it!

    While exploring the project I noticed a couple small things so I fixed them and put them in separate commits for you to review:

    Smaller one first: I factored the divider buttons to generate from a list. This should make it easier to add/remove buttons for new types of cells in the future and I think it's a tiny bit easier to read (also maybe an opportunity to generate from centrally registered cell types in some future version?). This is my first time using Svelte so I wanted to poke some (very!) minor low-hanging fruit first. 😸 Note that I moved the <hr> to the bottom of the div to reproduce identical button classes with :not(:first-child); I believe the result is the same.

    Second, I noticed that you had to be unusually precise to trigger the opacity hover in order to see the divider buttons, in fact I was confused for a few minutes trying to figure out how to add new cells before I discovered that you just have to hover in the right place. So I adjusted the styles to make the divider hover target easier to activate, specifically:

    • Remove CellDivider h-2 so the div takes up the whole space between cells
    • Remove margin on Cells
    • Use negative margin on divider to slightly overlap neighboring cells' padding. Keeps both compact spacing and easy hover target.
    • Increase divider z-index so the overlap works correctly

    With some added borders to visualize, before:

    image

    Here you can see that you have to hover the buttons directly or that tiny sliver around the <hr> for the buttons to become visible.

    After:

    image

    Now, hovering anywhere on the overlapped area triggers the hover effect of the divider which makes it much easier to find. I checked and none of the regular cells use this area so this change shouldn't impede their functionality. This does have a small effect on the vertical spacing of neighboring cells (3% taller based on the screenshots), but it is not very noticeable and worth the UX imo.

    I'm sorry for writing so many words for such a small PR but I don't have time to make it shorter, please forgive me! πŸ˜…πŸ™ What do you think?

    opened by infogulch 1
  • code cell with syntax error gets erased on toggle

    code cell with syntax error gets erased on toggle

    Steps to reproduce:

    1. Create a code cell.
    2. Write invalid code, such as: 1, 2, 3. The code block is highlighted in orange.
    3. Click the down arrow to toggle the visibility of the code block, rendering it not visible.
    4. Click the right arrow to toggle visibility of the code block, rendering it visible.
    5. The code block is empty, while expected to contain 1, 2, 3.
    opened by namin 1
  • Finish writing the Percival

    Finish writing the Percival "starter notebook" tutorial

    I will finish the tutorial and fill in the missing section with more complex data analyses, as well as nicer examples of data visualizations using Observable Plot.

    This starter notebook will be finished when I have time, and I'll advertise the project after that.

    See also: #5.

    (It would also be good to write a "How it works" section to the README explaining the Rust Datalog compilation and reactive WebAssembly components + Web Workers.)

    opened by ekzhang 1
  • Better error messages with a lexical grammar

    Better error messages with a lexical grammar

    Right now, the Percival compiler simply parses inputs as a character stream with parser combinators. Although this is simple and was sufficient for the purposes of the basic language, some error messages are a little bit suspect, especially ones that mention "expected ... in input." For example, the error below is misleading. It should suggest either an import, rule, or end-of-file.

    image

    By switching to a two-phase lexical and syntactic analysis step on a stream of tokens, rather than characters, the parser will be much more flexible, while also having clearer error messages.

    opened by ekzhang 0
  • Fix pre-bundling issue by updating Vite config

    Fix pre-bundling issue by updating Vite config

    Previously, I was using a hack in runtime.ts, adding some lines that were necessary because certain libraries were not present in the non-worker bundle. This makes tests get confused because they discover the libraries mid-execution and reload the page, breaking Puppeteer.

    By updating the optimizeDeps config option of Vite, I can tell Vite which entry points to crawl for pre-bundled dependencies at boot time, avoiding this reloading issue entirely.

    opened by ekzhang 0
  • Adjustments for mobile

    Adjustments for mobile

    I would love to run Datalog on my phone. This requires some small tweaks to the current UI:

    • [ ] Make all elements visible without hovering for mobile devices (or for all devices?). This includes the gutter and buttons.
    • [ ] Add a "run code" button. Shift + enter doesn't work on mobile; even if it did, a "run" button would be more convenient.
    opened by remysucre 0
  • `dev` cells throws module import error

    `dev` cells throws module import error

    I suspect this is something rather obvious to someone more familiar with the JavaScript build tools:

    image

    when running npm run dev.

    npm run serve allows cell execution (as expected) -- I'm not exactly sure why this throws during dev.

    opened by femtomc 3
  • Sidebar UX

    Sidebar UX

    Orient sidebar buttons vertically, as described in https://github.com/ekzhang/percival/pull/16#issuecomment-1120137337. This will conflict with 16; that PR should probably be merged first and I'll rebase this one on top of it after.

    Changes:

    • Orient buttons vertically, with the chevron in the top right corner. This makes it easier to add more buttons, and also makes the buttons clickable on mobile.
    • Adjust the spacing of the sidebar and Notebook so that the sidebar and buttons are always visible, even with a very narrow view. It's a tiny bit off-center at this width but worth actually being able to interact with the UI.
    • Add small border to the right of the sidebar, visible on cell hover, and small cell padding left to space it. This makes it easier to see which input and output views are part of the same cell; this was visually confusing at first and will get worse as we add more views to cells.
    • Remove FaChevronDown in favor of a conditional rotate-90 transform class when the input is shown. Combined with transition-all this nicely animates the chevron when you click on the sidebar.
    • Only show delete button when input is visible, this helps prevent taps on the invisible delete button on mobile (for closed cells at least), and works around an issue on closed markdown cells with one line where the delete button sticks out below the bottom of the sidebar.
    • Adjust the positioning calculation of the sidebar so that the left side reaches just to the edge of the view, not far past. This will make it easier to position items in the sidebar in general, and covers the case where the page is very zoomed out and 2000px is not enough. Generally this is minor but feels more 'correct' to me.
    opened by infogulch 0
  • Plot cell: allow async, multiple inputs, one output

    Plot cell: allow async, multiple inputs, one output

    This PR updates Plot cells to work as more fully-featured JS cells.

    • Plot cells can take multiple input dependencies: (dep1, dep2) => ... or no inputs, () => ....
    • Plot cells can be async, allowing them to access modules with await import(...): async dep => (await import('https://esm.sh/...'))(dep).
    • Plot cells can write a reactive result, using assignment syntax: resultName = () => [{ x: 1 }, { x: 2 }]. Currently this emits one relation, not a set of relations like code cells. A future change could use syntax like { table1, table2 } = () => ... to emit multiple relation outputs.
    • Cell outputs with outerHTML are rendered as HTML (as before). Array outputs are rendered as relations. Otherwise, the output is rendered as JSON. If #16 is merged, I'll replace the simple JSON renderer with the JSON tree inspector used there.

    After these changes "plot" vs "code" seem like misnomers - perhaps we should rename these concepts to "datalog" and "js". However, that's a bigger policy/design change.

    image
    opened by justjake 0
  • JS in Percival / Percival in JS

    JS in Percival / Percival in JS

    Percival makes Datalog so much more accessible! Now instead of needing to pick an implementation, install it, read the manual, figure out syntax highlighting in your editor, etc etc etc - we can just visit https://percival.ink. There's a bunch of examples, syntax highlighting, and a quick syntax to fetch data. That's great.

    But though you've broken the barrier to access a Datalog implementation, there's more we can do to make Datalog useable.

    1. Bring JS into the Percival notebook: We should add JS code blocks that can consume relations and emit new data to the DAG. This will allow learners to use Percival for analysis even if they're Datalog beginners. The user can attempt analysis in Datalog, and bail out to JS if they hit a roadblock. As they gain fluency, users gradually transition to Datalog.

      If we allow cells to return Promise, Percival becomes end-user extendable via ES6 modules. We can now import anything from NPM - not just CSV & JSON data.

      The simplest implementation of this idea would be to parse plot cells of the form output_name = (more, than, one, arg) => ....

    2. Bring Datalog rules out of Percival into the wider JS world: publish percival-wasm and/or src/lib/** as experimental packages on NPM. Advanced users can build on top of the work here incentivizes further contribution. The risk of publishing to NPM a backwards compatibility burden. But, given the niche nature of Datalog today, I don't expect this to be a substantial issue if we publish using alpha tags like 0.0.0-alpha001

    For both (1) and (2), I'm personally excited to explore integrating Percival <--> ObservableHQ.

    • We use Plot already. We could also use @observablehq/inspector and @observablehq/stdlib in plot/JS code blocks.
    • Observable's runtime is open source. We could investigate replacing the notebook toposort engine with @observablehq/runtime. This could allow importing Observable notebooks into Percival.
    • If we publish runtime.build to NPM, we can import Percival and run datalog within ObservableHQ notebooks. But, we won't have syntax highlighting :(
    opened by justjake 1
  • Load and save percival files in git repos

    Load and save percival files in git repos

    I think it would be neat to be able to load, save, edit, and commit percival files directly against a git repo.

    Something like isomorphic-git could be useful here:

    https://github.com/isomorphic-git/isomorphic-git

    It seems there are some potential complications to supporting a git workflow, especially related to CORS.

    Thoughts?

    opened by infogulch 3
Owner
Eric Zhang
Student at Harvard, competitive programmer, and researcher in machine learning & programming languages. Previously @get-convex @jumptrading @scaleapi @NVIDIA.
Eric Zhang
OpenAPI-based test coverage analysis tool that helps teams improve integration test coverage in CI/CD pipelines

Ready-to-use OpenAPI test coverage analysis tool that helps teams improve integration CoveAPI is an advanced test coverage analysis tool based on the

Yasser Tahiri 18 Aug 3, 2023
A Google-like web search engine that provides the user with the most relevant websites in accordance to his/her query, using crawled and indexed textual data and PageRank.

Mini Google Course project for the Architecture of Computer Systems course. Overview: Architecture: We are working on multiple components of the web c

Max 11 Aug 10, 2022
Actix Web is a powerful, pragmatic, and extremely fast web framework for Rust.

Actix Web is a powerful, pragmatic, and extremely fast web framework for Rust.

Actix 16.2k Jan 2, 2023
Sauron is an html web framework for building web-apps. It is heavily inspired by elm.

sauron Guide Sauron is an web framework for creating fast and interactive client side web application, as well as server-side rendering for back-end w

Jovansonlee Cesar 1.7k Dec 26, 2022
Hot reload static web server for deploying mutiple static web site with version control.

SPA-SERVER It is to provide a static web http server with cache and hot reload. δΈ­ζ–‡ README Feature Built with Hyper and Warp, fast and small! SSL with

null 7 Dec 18, 2022
Code template for a production Web Application using Axum: The AwesomeApp Blueprint for Professional Web Development.

AwesomeApp rust-web-app More info at: https://awesomeapp.dev/rust-web-app/ rust-web-app YouTube episodes: Episode 01 - Rust Web App - Course to Produc

null 45 Sep 6, 2023
A highly customizable, full scale web backend for web-rwkv, built on axum with websocket protocol.

web-rwkv-axum A axum web backend for web-rwkv, built on websocket. Supports BNF-constrained grammar, CFG sampling, etc., all streamed over network. St

Li Junyu 12 Sep 25, 2023
A Rust library to extract useful data from HTML documents, suitable for web scraping.

select.rs A library to extract useful data from HTML documents, suitable for web scraping. NOTE: The following example only works in the upcoming rele

Utkarsh Kukreti 829 Dec 28, 2022
Noria: data-flow for high-performance web applications

Noria: data-flow for high-performance web applications Noria is a new streaming data-flow system designed to act as a fast storage backend for read-he

MIT PDOS 4.5k Dec 28, 2022
Murasaki is a Fast, Secure, and Reliable Webkit based web browser.

Murasaki is a Fast, Secure, and Reliable Webkit based web browser. Table of Contents Goals Status Usage License Goals Security: Be secure, and not com

Moon Laboratories 5 Nov 17, 2021
Layers, extractors and template engine wrappers for axum based Web MVC applications

axum-template Layers, extractors and template engine wrappers for axum based Web MVC applications Getting started Cargo.toml [dependencies] axum-templ

Altair Bueno 11 Dec 15, 2022
axum-serde is a library that provides multiple serde-based extractors and responders for the Axum web framework.

axum-serde ?? Overview axum-serde is a library that provides multiple serde-based extractors / responses for the Axum web framework. It also offers a

GengTeng 3 Dec 12, 2023
A web application to configuration Caddy based on MoonZoon.

Cream A web application to configuration Caddy based on MoonZoon. MoonZoon is a Rust Fullstack Framework. Live demo Run on a local machine Check you'v

Tw 4 Sep 19, 2022
A GUI frontend in Rust based on web-view

neutrino I am not working anymore on this project. If you want to become a maintainer of neutrino, please answer to this issue. Preamble Docs | Repo |

null 287 Nov 25, 2022
Ergonomic and modular web framework built with Tokio, Tower, and Hyper

axum axum is a web application framework that focuses on ergonomics and modularity. More information about this crate can be found in the crate docume

Tokio 7.9k Dec 31, 2022
A simple authentication flow using Rust and Actix-web, with a PostgreSQL database and a sveltekit frontend.

Rust-auth-example This repository aims to represent a simple authentication flow using Rust and Actix-web, with a PostgreSQL database and a sveltekit

Kival Mahadew 4 Feb 19, 2023
A rust web framework with safety and speed in mind.

darpi A web api framework with speed and safety in mind. One of the big goals is to catch all errors at compile time, if possible. The framework uses

null 32 Apr 11, 2022
Volt - A powerful, fast and memory safe package manager for the web

Volt - A powerful, fast and memory safe package manager for the web

Volt Package Manager 811 Dec 30, 2022
Thruster - An fast and intuitive rust web framework

A fast, middleware based, web framework written in Rust

null 913 Dec 27, 2022