Terminal based, feature rich, interactive SQL tool

Overview

datafusion-tui (dft)

DataFusion-tui provides a feature rich terminal application, built with tui-rs, for using DataFusion (and eventually Ballista). It has drawn inspiration and several features from datafusion-cli. In contrast to datafusion-cli the objective of this tool is to provide a light SQL IDE experience for querying data with DataFusion. It is currently in early stages of development and as such there are likely to be bugs.

dft-recording.mov

Some of the current and planned features are listed here:

  • Tab management to provide clean and structured organization of DataFusion queries, results, and context
    • SQL editor
      • Text editor for writing SQL queries
      • Scrollable query results
      • Track memory usage during query (TODO)
      • Write query results to file (TODO)
      • Multiple SQL Editor tabs (TODO)
    • Query history
      • History of executed queries
    • ExecutionContext information (TODO)
      • Information from ExecutionContext / Catalog / ObjectStore / State / Config
    • Logs
      • Logs from dft and DataFusion
    • Help (TODO)
      • Documentation on functions / commands
  • Custom ObjectStore Support
    • S3 with AWS default credentials
    • S3 with custom endpoint / provider (i.e. MinIO)
    • HDFS (TODO)
    • ObjectStore explorer. I.e. able to list files in ObjectStore
  • Custom Table Providers (if supported in SQL by DataFusion)
    • Delta Table (TODO)
    • Big Table (TODO)
  • Preloading DDL from ~/.datafusion/.datafusionrc for local database available on startup

User Guide

The interface is split into several tabs so that relevant information can be viewed and controlled in a clean and organized manner. When not writing a SQL query keys can be entered to navigate and control the interface.

  • SQL Editor: where queries are entered and results can be viewed
    • Normal mode
      • q => quit datafusion-tui
      • e => start editing SQL Editor in Edit mode
      • c => clear contents of SQL Editor
      • Enter the tab number in brackets after a tabs name to navigate to that tab
      • If query results are longer or wider than screen, you can use arrow keys to scroll
    • Edit mode
      • Character keys to write queries
      • Backspace / tab / enter work same as normal
      • Add a semicolor after query and hit enter to execute query
      • esc to exit Edit mode and go back to Normal mode
  • Register custom ObjectStore
    • S3: run / install with --features=s3
      • If you want to use your default AWS credentials, then no further action is required. For example your credentials in ~/.aws/credentials will automatically be picked up.
      • If you want to use a custom S3 provider, such as MinIO, then you must create a s3.json configuration file in ~/.datafusion/object_stores/ with the fields endpoint, access_key_id, and secret_access_key.
Comments
  • No error message for invalid sql statements

    No error message for invalid sql statements

    Error description When executing an invalid query there is no feedback to the user that something went wrong. The result tab just shows the result of previous query. Also in the log tab, there is no indication that something went wrong.

    An invalid query can be

    • a wrong SQL statement
    • valid SQL but with wrong table name

    Expected behavior Some kind of error message/warning. Maybe for beginning, it is possible to show the error message from DataFusion itself?

    opened by seryj 12
  • Support BigTable

    Support BigTable

    Add ~/.datafusion/table_providers/bigtable.json which stores configuration options to create TableProvider and add optional BigTable feature. Tables defined in bigtable.json are loaded at startup.

    Config fields comes from the below plus a table name field.

    let bigtable_datasource = BigtableDataSource::new(
        "emulator".to_owned(),                               // project
        "dev".to_owned(),                                    // instance
        "weather_balloons".to_owned(),                       // table
        "measurements".to_owned(),                           // column family
        vec!["_row_key".to_owned()],                         // table_partition_cols
        "#".to_owned(),                                      // table_partition_separator
        vec![Field::new("pressure", DataType::Utf8, false)], // qualifiers
        true,                                                // only_read_latest
    ).await.unwrap();
    
    opened by matthewmturner 8
  • Feature/unit tests for editor

    Feature/unit tests for editor

    Few unit tests for testing the functionality in the editor tab. Right now, 4 tests out of (new) 8 are failing. 2 of them are related to handling of non-ascii characters. 2 are user-experience-related and are not that important (when in the beginning/end of the line, the cursor should jump to previous/next line when left/right buttons are pressed).

    opened by seryj 7
  • Page up/down and move execution to normal mode

    Page up/down and move execution to normal mode

    @seryj FYI adding page up/down and i did try shift+enter but im having trouble with it(the KeyModifier::SHIFT isnt being recognized for some reason). will need to come back to it or table for now.

    opened by matthewmturner 5
  • Editor navigation/usage issues

    Editor navigation/usage issues

    First of all, thanks for creating and working on this project! A good graphical interface will definitely help a lot of people in the future!

    Now, to my issues. While playing around, there were several situations when the application crashed/or behaved wrongly/not-expected in the editor window.

    Crashes:

    1. When using non-ascii characters (like German "umlaut" character ä, ö, ü), the application crashes when such a character is deleted. The issue is that non-ascii chars consume more then 1 byte but currently all operations in the editor directly change the bytes inside of the string (Lines) object.
    2. When the editor is opened and there is no text inside, moving the cursor crashes the application.

    Wrong behavior:

    1. When typing a character inside a line, the character appears in the end of the line. (I think this one was already fixed)

    Usability issues:

    1. In the end of the line, when moving a cursor to the right, the cursor stays in the same line instead of jumping to the next one
    2. In the beginning of a line, when moving a cursor to the left the cursor stays in the same line instead of jumping to the previous one
    3. End/Home buttons do not work

    While all issues except for non-ascii handling are not that big, handling non-ascii characters requires a re-implementation of almost all functions of the Editor object because direct changing of a byte inside the String object does not work anymore and one has to use Chars instead.

    This being said, I can offer an PR where above mentioned issues are addressed and appropriate unit tests are provided. But of course only if contribution to this project is desired. Alternatively, I could provide an PR with unit tests only. Whatever you prefer.

    Best regards

    bug enhancement 
    opened by seryj 3
  • Overflow when backspacing

    Overflow when backspacing

    When entering a single line of text, creating multiple new lines with enter, and then backspace to first line. Cursor also doesnt show at right position when going back to first line.

    opened by matthewmturner 1
  • Explore alternatives for executing a query

    Explore alternatives for executing a query

    I wonder if one needs a semicolon at all. Right now, the editor always contains a single query and unless this changes in the future, having a semicolon is only required to recognize that the user wants to execute a query. Thinking about other tools, the same problem is often solved by using another key combination e.g. "shift+enter". Could that maybe be a more user-friendly solution?

    On the pro-side, if chosen carefully, this "execute query" key combination could be used in the normal and edit modes such that the users will do less mistakes.

    On the contra-side, the complexity of key handling increases.

    How do you see it?

    Originally posted by @seryj in https://github.com/datafusion-contrib/datafusion-tui/issues/73#issuecomment-1094371360

    opened by matthewmturner 0
Owner
Community maintained unofficial extensions for Apache Arrow DataFusion
null
🖥 A feature rich terminal UI file transfer and explorer with support for SCP/SFTP/FTP/S3

?? A feature rich terminal UI file transfer and explorer with support for SCP/SFTP/FTP/S3

Christian Visintin 574 Jan 5, 2023
Rich terminals. follows Python's rich conventions

richterm use richterm::print; use richterm::text; use richterm::track; use richterm::progress; use std::{thread, time}; fn main() { let vec = ve

Abdur-Rahmaan Janhangeer 14 Aug 5, 2023
A lightweight but incredibly powerful and feature-rich BitTorrent tracker. Supports UDP + HTTP(S) and a private tracker mode.

Torrust Tracker Project Description Torrust Tracker is a lightweight but incredibly powerful and feature-rich BitTorrent tracker made using Rust. Feat

Torrust 162 Dec 31, 2022
Fast, minimal, feature-rich, extended formatting syntax for Rust!

Formatting Tools Fast, minimal, feature-rich, extended formatting syntax for Rust! Features include: Arbitrary expressions inside the formatting brace

Casper 58 Dec 26, 2022
Sleek is a CLI tool for formatting SQL. It helps you maintain a consistent style across your SQL code, enhancing readability and productivity.

Sleek: SQL Formatter ✨ Sleek is a CLI tool for formatting SQL. It helps you maintain a consistent style across your SQL code, enhancing readability an

Nick Rempel 40 Apr 20, 2023
Simple Interactive Terminal Todo App in Rust

todo-rs Simple Interactive Terminal Todo App in Rust Quick Start $ cargo run TODO Controls Keys Description k, j Move cursor up and down Shift+K, Shif

Tsoding 56 Dec 8, 2022
A simple interactive OSC (Open Sound Control) debugger for terminal

oscd, a simple interactive OSC debugger for the terminal by using nom as a lexer/parser, oscd offers auto type casting and support sending multiple osc arguments.

Karnpapon Boonput 5 Oct 19, 2022
A library that creates a terminal-like window with feature-packed drawing of text and easy input handling. MIRROR.

BearLibTerminal provides a pseudoterminal window with a grid of character cells and a simple yet powerful API for flexible textual output and uncompli

Tommy Ettinger 43 Oct 31, 2022
An interactive Bayesian Probability Calculator CLI that guides users through updating beliefs based on new evidence.

Bayesian Probability Calculator CLI Welcome to the Bayesian Probability Calculator CLI! This command-line tool is designed to help you update your bel

Ben Greenberg 4 Apr 25, 2023
An interactive cheatsheet tool for the command-line

navi An interactive cheatsheet tool for the command-line. navi allows you to browse through cheatsheets (that you may write yourself or download from

Denis Isidoro 12.2k Dec 30, 2022
Coinlive is an interactive command line tool that displays live cryptocurrency prices.

Coinlive is an interactive command line tool that displays live cryptocurrency prices. It can also display simple historical price charts.

Mayer Analytics 9 Dec 7, 2022
ruborute is an interactive command-line tool to get asphyxia@sdvx gaming data.

ruborute Are you 暴龍天 ?. The ruborute is an interactive command-line tool to get asphyxia@sdvx gaming data. asphyxia-core/plugins: https://github.com/a

RinChanNOW! 9 Sep 28, 2022
nvim-oxi provides safe and idiomatic Rust bindings to the rich API exposed by the Neovim text editor.

?? nvim-oxi nvim-oxi provides safe and idiomatic Rust bindings to the rich API exposed by the Neovim text editor. The project is mostly intended for p

Riccardo Mazzarini 655 Jul 13, 2023
Estratto is a powerful and user-friendly Rust library designed for extracting rich audio features from digital audio signals.

estratto 〜 An Audio Feature Extraction Library estratto is a powerful and user-friendly Rust library designed for extracting rich audio features from

Amber J Blue 5 Aug 25, 2023
AskBend: SQL-based Knowledge Base Search and Completion using Databend

AskBend: SQL-based Knowledge Base Search and Completion using Databend AskBend is a Rust project that utilizes the power of Databend and OpenAI to cre

Databend Labs 87 Apr 7, 2023
qsv - Performant CLI tool to query CSVs through SQL

qsv Performant CLI tool to query CSVs through SQL Installation After cloning the repository, you can install a binary locally using cargo install --pa

Dermot Haughey 3 Oct 28, 2021
A tool that allow you to run SQL-like query on local files instead of database files using the GitQL SDK.

FileQL - File Query Language FileQL is a tool that allow you to run SQL-like query on local files instead of database files using the GitQL SDK. Sampl

Amr Hesham 39 Mar 12, 2024
Configurable, extensible, interactive line reader

linefeed linefeed is a configurable, concurrent, extensible, interactive input reader for Unix terminals and Windows console. API Documentation linefe

Murarth 176 Jan 3, 2023