Rust Deduplicate-text-datasets Resources

Allfollow What is this? Take this flake's inputs, for example. inputs = { nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable"; rust-overla

Thongna 🌾 Thongna (ท้องนา) is a high-performance text processing library for the Thai language, built with Rust and exposed as a Python package. Insp

Held DragonOS/Linux Termial text editor 面向DragonOS和Linux系统的终端文本编辑器。 作者：Heyicong [email protected] 安装 默认安装路径为./install/bin/held Linux: make install

Markrequests A plain-text Issue and PR tracking system. An example of a file to track these is in prs/example.md Rationale I am unsatisfied with the c

smartcat (sc) Puts a brain behind cat! CLI interface to bring language models in the Unix ecosystem and allow power users to make the most out of llms

Bashtastic Visualizer TUI (Text User Interface) - Get Instant feedback for your sh commands. Explore and play with your queries 🚀. The idea of this p

Melba-stt A rust discord bot that joins a voice channel and transcribes spoken audio from each user. Running Install the rust toolchain With CUDA and

tui-big-text tui-big-text is a rust crate that renders large pixel text as a ratatui widget using the glyphs from the font8x8 crate. Installation carg

bevy_simple_text_input An unambitious single-line text input widget for bevy_ui. Usage See examples/basic.rs. Alternatives If you need more features,

summarize-cli Attempt to summarize text from stdin, using a large language model (locally and offline), to stdout. cargo build --release target/releas

align: a command line utility for aligning text. ⭐ Overview Aligns text within the terminal (or a specified number of columns). The text is treated as

epub_to_txt A simple (but really fast!) CLI tool written in Rust to convert an epub into a text file (in Markdown format). Usage epub_to_txt filename.

Siren Language I'm trying to make a simple programming language called Siren Language. This language will become a graphic and text programming langua

3FakeIM 📱️🚫️🌝️💾️ 3FakeIM is a joke program meant to imitate various fictional characters, and the "[CHARACTER] CALLED ME AT 3:00 AM" clickbait tre

🔗 nvim-oxi nvim-oxi provides safe and idiomatic Rust bindings to the rich API exposed by the Neovim text editor. The project is mostly intended for p

tui-textarea tui-textarea is a simple yet powerful text editor widget like textarea in HTML for tui-rs and ratatui. Multi-line text editor can be ea

🐌 snailshell A tiny library and application for animating text in the terminal. 🪄 It's so easy to add some flair to your text RPGs or console progra

ReefDB ReefDB is a minimalistic, in-memory and on-disk database management system written in Rust, implementing basic SQL query capabilities and full-

Secret Keeper A simple and secure command-line tool to protect your text by encrypting and decrypting it using the robust AES-256 algorithm. Built wit

Grug Grug is a command-line tool that provides a workflow for expanding, editing, diffing, and writing edits to files using vim-styled grep lines (suc

Large language models (LLMs) can be used for many tasks, but often have a limited context size that can be smaller than documents you might want to use. To use documents of larger length, you often have to split your text into chunks to fit within this context size.

cpu_usage-polybar A small script in rust to get the cpu usage in %'s with a gradient color for the text To get it to work on your PC you will have to

Silly WAT Linker SWL is a tool that makes writing WebAssembly Text files easier. It is future-proof and simple because it doesn’t actually understand

COSMIC Text Pure Rust multi-line text handling. COSMIC Text provides advanced text shaping, layout, and rendering wrapped up into a simple abstraction

Striving to create a great Application with full functions of learning languages by ChatGPT, TTS, STT and other awesome AI models, supports talking, speaking assessment, memorizing words with contexts, Listening test, so on.

llm-chain 🚀 llm-chain is a collection of Rust crates designed to help you work with Large Language Models (LLMs) more effectively. Our primary focus

piet-cosmic-text Implements piet's Text interface using the cosmic-text crate. License piet-cosmic-text is free software: you can redistribute it and/

hips-lib Performs text to image steganography by hinding and retrieving secret text within images or pixel arrays. This is achieved by encoding the se

rut A basic text editor, written in Rust (hence the name). Why, though? I just wanted a basic TUI text editor (like Nano) that could: Be used with all

aarty: A simple CLI tool to convert images to ASCII art with Rust 🦀 More screenshots Original image Original image Original image Original image Orig

📌 This program is designed to translate text into the correct layout when typing is incorrect. 📌 Example ghbdtn - привет Just (by default) pressing

STAM Library STAM is a data model for stand-off text annotation and described in detail here. This is a sofware library to work with the model, writte

A crate for converting an ASCII text string or file to a single unicode character. Also provides a macro to embed encoded source code into a Rust source file. Can also do the same to Python code while still letting the code run as before by wrapping it in a decoder.

console_static_text Crate for logging text that should stay in the same place in a console. This measures words to handle wrapping and has some consol

matchable matchable provides a convenient enum for checking if a piece of text is matching a string or a regex. The common usage of this crate is used

wordcloud-rs A Rust library to generate word-clouds from text and images! Example Code use std::collections::HashMap; use std::fs; use lazy_static::la

ik-rs ik-analyzer for Rust support Tantivy Usage Chinese Segment let mut ik = IKSegmenter::new(); let text = "中华人民共和国"; let tokens = ik.to

Cermic A little program written in Rust that prints random PNG images to the terminal in text grabbed from an image repo. NOTE: As of right now, Cermi

FastSpend Log your daily spending lightning fast with short text snippets! FastSpend is a tool to log your spending in seconds, powered by a lightning

Intuitive docs.rs Documentation Intuitive is a component-based library for creating text-based user interfaces (TUIs) easily. It is heavily inspired b

u8g2-fonts This crate is a pure Rust reimplementation of the font subsystem of U8g2. It is intended for the embedded-graphics ecosystem. Licensing Whi

msdfont WIP - school started so less updates from now on :(( Multi-channel signed distance field (MSDF) generator for fonts implemented in pure Rust.

MSDF Toolkit Higher-level toolkit for MSDF text rendering About MSDF - an abbreviation of Multi-channel Signed Distance Field. In short, an efficient

Calculator A calculator working purely with text inputs. Downloading Desktop Version (Windows + Mac) available in the Releases Tab Web Version availab

kaka (An attempt to write) a modal text editor. NOTE: The project is very young and certainly not ready for use. Current project goals keymap and mode

Kana About Kana is a small CLI program for transliterating romaji text to either hiragana (ひらがな) or katakana (カタカナ). How it works Internally, it uses

python-daachorse daachorse is a fast implementation of the Aho-Corasick algorithm using the compact double-array data structure. This is a Python wrap

A kakoune / neovim inspired editor, written in Rust. The editing model is very heavily based on kakoune; during development I found myself agree

Ruplacer - Find and replace text in source files

Bunt: simple macro-based terminal colors and styles bunt offers macros to easily print colored and formatted text to a terminal. It is just a convenie

Archeum About The Project Archeum is a minimalist text editor that is really usefull if you've been in the vim psychosis for to long. Reject plugins,

Bevy Debug Text Overlay A proof of concept for adding a very convenient text overlay macro to the bevy game engine. This is derived from the code I us

Tantivy is a full-text search engine library written in Rust. It is closer to Apache Lucene than to Elasticsearch or Apache Solr in the sense it is no

Wealthy Rich ported to Rust! This aims to be a crate for rich text and beautiful formatting in the terminal

🦅 glyphon 🦁 Fast, simple 2D text rendering for wgpu What is this? This crate provides a simple way to render 2D text with wgpu by: rasterizing glyph

uwuifyy A robust, customizable, blazingly-fast, efficient and easy-to-use command line application to uwu'ify your text! Logo Credits: Jade Nelson Tab

This project has moved upstream to the GStreamer Rust Plugins: https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/729 Vosk Speec

wgpu-text wgpu-text is a wrapper over glyph-brush for fast and easy text rendering in wgpu. This project was inspired by and is similar to wgpu_glyph,

A blazingly fast command-line tool for converting Chinese punctuations to English punctuations

lipsum-cli is a small terminal application written in Rust language. It's used for generating pseudo-Latin lorem ipsum filler text in terminal.

arpchat so... you know arp? the protocol your computer uses to find the mac addresses of other computers on your network? yeah. that. i thought it wou

dip editor Multi-platform Text editor purely written in Rust, supercharged by Bevy game engine and Dioxus UI framework. heavily in development Why Gam

crispmacs crispmacs is a WIP implementation of Emacs from scratch in Rust. It consists of two parts: crisp and the editor. Crisp crisp is a Lisp that'

cutters A rule based sentence segmentation library. 🚧 This library is experimental. 🚧 Features Full UTF-8 support. Robust parsing. Language specific

Tantivy is a full text search engine library written in Rust. It is closer to Apache Lucene than to Elasticsearch or Apache Solr in the sense it is no

hj hj is a command line tool to convert HTTP/1-style text into JSON. This command is inspired by yusukebe/rj, which is a standalone HTTP client that s

Iridescent Features iridescent is a library for styling terminal text easily. It supports basic ANSI sequences, Xterm-256 colors, and RGB. You can ope

pithy 0.1.0 - an absurdly fast, strangely accurate, summariser Quick example: pithy -f your_file_here.txt --sentences 4 --help: Print this help messa

A Rust wrapper for the Text synthesization service TextSynth API

CinPatent: Datasets for Patent Classification We release two datasets for patent classification in English and Japanese at Google Drive. The data fold

Sentences 2 svg Does what it says on the tin. This takes in a file with some sentences and outputs numbered svgs. There are 3 arguments to keep in min

Deduplicating Training Data Makes Language Models Better This repository contains code to deduplicate language model datasets as descrbed in the paper

probly-search · A full-text search library, optimized for insertion speed, that provides full control over the scoring calculations. This start initia

Making Postgres and Elasticsearch work together like it's 2021 Readme ZomboDB brings powerful text-search and analytics features to Postgres by using

Cursive Cursive is a TUI (Text User Interface) library for rust. It uses ncurses by default, but other backends are available. It allows you to build

UNIC: Unicode and Internationalization Crates for Rust https://github.com/open-i18n/rust-unic UNIC is a project to develop components for the Rust pro

Ginkgo is a text editor built entirely in Rust. It supports cursor movements, CTRL commands, select vim commands, insert vs. normal modes, and more. Ginkgo is based on my text editor JED, which itself was based on the popular online editor Kilo.

Afterglow IntelliJ This theme for IntelliJ is based on the the Afterglow Sublime Text theme, and replaces the default sidebar icons and colour of Inte

README A tool that generates a Sublime Text project file that helps you get started using Scoggle. While Scoggle-Gen may not find every single source

Dye dye is a tool to easily color text in shell. Usage See the gif below to see these commands in action. echo $(dye --red WARN) This tool will knock

rite-cloud A cloud backup solution for the rite text editor. Written in Rust using Tide. Usage You can try out rite-cloud using the instance hosted he

Glyph This is my personal code editor that I am building for fun and to get more familiar with OpenGL. Glyph currently supports Vim keybinds, syntax h

Makdown-like text parser.

WriteForAll: tips to make text better WriteForAll is a text file style checker, that compares text documents with editorial tips to make text better.

Subtext: markup for note-taking Subtext is a text-based, block-oriented hypertext format. It is designed with note-taking in mind. It has a simple, pe

ROAPI ROAPI automatically spins up read-only APIs for static datasets without requiring you to write a single line of code. It builds on top of Apache

Getting Started This is a minimal grep command-line utility built on Rust. It provides searching for plain-text files for lines that match a given str

☑️ Todome （日本語版はこちら） Todome is a notation developed for making and editing to-do lists. It is inspired by Todo.txt, and in fact some of the todome not

lingua-py lingua-rs Python binding. An accurate natural language detection library, suitable for long and short text alike. Installation pip install l

lipi Lipi (Sanskrit for 'writing, letters, alphabet') is a pure Rust crate that provides font independent text analysis support for shaping and layout

🐎 daachorse Daac Horse: Double-Array Aho-Corasick Overview A fast implementation of the Aho-Corasick algorithm using Double-Array Trie. Examples use

Indicium Search 🔎 A simple in-memory search for collections (Vec, HashMap, BTreeMap, etc) and key-value stores. Features autocompletion. There are ma

Colorstyle colorStyle is a library of styles for command-line text. Inspired by flylog/colorstyle (golang) Example let text = colorstyle::green("gre

nom-text Goal: a library that extends nom to provide better tools for text formats (programming languages, configuration files). current needs Recogni

Swash is a pure Rust, cross-platform crate that provides font introspection, complex text shaping and glyph rendering. Goals This crate aims to

buke full text search manpages cargo run --release -- --build builds an sqlite3 database out of all manpages in your $MANPATH cargo run --release -- "

Source text parsing, lexing, and AST related functionality for Deno.

The fastest way to identify anything lemmeknow ⚡ Identify any mysterious text or analyze strings from a file, just ask lemmeknow. lemmeknow can be use

mdzk A lovingly designed system and static publishing tool for your plain text Zettelkasten mdzk is a plain text Zettelkasten system that is based on

ztop Display ZFS datasets' I/O in real time Overview ztop is like top, but for ZFS datasets. It displays the real-time activity for datasets. The buil