Base 32 + 64 encoding and decoding identifiers + bytes in rust, quickly

Related tags

Command-line fast32
Overview

fast32

Base32 and base64 encoding in Rust. Primarily for integer (u64, u128) and UUID identifiers (behind feature uuid), as well as arbitrary byte arrays. And do it all very quickly (more on this below).

The crate has installation instructions, and the documentation has extensive examples.

Encoding integers

Note that by default, encoding an integer into base32 or base64 via normal algorithms does not "look like" a number -- notably the rightmost character usually looks off, and there are sometimes more characters than there needs to be. This might be a plus for obfuscation, barely, but it makes them hard to reason about quickly, and it's also more efficient to process them as integers rather than arbitrary arrays of bytes (because we know upfront that integers are always a small size).

For example, the normal/base10 integer 31 processed normally, as bytes, into official RFC 4648 base32 hex, without padding, will come out as "D4". In contrast, processing it as an integer, as this library can, will come out as "Z" (in Crockford's base32 alphabet) which is more intuitively one less than 32 at "10", as one might hope (vs "EA" in base32 hex -- note that's an A not a 4 so the string changed nonintuitively for an increment of 1). This is helpful with "nice looking" urls of base32 encodings of identifiers, etc.

Speed

This is intended to be as fast as basically possible, while still keeping an intuitive interface.

It is, per the bench comparisons in this repo, on my machine, about 65-100%+ faster than the closest alternative for decoding u64s, and ~15-50%+ faster for encoding u64s, with bigger percent improvements for larger numbers. (That repo does not offer u128 or uuid support to compare against.)

It is also ~35-40% faster on decoding and encoding raw bytes vs the closest other alternative that is more mature but does not offer this crate's integer encoding/decoding, on top of the slower performance. There is an earlier and seemingly long abandoned alternative that is generally about 3x slower than this crate (and thus ~2x slower than the other alternative as well), but still somewhat popular.

Summary

In short, this crate should do everything you want for base32 and base64 encoding (please raise an issue if it doesn't!) while doing all of it very quickly and conveniently.

You might also like...
A command line tool to control the power state of Valve Base Stations 2.0.

lighthousectl A command line tool to control the power state of Valve Base Stations 2.0. Usage Scan All Base Stations It scans endlessly. You can stop

Mod for Mega Man Battle Network Legacy Collection to restore the WWW base music in the postgame.

MMBNLC Postgame WWW Base Music mod This is a mod for Mega Man Battle Network Legacy Collection Vol. 2 adjusts the field music played in the WWW base i

A rust crate to view a structure as raw bytes (&[u8])

rawbytes A Rust crate to view a structure as a plain byte array (&[u8]). Super simple. Tiny. Zero dependencies. This is a safer interface to slice::fr

hexyl is a simple hex viewer for the terminal. It uses a colored output to distinguish different categories of bytes
hexyl is a simple hex viewer for the terminal. It uses a colored output to distinguish different categories of bytes

hexyl is a simple hex viewer for the terminal. It uses a colored output to distinguish different categories of bytes (NULL bytes, printable ASCII characters, ASCII whitespace characters, other ASCII characters and non-ASCII).

A terminal UI to edit bytes by the nibble.
A terminal UI to edit bytes by the nibble.

heh The HEx Helper is a cross-platform terminal UI used for modifying file data in hex or ASCII. It aims to replicate some of the look of hexyl while

A tool to convert old and outdated "characters" into the superior Rustcii-Encoding.

rustcii A tool to convert old and outdated "characters" into the superior Rustcii-Encoding. Speak your mind. Blazingly ( 🦀 ) fast ( 🚀 ). Github | cr

a universal meta-transliterator that can decipher arbitrary encoding schemas, built in pure Rust
a universal meta-transliterator that can decipher arbitrary encoding schemas, built in pure Rust

transliterati a universal meta-transliterator that can decipher arbitrary encoding schemas, built in pure Rust what does it do? You give it this: Барл

Truly universal encoding detector in pure Rust - port of Python version

Charset Normalizer A library that helps you read text from an unknown charset encoding. Motivated by original Python version of charset-normalizer, I'

tmplt is a command-line interface tool that allows you to quickly and easily set up project templates for various programming languages and frameworks
tmplt is a command-line interface tool that allows you to quickly and easily set up project templates for various programming languages and frameworks

tmplt A User Friendly CLI Tool For Creating New Projects With Templates About tmplt is a command-line tool that lets users quickly create new projects

Comments
  • Provide lowercase alphabets

    Provide lowercase alphabets

    I want to encode into lowercase crockford Base32. Currently I use to_ascii_lowercase to do this in-place. Although it avoids character tables and doesn't support crockford encoding, an alternative project, base32ct, provides a means of encoding directly into lowercase base32.

    opened by russellbanks 4
  • Unexpected/unhandled error when decoding.

    Unexpected/unhandled error when decoding.

    Hello, I was using this library to check if a number was in base32 using the decode function.

    fn is_base32_number(s: &str) -> bool {
        match RFC4648.decode(s.as_bytes()) {
            Ok(_) => true,
            Err(_) => false,
        }
    }
    

    When I pass a string that has a length less than 6 to the function, it panics with a "attempt to subtract with overflow" without throwing a DecodeError, which is caused by the following function: https://github.com/rogusdev/fast32/blob/e9d804855ec8659e056c0e760ba88d8b5c65f6ea/src/base32/alphabet.rs#L154, more specifically where it tries to index a[len - 6].

    I made what i feel like is a bandaid fix, but I have only been writing rust for like a week or 2 so I am not sure if it follows the best conventions and guidelines.

    I am sure this is not an intended function of decode method, but from what I have been reading it would follow Rust conventions to make it as safe as possible.

    opened by enlightened 1
Releases(v1.0.2)
Owner
Chris Rogus
Chris Rogus
Costless typed identifiers backed by UUID, with kind readable in serialized versions

Why kind ? With kind, you use typed identifiers in Rust, with no overhead over Uuid have the type be human readable and obvious in JSON and any export

Wingback 67 Mar 24, 2024
Databento Binary Encoding (DBZ) - Fast message encoding and storage format for market data

dbz A library (dbz-lib) and CLI tool (dbz-cli) for working with Databento Binary Encoding (DBZ) files. Python bindings for dbz-lib are provided in the

Databento, Inc. 15 Nov 4, 2022
An ebpf knowledge base, based on llama_index and bpf-developer-tutorial

ebpf-knowledge-base An ebpf knowledge base, based on llama_index and bpf-developer-tutorial Usage First, you need to clone this repo: git clone --recu

eunomia-bpf 7 Apr 1, 2023
AskBend: SQL-based Knowledge Base Search and Completion using Databend

AskBend: SQL-based Knowledge Base Search and Completion using Databend AskBend is a Rust project that utilizes the power of Databend and OpenAI to cre

Databend Labs 87 Apr 7, 2023
Encode and decode dynamically constructed values of arbitrary shapes to/from SCALE bytes

scale-value · This crate provides a Value type, which is a runtime representation that is compatible with scale_info::TypeDef. It somewhat analogous t

Parity Technologies 15 Jun 24, 2023
Patch binary file using IDA signatures and defined replacement bytes in YAML.

fabricbin Patch binary file using IDA signatures and defined replacement bytes in YAML. Install: cargo install --git https://github.com/makindotcc/fab

makin 3 Oct 24, 2023
Rust CS:GO base

Helveta Counter-Strike: Global Offensive Cheat base written in Rust. NOT COMPLETE! In active development. Disclaimer There's an alternative, currently

cristei 3 Sep 4, 2021
Rust implementation of custom numeric base conversion.

base_custom Use any characters as your own numeric base and convert to and from decimal. This can be taken advantage of in various ways: Mathematics:

Daniel P. Clark 5 Dec 28, 2021
This repository presents a numbers vizualizer in a polar base. This small project has been entirely made in Rust !

NumbersRepresentation This repository presents a numbers vizualizer in a polar base. This small project has been entirely made in Rust ! This is an id

Lilian 'S3l4h' Schall 3 Apr 12, 2022
CLI tool to convert numbers from one base to another

changebase A CLI tool for changing the base of numbers. > changebase -h numeric base converter USAGE: changebase [FLAGS] [OPTIONS] <value> FLAG

null 2 Oct 14, 2022