the official Rust and C implementations of the BLAKE3 cryptographic hash function

Related tags

Cryptography BLAKE3
Overview

BLAKE3

BLAKE3 is a cryptographic hash function that is:

  • Much faster than MD5, SHA-1, SHA-2, SHA-3, and BLAKE2.
  • Secure, unlike MD5 and SHA-1. And secure against length extension, unlike SHA-2.
  • Highly parallelizable across any number of threads and SIMD lanes, because it's a Merkle tree on the inside.
  • Capable of verified streaming and incremental updates, again because it's a Merkle tree.
  • A PRF, MAC, KDF, and XOF, as well as a regular hash.
  • One algorithm with no variants, which is fast on x86-64 and also on smaller architectures.

The chart below is an example benchmark of 16 KiB inputs on modern server hardware (a Cascade Lake-SP 8275CL processor). For more detailed benchmarks, see the BLAKE3 paper.

performance graph

BLAKE3 is based on an optimized instance of the established hash function BLAKE2 and on the original Bao tree mode. The specifications and design rationale are available in the BLAKE3 paper. The default output size is 256 bits. The current version of Bao implements verified streaming with BLAKE3.

This repository is the official implementation of BLAKE3. It includes:

  • The blake3 Rust crate, which includes optimized implementations for SSE2, SSE4.1, AVX2, AVX-512, and NEON, with automatic runtime CPU feature detection on x86. The rayon feature provides multithreading.

  • The b3sum Rust crate, which provides a command line interface. It uses multithreading by default, making it an order of magnitude faster than e.g. sha256sum on typical desktop hardware.

  • The C implementation, which like the Rust implementation includes SIMD code and runtime CPU feature detection on x86. Unlike the Rust implementation, it's not currently multithreaded. See c/README.md.

  • The reference implementation, which is discussed in Section 5.1 of the BLAKE3 paper. This implementation is much smaller and simpler than the optimized ones above. If you want to see how BLAKE3 works, or you're writing a port that doesn't need multithreading or SIMD optimizations, start here.

  • A set of test vectors that covers extended outputs, all three modes, and a variety of input lengths.

  • Actions Status

BLAKE3 was designed by:

The development of BLAKE3 was sponsored by Electric Coin Company.

NOTE: BLAKE3 is not a password hashing algorithm, because it's designed to be fast, whereas password hashing should not be fast. If you hash passwords to store the hashes or if you derive keys from passwords, we recommend Argon2.

Usage

The b3sum utility

The b3sum command line utility prints the BLAKE3 hashes of files or of standard input. Prebuilt binaries are available for Linux, Windows, and macOS (requiring the unidentified developer workaround) on the releases page. If you've installed Rust and Cargo, you can also build b3sum yourself with:

cargo install b3sum

If rustup didn't configure your PATH for you, you might need to go looking for the installed binary in e.g. ~/.cargo/bin. You can test out how fast BLAKE3 is on your machine by creating a big file and hashing it, for example:

# Create a 1 GB file.
head -c 1000000000 /dev/zero > /tmp/bigfile
# Hash it with SHA-256.
time openssl sha256 /tmp/bigfile
# Hash it with BLAKE3.
time b3sum /tmp/bigfile

The blake3 crate docs.rs

To use BLAKE3 from Rust code, add a dependency on the blake3 crate to your Cargo.toml. Here's an example of hashing some input bytes:

// Hash an input all at once.
let hash1 = blake3::hash(b"foobarbaz");

// Hash an input incrementally.
let mut hasher = blake3::Hasher::new();
hasher.update(b"foo");
hasher.update(b"bar");
hasher.update(b"baz");
let hash2 = hasher.finalize();
assert_eq!(hash1, hash2);

// Extended output. OutputReader also implements Read and Seek.
let mut output = [0; 1000];
let mut output_reader = hasher.finalize_xof();
output_reader.fill(&mut output);
assert_eq!(&output[..32], hash1.as_bytes());

// Print a hash as hex.
println!("{}", hash1);

Besides hash, BLAKE3 provides two other modes, keyed_hash and derive_key. The keyed_hash mode takes a 256-bit key:

// MAC an input all at once.
let example_key = [42u8; 32];
let mac1 = blake3::keyed_hash(&example_key, b"example input");

// MAC incrementally.
let mut hasher = blake3::Hasher::new_keyed(&example_key);
hasher.update(b"example input");
let mac2 = hasher.finalize();
assert_eq!(mac1, mac2);

The derive_key mode takes a context string and some key material (not a password). The context string should be hardcoded, globally unique, and application-specific. A good default format for the context string is "[application] [commit timestamp] [purpose]":

// Derive a couple of subkeys for different purposes.
const EMAIL_CONTEXT: &str = "BLAKE3 example 2020-01-07 17:10:44 email key";
const API_CONTEXT: &str = "BLAKE3 example 2020-01-07 17:11:21 API key";
let input_key_material = b"usually at least 32 random bytes, not a password";
let email_key = blake3::derive_key(EMAIL_CONTEXT, input_key_material);
let api_key = blake3::derive_key(API_CONTEXT, input_key_material);
assert_ne!(email_key, api_key);

The C implementation

See c/README.md.

Other implementations

We post links to third-party bindings and implementations on the @BLAKE3team Twitter account whenever we hear about them. Some highlights include an optimized Go implementation, Wasm bindings for Node.js and browsers, binary wheels for Python, .NET bindings, and JNI bindings.

Contributing

Please see CONTRIBUTING.md.

Intellectual property

The Rust code is copyright Jack O'Connor, 2019-2020. The C code is copyright Samuel Neves and Jack O'Connor, 2019-2020. The assembly code is copyright Samuel Neves, 2019-2020.

This work is released into the public domain with CC0 1.0. Alternatively, it is licensed under the Apache License 2.0.

Miscellany

Comments
  • impl digest::* for Blake3?

    impl digest::* for Blake3?

    It'd be really nice for BLAKE3 to follow RustCrypto's digest crate to allow for BLAKE3 to substitute other functions in generic code.

    ~~Hasher is a poor name here, Blake3 would be cleaner. I expect to see many instances of use blake3::Hasher as Blake3. A type alias suffices to avoid a breaking change type Hasher = Blake3~~

    These traits can be implemented without a breaking change. However, it'd be cleaner to only implement digest's traits and any other breaking changes could be batched. The release is early and much more usage will come with the digest traits implemented.

    Great work BLAKE3-team!

    opened by WildCryptoFox 31
  • Hasher::new_derive_key accepts a str, why not [u8]?

    Hasher::new_derive_key accepts a str, why not [u8]?

    There is no reason for UTF-8 here. Please s/str/[u8]/ here. Alternatively provide another function to avoid the breaking change. Though, as I've mentioned in #11 I'm not against a breaking change for consistency with other crates, especially this early.

    https://docs.rs/blake3/0.1.0/blake3/struct.Hasher.html#method.new_derive_key

    pub fn new_derive_key(context: &str) -> Self
    
    opened by WildCryptoFox 21
  • Compilation error on Windows

    Compilation error on Windows

    Similar to issue https://github.com/BLAKE3-team/BLAKE3/issues/79, I'm getting compilation errors when building blake3 v0.3.7 on Windows:

    [2021-01-13T09:05:58.008Z] The following warnings were emitted during compilation:
    [2021-01-13T09:05:58.008Z] 
    [2021-01-13T09:05:58.008Z] warning: The C compiler "C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin\\amd64\\cl.exe" does not support /arch:AVX512.
    [2021-01-13T09:05:58.008Z] 
    [2021-01-13T09:05:58.008Z] error: failed to run custom build command for `blake3 v0.3.7`
    [2021-01-13T09:05:58.008Z] 
    [2021-01-13T09:05:58.008Z] Caused by:
    [2021-01-13T09:05:58.008Z]   process didn't exit successfully: `C:\Users\ci\AppData\Local\Temp\cargo-installMihItV\release\build\blake3-98f585247d671863\build-script-build` (exit code: 1)
    [2021-01-13T09:05:58.008Z]   --- stdout
    [2021-01-13T09:05:58.008Z]   cargo:rerun-if-env-changed=CARGO_FEATURE_PURE
    [2021-01-13T09:05:58.008Z]   TARGET = Some("x86_64-pc-windows-msvc")
    [2021-01-13T09:05:58.008Z]   HOST = Some("x86_64-pc-windows-msvc")
    [2021-01-13T09:05:58.008Z]   CC_x86_64-pc-windows-msvc = None
    [2021-01-13T09:05:58.008Z]   CC_x86_64_pc_windows_msvc = None
    [2021-01-13T09:05:58.008Z]   HOST_CC = None
    [2021-01-13T09:05:58.008Z]   CC = None
    [2021-01-13T09:05:58.008Z]   CFLAGS_x86_64-pc-windows-msvc = None
    [2021-01-13T09:05:58.008Z]   CFLAGS_x86_64_pc_windows_msvc = None
    [2021-01-13T09:05:58.008Z]   HOST_CFLAGS = None
    [2021-01-13T09:05:58.008Z]   CFLAGS = None
    [2021-01-13T09:05:58.008Z]   CRATE_CC_NO_DEFAULTS = None
    [2021-01-13T09:05:58.008Z]   CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
    [2021-01-13T09:05:58.008Z]   OPT_LEVEL = Some("3")
    [2021-01-13T09:05:58.008Z]   CC_x86_64-pc-windows-msvc = None
    [2021-01-13T09:05:58.008Z]   CC_x86_64_pc_windows_msvc = None
    [2021-01-13T09:05:58.008Z]   HOST_CC = None
    [2021-01-13T09:05:58.008Z]   CC = None
    [2021-01-13T09:05:58.008Z]   CFLAGS_x86_64-pc-windows-msvc = None
    [2021-01-13T09:05:58.008Z]   CFLAGS_x86_64_pc_windows_msvc = None
    [2021-01-13T09:05:58.008Z]   HOST_CFLAGS = None
    [2021-01-13T09:05:58.008Z]   CFLAGS = None
    [2021-01-13T09:05:58.008Z]   CRATE_CC_NO_DEFAULTS = None
    [2021-01-13T09:05:58.009Z]   CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
    [2021-01-13T09:05:58.009Z]   DEBUG = Some("false")
    [2021-01-13T09:05:58.009Z]   cargo:warning=The C compiler "C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin\\amd64\\cl.exe" does not support /arch:AVX512.
    [2021-01-13T09:05:58.009Z]   cargo:rerun-if-env-changed=BLAKE3_CI
    [2021-01-13T09:05:58.009Z]   cargo:rerun-if-env-changed=CARGO_FEATURE_PREFER_INTRINSICS
    [2021-01-13T09:05:58.009Z]   cargo:rerun-if-env-changed=CARGO_FEATURE_PURE
    [2021-01-13T09:05:58.009Z]   cargo:rustc-cfg=blake3_sse2_ffi
    [2021-01-13T09:05:58.009Z]   cargo:rustc-cfg=blake3_sse41_ffi
    [2021-01-13T09:05:58.009Z]   cargo:rustc-cfg=blake3_avx2_ffi
    [2021-01-13T09:05:58.009Z]   TARGET = Some("x86_64-pc-windows-msvc")
    [2021-01-13T09:05:58.009Z]   OPT_LEVEL = Some("3")
    [2021-01-13T09:05:58.009Z]   HOST = Some("x86_64-pc-windows-msvc")
    [2021-01-13T09:05:58.009Z]   CC_x86_64-pc-windows-msvc = None
    [2021-01-13T09:05:58.009Z]   CC_x86_64_pc_windows_msvc = None
    [2021-01-13T09:05:58.009Z]   HOST_CC = None
    [2021-01-13T09:05:58.009Z]   CC = None
    [2021-01-13T09:05:58.009Z]   CFLAGS_x86_64-pc-windows-msvc = None
    [2021-01-13T09:05:58.009Z]   CFLAGS_x86_64_pc_windows_msvc = None
    [2021-01-13T09:05:58.009Z]   HOST_CFLAGS = None
    [2021-01-13T09:05:58.009Z]   CFLAGS = None
    [2021-01-13T09:05:58.009Z]   CRATE_CC_NO_DEFAULTS = None
    [2021-01-13T09:05:58.009Z]   CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
    [2021-01-13T09:05:58.009Z]   DEBUG = Some("false")
    [2021-01-13T09:05:58.009Z]   running: "C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin\\amd64\\ml64.exe" "-nologo" "-FoC:\\Users\\ci\\AppData\\Local\\Temp\\cargo-installMihItV\\release\\build\\blake3-86dd5a2a633273e8\\out\\c/blake3_sse2_x86-64_windows_msvc.o" "-c" "c/blake3_sse2_x86-64_windows_msvc.asm"
    [2021-01-13T09:05:58.009Z]    Assembling: c/blake3_sse2_x86-64_windows_msvc.asm
    [2021-01-13T09:05:58.009Z]   c/blake3_sse2_x86-64_windows_msvc.asm(2057)pecified size
    [2021-01-13T09:05:58.009Z]   c/blake3_sse2_x86-64_windows_msvc.asm(2057) : error A2070:invalid instrucc/blake3_sse2_x86-64_windows_msvc.asm(2058)pecified size
    [2021-01-13T09:05:58.009Z]   c/blake3_sse2_x86-64_windows_msvc.asm(2058) : error A2070:invalid instrucc/blake3_sse2_x86-64_windows_msvc.asm(2189)pecified size
    [2021-01-13T09:05:58.009Z]   c/blake3_sse2_x86-64_windows_msvc.asm(2189) : error A2070:invalid instrucc/blake3_sse2_x86-64_windows_msvc.asm(2190)pecified size
    [2021-01-13T09:05:58.009Z]   c/blake3_sse2_x86-64_windows_msvc.asm(2190) : error A2070:invalid instrucexit code: 1
    [2021-01-13T09:05:58.009Z]   running: "C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin\\amd64\\ml64.exe" "-nologo" "-FoC:\\Users\\ci\\AppData\\Local\\Temp\\cargo-installMihItV\\release\\build\\blake3-86dd5a2a633273e8\\out\\c/blake3_sse41_x86-64_windows_msvc.o" "-c" "c/blake3_sse41_x86-64_windows_msvc.asm"
    [2021-01-13T09:05:58.009Z]    Assembling: c/blake3_sse41_x86-64_windows_msvc.asm
    [2021-01-13T09:05:58.009Z]   c/blake3_sse41_x86-64_windows_msvc.asm(1820)specified size
    [2021-01-13T09:05:58.009Z]   c/blake3_sse41_x86-64_windows_msvc.asm(1820) : error A2070:invalid instruc/blake3_sse41_x86-64_windows_msvc.asm(1821)specified size
    [2021-01-13T09:05:58.009Z]   c/blake3_sse41_x86-64_windows_msvc.asm(1821) : error A2070:invalid instruc/blake3_sse41_x86-64_windows_msvc.asm(1941)specified size
    [2021-01-13T09:05:58.009Z]   c/blake3_sse41_x86-64_windows_msvc.asm(1941) : error A2070:invalid instruc/blake3_sse41_x86-64_windows_msvc.asm(1942)specified size
    [2021-01-13T09:05:58.009Z]   c/blake3_sse41_x86-64_windows_msvc.asm(1942) : error A2070:invalid instruexit code: 1
    

    I'm actually trying to build sccache 0.2.15, and blake3 is a dependency (or transient dependency?) of that, so forgive me for my unfamiliarity with this project. I've checked the open issues/PRs and didn't see this reported yet.

    opened by nre-ableton 18
  • AVX-512 compile error on macOS Clang

    AVX-512 compile error on macOS Clang

    Describe the bug

    Clean clone of Wasmer fails to compile on MacOS due to AVX-512 error when compiling assembly code.

    The actual compilation command that fails (full compilation log below):

    "cc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c11" "-o" "wasmer/target/debug/build/blake3-da163d92031ce8e8/out/c/blake3_avx512_x86-64_unix.o" "-c" "c/blake3_avx512_x86-64_unix.S"

    cargo:warning=c/blake3_avx512_x86-64_unix.S:2315:9: error: instruction requires: AVX-512 ISA AVX-512 VL ISA
    cargo:warning=        vpxord xmm3, xmm3, xmm0
    

    Compilation works if -march=skylake-avx512 is added to command above, the error happens because Clang by default does not enable AVX-512 intrinsics since they are not supported by my CPU.

    Additional context

    Full compilation log:

    error: failed to run custom build command for `blake3 v0.3.1`
    
    Caused by:
      process didn't exit successfully: `wasmer/target/debug/build/blake3-32d62a0f85ffa978/build-script-build` (exit code: 1)
    --- stdout
    cargo:rerun-if-env-changed=CARGO_FEATURE_PURE
    cargo:rerun-if-env-changed=CARGO_FEATURE_PURE
    TARGET = Some("x86_64-apple-darwin")
    HOST = Some("x86_64-apple-darwin")
    CC_x86_64-apple-darwin = None
    CC_x86_64_apple_darwin = None
    HOST_CC = None
    CC = None
    CFLAGS_x86_64-apple-darwin = None
    CFLAGS_x86_64_apple_darwin = None
    HOST_CFLAGS = None
    CFLAGS = None
    CRATE_CC_NO_DEFAULTS = None
    CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2,sse3,ssse3")
    CC_x86_64-apple-darwin = None
    CC_x86_64_apple_darwin = None
    HOST_CC = None
    CC = None
    CFLAGS_x86_64-apple-darwin = None
    CFLAGS_x86_64_apple_darwin = None
    HOST_CFLAGS = None
    CFLAGS = None
    CRATE_CC_NO_DEFAULTS = None
    CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2,sse3,ssse3")
    cargo:rerun-if-env-changed=CARGO_FEATURE_PREFER_INTRINSICS
    TARGET = Some("x86_64-apple-darwin")
    HOST = Some("x86_64-apple-darwin")
    CC_x86_64-apple-darwin = None
    CC_x86_64_apple_darwin = None
    HOST_CC = None
    CC = None
    CFLAGS_x86_64-apple-darwin = None
    CFLAGS_x86_64_apple_darwin = None
    HOST_CFLAGS = None
    CFLAGS = None
    CRATE_CC_NO_DEFAULTS = None
    CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2,sse3,ssse3")
    CC_x86_64-apple-darwin = None
    CC_x86_64_apple_darwin = None
    HOST_CC = None
    CC = None
    CFLAGS_x86_64-apple-darwin = None
    CFLAGS_x86_64_apple_darwin = None
    HOST_CFLAGS = None
    CFLAGS = None
    CRATE_CC_NO_DEFAULTS = None
    CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2,sse3,ssse3")
    cargo:rerun-if-env-changed=CARGO_FEATURE_PURE
    cargo:rerun-if-env-changed=CARGO_FEATURE_PREFER_INTRINSICS
    cargo:rustc-cfg=blake3_sse41_ffi
    cargo:rustc-cfg=blake3_avx2_ffi
    cargo:rustc-cfg=blake3_avx512_ffi
    TARGET = Some("x86_64-apple-darwin")
    OPT_LEVEL = Some("0")
    HOST = Some("x86_64-apple-darwin")
    CC_x86_64-apple-darwin = None
    CC_x86_64_apple_darwin = None
    HOST_CC = None
    CC = None
    CFLAGS_x86_64-apple-darwin = None
    CFLAGS_x86_64_apple_darwin = None
    HOST_CFLAGS = None
    CFLAGS = None
    CRATE_CC_NO_DEFAULTS = None
    DEBUG = Some("true")
    CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2,sse3,ssse3")
    running: "cc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c11" "-o" "wasmer/target/debug/build/blake3-da163d92031ce8e8/out/c/blake3_sse41_x86-64_unix.o" "-c" "c/blake3_sse41_x86-64_unix.S"
    exit code: 0
    running: "cc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c11" "-o" "wasmer/target/debug/build/blake3-da163d92031ce8e8/out/c/blake3_avx2_x86-64_unix.o" "-c" "c/blake3_avx2_x86-64_unix.S"
    exit code: 0
    running: "cc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c11" "-o" "wasmer/target/debug/build/blake3-da163d92031ce8e8/out/c/blake3_avx512_x86-64_unix.o" "-c" "c/blake3_avx512_x86-64_unix.S"
    cargo:warning=c/blake3_avx512_x86-64_unix.S:28:9: error: instruction requires: AVX-512 ISA
    cargo:warning=        kmovw k1, r9d
    cargo:warning=        ^
    cargo:warning=c/blake3_avx512_x86-64_unix.S:38:9: error: instruction requires: AVX-512 ISA AVX-512 VL ISA
    cargo:warning=        vpcmpltud k2, ymm2, ymm0
    cargo:warning=        ^
    
    rustc 1.42.0 (b8cedc004 2020-03-09) | x86_64
    

    Output of sysctl -a | grep machdep.cpu.features:

    machdep.cpu.features: FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE SSE3 PCLMULQDQ DTES64 MON DSCPL VMX EST TM2 SSSE3 FMA CX16 TPR PDCM SSE4.1 SSE4.2 x2APIC MOVBE POPCNT AES PCID XSAVE OSXSAVE SEGLIM64 TSCTMR AVX1.0 RDRAND F16C
    

    To note, no support for AVX-512.

    Related issue from Wasmer: https://github.com/wasmerio/wasmer/issues/1373

    opened by tritao 18
  • Excessive futex syscalling with no-mmap

    Excessive futex syscalling with no-mmap

    Problem: b3sum --no-mmap is using a tiny read buffer size (8KB) and syscalls read(2) and futex(2) aggressively. I find that hashing at only 110 MB/s maxes out all 24 cores, doing about 10k syscalls per second (tail -f strace.log | pv -l >/dev/null).

    top

    CPU% MEM%   TIME+  Command                                                                                                                                                                                                           
    2025  0.0 22:58.81 b3sum --no-mmap idsab.tar.zst
    

    strace.log

    futex(0x563438b1804c, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0x563438b17ff0, FUTEX_WAKE_PRIVATE, 1) = 0
    read(3, "\3\352\t\225\213r\371\3421\231\361\303\306\237\252\362\274\212>\33\t\f&\240u\31\252\331\310\242\265T"..., 8192) = 8192
    futex(0x563438b14028, FUTEX_WAKE_PRIVATE, 2147483647) = 2
    futex(0x563438b13fd0, FUTEX_WAKE_PRIVATE, 1) = 1
    read(3, "9\22\373\5\17\357\35\0364\35\371{\324\220\275J~B0$\273\332\264\20\v\352\304\23\301l\316\320"..., 8192) = 8192
    futex(0x563438b18048, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
    futex(0x563438b17ff0, FUTEX_WAKE_PRIVATE, 1) = 0
    read(3, "\225/\247\216\367Q\375,E\357\305\367\323\321\2\223f\266\n\222\257\305\260qLJ\f\247\370\202\6\306"..., 8192) = 8192
    futex(0x563438b1402c, FUTEX_WAKE_PRIVATE, 2147483647) = 1
    futex(0x563438b13fd0, FUTEX_WAKE_PRIVATE, 1) = 1
    read(3, "7\365\30n\370K\f\265\200\363r7\35E\321\347_\226\351-\331\24\232\200\352\233\316W\20\2121\264"..., 8192) = 8192
    futex(0x563438b14028, FUTEX_WAKE_PRIVATE, 2147483647) = 2
    futex(0x563438b13fd0, FUTEX_WAKE_PRIVATE, 1) = 1
    read(3, "\234\235\250\352\315\343*\t\215\205o(\231\230\325\f\222\257o\302A3\315z\240}\364'\327\33Wj"..., 8192) = 8192
    ... and so on
    

    Environment:

    • Built with cargo install b3sum (as of 2020-02-03)
    • Stable cargo 1.41.0 (626f0f40e 2019-12-03)
    • ZFS on Linux
    • Ubuntu 20.04
    • Kernel 5.4.0-9-generic
    opened by terorie 18
  • OID Value

    OID Value

    Currently I use this fake OID value (next available one after blake2 family) but we probably need an official one to be defined

    1 3 6 1 4 1 1722 12 2 3 8 : BLAKE3 : blake3

    Reference: http://www.oid-info.com/cgi-bin/display?oid=1+3+6+1+4+1+1722+12+2&action=display

    opened by dlegaultbbry 15
  • Attempts to implement the blake3 hash algorithm using the standard 2020 programming language, c++, have consistently turned out to be incorrect.

    Attempts to implement the blake3 hash algorithm using the standard 2020 programming language, c++, have consistently turned out to be incorrect.

    I am a cryptography-loving c++ technician, and I have always wanted to implement a simple and portable future c++ version of the Blake3 hash algorithm. But I don't know much about the Rust programming language, and other people only want to use this code base, or just do the code adaptation of the programming language binding interface, but few have understood the Blake3 hash algorithm and contributed code to the open source community. I hope that the official or others can point out my mistakes in the implementation of the Blake3 hash algorithm code understanding, I will contribute my own small capacity, and thank everyone who helped me.

    If you are interested in my project please follow my Include code base for Applied Cryptography https://github.com/Twilight-Dream-Of-Magic/TDOM-EncryptOrDecryptFile-Reborn

    @oconnor663 Could you please explain the question I asked

    opened by Twilight-Dream-Of-Magic 14
  • BLAKE3 collision resistance

    BLAKE3 collision resistance

    How collision resistant are BLAKE3 hashes of different length comparing to the following table from Wikipedia?

    Hash function | Security claim | Best attack | Publish date | Comment -- | -- | -- | -- | -- SHA256 | 2128 | 31 of 64 rounds (265.5) | 2013-05-28 | Two-block collision.[3] SHA512 | 2256 | 24 of 80 rounds (232.5) | 2008-11-25 | Paper.[4] SHA-3 | Up to 2512 | 6 of 24 rounds (250) | 2017 | Paper.[5] BLAKE2s | 2128 | 2.5 of 10 rounds (2112) | 2009-05-26 | Paper.[6] BLAKE2b | 2256 | 2.5 of 12 rounds (2224) | 2009-05-26 | Paper.[6]

    I am especially interested to compare BLAKE3 128 to BLAKE2s 128 as it is the digest size (#107) I'd like to use.

    opened by abitrolly 14
  • Fix differing behavior in C implementation on big-endian platforms

    Fix differing behavior in C implementation on big-endian platforms

    Hello,

    J-P did not express much interest for “exotic” platforms, so @jakub-zwolakowski only configured the C implementation to be interpreted on a 32-bit big-endian and on a 64-bit big-endian platform, in addition to x86 and x86-64. The chosen platforms, ppc32 and ppc64, have the advantage of defining char as an unsigned integer type, so they provide one more portability check for the same price (BLAKE3 is not written to rely on the signedness of char, but since the type char appears in the prototypes of standard functions, it is difficult to be absolutely sure).

    Long story short, there was no Undefined Behavior for these big-endian platforms but they were computing different digests than the little-endian platforms. One possibility is that a bug in TrustInSoft CI's emulation of these big-endian architectures causes the execution to be imperfectly emulated, but that does not seem very likely, because any ordinary bug would already be visible for shorter plaintexts. Another possibility is that although the C implementation of BLAKE3 is written to behave the same on little- and big-endian platforms with functions named load32 and store32, one spot where these functions were necessary was missed.

    The tests that show the wrong digest being computed are: https://dev.ci.trust-in-soft.com/projects/jakub-zwolakowski/BLAKE3/37?page=1

    The same tests, with the change in this PR applied, compute the correct digest in: https://ci.trust-in-soft.com/projects/pascal-cuoq/BLAKE3/2?page=4

    I expect the change to make more sense to you when you see it than it did to me when writing it. If the change does not seem to make sense, then we can investigate further.

    opened by pascal-cuoq 14
  • b3sum incredibly slow

    b3sum incredibly slow

    b3sum runs incredibly slow.

    PC: Windows 10 Pro i7-6700 32GB RAM

    b3sum built using: cargo install b3sum (v0.1.2)

    Test File: 2.88GB

    Results:

    md5sum: 10 seconds
    b2sum: 5 seconds
    b3sum: 4 Minutes 37 Seconds
    
    opened by tERyceNzAchE 13
  • Fake warning on Fedora 34 (in fact newer gcc version 11.2.1 20210728)

    Fake warning on Fedora 34 (in fact newer gcc version 11.2.1 20210728)

    I found "classical" fake warning on newer gcc

    [root@fedora 47]# make
    g++ -Dunix -O3 -march=native zpaqfranz.cpp -o zpaqfranz -pthread -static
    In function ‘size_t compress_parents_parallel(const uint8_t*, size_t, const uint32_t*, uint8_t, uint8_t*)’,
        inlined from ‘void compress_subtree_to_parent_node(const uint8_t*, size_t, const uint32_t*, uint64_t, uint8_t, uint8_t*)’ at zpaqfranz.cpp:9425:34,
        inlined from ‘void blake3_hasher_update.part.0(blake3_hasher*, const void*, size_t)’ at zpaqfranz.cpp:9609:38:
    zpaqfranz.cpp:9313:11: warning: ‘void* memcpy(void*, const void*, size_t)’ writing 32 bytes into a region of size 0 overflows the destination [-Wstringop-overflow=]
     9313 |     memcpy(&out[parents_array_len * BLAKE3_OUT_LEN],
          |     ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     9314 |            &child_chaining_values[2 * parents_array_len * BLAKE3_OUT_LEN],
          |            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     9315 |            BLAKE3_OUT_LEN);
          |            ~~~~~~~~~~~~~~~
    zpaqfranz.cpp: In function ‘void blake3_hasher_update.part.0(blake3_hasher*, const void*, size_t)’:
    zpaqfranz.cpp:9422:11: note: at offset 32 into destination object ‘out_array’ of size 32
     9422 |   uint8_t out_array[MAX_SIMD_DEGREE_OR_2 * BLAKE3_OUT_LEN / 2];
          |           ^~~~~~~~~
    In function ‘void compress_subtree_to_parent_node(const uint8_t*, size_t, const uint32_t*, uint64_t, uint8_t, uint8_t*)’,
        inlined from ‘void blake3_hasher_update.part.0(blake3_hasher*, const void*, size_t)’ at zpaqfranz.cpp:9609:38:
    zpaqfranz.cpp:9426:11: warning: ‘void* memcpy(void*, const void*, size_t)’ reading between 64 and 96 bytes from a region of size 32 [-Wstringop-overread]
     9426 |     memcpy(cv_array, out_array, num_cvs * BLAKE3_OUT_LEN);
          |     ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    zpaqfranz.cpp: In function ‘void blake3_hasher_update.part.0(blake3_hasher*, const void*, size_t)’:
    zpaqfranz.cpp:9422:11: note: source object ‘out_array’ of size 32
     9422 |   uint8_t out_array[MAX_SIMD_DEGREE_OR_2 * BLAKE3_OUT_LEN / 2];
          |           ^~~~~~~~~
    

    may I suggest some code refactoring (just to do not get those spurious warnings)?

    It happens quite frequently that I have to change perfectly functional code so as not to have to "fight" with compilers often full of bugs in detecting "memory" errors.
    It's not that important, just a hint

    PS Newer Linux-based g++ runs BLAKE3 two time faster then "not so old" g++, on AMD.
    Yes, two time faster, not 2%!

    opened by fcorbelli 12
  • Proofs of reads and edits to hashed values.

    Proofs of reads and edits to hashed values.

    This is more discussion than issue.

    Interestingly, blake3 could support merkle proofs of "edits" to hashed values.

    I suppose one could implement this vaguely like:

    struct CoPathElement {
        depth: u8,
        right_v_left: bool,
        intermediate_hash: [u8; 32],
    }
    
    pub struct WireProof {
        b///Segments runs from key to key+value.len()
        read_segments: BTreeMap<u64,Box<[u8]>>,
        copath_accumulator: Vec<CoPathElement>,
    }
    
    pub struct HasherVerifier {
        proof: WireProof,
        written_segments: BTreeMap<u64,Box<[u8]>>,
    }
    
    pub struct HasherProver {
        hasher: Hasher,
        proof: HasherVerifier,
        unfinished_read_segments: ...
    }
    

    HasherProver has most of the methods of Hasher, except update methods add an argument reading: bool which determines if they save copath elements and read_segments.

    HasherProver and HasherVerifier both permits reads freely from their read_segments, but this method preferentially returns the equivalent written_segments array, if one exists. Also both permit writes into written_segments, cloning the read_segments element when none exists, but they error if you write into places for which no read_segment exists.

    HasherProver and HasherVerifier both have append and append_at methods that reconstruct and return a Hasher, at the end or at some internal segment location by working off copath data. You can finalize from these.

    HasherProver has the standard finalize methods, as well as finalize_at methods that work off the copath data and segments.

    HasherProver has some export WireProof method, which adds copath elements for the final state.

    I've maybe missed some things here, but it does not look as complex as one might think. A HasherProver must process all data being hashed here of course. One could avoid this with more complex internal representation that cashes intermediate nodes, but that's a radical change, whereas this flavor could be bolted onto blake3 as is.

    opened by burdges 0
  • Is it possible to get a complete go implementation by using golang call rust?

    Is it possible to get a complete go implementation by using golang call rust?

    Hello, I know there is an incomplete implementation of blakes by golang, well it doesn't support mutli-thread, nor full interfaces. I' am wondering if I can implement blake3 of golang by using golang to call rust, with FFI or CGO. I am going to use BLAKE3 in a golang project receiving file stream. I am not familiar with rust, Is there any difficulty to do that?

    opened by shootingstar123 5
  • Use global_asm to include the sse/avx impls

    Use global_asm to include the sse/avx impls

    This PR replaces uses Rust's global_asm to include the sse2/sse41/avx2/avx512 assembly functions used by blake3. This is useful to reduce the toolchain requirements of this crates' user, which is especially nice for cross-compilation purposes. We no longer require a working C compiler/assembler to get the fast asm version of blake3 on x86_64!

    You'll notice this MR commits a certain amount of code crimes to get it working:

    1. It will always take the windows_gnu.S version of the code, as it doesn't require running the preprocessor on it. However, this means it always uses the win64 ABI, instead of the C ABI. As such, the FFI definitions have to be changed to show this (e.g. the extern "win64"). While this works, it feels terrible.
    2. The asm is modified by the build.rs to fixup the assembly so it can be included in global_asm without issues. In particular, the initial .intel_syntax is removed (that's the default in global_asm), and the {s and }s are doubled to escape them (as a {} is used for formatting).
    3. When compiling for apple, we replace .section .text with .text and .section .rodata with .static_data, to make it compile.

    While I think most of this is fine, fixing the ABI so we can keep using extern "C" ABIs would be a bit nicer.

    opened by roblabla 2
  • Missing unwinding codes on Windows

    Missing unwinding codes on Windows

    Both the GNU as and MASM implementations for Windows x64 miss the unwinding codes that are required by the ABI (similar to calling conventions). In addition, the prologue does not follow the rules of a canonical prologue, as the saving of the xmm registers happens after the dynamic stack allocation (that is, the extra alignment). As for how to add unwind specifiers, there's docs for MASM and for GNU as

    opened by namazso 1
  • constant time Ord and PartialOrd implementations

    constant time Ord and PartialOrd implementations

    Fixes #202.

    This does not follow the recommended path of utilizing subtle as it does not provide any helpers to do constant time ordering on containers.

    The overall method is to compare each u32 in the hashes, turn each of those into -1, 0, or 1, accumulate them into a single signed word, and then finally compare that word to 0.

    The optimized object code on x86-64 is somewhat large but appears to be constant time - no jumps, early returns, or anything like that. This is, of course, not a guarantee that it will never face optimization in hardware or software in the future.

    opened by imuli 3
Releases(1.3.3)
  • 1.3.3(Nov 26, 2022)

  • 1.3.2(Nov 20, 2022)

  • 1.3.1(Feb 14, 2022)

  • 1.3.0(Jan 8, 2022)

  • 1.2.0(Nov 5, 2021)

    version 1.2.0

    Changes since 1.1.0:

    • SECURITY FIX: Fixed an instance of undefined behavior in the Windows SSE2 assembly implementations, which affected both the Rust and C libraries in their default build configurations. See https://github.com/BLAKE3-team/BLAKE3/issues/206. The cause was a vector register that wasn't properly saved and restored. This bug has been present since SSE2 support was initially added in v0.3.7. The effects of this bug depend on surrounding code and compiler optimizations; see test_issue_206_windows_sse2 for an example of this bug causing incorrect hash output. Note that even when surrounding code is arranged to trigger this bug, the SSE2 implementation is normally only invoked on CPUs where SSE4.1 (introduced in 2007) isn't supported. One notable exception, however, is if the Rust library is built in no_std mode, with default_features = false or similar. In that case, runtime CPU feature detection is disabled, and since LLVM assumes that all x86-64 targets support SSE2, the SSE2 implementation will be invoked. For that reason, Rust callers who build blake3 in no_std mode for x86-64 Windows targets are the most likely to trigger this bug. We found this bug in internal testing, and we aren't aware of any callers encountering it in practice.
    • Added the Hasher::count() method.
    Source code(tar.gz)
    Source code(zip)
    b3sum_linux_x64_bin(4.58 MB)
    b3sum_macos_x64_bin(1.30 MB)
    b3sum_windows_x64_bin.exe(935.50 KB)
  • 1.1.0(Oct 21, 2021)

    version 1.1.0

    Changes since 1.0.0:

    • The NEON implementation is now enabled by default on AArch64 targets. Previously it was disabled without the "neon" Cargo feature in Rust or the "BLAKE3_USE_NEON=1" preprocessor flag in C. This is still the case on ARM targets other than AArch64, because of the lack of dynamic CPU feature detection on ARM. Contributed by @rsdy.
    • The previous change leads to some build incompatibilities, particularly in C. If you build the C implementation for AArch64 targets, you now need to include blake3_neon.c, or else you'll get a linker error like "undefined reference to `blake3_hash_many_neon'". If you don't want the NEON implementation, you need to explicitly set "BLAKE3_USE_NEON=0". On the Rust side, AArch64 targets now require the C toolchain by default. build.rs includes workarounds for missing or very old C compilers for x86, but it doesn't currently include such workarounds for AArch64. If we hear about build breaks related to this, we can add more workarounds as appropriate.
    • C-specific Git tags ("c-0.3.7" etc.) have been removed, and all the projects in this repo (Rust "blake3", Rust "b3sum", and the C implementation) will continue to be versioned in lockstep for the foreseeable future.
    Source code(tar.gz)
    Source code(zip)
    b3sum_linux_x64_bin(4.58 MB)
    b3sum_macos_x64_bin(1.30 MB)
    b3sum_windows_x64_bin.exe(935.50 KB)
  • 1.0.0(Jul 25, 2021)

    version 1.0.0

    Changes since 0.3.8:

    • Add Hash::from_hex() and implement FromStr for Hash.
    • Implement Display for Hash, equivalent to Hash::to_hex().
    • Implement PartialEq<[u8]> for Hash, using constant_time_eq.
    • Change derive_key() to return a 32-byte array. As with hash() and keyed_hash(), callers who want a non-default output length can use Hasher::finalize_xof().
    • Replace Hasher::update_with_join() with Hasher::update_rayon(). The former was excessively generic, and the Join trait leaked implementation details. As part of this change, the Join trait is no longer public.
    • Upgraded arrayvec to 0.7.0, which uses const generics. This bumps the minimum supported Rust compiler version to 1.51.
    • Gate the digest and crypto-mac trait implementations behind an unstable feature, "traits-preview". As part of this change upgrade crypto-mac to 0.11.0.
    Source code(tar.gz)
    Source code(zip)
    b3sum_linux_x64_bin(4.61 MB)
    b3sum_macos_x64_bin(1.29 MB)
    b3sum_windows_x64_bin.exe(937.00 KB)
  • 0.3.8(May 25, 2021)

  • 0.3.7(Oct 1, 2020)

    version 0.3.7

    Changes since 0.3.6:

    • BUGFIX: The C implementation was incorrect on big endian systems for inputs longer than 1024 bytes. This bug affected all previous versions of the C implementation. Little endian platforms like x86 were unaffected. The Rust implementation was also unaffected. @jakub-zwolakowski and @pascal-cuoq from TrustInSoft reported this bug: https://github.com/BLAKE3-team/BLAKE3/pull/118
    • BUGFIX: The C build on x86-64 was producing binaries with an executable stack. @tristanheaven reported this bug: https://github.com/BLAKE3-team/BLAKE3/issues/109
    • @mkrupcale added optimized implementations for SSE2. This improves performance on older x86 processors that don't support SSE4.1.
    • The C implementation now exposes the blake3_hasher_init_derive_key_raw function, to make it easier to implement language bindings. Added by @k0001.
    Source code(tar.gz)
    Source code(zip)
    b3sum_linux_x64_bin(4.18 MB)
    b3sum_macos_x64_bin(1.16 MB)
    b3sum_windows_x64_bin.exe(984.00 KB)
  • 0.3.6(Jul 29, 2020)

  • 0.3.5(Jul 10, 2020)

  • 0.3.4(May 23, 2020)

    version 0.3.4

    Changes since 0.3.3:

    • b3sum now supports the --check flag. This is intended to be a drop-in replacement for e.g. md5sum --check from Coreutils. The behavior is somewhat stricter than Coreutils with respect to invalid Unicode in filenames. For a complete description of how --check works, see the file b3sum/what_does_check_do.md.
    • To support the --check feature, backslashes and newlines that appear in filenames are now escaped in the output of b3sum. This is done the same way as in Coreutils.
    • To support --check interoperability between Unix and Windows, backslashes in filepaths on Windows are now replaced with forward slashes in the output of b3sum. Note that this is different from Coreutils.
    Source code(tar.gz)
    Source code(zip)
    b3sum_linux_x64_bin(3.87 MB)
    b3sum_macos_x64_bin(1.12 MB)
    b3sum_windows_x64_bin.exe(950.50 KB)
  • 0.3.3(Apr 28, 2020)

Owner
BLAKE3 team
The BLAKE3 cryptographic hash function
BLAKE3 team
Collection of cryptographic hash functions written in pure Rust

RustCrypto: hashes Collection of cryptographic hash functions written in pure Rust. All algorithms reside in the separate crates and implemented using

Rust Crypto 1.2k Jan 8, 2023
Pure-Rust traits and utilities for constant-time cryptographic implementations.

subtle Pure-Rust traits and utilities for constant-time cryptographic implementations. It consists of a Choice type, and a collection of traits using

dalek cryptography 196 Dec 13, 2022
Fastmurmur3 - Fast non-cryptographic hash, with the benchmarks to prove it.

Fastmurmur3 Murmur3 is a fast, non-cryptographic hash function. fastmurmur3 is, in my testing, the fastest implementation of Murmur3. Usage let bytes:

Kurt Wolf 13 Dec 2, 2022
Lockstitch is an incremental, stateful cryptographic primitive for symmetric-key cryptographic operations in complex protocols.

Lockstitch is an incremental, stateful cryptographic primitive for symmetric-key cryptographic operations (e.g. hashing, encryption, message authentication codes, and authenticated encryption) in complex protocols.

Coda Hale 3 Dec 27, 2022
A Boring(SSL)-compatible API abstraction for Rust cryptographic implementations.

A Boring(SSL)-compatible API abstraction for Rust cryptographic implementations. What is Superboring? Superboring hides the complexity, diversity and

Frank Denis 7 Dec 29, 2023
std::simd implementation of BLAKE3

BLAKE3-STD the first blake3 implementation on std::simd ONLY COMPILES WITH NIGHTLY [dependencies] blake3-std = "0.0.1" OFFICIAL DOC BLAKE3 is a crypto

LemonHX 16 Nov 23, 2022
Bessie - an authenticated, chunked cipher based on BLAKE3

Bessie Bessie is an authenticated, chunked cipher based on BLAKE3. Right now it's in the early design stages. See design.md. Although the Bessie ciphe

Jack O'Connor 12 Dec 9, 2022
Easy to use cryptographic framework for data protection: secure messaging with forward secrecy and secure data storage. Has unified APIs across 14 platforms.

Themis provides strong, usable cryptography for busy people General purpose cryptographic library for storage and messaging for iOS (Swift, Obj-C), An

Cossack Labs 1.6k Dec 30, 2022
Dexios-Core is a library used for managing cryptographic functions and headers that adhere to the Dexios format.

What is it? Dexios-Core is a library used for managing cryptographic functions and headers that adhere to the Dexios format. Security Dexios-Core uses

brxken 3 Jul 4, 2022
Key derivation and cryptographic signing functionality for Ethereum applications (ethers-rs)

ethers-signer-factory ethers-signer-factory is a Rust crate that provides functions for key derivation and signing of Ethereum transactions and messag

Ilia 3 Sep 27, 2023
A (mostly) pure-Rust implementation of various cryptographic algorithms.

Rust-Crypto A (mostly) pure-Rust implementation of various common cryptographic algorithms. Rust-Crypto seeks to create practical, auditable, pure-Rus

null 1.2k Dec 27, 2022
Sodium Oxide: Fast cryptographic library for Rust (bindings to libsodium)

sodiumoxide |Crate|Documentation|Gitter| |:---:|:-----------:|:--------:|:-----:|:------:|:----:| |||| NaCl (pronounced "salt") is a new easy-to-use h

sodiumoxide 642 Dec 17, 2022
Secure storage for cryptographic secrets in Rust

secrets secrets is a library to help Rust programmers safely held cryptographic secrets in memory. It is mostly an ergonomic wrapper around the memory

Stephen Touset 165 Dec 22, 2022
Pure Rust implementation of the RNCryptor cryptographic format by Rob Napier

rncryptor Rust Implementation of the RNCryptor spec This library implements the specification for the RNCryptor encrypted file format by Rob Napier. d

null 7 Jun 29, 2022
A rust binding for nodejs to generate md5 hash value

Hasher A rust binding for creating node module to generate md5 hash value This project was bootstrapped by create-neon. Installing hasher Installing h

Md. Al-Amin 0 Nov 7, 2021
MD5/SHA256 HASH ATTACK IN RUST

hashraccoon Installation Install cargo curl https://sh.rustup.rs -sSf | sh Install the hashraccoon crate cargo install hashraccoon Download the rockyo

null 3 Nov 5, 2022
Pure Rust implementations of the key-committing (and context-committing) AEADs

kc-aeads Pure Rust implementations of the key-committing (and context-committing) AEADs defined in Bellare and Hoang '22. Crash course on the paper: T

Michael Rosenberg 2 Aug 10, 2022
Highly modular & configurable hash & crypto library

Octavo Highly modular & configurable hash & crypto library written in pure Rust. Installation [dependencies] octavo = { git = "https://github.com/libO

Octavo Developers 139 Dec 29, 2022
Modern Cryptographic Firmware

Trussed® Modern Cryptographic Firmware Status Very much WIP. Actively developed. Unstable APIs.

Trussed® 300 Dec 16, 2022