leveldb LevelDB LevelDB skade/leveldb — LevelDB bindings

Florian Gilcher

Last update: Nov 20, 2022

Related tags

Database leveldb

Overview

Rust leveldb bindings

Almost-complete bindings for leveldb for Rust.

Documentation

Rust version policy

leveldb is built and tested on stable releases of Rust. This are currently 1.31.0 and 1.43.1. Nightlies might not build at any point and failures are allowed. There are no known issues with nightlies, though.

Prerequisites

snappy and leveldb need to be installed. On Ubuntu, I recommend:

sudo apt-get install libleveldb-dev libsnappy-dev

Usage

If your project is using Cargo, drop the following lines in your Cargo.toml:

[dependencies]

leveldb = "0.8"

Development

Make sure you have all prerequisites installed. Run

$ cargo build

for building and

$ cargo test

to run the test suite.

Examples

extern crate tempdir;
extern crate leveldb;

use tempdir::TempDir;
use leveldb::database::Database;
use leveldb::iterator::Iterable;
use leveldb::kv::KV;
use leveldb::options::{Options,WriteOptions,ReadOptions};

fn main() {
  let tempdir = TempDir::new("demo").unwrap();
  let path = tempdir.path();

  let mut options = Options::new();
  options.create_if_missing = true;
  let mut database = match Database::open(path, options) {
      Ok(db) => { db },
      Err(e) => { panic!("failed to open database: {:?}", e) }
  };

  let write_opts = WriteOptions::new();
  match database.put(write_opts, 1, &[1]) {
      Ok(_) => { () },
      Err(e) => { panic!("failed to write to database: {:?}", e) }
  };

  let read_opts = ReadOptions::new();
  let res = database.get(read_opts, 1);

  match res {
    Ok(data) => {
      assert!(data.is_some());
      assert_eq!(data, Some(vec![1]));
    }
    Err(e) => { panic!("failed reading data: {:?}", e) }
  }

  let read_opts = ReadOptions::new();
  let mut iter = database.iter(read_opts);
  let entry = iter.next();
  assert_eq!(
    entry,
    Some((1, vec![1]))
  );
}

Open issues

Filter policies are missing
Iterators with arbirary start and end points are unsupported

License

MIT, see LICENSE

Comments

Compilation error on macOS

Getting the following compilation error on macOS 10.14.4:

❯❯❯ RUST_BACKTRACE=1 cargo build                                                                                                                                                                       
warning: An explicit [[test]] section is specified in Cargo.toml which currently
disables Cargo from automatically inferring other test targets.
This inference behavior will change in the Rust 2018 edition and the following
files will be included as a test target:

* /private/tmp/leveldb/tests/cache.rs
* /private/tmp/leveldb/tests/database.rs
* /private/tmp/leveldb/tests/comparator.rs
* /private/tmp/leveldb/tests/snapshots.rs
* /private/tmp/leveldb/tests/binary.rs
* /private/tmp/leveldb/tests/writebatch.rs
* /private/tmp/leveldb/tests/concurrent_access.rs
* /private/tmp/leveldb/tests/management.rs
* /private/tmp/leveldb/tests/compaction.rs
* /private/tmp/leveldb/tests/utils.rs
* /private/tmp/leveldb/tests/iterator.rs

This is likely to break cargo build or cargo test as these files may not be
ready to be compiled as a test target today. You can future-proof yourself
and disable this warning by adding `autotests = false` to your [package]
section. You may also move the files to a location where Cargo would not
automatically infer them to be a target, such as in subfolders.

For more information on this warning you can consult
https://github.com/rust-lang/cargo/issues/5330
   Compiling leveldb-sys v2.0.3
thread 'rustc' panicked at 'called `Option::unwrap()` on a `None` value', src/libcore/option.rs:347:21
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::_print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: rustc::util::common::panic_hook
   5: std::panicking::rust_panic_with_hook
   6: std::panicking::continue_panic_fmt
   7: rust_begin_unwind
   8: core::panicking::panic_fmt
   9: core::panicking::panic
  10: <rustc_codegen_llvm::back::archive::LlvmArchiveBuilder as rustc_codegen_ssa::back::archive::ArchiveBuilder>::build
  11: <rustc_codegen_llvm::LlvmCodegenBackend as rustc_codegen_utils::codegen_backend::CodegenBackend>::join_codegen_and_link::{{closure}}
  12: rustc::util::common::time
  13: <rustc_codegen_llvm::LlvmCodegenBackend as rustc_codegen_utils::codegen_backend::CodegenBackend>::join_codegen_and_link
  14: rustc_interface::queries::Query<T>::compute
  15: rustc_interface::queries::<impl rustc_interface::interface::Compiler>::link
  16: rustc_interface::interface::run_compiler_in_existing_thread_pool
  17: std::thread::local::LocalKey<T>::with
  18: scoped_tls::ScopedKey<T>::set
  19: syntax::with_globals
query stack during panic:
end of query stack

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.36.0 (a53f9df32 2019-07-03) running on x86_64-apple-darwin

note: compiler flags: -C debuginfo=2 --crate-type lib

note: some of the compiler flags provided by cargo are hidden

error: Could not compile `leveldb-sys`.

To learn more, run the command again with --verbose.

Installed the dependencies like this before:

brew install snappy leveldb

As a side note, building leveldb-sys (v2.0.3) works.

opened by mre 7

Added `Bytes` and `get_bytes` method.

These are useful for faster retrieval of data in some cases, because they skip conversion to Vec (and thus allocation and copying).

I'm not sure if the method should be called get_bytes. Maybe get_native?

opened by Kixunil 5
Use portable c_char instead of i8

Some architectures, such as ARM, use unsigned 8-bit integers to represent C char type.

In order to make the code work on architectures with both signed and unsigned chars, this PR replaces i8 by c_char in places where it corresponds to C char type.

opened by a-rodin 3
How to use in multiple threads?

Database implements no Clone, Send, nor Sync. The documentation however mentions:

Multiple Database objects can be kept around, as leveldb synchronises internally.

opened by astro 3

fix deny(missing_docs) error

quick comment to fix build failing because of the #![deny(missing_docs)]

am I correct in assuming this will fix the error?

/home/leshow/.cargo/registry/src/github.com-1ecc6299db9ec823/leveldb-0.3.9/src/database/comparator.rs:23:11: 23:12 error: missing documentation for an associated type
/home/leshow/.cargo/registry/src/github.com-1ecc6299db9ec823/leveldb-0.3.9/src/database/comparator.rs:23      type K: Key;
                                                                                                                   ^
/home/leshow/.cargo/registry/src/github.com-1ecc6299db9ec823/leveldb-0.3.9/src/lib.rs:43:9: 43:21 note: lint level defined here
/home/leshow/.cargo/registry/src/github.com-1ecc6299db9ec823/leveldb-0.3.9/src/lib.rs:43 #![deny(missing_docs)]

opened by leshow 3

Error building leveldb-sys v2.0.4 in macOS Catalina

Error building leveldb-sys v2.0.4 in MacOS Catalina 10.15.1 I also tried to install leveldb using homebrew. No difference. Who is to blame here?

error: failed to run custom build command for `leveldb-sys v2.0.4`

Caused by:
  process didn't exit successfully: `/[...]/target/debug/build/leveldb-sys-6dc376d2aa14157c/build-script-build` (exit code: 101)
--- stdout
[build] Started
[snappy] Cleaning
test -z "libsnappy.la" || rm -f libsnappy.la
rm -f "./so_locations"
rm -rf .libs _libs
 rm -f snappy_unittest
rm -f *.o
rm -f *.lo
rm -f *.tab.c
test -z "snappy-stubs-public.h" || rm -f snappy-stubs-public.h
test . = "." || test -z "" || rm -f
rm -f config.h stamp-h1
rm -f libtool config.lt
rm -f TAGS ID GTAGS GRTAGS GSYMS GPATH tags
rm -f config.status config.cache config.log configure.lineno config.status.lineno
rm -rf ./.deps
rm -f Makefile
[snappy] Configuring
checking for a BSD-compatible install... /usr/local/bin/ginstall -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/local/bin/gmkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking build system type... x86_64-apple-darwin19.0.0
checking host system type... x86_64-apple-darwin19.0.0
checking how to print strings... printf
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking dependency style of gcc... gcc3
checking for a sed that does not truncate output... /usr/local/bin/gsed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... no
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 196608
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking how to convert x86_64-apple-darwin19.0.0 file names to x86_64-apple-darwin19.0.0 format... func_convert_file_noop
checking how to convert x86_64-apple-darwin19.0.0 file names to toolchain format... func_convert_file_noop
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @FILE support... no
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for mt... no
checking if : is a manifest tool... no
checking for dsymutil... dsymutil
checking for nmedit... nmedit
checking for lipo... lipo
checking for otool... otool
checking for otool64... no
checking for -single_module linker flag... yes
checking for -exported_symbols_list linker flag... yes
checking for -force_load linker flag... yes
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... no
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fno-common -DPIC
checking if gcc PIC flag -fno-common -DPIC works... yes
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... darwin19.0.0 dyld
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... no
checking whether to build static libraries... yes
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking dependency style of g++... gcc3
checking how to run the C++ preprocessor... g++ -E
checking for ld used by g++... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... no
checking whether the g++ linker (/usr/bin/ld) supports shared libraries... yes
checking for g++ option to produce PIC... -fno-common -DPIC
checking if g++ PIC flag -fno-common -DPIC works... yes
checking if g++ static flag -static works... no
checking if g++ supports -c -o file.o... yes
checking if g++ supports -c -o file.o... (cached) yes
checking whether the g++ linker (/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... darwin19.0.0 dyld
checking how to hardcode library paths into programs... immediate
checking whether byte ordering is bigendian... no
checking for size_t... yes
checking for ssize_t... yes
checking for stdint.h... (cached) yes
checking stddef.h usability... yes
checking stddef.h presence... yes
checking for stddef.h... yes
checking sys/mman.h usability... yes
checking sys/mman.h presence... yes
checking for sys/mman.h... yes
checking sys/resource.h usability... yes
checking sys/resource.h presence... yes
checking for sys/resource.h... yes
checking windows.h usability... no
checking windows.h presence... no
checking for windows.h... no
checking byteswap.h usability... no
checking byteswap.h presence... no
checking for byteswap.h... no
checking sys/byteswap.h usability... no
checking sys/byteswap.h presence... no
checking for sys/byteswap.h... no
checking sys/endian.h usability... no
checking sys/endian.h presence... no
checking for sys/endian.h... no
checking sys/time.h usability... yes
checking sys/time.h presence... yes
checking for sys/time.h... yes
checking for mmap... yes
checking for 'gtest-config'... checking for gtest-config... no
no
checking for pkg-config... /usr/local/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for gflags... no
checking if the compiler supports __builtin_expect... yes
checking if the compiler supports __builtin_ctzll... yes
checking for zlibVersion in -lz... yes
checking for lzo1x_1_15_compress in -llzo2... no
checking for lzf_compress in -llzf... no
checking for fastlz_compress in -lfastlz... no
checking for qlz_compress in -lquicklz... no
configure: creating ./config.status
config.status: creating Makefile
config.status: creating snappy-stubs-public.h
config.status: creating config.h
config.status: executing depfiles commands
config.status: executing libtool commands

--- stderr
thread 'main' panicked at 'configure failed', src/libcore/option.rs:1036:5
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

opened by tuler 2

Make crate work ok aarch64
Both cargo build was failing with 16 errors. All of the type expected i8, found u8.

I made it work on aarch64 and x64 following:

https://github.com/sfackler/rust-openssl/commit/f54af75eb7f8c861383119722f548cb2866ca813

https://github.com/rust-lang/rust/issues/60226

Im not a pro on this topic, but I hope that this fix is correct :)
opened by b-m-f 2
Do not take ownership of key on get/delete

I've noticed that on get and delete, the method takes ownership of they key. This is somewhat unexpected, since the value can be borrowed on put, but the key can't. This may require calling the library with key.clone(), which I'd like to avoid. :)

I've looked into the code, while I think it should be safe to change that to a borrow, I'd like to avoid touching the unsafe parts of the crate myself.

https://github.com/skade/leveldb/blob/118359e61210dfe76f64f7fc325c366643c64384/src/database/kv.rs#L96-L126

Thanks!

opened by kpcyrd 2
Relicense under dual MIT/Apache-2.0
Why?

The MIT license requires reproducing countless copies of the same copyright header with different names in the copyright field, for every MIT library in use. The Apache license does not have this drawback, and has protections from patent trolls and an explicit contribution licensing clause. However, the Apache license is incompatible with GPLv2. This is why Rust is dual-licensed as MIT/Apache (the "primary" license being Apache, MIT only for GPLv2 compat), and doing so would be wise for this project. This also makes this crate suitable for inclusion in the Rust standard distribution and other project using dual MIT/Apache.

How?

To do this, get explicit approval from each contributor of copyrightable work (as not all contributions qualify for copyright) and then add the following to your README:

## License Licensed under either of * Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0) * MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT) at your option. ### Contribution Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you shall be dual licensed as above, without any additional terms or conditions.

and in your license headers, use the following boilerplate (based on that used in Rust):

// Copyright (c) 2015 t developers // Licensed under the Apache License, Version 2.0 // <LICENSE-APACHE or // http://www.apache.org/licenses/LICENSE-2.0> or the MIT // license <LICENSE-MIT or http://opensource.org/licenses/MIT>, // at your option. All files in the project carrying such // notice may not be copied, modified, or distributed except // according to those terms.

And don't forget to update the license metadata in your Cargo.toml!

Contributor checkoff

[ ] @skade

[ ] @astro

[x] @aij
opened by emberian 2
core::fmt::String missing for error
running the example code I'm getting a

the trait `core::fmt::String` is not implemented for the type `leveldb::database::error::Error` [E0277] src/main.rs:45 Err(e) => { panic!("failed to open database: {}", e) }

I believe error is just missing a #[derive(Display)] in order for it to be printed to the console, am I correct?
opened by leshow 2
Add Github Actions.

Adds automated CI for OS which GA officially supports. No fmt/clippy since they didn't pass.

Windows is fixed in https://github.com/skade/leveldb-sys/pull/18.

Example results: https://github.com/timberio/leveldb/runs/1098011055?check_suite_focus=true

opened by Hoverbear 1

AddressSanitizer detects buffer overflows, comparator names must be null-terminated

You can reproduce this with

RUSTFLAGS=-Zsanitizer=address cargo +nightly test -Zbuild-std --target=x86_64-unknown-linux-gnu

And you should see an error that looks like this:

==155398==ERROR: AddressSanitizer: global-buffer-overflow on address 0x56533814bf07 at pc 0x565337537676 bp 0x7fcf7a4f4d20 sp 0x7fcf7a4f44e8
READ of size 8 at 0x56533814bf07 thread T7 (comparator::com)
    #0 0x565337537675 in strlen /rustc/llvm/src/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:389:5
    #1 0x565337999bd6 in leveldb::Slice::Slice(char const*) /home/ben/.cargo/registry/src/github.com-1ecc6299db9ec823/leveldb-sys-2.0.9/deps/leveldb-1.22/include/leveldb/slice.h:40:48
    #2 0x56533799de56 in leveldb::DBImpl::NewDB() /home/ben/.cargo/registry/src/github.com-1ecc6299db9ec823/leveldb-sys-2.0.9/deps/leveldb-1.22/db/db_impl.cc:184:27
    #3 0x56533799ea20 in leveldb::DBImpl::Recover(leveldb::VersionEdit*, bool*) /home/ben/.cargo/registry/src/github.com-1ecc6299db9ec823/leveldb-sys-2.0.9/deps/leveldb-1.22/db/db_impl.cc:295:16
    #4 0x5653379a4f10 in leveldb::DB::Open(leveldb::Options const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, leveldb::DB**) /home/ben/.cargo/registry/src/github.com-1ecc6299db9ec823/leveldb-sys-2.0.9/deps/leveldb-1.22/db/db_impl.cc:1482:49
    #5 0x565337999f9e in leveldb_open /home/ben/.cargo/registry/src/github.com-1ecc6299db9ec823/leveldb-sys-2.0.9/deps/leveldb-1.22/db/c.cc:167:33
    #6 0x5653375d0ac7 in leveldb::database::Database$LT$K$GT$::open_with_comparator::h8a9ab6d14aae33c4 /tmp/leveldb-0.8.6/src/database/mod.rs:151:22
    #7 0x5653375e1cd4 in tests::comparator::comparator::test_comparator::h968093db9d08b506 /tmp/leveldb-0.8.6/tests/comparator.rs:37:29

The problem is that leveldb is eventually going to call strlen on the comparator name pointers. This is a soundness bub because the Comparator trait is safe to implement.

Normally I'd make some attempt to patch this but I'm not really sure what to do here, and I think this is project is pretty abandoned. Figured it would be good to let you know anyway :)

opened by saethlin 0

Verify that concurrent write wrote correctly

Summary

Test tests/concurrent_access.rs spins up 10 threads, each writing to their own key concurrently. However, there is no verificiation that the values were actually written.

#Solution This commit waits for the threads to join and then verifies all 10 values.

#Byproducts: There were some warnings (mostly missing dyns and #[inline]s in function prototypes) which this commit also fixes.

opened by bsdinis 1

Expose `Key` trait as public to allow custom data types as leveldb keys

Currently there is no way to use our own data types as keys because the module is private, for example this will not work

// ...
use leveldb::database::key::Key; // error[E0603]: crate `key` is private

#[derive(Clone)]
struct MyData {
    key: String,
}

impl Key for MyData {
    fn from_u8(key: &[u8]) -> Self {
        MyData {
            key: str::from_utf8(key).unwrap().into(),
        }
    }

    fn as_slice<T, F: Fn(&[u8]) -> T>(&self, f: F) -> T {
        let dst = self.key.as_bytes();
        f(&dst)
    }
}
// ...

This PR creates a simple fix by removing the db-key dependency and adds the Key trait as public in src/database/key.rs.

A full example is here

opened by cl2089 0

Updated dependency db-key to version 0.1

Updated dependency db-key to version 0.1 and made some necessary modification to fit the new version. But I haven't updated tests and the documentation yet.

opened by Fancyflame 0

Owner

Florian Gilcher

CEO of @Asquera and @ferroussystems. Heavily invested in the @ruby, @rust-lang and @elastic/@solr communities. Organiser @eurucamp/@jrubyconf/@rustfest.

GitHub

Rust bindings for LMDB

lmdb-rs Rust bindings for LMDB Documentation (master branch) Building LMDB is bundled as submodule so update submodules first: git submodule update --

104 Dec 8, 2022

Ergonomic bindings to SQLite for Rust

Rusqlite Rusqlite is an ergonomic wrapper for using SQLite from Rust. It attempts to expose an interface similar to rust-postgres. use rusqlite::{para

1.9k Jan 5, 2023

Rustic bindings for sqlite3

57 Sep 26, 2022

"KakaoTalk" internal api bindings

talk-api-client "KakaoTalk" internal api bindings Examples Auth See examples/auth.rs To run example, clone this repository and run command cargo run -

5 Sep 7, 2022

SQLite3 Bindings for Rust

SQLite3 Bindings for Rust To compile use rustc src/sqlite3.rs or if you have Cargo installed cargo build. The interface is currently evolving both alo

74 Aug 6, 2022

LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.

LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values. Authors: Sanjay Ghem