Rust language bindings for Faiss

Overview

Faiss-rs

faiss at crates.io Continuous integration status Minimum Rust Version Stable dependency status

This project provides Rust bindings to Faiss, the state-of-the-art vector search and clustering library.

Installing as a dependency

Currently, this crate does not build Faiss automatically for you. The dynamic library needs to be installed manually to your system.

  1. Follow the instructions here to build Faiss using CMake, enabling the variables FAISS_ENABLE_C_API and BUILD_SHARED_LIBS. The crate is currently only compatible with version v1.7.1. Consider building Faiss from this fork, c_api_head branch, which will contain the latest bindings to the C interface. For example:
    cmake -B . -DFAISS_ENABLE_C_API=ON -DBUILD_SHARED_LIBS=ON -DCMAKE_BUILD_TYPE=Release
    
    This will result in the dynamic library faiss_c ("c_api/libfaiss_c.so" on Linux), which needs to be installed in a place where your system will pick up (in Linux, try somewhere in the LD_LIBRARY_PATH environment variable, such as "/usr/lib", or try adding a new path to this variable). For GPU support, don't forget to enable the option FAISS_ENABLE_GPU. Note: faiss_c might link dynamically to the native faiss library, which in that case you will need to install the main shared object (faiss/libfaiss.so) as well.
  2. You are now ready to include this crate as a dependency:
[dependencies]
"faiss" = "0.10.0"

If you have built Faiss with GPU support, you can include the "gpu" feature in the bindings:

[dependencies]
"faiss" = {version = "0.10.0", features = ["gpu"]}

Using

A basic example is seen below. Please check out the documentation for more.

use faiss::{Index, index_factory, MetricType};

let mut index = index_factory(64, "Flat", MetricType::L2)?;
index.add(&my_data)?;

let result = index.search(&my_query, 5)?;
for (i, (l, d)) in result.labels.iter()
    .zip(result.distances.iter())
    .enumerate()
{
    println!("#{}: {} (D={})", i + 1, *l, *d);
}

License and attribution notice

Licensed under either of

at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

This work is not affiliated with Facebook AI Research or the main Faiss software.

Comments
  • Slow training IVF indexes

    Slow training IVF indexes

    I tried comparison two kinds of program python and c_api

    Both test lib use OpenBlas and LAPACK.

    OS: Oracle Linux 8.3

    Python results

    faiss lib from anaconda channel conda-forge

    In python on GPU

    import numpy as np
    import faiss
    
    d = 128
    res = faiss.StandardGpuResources()
    
    index = faiss.index_factory(d, "IVF16384,Flat")
    index.verbose = True
    select_gpu = 0
    
    xt = faiss.rand((1000000, d))
    index_ivf = faiss.extract_index_ivf(index)
    clustering_index = faiss.index_cpu_to_gpu(res, select_gpu, faiss.IndexFlatL2(d))
    index_ivf.clustering_index = clustering_index
    
    index.train(xt)
    
    
    time python3 train-gpu-rand.py 
    Training level-1 quantizer
    Training level-1 quantizer on 1000000 vectors in 128D
    Training IVF residual
    IndexIVF: no residual training
    
    real    0m8.057s
    user    1m59.744s
    sys     0m17.446s
    

    In python on CPU

    import numpy as np
    import faiss
    
    d = 128
    index = faiss.index_factory(d, "IVF16834,Flat")
    index.verbose = True
    
    xt = faiss.rand((1000000, d))
    index.train(xt)
    
    
    time python3 train-cpu-rand.py 
    Training level-1 quantizer
    Training level-1 quantizer on 1000000 vectors in 128D
    Training IVF residual
    IndexIVF: no residual training
    
    real    7m13.904s
    user    403m29.746s
    sys     519m31.349s
    

    Rust bindings

    in Rust on CPU use faiss-rsv0.10

    use rand::prelude::*;
    use faiss::Index;
    
    fn main() {
        let d:usize = 128;
        let mut rng = rand::thread_rng();
        let mut xt:Vec<f32> = Vec::with_capacity(1_000_000 * d);
        for _ in 0..(1_000_000 * d) {
            xt.push(rng.gen());
        }
    
        let mut index = faiss::index_factory(
            d as u32,
            "IVF16834,Flat",
            faiss::MetricType::L2,
        )
        .unwrap();
    
        index.set_verbose(true);
        index.train(&xt).unwrap();
    }
    
    

    build from source faiss lib https://github.com/facebookresearch/faiss/pull/1838 from branch https://github.com/ava57r/faiss/tree/c_api_avx2 commit https://github.com/ava57r/faiss/commit/02ecc5be906fe787079054e236d2f58119c373e2

    options

    cmake -B build \
    -DBUILD_TESTING=OFF \
    -DFAISS_ENABLE_GPU=OFF \
    -DFAISS_OPT_LEVEL=avx2 \
    -DFAISS_ENABLE_C_API=ON \
    -DFAISS_ENABLE_PYTHON=OFF \
    -DBUILD_SHARED_LIBS=ON \
    -DCMAKE_BUILD_TYPE=Release .
    make -C build -j
    

    results I kill the process

    first time

       Compiling train-faiss-cpu v0.1.0 (/home/faiss-dev/train-faiss-cpu)
        Finished release [optimized] target(s) in 0.36s
         Running `target/release/train-faiss-cpu`
    Training level-1 quantizer
    Training level-1 quantizer on 1000000 vectors in 128D
    ^C
    
    real    70m43.693s
    user    1032m34.066s
    sys     5m17.008s
    

    second time

         Running `target/release/train-faiss-cpu`
    Training level-1 quantizer
    Training level-1 quantizer on 1000000 vectors in 128D
    ^C
    
    real    16m37.219s
    user    242m21.940s
    sys     1m22.869s
    

    I can't find a reason this behavior.

    Could We need to add benckmark(s)?

    opened by ava57r 13
  • faiss_ParameterSpace_set_index_parameters usage

    faiss_ParameterSpace_set_index_parameters usage

    I want to change the parameter "nprobe" in the index.

    According to https://github.com/facebookresearch/faiss/wiki/FAQ you need to use the set_index_parameter function (I found faiss_ParameterSpace_set_index_parameters)

    But I do not understand how to create an instance of FaissParameterSpace and how to work with it

    enhancement good first issue 
    opened by guskovd 7
  • panicked at 'too large index value provided to Idx::new'

    panicked at 'too large index value provided to Idx::new'

    When you use a large integer (greater than 2^63) as an ID, faiss-rs will panic. This is a bit unexpected as it's not mentioned anywhere in the documentation, nor do I think faiss has this limitation. If the intent is to represent null ids, isn't Option<faiss::idx_t> a better choice?

    opened by nemosupremo 4
  • Memory leak detected.

    Memory leak detected.

    Hello,

    I use the crate-0.10.0 for our embedding recall, and have found some tips: a. IndexImpl has not impl concurrentIndex, so IVFScalarQuantizerIndexImpl returned by into_ivf_scalar_quantizer builds error; b. search increase the leaking of memory without upbound limit, so does the read_index, dropped but no release the RES memory.

    What would the problem be? Anybody encount?

    Thanks

    opened by jtong11 4
  • Convert IndexImpl to IdMap<IndexImpl>

    Convert IndexImpl to IdMap

    Hello.

    Example

    let index = FlatIndexImpl::new_l2(4).unwrap();
    let id_map = IdMap::new(index).unwrap();
    write_index(&id_map, "/tmp/id_map_ex.dat").unwrap();
    let new_index = read_index("/tmp/id_map_ex.dat").unwrap();
    

    new_index is IndexImpl

    opened by ava57r 4
  • Why &mut self for fn search in trait Index

    Why &mut self for fn search in trait Index

    Hello.

    Faiss binding required only *const FaissIndex https://github.com/Enet4/faiss-rs/blob/master/faiss-sys/src/bindings.rs#L108

    but here &mut self https://github.com/Enet4/faiss-rs/blob/master/src/macros.rs#L71 https://github.com/Enet4/faiss-rs/blob/master/src/macros.rs#L76

    question 
    opened by ava57r 4
  • 49 - allow read_index_with_flags for memory mapping some index types

    49 - allow read_index_with_flags for memory mapping some index types

    I think using pub mod consts here achieves least surprise for users and minimizes boilerplate to make enum like a bitmask. (There's this crate, and this crate, but it seems a lot just to make enum have something like impl BitOr / allow for "and"ing options.)

    I could attempt to test this by creating a temporary index file and open it with these flags, too.

    Not sure if you'd really want to version bump on this.

    opened by mooreniemi 3
  • read_index io flag support

    read_index io flag support

    Reading Faiss documentation and related issues:

    A standard index can be memory-mapped as an on-disk index with using the I/O flag IO_FLAG_MMAP. This makes it possible to load many indexes that would not otherwise fit in RAM.

    So I wanted to send an I/O flag in to read_index, as in this example.

    But currently, that doesn't appear possible with this crate's read_index method. Am I missing some other way of accomplishing this? Would you accept a PR to make available?

    opened by mooreniemi 3
  • Relax ConcurrentIndex trait bounds

    Relax ConcurrentIndex trait bounds

    Hello.

    When we use composite indexes, we pass a pointer only to the main index through the C API. We should not be interested in how Fais works internally with the subindex.

    opened by ava57r 2
  • use cpp interface instead of c API

    use cpp interface instead of c API

    Having played with accessing faiss via the cxx crate it was reasonably straightforward to create interoperability between rust and faiss.

    Is there a reason why the use of the C api is preferred for interacting with faiss?

    opened by mpetri 2
  • "Linking with cc failed" -- issue compiling?

    Hey folks! I'm having what appears to be some issue compiling. Do you have any thoughts or suggestions about what might be going on here?

    The closest I can find seems to point to a cargo or mac error: https://github.com/rust-lang/rust/issues/25289#issuecomment-101352844
    https://github.com/rust-lang/rust/issues/60149#issuecomment-485253621

    I'm really quite a newbie at rust, so any debugging advice is appreciated!

    Code :

    use faiss::{index_factory, MetricType};
    
    pub fn run_query() -> Option<f32>{
        let mut index = index_factory(3, "Flat", MetricType::L2).unwrap();
        Some(0.0)
    }
    

    Dependency:

    [dependencies]
    "faiss" = {version = "0.6.0"}
    

    Error:

    error: linking with `cc` failed: exit status: 1
      |
      = note: "cc" "-m64" "-arch" "x86_64" "-L" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/thistle.1npak595a7css8z4.rcgu.o" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/thistle.20hce9hmqhl203kn.rcgu.o" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/thistle.2m6hiy95l4iyrwu5.rcgu.o" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/thistle.2wd4t96zh75gv0va.rcgu.o" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/thistle.3aclf5rjqxhrola4.rcgu.o" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/thistle.chnosn5cykoslmi.rcgu.o" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/thistle.sf29mzngf3nyfuf.rcgu.o" "-o" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/thistle" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/thistle.1vtd5t81sh40jhul.rcgu.o" "-Wl,-dead_strip" "-nodefaultlibs" "-L" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps" "-L" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/build/torch-sys-ff3e9973ea28b04e/out/libtorch/libtorch/lib" "-L" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/build/torch-sys-ff3e9973ea28b04e/out" "-L" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/build/bzip2-sys-7fb991f7dbc50a4c/out/lib" "-L" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libthistle-d4a6c2bff82dd787.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librust_bert-31fa07c451540f80.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libitertools-1dfefb650b8cd53d.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librust_tokenizers-c31d0d587cde9540.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libcsv-ed679ac6734a6098.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libcsv_core-d75bfe6d0b9a8acb.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libbstr-c0070e4a1cc2d79c.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libregex_automata-87878d88ec463ae2.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libserde_json-21d54dcd579fec67.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libryu-90cf68ddeda11aee.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libitoa-abc241418b49187a.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libunicode_normalization-b2ce0f79e7c9edf9.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libtinyvec-1c50f201c52e47ed.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libtinyvec_macros-58d800761f45510a.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libregex-7a5e938964f73b44.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libregex_syntax-5efabd26812b2a98.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libaho_corasick-35270916c15955fe.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libmemchr-3f7b52af37b6f9bd.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libitertools-ae0eacccf85a2ec4.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librayon-bc7a015cc78390f5.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librayon_core-99e1dad4e52206ec.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libnum_cpus-0cc61c5a851016e3.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libcrossbeam_deque-8134651f99d6d55c.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libcrossbeam_epoch-5f4b888d78829961.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libmemoffset-6a6eb427edffc44b.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libscopeguard-31f20504eb2703cc.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libcrossbeam_channel-6756a1e59e54d8c6.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libcrossbeam_utils-a78a168a13cd1267.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libeither-562e7f0a923e8f49.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libordered_float-51fc742f3bcfcf51.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libtch-574ba415f0a7b978.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libndarray-20eeeae1632d9fd8.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libmatrixmultiply-f4c1da960d45074e.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libnum_complex-d55c58eb6e91e88f.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libnum_integer-e553cc3e4c6712d2.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libnum_traits-a75e93e9dc909b7f.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librawpointer-af04cc4bb8e47631.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libtorch_sys-2d9d21974de55caa.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libzip-c44b3f38577791de.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libtime-b0c05a3d813ace7b.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libthiserror-68a9b42d4132521f.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libbzip2-896046b3e15dd93a.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libbzip2_sys-e17a3ea19e42b8cc.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libflate2-6e853baf9c8b9e50.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libminiz_oxide-8509711269a1300e.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libadler32-51f964c75e3afbf2.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libcfg_if-892e7cee77ae3329.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libbyteorder-be28ddbc530eab12.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libcrc32fast-6110fcd70cf2e9da.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libfailure-ae0aa4a153df8b9f.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libbacktrace-093e0afaa529e2b3.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libobject-c15f758dd1367b9c.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libaddr2line-1fc0fb83f501e321.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libgimli-6174ef008c83096e.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libcfg_if-a42ba01468c6d92e.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librustc_demangle-9cc4760bb3afbdef.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/liblazy_static-98ea06d08472431a.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libfaiss-c6a410c6f8ab2423.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libfaiss_sys-74aeabac60e0cbd0.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libuuid-7e702774b0c69b2c.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libserde-7d7b34b7398292db.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librand-90af94b665a55644.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librand_xorshift-71f5c0bf8db7f494.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librand_pcg-c9f195549219f059.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librand_hc-7600fcdf91228ccb.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librand_chacha-f72b2299b81d26aa.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librand_isaac-8e1f8597c5ec4d3f.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librand_core-4170c848b672f583.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librand_os-5e84c8953b0ba2d6.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librand_jitter-3cd042c23570d8ca.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/liblibc-c2fdc019e3d38deb.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librand_core-92e42927a2b53f93.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libstd-46a061b08cc06738.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libpanic_unwind-7bebd2f04d2b9056.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libobject-8022bf77ecaec932.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libaddr2line-40998048b1ba428e.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libgimli-bd2a545ac887ab20.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libstd_detect-c0832a78f281aa17.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/librustc_demangle-e84927fb7a338bcb.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libhashbrown-80e6dd310746668f.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_alloc-70160232b7f5a1eb.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libunwind-dc1f6511b0655580.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcfg_if-7606550a486f0fa7.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/liblibc-d71be2efb2f94c3f.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/liballoc-6bbd477ed289203f.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_core-c7c7638eea6d267e.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcore-ac475f5c856023a5.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcompiler_builtins-35393beafda280dd.rlib" "-lc++" "-ltorch" "-ltorch_cpu" "-lc10" "-lfaiss_c" "-framework" "Security" "-liconv" "-lSystem" "-lresolv" "-lc" "-lm" "-liconv"
      = note: ld: library not found for -lfaiss_c
    
    opened by bwindsor22 2
  • Write Index into memory or Read Index from memory

    Write Index into memory or Read Index from memory

    Hello.

    C++ Faiss supports some methods for saving a index into memory.

    write into IOWriter https://github.com/facebookresearch/faiss/blob/main/faiss/index_io.h#L40 and impl VectorIOWriter https://github.com/facebookresearch/faiss/blob/main/faiss/impl/io.h#L59

    read from IOReader https://github.com/facebookresearch/faiss/blob/main/faiss/index_io.h#L61 and impl VectorIOReader https://github.com/facebookresearch/faiss/blob/main/faiss/impl/io.h#L53

    Could We add these impls into rust library?

    opened by ava57r 1
Releases(v0.11.0)
  • v0.11.0(Mar 28, 2022)

    Changes

    • faiss-sys has been updated to link against Faiss version 1.7.2. Please do not forget to update your faiss and faiss_c libraries by building them against the revision pointed by tag v1.7.2. Linking against v1.7.1 may appear to work in some cases, but the API is not backwards compatible! (#52)
    • TryFromInnerPtr::try_from_inner_ptr is now unsafe, since it cannot provide the checks required to ensure memory safety. (#53)

    New

    • io method read_index_with_flags allows for memory mapping to some index types (#50 @mooreniemi)
    • TryClone trait, for cloning an index (#44 @ava57r)
    • UpcastIndex trait (#43 @ava57r)

    Fixes and Enhancements

    • Fixed ownership mistakes in IVFFlatIndexImpl (#42 @ava57r)
    • Implemented to_gpu for some of the new index types (#39 @ava57r)
    • Added Box impl for index types (#41 @ava57r)
    • Used impl_concurrent_index macro for implementing the trait with reduced boilerplate (#34 @ava57r)
    • Fixed README (#32 @ava57r)

    Full Changelog: https://github.com/Enet4/faiss-rs/compare/v0.10.0...v0.11.0

    Source code(tar.gz)
    Source code(zip)
  • v0.10.0(Jun 21, 2021)

    Changes

    • The bindings now depend exclusively on version 1.17.1 of the native Faiss library (#16 #28)
      • See the branch Enet4/c_api_v1.7.1 for a compatible pinned revision
      • look out for the native shared objects, as libfaiss_c.so might no longer statically embed the native faiss objects, and so will need to be copied alongside it.

    New

    • New TryFromInnerPtr trait, for obtaining an index from a pointer in a way which can fail (#30 @ava57r)
    • Support for multiple GPUs (#12)
      • via to_gpu_multiple methods
    • RefineFlatIndex (#25 @ava57r)
    • PreTransformIndex (#24 @ava57r)
    • Added getters and setters for verbose property (#23 @ava57r)
    • ScalarQuantizerIndex (#21 #30 @ava57r)
    • IVFFlatIndex (#17 @ava57r)
    • Access to index parameters via ParameterSpace (#14 @ava57r)

    Chores

    • Migrated and updated CI using GitHub Actions (#29)
    Source code(tar.gz)
    Source code(zip)
  • v0.9.0(Sep 21, 2020)

    Breaking changes

    • The crate now depends on Faiss v1.6.3 with the C API patches (branch c_api_v1.6.3). Please update your dynamic libraries before using. (#8 @gensmusic )
    • Updated the clustering API in line with the changes in the native library: see the method iteration_stats for retrieving the stats of the clustering process (#8 @gensmusic)
    • Implementations of Error::description were removed.
    • Removed NativeError::from_last_error from the public API.
    Source code(tar.gz)
    Source code(zip)
  • v0.8.0(Aug 23, 2019)

    This is a maintenance release, for improved stability.

    Breaking changes

    • Low-level bindings have been updated for Faiss 1.5. Please update your native faiss_c objects.
    • New abstraction for vector identifiers Idx, a newtype which makes a distinction between an identifier and no identifier (= -1).
    • set_temp_memory_fraction has been removed from GpuResources, since it is also no longer available in the native API.
    • MetricType::code takes self by move.
    • Methods as_flat and as_lsh have been renamed to into_flat and into_lsh, old names are deprecated.

    Other

    • The crate faiss has been migrated to the 2018 edition.
    Source code(tar.gz)
    Source code(zip)
  • v0.7.0(Dec 10, 2018)

    New

    • Added method remove_ids to Index (@justinaustin)
    • Updated faiss-sys bindings to include native multi-gpu functions.

    Other

    • A few documentation improvements.
    Source code(tar.gz)
    Source code(zip)
  • v0.6.0(Jun 13, 2018)

    New

    • New wrapper IdMap augments an existing index with arbitrary ID mapping.
    • A CPU-backed index can now be cloned with try_clone (if index cloning is supported).

    Breaking changes

    • Internal macros are no longer exported. Hopefully no one was using them in the first place.

    Minor

    • IndexLsh is now re-exported at the root.
    • Documentation was significantly improved!
    Source code(tar.gz)
    Source code(zip)
  • v0.5.0(May 12, 2018)

    New

    • Index type LshIndex for locality sensitive hashing indexing.
    • index::io module for reading and writing indices from/to disk.

    Breaking changes

    • New error variant BadFilePath for incompatible file path strings in io module.
    • Relies on an updated C API of faiss_c, so you may need to rebuild Faiss with the latest version.

    Minor tweaks

    • implement Debug for StandardGpuResources
    Source code(tar.gz)
    Source code(zip)
  • v0.4.0(Apr 3, 2018)

    New

    • to_gpu and to_cpu methods were brought back, but without consuming the original index. Unlike in into_gpu, both indexes can be used separately and are independent from each other.
    • Added slicker FlatIndex::new_l2 and FlatIndex::new_ip constructors.
    • FlatIndex is now reachable at the crate's root.

    Breaking Change

    • GpuIndexImpl::from_cpu is no longer in the public API (its use was already ill advised).
    • Invalid description input to index_factory will return a new error variant instead of panicking.

    Fixes

    • Documentation on indexes and GPU support was significantly improved.
    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(Mar 21, 2018)

    New

    • StandardGpuResources can now be used by other indices through an immutable reference, thus allowing multiple GPU indices to share the same resources. The type is also Send now (but will never be Sync).
    • The new supertrait ConcurrentIndex contains methods for concurrent searching over an index. In this extended interface, search, range_search, and assign take &self instead of &mut self. FlatIndex implements ConcurrentIndex, more may come in the future.

    Breaking Changes

    • renamed to_gpu and to_cpu methods to into_gpu and into_cpu, to comply with conventional Rust method naming.
    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Feb 14, 2018)

    New:

    • Added the Clustering API. Safely perform k-means clustering and retrieve the centroids of each cluster.
    • Added a more type-safe FlatIndex API, which enables fetching the indexed data and calculating distances on subsets of the index.

    Breaking changes:

    • The main error type Error is now an enum rather than an alias to NativeError.
    • Fetching the pointer to the underlying implementation is always done through inner_ptr(), inner_ptr_mut() no longer exists.

    Other:

    • Improved implementation of GPU index: going back and forth between CPU and GPU retains static typing.
    • Improved documentation.
    Source code(tar.gz)
    Source code(zip)
Owner
Eduardo Pinho
Software Engineer @ BMD Software; PhD in computer science; medical imaging informatics; Rust, JavaScript, C++ enthusiast.
Eduardo Pinho
Rust language bindings for the LIBLINEAR C/C++ library.

liblinear-rs Rust bindings for the liblinear C/C++ library. Provides a thin (but rustic) wrapper around the original C-interface exposed by the librar

Madeesh Kannan 8 Sep 22, 2022
Rust bindings for the C++ api of PyTorch.

tch-rs Rust bindings for the C++ api of PyTorch. The goal of the tch crate is to provide some thin wrappers around the C++ PyTorch api (a.k.a. libtorc

Laurent Mazare 2.3k Jan 1, 2023
Rust bindings for TensorFlow Lite

Rust bindings for TensorFlow Lite This crates provides TensorFlow Lite APIs. Please read the API documentation on docs.rs Using the interpreter from a

Boncheol Gu 84 Dec 11, 2022
Rust bindings for XGBoost.

rust-xgboost Rust bindings for the XGBoost gradient boosting library. Documentation Basic usage example: extern crate xgboost; use xgboost::{paramete

Dave Challis 79 Nov 28, 2022
Locality Sensitive Hashing in Rust with Python bindings

lsh-rs (Locality Sensitive Hashing) Locality sensitive hashing can help retrieving Approximate Nearest Neighbors in sub-linear time. For more informat

Ritchie Vink 65 Jan 2, 2023
OpenAI Gym bindings for Rust

gym-rs OpenAI gym binding for Rust. Actively maintained! If you have any problem just create an issue. Install Just install the requierements layed ou

Mr.Robb 45 Dec 11, 2022
High-level non-blocking Deno bindings to the rust-bert machine learning crate.

bertml High-level non-blocking Deno bindings to the rust-bert machine learning crate. Guide Introduction The ModelManager class manages the FFI bindin

Carter Snook 14 Dec 15, 2022
An example of using TensorFlow rust bindings to serve trained machine learning models via Actix Web

Serving TensorFlow with Actix-Web This repository gives an example of training a machine learning model using TensorFlow2.0 Keras in python, exporting

Kyle Kosic 39 Dec 12, 2022
Docker for PyTorch rust bindings `tch`. Example of pretrain model.

tch-rs-pretrain-example-docker Docker for PyTorch rust bindings tch-rs. Example of pretrain model. Docker files support the following install libtorch

vaaaaanquish 5 Oct 7, 2022
mxnet Rust Bindings

mxnet Rust Bindings This is a work in progress. Contributions gladly accepted! The mxnet crate defines a high-level Rust API for mxnet using the mxnet

Jacob Lee 5 Sep 17, 2022
Rust bindings for darknet

Rust bindings for darknet Darknet: Convolutional Neural Networks todo rewrite the demo function used in yolo.c in rust Examples Link existing files an

Oliver Funk 8 Jul 11, 2021
SlintDotnet is a C# bindings project to enable developers to use Slint UI with .NET C#

SlintDotnet (Alpha) Slint is a UI toolkit that supports different programming languages. SlintDotnet is the integration with .NET C#. ⚠️ This is exper

Matheus Castello 9 Oct 2, 2023
A tour of rust's language features

Tour of Rust Welcome to the source repo of Tour of Rust. Goals This project is meant to give an experienced programmer a swift introduction to Rust as

RICHΛRD ΛNΛYΛ 693 Jan 6, 2023
Rust-like syntax for OpenGL Shading Language

Rust-like syntax for GLSL glassful translates a small subset of Rust to OpenGL Shading Language. Besides one's personal preferences regarding Rust-lik

Keegan McAllister 158 Sep 22, 2022
🐉 Making Rust a first-class language and ecosystem for GPU shaders 🚧

?? rust-gpu Rust as a first-class language and ecosystem for GPU graphics & compute shaders Current Status ?? Note: This project is still heavily in d

Embark 5.5k Jan 9, 2023
A collection of CC-BY-SA course material to teach the Rust programming language, in different formats, levels, and focus points

A collection of CC-BY-SA course material to teach the Rust programming language, in different formats, levels, and focus points. Contact me for remote and on-site trainings!

Katharina Fey 13 Apr 13, 2023
[WIP] An experimental Java-like language and it's virtual machine, for learning Java and JVM.

Sky VM An experimental Java-like language and it's virtual machine, for learning Java and JVM. Dependencies Rust (rust-lang/rust) 2021 Edition, dual-l

Kk Shinkai 2 Jan 3, 2022
Masked Language Model on Wasm

Masked Language Model on Wasm This project is for OPTiM TECH BLOG. Please see below: WebAssemblyを用いてBERTモデルをフロントエンドで動かす Demo Usage Build image docker

OPTiM Corporation 20 Sep 23, 2022
Deduplicating Training Data Makes Language Models Better

Deduplicating Training Data Makes Language Models Better This repository contains code to deduplicate language model datasets as descrbed in the paper

Google Research 431 Dec 27, 2022