Hora
[Homepage] [Document] [Examples]
Hora Search Everywhere!
Hora is an approximate nearest neighbor search algorithm (wiki) library. We implement all code in Rust๐ฆ for reliability, high level abstraction and high speeds comparable to C++.
Hora, ใใปใใ in Japanese, sounds like [hลlษ], and means Wow, You see! or Look at that!. The name is inspired by a famous Japanese song ใๅฐใใชๆใฎใใใ.
Demos
Features
-
Performant
โก๏ธ - SIMD-Accelerated (packed_simd)
- Stable algorithm implementation
- Multiple threads design
-
Supports Multiple Languages
โ๏ธ PythonJavascriptJavaGo(WIP)Ruby(WIP)Swift(WIP)R(WIP)Julia(WIP)- Can also be used as a service
-
Supports Multiple Indexes
๐ -
Portable
๐ผ - Supports
WebAssembly - Supports
Windows,LinuxandOS X - Supports
IOSandAndroid(WIP) - Supports
no_std(WIP, partial) - No heavy dependencies, such as
BLAS
- Supports
-
Reliability
๐ Rustcompiler secures all code- Memory managed by
Rustfor all language libraries such asPython's - Broad testing coverage
-
Supports Multiple Distances
๐งฎ Dot Product DistanceEuclidean DistanceManhattan DistanceCosine Similarity
-
Productive
โญ - Well documented
- Elegant, simple and easy to learn API
Installation
Rust
in Cargo.toml
[dependencies]
hora = "0.1.1"
Python
$ pip install horapy
Javascript (WebAssembly)
$ npm i horajs
Building from source
$ git clone https://github.com/hora-search/hora
$ cargo build
Benchmarks
by aws t2.medium (CPU: Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz) more information
Examples
Rust example [more info]
use hora::core::ann_index::ANNIndex;
use rand::{thread_rng, Rng};
use rand_distr::{Distribution, Normal};
pub fn demo() {
let n = 1000;
let dimension = 64;
// make sample points
let mut samples = Vec::with_capacity(n);
let normal = Normal::new(0.0, 10.0).unwrap();
for _i in 0..n {
let mut sample = Vec::with_capacity(dimension);
for _j in 0..dimension {
sample.push(normal.sample(&mut rand::thread_rng()));
}
samples.push(sample);
}
// init index
let mut index = hora::index::hnsw_idx::HNSWIndex::<f32, usize>::new(
dimension,
&hora::index::hnsw_params::HNSWParams::<f32>::default(),
);
for (i, sample) in samples.iter().enumerate().take(n) {
// add point
index.add(sample, i).unwrap();
}
index.build(hora::core::metrics::Metric::Euclidean).unwrap();
let mut rng = thread_rng();
let target: usize = rng.gen_range(0..n);
// 523 has neighbors: [523, 762, 364, 268, 561, 231, 380, 817, 331, 246]
println!(
"{:?} has neighbors: {:?}",
target,
index.search(&samples[target], 10) // search for k nearest neighbors
);
}
Python example [more info]
(dimension: 50, dtype: usize, max_item: 1000000, n_neigh: 32, n_neigh0: 64, ef_build: 20, ef_search: 500, has_deletion: False)
# has neighbors: [410, 736, 65, 36, 631, 83, 111, 254, 990, 161]
print("{} in {} \nhas neighbors: {}".format(
target, index, index.search(samples[target], 10))) # search
">
import numpy as np from horapy import HNSWIndex dimension = 50 n = 1000 # init index instance index = HNSWIndex(dimension, "usize") samples = np.float32(np.random.rand(n, dimension)) for i in range(0, len(samples)): # add node index.add(np.float32(samples[i]), i) index.build("euclidean") # build index target = np.random.randint(0, n) # 410 in Hora ANNIndex(dimension: 50, dtype: usize, max_item: 1000000, n_neigh: 32, n_neigh0: 64, ef_build: 20, ef_search: 500, has_deletion: False) # has neighbors: [410, 736, 65, 36, 631, 83, 111, 254, 990, 161] print("{} in {} \nhas neighbors: {}".format( target, index, index.search(samples[target], 10))) # search
JavaScript example [more info]
{
const dimension = 50;
var bf_idx = horajs.BruteForceIndexUsize.new(dimension);
// var hnsw_idx = horajs.HNSWIndexUsize.new(dimension, 1000000, 32, 64, 20, 500, 16, false);
for (var i = 0; i < 1000; i++) {
var feature = [];
for (var j = 0; j < dimension; j++) {
feature.push(Math.random());
}
bf_idx.add(feature, i); // add point
}
bf_idx.build("euclidean"); // build index
var feature = [];
for (var j = 0; j < dimension; j++) {
feature.push(Math.random());
}
console.log("bf result", bf_idx.search(feature, 10)); //bf result Uint32Array(10) [704, 113, 358, 835, 408, 379, 117, 414, 808, 826]
}
(async () => {
await horajs.default();
await horajs.init_env();
demo();
})();
">
import * as horajs from "horajs"; const demo = () => { const dimension = 50; var bf_idx = horajs.BruteForceIndexUsize.new(dimension); // var hnsw_idx = horajs.HNSWIndexUsize.new(dimension, 1000000, 32, 64, 20, 500, 16, false); for (var i = 0; i < 1000; i++) { var feature = []; for (var j = 0; j < dimension; j++) { feature.push(Math.random()); } bf_idx.add(feature, i); // add point } bf_idx.build("euclidean"); // build index var feature = []; for (var j = 0; j < dimension; j++) { feature.push(Math.random()); } console.log("bf result", bf_idx.search(feature, 10)); //bf result Uint32Array(10) [704, 113, 358, 835, 408, 379, 117, 414, 808, 826] } (async () => { await horajs.default(); await horajs.init_env(); demo(); })();
Java example [more info]
public void demo() {
final int dimension = 2;
final float variance = 2.0f;
Random fRandom = new Random();
BruteForceIndex bruteforce_idx = new BruteForceIndex(dimension); // init index instance
List
Roadmap
- Full test coverage
- Implement EFANNA algorithm to achieve faster KNN graph building
- Swift support and iOS/macOS deployment example
- Support
R - support
mmap
Related Projects and Comparison
-
Hora's implementation is strongly inspired by these libraries.Faissfocuses more on the GPU scenerio, andHorais lighter than Faiss (no heavy dependencies).Horaexpects to support more languages, and everything related to performance will be implemented by Rust๐ฆ .Annoyonly supports theLSH (Random Projection)algorithm.ScaNNandFaissare less user-friendly, (e.g. lack of documentation).- Hora is ALL IN RUST
๐ฆ .
-
MilvusandValdalso support multiple languages, but serve as a service instead of a libraryMilvusis built upon some libraries such asFaiss, whileHorais a library with all the algorithms implemented itself
Contribute
We appreciate your help!
We are glad to have you participate, any contributions are welcome, including documentations and tests. You can create a Pull Request or Issue on GitHub, and we will review it as soon as possible.
We use GitHub issues for tracking suggestions and bugs.
Clone the repo
git clone https://github.com/hora-search/hora
Build
cargo build
Test
cargo test --lib
Try the changes
cd examples
cargo run
License
The entire repository is licensed under the Apache License.
283 Dec 21, 2022
6 Oct 19, 2022
7 Oct 9, 2021
13 Mar 10, 2022
2 Apr 1, 2022
28 Aug 20, 2022
2 Nov 22, 2022
8 Sep 23, 2022
2 Aug 7, 2022
1 May 18, 2021
0 Mar 19, 2020
2 Apr 28, 2021
4 Sep 15, 2021
3 Dec 25, 2021
2 Mar 14, 2021
11 Oct 16, 2022
6 Oct 21, 2022
6 Feb 24, 2023