Rust Vector for large amounts of data, that does not copy when growing, by using full `mmap`'d pages.

Overview

Large Vector

Rust Vector for large amounts of data, that does not copy when growing, by using full mmap'd pages.

Maturity

I made ths to learn about mmap.

This is a first draft, testing out my idea. I copied the doctests from alloc::vec::Vec and they all pass.

I only have one test, where it grows until the virtual address changes , and data still look good.

Next Steps

  • On shrinking, madvise(MADV_FREE)
  • Performance Test

Memory Use

Creating huge vectors does not increase the memory usage immediately. The following line only makes the OS assign 8GB of virtual address in the Process.

let mut vec = LVec::<usize>::with_capacity(1024 * 1000 * 1000);

physical memory is only allocated when you actually use the page.

The same is true for alloc::vec::Vec, so it does not seem that this crate provides any benefit over standard vector.

License

Licensed under either of

at your option.

You might also like...
The Fast Vector Similarity Library is designed to provide efficient computation of various similarity measures between vectors.
The Fast Vector Similarity Library is designed to provide efficient computation of various similarity measures between vectors.

Fast Vector Similarity Library Introduction The Fast Vector Similarity Library is designed to provide efficient computation of various similarity meas

The first compute-centric vector graphic video game
The first compute-centric vector graphic video game

🕹️ Vong This repository contains source code for the first native use of a compute-centric vector graphics video game, inspired by Pong. ✍️ Authors @

An AI-native lightweight, reliable, and high performance open-source vector database.
An AI-native lightweight, reliable, and high performance open-source vector database.

What is OasysDB? OasysDB is a vector database that can be used to store and query high-dimensional vectors. Our goal is to make OasysDB fast and easy

A personally annotated copy of the
A personally annotated copy of the "The Rust Programming Language"

Rust Book This is a personally annotated copy of the "The Rust Programming Language"1. Why Rust For me, I've never really been exposed to low-level sy

A minimal and fast zero-copy parser for the PE32+ file format.

peview A minimal and fast zero-copy parser for the PE32+ file format. Goal This project aims to offer a more light weight and easier to use alternativ

Zero-copy, no-std proquint encoding and decoding

proqnt A pronounceable quintuplet, or proquint, is a pronounceable 5-letter string encoding a unique 16-bit integer. Proquints may be used to encode b

Additional Rust collections not found in std::collections

More collections Rust crate with additional collections not found in std::collections. Multimaps Completion Name Behaves like 🟩 🟩 🟨 ⬜️ ⬜️ HashSetMu

Rust libraries for Bluesky's AT Protocol services. NOT STABLE (yet)

ATrium ATrium is a collection of Rust libraries designed to work with the AT Protocol, providing a versatile and coherent ecosystem for developers. Th

Indeed, an ORM library, not a framework, written in Rust

Ormlib Indeed, an ORM library, not a framework, written in Rust Features The main idea that I put into my ORM library is a minimum of stupid code and

Owner
Wonko der Verständige
🧉
Wonko der Verständige
Rust library for concurrent data access, using memory-mapped files, zero-copy deserialization, and wait-free synchronization.

mmap-sync mmap-sync is a Rust crate designed to manage high-performance, concurrent data access between a single writer process and multiple reader pr

Cloudflare 97 Jun 26, 2023
A HashMap/Vector hybrid: efficient, ordered key-value data storage in Rust.

hashvec A HashVec is a hash map / dictionary whose key-value pairs are stored (and can be iterated over) in a fixed order, by default the order in whi

Skye Terran 2 May 16, 2022
`memory_pages` is a small library provinig a cross-platform API to request pages from kernel with certain premisions

memory_pages: High level API for low level memory management While using low-level memory management in a project can provide substantial benefits, it

null 14 Mar 30, 2023
A special rope, designed to work with any data type that is not String

AnyRope AnyRope is an arbitrary data type rope for Rust, designed for similar operations that a rope would do, but targeted at data types that are not

ahoyiski 27 Mar 22, 2023
Multi-platform desktop app to download and run Large Language Models(LLM) locally in your computer.

Multi-platform desktop app to download and run Large Language Models(LLM) locally in your computer ?? Download | Give it a Star ⭐ | Share it on Twitte

Julio Andres 73 Jun 15, 2023
Powerful math lib for Vector, Matrix and Quaternion operations

An opinionated, powerful math lib for Vector2, Vector3, Matrix and Quaternion operations Vector2 Add, Sub, Div, Mul, Eq Distance Move towards target a

O(ʒ) 4 Mar 28, 2022
Powerful math lib for Vector, Matrix and Quaternion operations

An opinionated, powerful math lib for Vector2, Vector3, Matrix and Quaternion operations Vector2 Add, Sub, Div, Mul, Eq Distance Move towards target a

O(ʒ) 4 Mar 28, 2022
Type erased vector. All elements have the same type.

Type erased vector. All elements have the same type. Designed to be type-erased as far as possible - most of the operations does not know about concre

null 7 Dec 3, 2022
Compact, clone-on-write vector and string.

ecow Compact, clone-on-write vector and string. Types An EcoVec is a reference-counted clone-on-write vector. It takes up two words of space (= 2 usiz

Typst 78 Apr 18, 2023
Vemcache is an in-memory vector database.

Vemcache Vemcache is an in-memory vector database. Vemcache can be thought of as the Redis equivalent for vector databases. Getting Started Prerequisi

Faizaan Chishtie 8 May 21, 2023