Serde support for n-dimensional arrays from self-describing formats

Overview

serde-ndim

Crates.io Documentation

Overview

This crate provides a way to serialize and deserialize arrays of arbitrary dimensionality from self-described formats such as JSON where no out-of-band data is provided about the shape of the resulting array.

This is useful for some data sources (e.g. in astronomical applications), but not the format supported by the built-in Serde integration of popular crates like ndarray or nalgebra.

Consider input like the following:

[
    [
        [1, 2, 3, 4],
        [4, 5, 6, 7]
    ],
    [
        [7, 8, 9, 10],
        [10, 11, 12, 13]
    ],
    [
        [13, 14, 15, 16],
        [16, 17, 18, 19]
    ]
]

This should deserialize into a 3-dimensional array of shape [3, 2, 4]. This crate provides serialize and deserialize functions that can be used via #[serde(with = "serde_ndim")] that do just that.

Deserialization

The tricky bit is that deserialization is built to learn and ensure internal consistency while reading the data:

  1. During the first descent, it waits until it reaches a leaf number (1) to determine number of dimensions from recursion depth (3 in example above).
  2. It unwinds from the number one step up and reads the sequence [1, 2, 3, 4], learning its length (4). Now it remembers the expected shape as [unknown, unknown, 4] - it hasn't seen the lengths of the upper dimensions, but at least it knows there are 3 dimensions and the last one has length 4.
  3. It unwinds a step up, recurses into the next sequence, and reads [4, 5, 6, 7]. This time it knows it's not the first descent to this dimension, so instead of learning it, it validates the new length against the stored one (4 == 4, all good).
  4. It reached the end of this sequence of sequences, so now it knows and stores the expected shape as [unknown, 2, 4].
  5. By repeating the process, it eventually learns and validates the shape of the whole array as [3, 2, 4].
  6. All this time it was collecting raw numbers into a flat Vec<_> traditionally as an optimised storage of multidimensional arrays. Now it just needs to call a function that constructs a multidimensional array from the shape and flat data.

Note: The resulting array will be in the standard column-major layout.

Constructors for ndarray::Array and nalgebra::DMatrix are provided out of the box under the ndarray and nalgebra features respectively, so you can use them like this:

use serde::{Deserialize, Serialize};

#[derive(Deserialize, Serialize)]
struct MyStruct {
    #[serde(with = "serde_ndim")]
    ndarray: ndarray::ArrayD<f32>,
    /* ... */
}

You can also reuse deserialization for custom types by implementing the serde_ndarray::de::MakeNDim trait.

Serialization

Serialization is also provided. Its implementaton is much simpler, so I won't go into details here, feel free to check out the code if you want.

It's also provided for ndarray::Array and nalgebra::DMatrix, but if you want to serialize custom types, you can do so by implementing the serde_ndarray::ser::NDim trait.

You might also like...
Rust Library for controlling divoom devices that support REST APIs, such as pixoo-64.
Rust Library for controlling divoom devices that support REST APIs, such as pixoo-64.

Divoom Rust Library for controlling divoom devices that support REST APIs, such as pixoo-64 (and from how divoom's api/doc organizes, maybe more in th

Support SIMD low-memory overhead and high-performance adaptive radix tree.

Artful Artful is an adaptive radix tree library for Rust. At a high-level, it's like a BTreeMap. It is based on the implementation of paper, see The A

A tiny service that downloads files over HTTP links, with resume and restart support.
A tiny service that downloads files over HTTP links, with resume and restart support.

Http Drogue Http Drogue is a tiny service that downloads files over HTTP from links you provide. It can restart and resume interrupted downloads. Http

Rust based magic-string with source map chains support

enhanced-magic-string Rust implementation of https://www.npmjs.com/package/magic-string with original sourcemap chain support. license. This project i

Extended Kinect support for Garry's Mod
Extended Kinect support for Garry's Mod

🦵 gmcl_rekinect This is a reimplementation of Kinect support for Garry's Mod, allowing you to do stuff like this on more platforms. Features Support

concat-arrays: a rust macro for concatenating fixed-size arrays

concat-arrays: a rust macro for concatenating fixed-size arrays This crate defines concat_arrays!, a macro that allows you to concatenate arrays.

🧵 Generate self-describing strings of a given length to help aid software testing
🧵 Generate self-describing strings of a given length to help aid software testing

rust-counter-strings Counter strings generator written in rust to help aid software testing What is a counterstring? "A counterstring is a graduated s

Rust implementations of Fast Fourier Transform convolution and correlation for n-dimensional arrays

fftconvolve Rust implementations of Fast Fourier Transform convolution and correlation for n-dimensional arrays Examples 1-dimensional use fftconvolve

Easy c̵̰͠r̵̛̠ö̴̪s̶̩̒s̵̭̀-t̶̲͝h̶̯̚r̵̺͐e̷̖̽ḁ̴̍d̶̖̔ ȓ̵͙ė̶͎ḟ̴͙e̸̖͛r̶̖͗ë̶̱́ṉ̵̒ĉ̷̥e̷͚̍ s̷̹͌h̷̲̉a̵̭͋r̷̫̊ḭ̵̊n̷̬͂g̵̦̃ f̶̻̊ơ̵̜ṟ̸̈́ R̵̞̋ù̵̺s̷̖̅ţ̸͗!̸̼͋

Rust S̵̓i̸̓n̵̉ I̴n̴f̶e̸r̵n̷a̴l mutability! Howdy, friendly Rust developer! Ever had a value get m̵̯̅ð̶͊v̴̮̾ê̴̼͘d away right under your nose just when

Rust libraries and tools to help with interoperability and testing of serialization formats based on Serde.

The repository zefchain/serde-reflection is based on Facebook's repository novifinancial/serde-reflection. We are now maintaining the project here and

axum-serde is a library that provides multiple serde-based extractors and responders for the Axum web framework.

axum-serde 📑 Overview axum-serde is a library that provides multiple serde-based extractors / responses for the Axum web framework. It also offers a

Fast suffix arrays for Rust (with Unicode support).
Fast suffix arrays for Rust (with Unicode support).

suffix Fast linear time & space suffix arrays for Rust. Supports Unicode! Dual-licensed under MIT or the UNLICENSE. Documentation https://docs.rs/suff

Flou is a domain-specific language (DSL) for describing flowcharts.

Flou is a domain-specific language (DSL) for describing flowcharts. It is also a CLI of the same name that renders the previously mentioned flowchart description into an SVG file.

a super fast scientific calculator with dimensional analysis support written in Rust 🦀

larvae a super fast scientific calculator with dimensional analysis support written in Rust 🦀 🐛 heavily inspired from insect Usage: Command mode: $

A CSV parser for Rust, with Serde support.

csv A fast and flexible CSV reader and writer for Rust, with support for Serde. Dual-licensed under MIT or the UNLICENSE. Documentation https://docs.r

Serde support for encoding/decoding rusty_v8 values

Serde support for encoding/decoding rusty_v8 values

Serde support for (rusty_)v8

serde_v8 Serde support for (rusty_)v8 WIP: see denoland/deno#9540 TODO Experiment with KeyCache to optimize struct keys Experiment with external v8 st

serde support for http crate types Request, Response, Uri, StatusCode, HeaderMap

serde extensions for the http crate types Allows serializing and deserializing the following types from http: Response Request HeaderMap StatusCode Ur

Owner
Ingvar Stepanyan
Ukrainian developer focusing on D2D (development tools & specs).
Ingvar Stepanyan
`N` dimensional cubic chunks crate

Hypercube Todo create OffsetShape<S: Shape, const D: usize> a Shape offset by some amount Api positions(self) -> OffsetPositions<Self> create OffsetPo

null 3 Jan 15, 2023
Prometheus exporter that scrapes data in different formats

data-exporter A prometheus exporter that scrapes remote data or local files and converts them to prometheus metrics. It is similar to json_exporter, b

Fredrik Enestad 5 Sep 27, 2022
DWARF packaging utility, written in Rust, supporting GNU extension and DWARF 5 package formats.

thorin thorin is an DWARF packaging utility for creating DWARF packages (*.dwp files) out of input DWARF objects (*.dwo files; or *.o files with .dwo

The Rust Programming Language 19 Nov 16, 2022
Serialize & deserialize device tree binary using serde

serde_device_tree Use serde framework to deserialize Device Tree Blob binary files; no_std compatible. Use this library Run example: cargo run --examp

Luo Jia 20 Aug 20, 2022
Macro for fast implementing serialize methods in serde::Serializer trait

impl_serialize! This library provides a simple procedural macro for fast implementing serialize methods in serde::Serializer trait. [dependencies] imp

Eduard Baturin 2 Sep 6, 2022
Procedural macro to derive Serde serializer-deserializer for Prost

prost-serde-derive prost-serde-derive is a procedural macro to generate Serde serializers and deserializers for Prost-generated structs. Rationale Cur

Park Joon-Kyu 4 Dec 15, 2022
A benchmark of Rust/serde deserializers on configuration files

This program compares the time some serde deserializers take to deserialize some string into a configuration-like struct deriving Deserialize. The ben

Denys Séguret 4 Oct 30, 2023
Dataflow system for building self-driving car and robotics applications.

ERDOS ERDOS is a platform for developing self-driving cars and robotics applications. Getting started The easiest way to get ERDOS running is to use o

ERDOS 163 Dec 29, 2022
Linked Atomic Random Insert Vector: a thread-safe, self-memory-managed vector with no guaranteed sequential insert.

Linked Atomic Random Insert Vector Lariv is a thread-safe, self-memory-managed vector with no guaranteed sequential insert. It internally uses a linke

Guillem Jara 8 Feb 1, 2023
A Rust implementation of generic prefix tree (trie) map with wildcard capture support

prefix_tree_map A Rust implementation of generic prefix tree (trie) map with wildcard capture support. Design Trie is a good data structure for storin

EAimTY 3 Dec 6, 2022