High-level netCDF bindings for Rust

Related tags

Geospatial netcdf
Overview

netcdf

Docs Build Status Crates.io codecov Crates.io

Medium-level netCDF bindings for Rust, allowing easy reading and writing of array-like structures to a file. netCDF can read and write hdf5 files, which is a commonly used file format in scientific computing.

Status

Supported:

  • Variables
  • Normal Dimensions
  • Attributes
  • Subgroups
  • Open/Append/Create modes
  • Reading from memory (read only for now)
  • Unlimited dimensions
  • string variables
  • user defined types (variable length, enum, compound, opaque)

Not (yet) supported:

  • some exotic user defined types

All variable data is read into a contiguous buffer, or inta an ndarray if the ndarray feature is activated.

Building

This crate depends on libnetcdf, but a static build from source is also supported, which can be enabled using the static feature.

The crate is built on several platforms using github actions, and is currently known to build form from source on all major platforms (linux, macos, windows (gnu+msvc)), and through the package installers conda and apt.

Documentation

Some examples of usage can be found in the tests/lib.rs file. The documentation can also be found using cargo doc.

Thread safety

The netcdf-c library is not threadsafe. To render a safe interface, a global mutex is used to serialize access to the underlying library. If performance is needed, consider using a non threadsafe version of hdf5, so double locking is avoided.

Use of netcdf-sys is not thread-safe. Users of this library must take care that calls do not interfere with simultaneous use of e.g. netcdf. Using the hdf5-sys library could also pose a problem, as this library is used throughout netCDF-c and internal state may be disrupted.

License

Licensed under either of

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Comments
  • static build of netcdf

    static build of netcdf

    Introduces feature static which builds both hdf5 and netcdf from source. Adds CI for conda and static builds on the three major platforms.

    Closes #29 Closes #46

    opened by magnusuMET 24
  • Move repo into organization?

    Move repo into organization?

    @milesgranger suggested in #22 that we create an org for this repo or find another org willing to take ownership of it. I'm in support of this - my time is limited and I haven't been keeping up with Rust since I wrote the original version of this crate in 2015.

    Any suggestions or opinions on the details here?

    • Where exactly should we move it?
    • Who wants to take on a maintainer role?
    opened by mhiley 23
  • Investigate where netcdf-c breaks thread safety

    Investigate where netcdf-c breaks thread safety

    The global mutex is in no way ideal when reading/writing from multiple threads, and could be split into several (global/per file), or replaced by RWlocks.

    This requires investigation into where netcdf does something thread unsafe, and limit the locking to this part. Should also investigate where HDF5 might be problematic

    One could also integrate bindings to https://github.com/Parallel-NetCDF/PnetCDF, but this limits the formats to CLASSIC

    https://github.com/Unidata/netcdf-c/issues/1373

    opened by magnusuMET 22
  • benchmark concurrent access to netcdf file

    benchmark concurrent access to netcdf file

    Test concurrent access to netcdf file using varying number of threads and varying number of open files. At the moment opening several files should not have any effect since there is a library-global mutex.

    This introduces a slightly larger test file (3.1 MB) retrieved from http://test.opendap.org/dap/data/nc/coads_climatology.nc

    Run with:

    $ cargo bench --tests conc
    
    opened by gauteh 19
  • ergonomics and added features

    ergonomics and added features

    This PR makes the crate more ergonomic for users, and adds some missing functionality such as unlimited dimensions.

    Removed:

    • mutex from netcdf-sys
    • direct access to internal structures

    Changed:

    • getters and setters for dimensions/variables/attributes
    • funtions returns a netcdf::Result
    • functions now take Option<&[usize]> for indexes/sizes
    • ndarray is now feature gated
    • tests produces files in a temporary directory

    Added:

    • can open from any Path-like item
    • Deref and DerefMutfor the file makes adding and accessing variables/dimensions easier
    • unlimited dimensions
    • read from a memory slice (feature gated due to travis netcdf version)
    • tests
    opened by magnusuMET 16
  • thread 'main' panicked at 'Unable to locate HDF5 root directory and/or headers.'

    thread 'main' panicked at 'Unable to locate HDF5 root directory and/or headers.'

    Hello, I've been having this error thrown by rust-analyser since trying to use this crate. I'm not overly familiar with netcdf files so I'm not exactly sure what I need to install/ where to install it from. (honestly when I look up HDF5 I'm not sure which site to even open up haha)

    opened by Butterygames 10
  • package version

    package version

    The "crates.io" version of this package (0.1.0) is from 2015. And still the current version is 0.1.0

    Maybe version in Cargo.toml should be increased? At least to 0.2.0? There are new methods such as 'Variable::values_at', etc. compared to version of 2015.

    opened by alllel 8
  • Change header name in netcdf-sys

    Change header name in netcdf-sys

    Hello,

    I wasn't able to build netcdf-sys without renaming "netcdf_v4.3.3.1.h" to "netcdf.h", and without changing the gcc call in "netcdf-sys/build.rs" (to include "netcdf.h"). I've made a fork that I could compile, it does work but some units tests fail. I assume it's caused by my version of libnetcdf-dev. (1:4.1.3-7ubuntu2 amd64).

    I'm both new to rust and to the netcdf file format :S, I really hope I'm not wasting your time. Best regards,

    evomassiny

    opened by evomassiny 8
  • Add rustdoc

    Add rustdoc

    Hi,

    I was wondering if you would be open to pull requests if I were to help write up some documentation. I figure it would be generated using rustdoc and be hosted using github-pages.

    I'm new to Rust, but I thought I could at least do this much to help out. :smile:

    opened by richli 6
  • Troubles reading xarray netcdf files

    Troubles reading xarray netcdf files

    Hello !

    I'm trying the examples/ncdump.rs bin to read files generated from python by xarray.Dataset.to_netcdf : https://xarray.pydata.org/en/stable/generated/xarray.Dataset.to_netcdf.html

    https://github.com/georust/netcdf/blob/ca0970edf8f28517319b0a08b010eba449c6ec94/examples/ncdump.rs#L71

    thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Netcdf(-45)', netcdf-0.5.2/src/variable.rs:138:56
    

    The -45 being NC_EBADTYPE from https://github.com/georust/netcdf/blob/71b41fe/netcdf-sys/src/netcdf_const.rs#L109, I suppose.

    I added a catch_unwind to get more details: it seems to fail on types coming from numpy: fixed-length sequence of chars (dtype('S1'), dtype('S4'), etc.)

    Any idea on where to go from there ?

    Thanks a lot !

    opened by NotSqrt 5
  • [WIP] Open from bytes slice

    [WIP] Open from bytes slice

    Will close #13

    Disclosure, I'm very new to interfacing Rust with C.

    With that said, I've gotten as far as I can, and so far I see it, the test I've added should pass, but I get an error code 2 back from the nc_open_mem as suggested in the original issue. So I must be doing a silly thing here and there.

    Willing to go further with a few hints. :+1:

    opened by milesgranger 5
  • CF Time attribute

    CF Time attribute

    Hello, thank you for the work on this project !

    By any chance, is there a plan to add a CF time attribute reading/parsing to handle the datetime type ? I can't find anything on the docs nor on crates.io I think of something like Julia CFtime or Python CFtime

    I believe it would be a great feature for the geophysical field

    enhancement help wanted good first issue 
    opened by antscloud 12
  • Add derive macros for more advanced types

    Add derive macros for more advanced types

    Look to hdf5-rust for how this could look like, where the user can specify arbtrary types and derive NcType for serialisation and deserialisation to netcdf files

    enhancement help wanted 
    opened by magnusuMET 0
  • Chunks iterators

    Chunks iterators

    When reading values from a variable, it should be possible to get a lazy-loading iterator over chunks.

    Implementation details: new method: fn values_chunked(start, chunklength, &mut buffer) -> ChunkIterator { }

    struct ChunkIterator { start: () buflen: () buffer: () }

    enhancement help wanted good first issue 
    opened by magnusuMET 0
  • Parallel read

    Parallel read

    Hi, I think both netcdf and hdf5 supports parallel reads. Glimpsing at the source of this project there seem to be some global locks. Is parallel netcdf supported?

    I am interested in the raw xdr encoded data for a dap server. Is it possible to access the raw bytes through this library?

    opened by gauteh 6
Releases(v0.7.0)
  • v0.7.0(Jan 28, 2022)

  • v0.6.0(Jan 8, 2021)

  • v0.5.0(Aug 10, 2020)

    This release features the following:

    • Correctness fixes for seriial access to nc functions
    • Support for user defined types
    • Using github actions for all CI
    • Fix for buffer overflow for path on File
    • New feature static which allows for building netCDF and hdf5 from source
    Source code(tar.gz)
    Source code(zip)
Owner
GeoRust
A collection of geospatial tools and libraries written in Rust
GeoRust
Rust bindings for the latest stable release of PROJ

PROJ Coordinate transformation via bindings to the PROJ v7.2.1 API. Two coordinate transformation operations are currently provided: projection (and i

GeoRust 96 Dec 21, 2022
Rust bindings for GEOS

geos Rust bindings for GEOS C API. The supported geos version is >= 3.5 Disclaimer GEOS can be a tad strict on the validity on the input geometry and

GeoRust 75 Dec 11, 2022
Rust bindings for the latest stable release of PROJ

PROJ Coordinate transformation via bindings to the PROJ v7.2.1 API. Two coordinate transformation operations are currently provided: projection (and i

GeoRust 96 Dec 21, 2022
Rust bindings for GDAL

gdal [] GDAL bindings for Rust. So far, you can: open a raster dataset for reading/writing get size and number of bands get/set projection and geo-tra

GeoRust 208 Dec 27, 2022
Rust crate for performing coordinate transforms

Synopsis A Rust crate use for performing coordinate transformations. The crate relies on nalgebra vectors to perform the coordinate transformations. C

Dave 25 Aug 20, 2022
An fast, offline reverse geocoder (>1,000 HTTP requests per second) in Rust.

Rust Reverse Geocoder A fast reverse geocoder in Rust. Inspired by Python reverse-geocoder. Links Crate 2.0.0 Docs 1.0.1 Docs Description rrgeo takes

Grant Miner 91 Dec 29, 2022
Geospatial primitives and algorithms for Rust

geo Geospatial Primitives, Algorithms, and Utilities The geo crate provides geospatial primitive types such as Point, LineString, and Polygon, and pro

GeoRust 989 Dec 29, 2022
Geohash for Rust

Rust-Geohash Rust-Geohash is a Rust library for Geohash algorithm. Ported from node-geohash module. Documentation Docs Check the API doc at docs.rs Li

GeoRust 74 Sep 8, 2022
Rust read/write support for well-known text (WKT)

wkt Rust read/write support for well-known text (WKT). License Licensed under either of Apache License, Version 2.0 (LICENSE-APACHE or http://www.apac

GeoRust 40 Dec 11, 2022
Google Encoded Polyline encoding & decoding in Rust.

polyline Google Encoded Polyline encoding & decoding in Rust. A Note on Coordinate Order This crate uses Coordinate and LineString types from the geo-

GeoRust 14 Dec 11, 2022
Geocoding library for Rust.

geocoding Rust utilities to enrich addresses, cities, countries, and landmarks with geographic coordinates through third-party geocoding web services.

GeoRust 55 Dec 12, 2022
Rust read/write support for GPS Exchange Format (GPX)

gpx gpx is a library for reading and writing GPX (GPS Exchange Format) files. It uses the primitives provided by geo-types to allow for storage of GPS

GeoRust 63 Dec 5, 2022
Geospatial primitives and algorithms for Rust

geo Geospatial Primitives, Algorithms, and Utilities The geo crate provides geospatial primitive types such as Point, LineString, and Polygon, and pro

GeoRust 990 Jan 1, 2023
Spatial Data Structures for Rust

spade Documentation Using spade Examples Project state Performance License Spade (SPAtial DatastructurEs, obviously!) implements a few nifty data stru

Stefan Altmayer 195 Dec 21, 2022
Rust implementation of the Martinez-Rueda Polygon Clipping Algorithm

Boolean operations on geo shapes This is an implementation of the Martinez-Rueda Polygon Clipping Algorithm in rust to integrate smoothly into the alr

21re 70 Nov 28, 2022
Fast 2D Delaunay triangulation in Rust. A port of Delaunator.

delaunator-rs A very fast static 2D Delaunay triangulation library for Rust. A port of Delaunator. Documentation Example use delaunator::{Point, trian

Vladimir Agafonkin 123 Dec 20, 2022
port of MapBox's earcut triangulation code to Rust language

Earcutr This is a port of the MapBox company's Earcut computer code, which triangulates polygons. Please see https://github.com/mapbox/earcut for more

don bright 41 Dec 26, 2022
Rust read/write support for GPS Exchange Format (GPX)

gpx gpx is a library for reading and writing GPX (GPS Exchange Format) files. It uses the primitives provided by geo-types to allow for storage of GPS

GeoRust 63 Dec 5, 2022
Reading GeoTIFFs in Rust, nothing else!

A TIFF Library for Rust I needed this library to import elevation models for a routing library. As elevation models usually come in GeoTIFF format, bu

GeoRust 29 Dec 14, 2022