Open Data Access Layer that connect the whole world together

Related tags

Database data storage s3
Overview

OpenDAL

Open Data Access Layer that connect the whole world together.

Status

OpenDAL is in alpha stage and has been early adopted by databend. Welcome any feedback at Discussions!

Quickstart

use opendal::Operator;
use opendal::services::fs;
use anyhow::Result;

#[tokio::main]
async fn main() -> Result<()>{
    let mut op = Operator::new(fs::Backend::build().finish().await?);
    
    // Write data into file.
    let n = op.write("/path/to/file.txt", 1024).run(buf).await?;
    
    // Read data from file.
    let mut r = op.read("/path/to/file.txt").run().await?;
    r.read_to_end(&mut buf).await?;
    
    // Get file metadata.
    let o = op.stat("/path/to/file.txt").run().await?;
    
    // Delete file.
    op.delete("/path/to/file.txt").run().await?;
}

License

OpenDAL is licensed under Apache 2.0.

Comments
  • refactor: replace error::other with new_other_object_error

    refactor: replace error::other with new_other_object_error

    Signed-off-by: STRRL [email protected]

    I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

    Summary

    close: https://github.com/datafuselabs/opendal/issues/563

    • I created 2 functions new_other_object_error and new_other_backend_error which could return a io:Error diectlly.
    opened by STRRL 15
  • Tracking Issues of OpenDAL Observability

    Tracking Issues of OpenDAL Observability

    We have moved all our observability code into layers: https://opendal.databend.rs/opendal/layers/index.html

    But we didn't track the structs returned by our APIs like reader and writer.

    This issue is used to track this problem.

    Tasks

    Logging

    • [ ] Add log for Reader returned by read
    • [ ] Add log for Reader inputed in write
    • [ ] Add log for DirStreamer returned by list

    Tracing

    • [ ] Add trace for Reader returned by read
    • [ ] Add trace for Reader inputed in write
    • [ ] Add trace for DirStreamer returned by list

    Metrics

    • [ ] Add metrics for Reader returned by read
    • [ ] Add metrics for Reader inputed in write
    • [ ] Add metrics for DirStreamer returned by list
    good first issue help wanted 
    opened by Xuanwo 10
  • ipfs should support init from existing CID/IPFS/IPNS

    ipfs should support init from existing CID/IPFS/IPNS

    Our current ipfs implementation uses the MFS API.

    To interact with existing CID, we need to cp /ipfs/<cid> / before starting:

    Content added with "ipfs add" (which by default also becomes pinned) is not added to MFS. Any content can be lazily referenced from MFS with the command. "ipfs files cp /ipfs/ /some/path/" (see ipfs files cp --help).

    Ref: https://docs.ipfs.tech/reference/kubo/cli/#ipfs-files-cp

    opened by Xuanwo 9
  • Support concurrent requests in `memory` backend.

    Support concurrent requests in `memory` backend.

    Currently memory backend use Arc<Mutex<HashMap<_, _>>> as it's inner store. Thus it doesn't allow concurrent requests at all (even on different inmem files). And blocks entire store while serving a single request on a single file.

    Along with being non-performant, it doesn't allow to test concurrency behaviour of application using inmem backend (memory backend can be often used for such tests).

    Much appropriate will be to use standard concurrent safe hashmap like dashmap, or evmap, which distributes locks at inner bucket level.

    opened by damooo 8
  • oli: Implement basic cp command

    oli: Implement basic cp command

    I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

    Summary

    Related to #675

    Implement basic cp for oli:

    oli cp <source_file> <target_file>
    
    • [x] Parse oli profiles from env, and then build accessor objects.
    • [x] Parse source and target file path, and then find the right accessor.
    • [x] Read from source and then write to target.
    • [ ] Add docs.
    opened by eastfisher 8
  • feat: implemented ftp backend

    feat: implemented ftp backend

    I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

    Summary

    Summary about this PR I've implemented ftp backend. I tested it locally with vsftpd and it works fine. I didn't make too much tests on tls mode, but it should be working as supftp almost handle everything for us.

    It also pass all the behavior tests:

    czy@czy-Inspiron-5680:~/opendal$ cargo test ftp
       Compiling opendal v0.13.1 (/home/czy/opendal)
        Finished test [unoptimized + debuginfo] target(s) in 9.02s
         Running unittests src/lib.rs (target/debug/deps/opendal-5b46cd0e7a5789ec)
    
    running 0 tests
    
    test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 29 filtered out; finished in 0.00s
    
         Running tests/behavior/main.rs (target/debug/deps/behavior-516c49dc2e664db6)
    
    running 42 tests
    test behavior::services_ftp::test_create_dir ... ok
    test behavior::services_ftp::test_create_file_existing ... ok
    test behavior::services_ftp::test_create_file_with_special_chars ... ok
    test behavior::services_ftp::test_delete_empty_dir ... ok
    test behavior::services_ftp::test_create_file ... ok
    test behavior::services_ftp::test_list_dir_with_file_path ... ok
    test behavior::services_ftp::test_delete ... ok
    test behavior::services_ftp::test_list_dir ... ok
    test behavior::services_ftp::test_delete_not_existing ... ok
    test behavior::services_ftp::test_check ... ok
    test behavior::services_ftp::test_create_dir_exising ... ok
    test behavior::services_ftp::test_delete_with_special_chars ... ok
    test behavior::services_ftp::test_list_nested_dir ... ok
    test behavior::services_ftp::test_list_empty_dir ... ok
    test behavior::services_ftp::test_list_sub_dir ... ok
    test behavior::services_ftp::test_object_id ... ok
    test behavior::services_ftp::test_object_name ... ok
    test behavior::services_ftp::test_multipart_abort ... ok
    test behavior::services_ftp::test_multipart_complete ... ok
    test behavior::services_ftp::test_object_path ... ok
    test behavior::services_ftp::test_presign_read ... ok
    test behavior::services_ftp::test_presign_write ... ok
    test behavior::services_ftp::test_presign_write_multipart ... ok
    test behavior::services_ftp::test_metadata ... ok
    test behavior::services_ftp::test_read_full ... ok
    test behavior::services_ftp::test_read_not_exist ... ok
    test behavior::services_ftp::test_read_range ... ok
    test behavior::services_ftp::test_read_with_dir_path ... ok
    test behavior::services_ftp::test_stat_dir ... ok
    test behavior::services_ftp::test_stat_not_cleaned_path ... ok
    test behavior::services_ftp::test_read_with_special_chars ... ok
    test behavior::services_ftp::test_remove_all ... ok
    test behavior::services_ftp::test_stat_not_exist ... ok
    test behavior::services_ftp::test_stat_root ... ok
    test behavior::services_ftp::test_stat_with_special_chars ... ok
    test behavior::services_ftp::test_walk_bottom_up ... ok
    test behavior::services_ftp::test_write ... ok
    test behavior::services_ftp::test_write_with_special_chars ... ok
    test behavior::services_ftp::test_write_with_dir_path ... ok
    test behavior::services_ftp::test_stat ... ok
    test behavior::services_ftp::test_walk_top_down ... ok
    test behavior::services_ftp::test_walk_top_down_within_empty_dir ... ok
    
    test result: ok. 42 passed; 0 failed; 0 ignored; 0 measured; 252 filtered out; finished in 0.01s
    

    I created a example file in examples/ if you want to try. The error handling looks kind cluttered tho, and Im trying to improve it.

    opened by ArberSephirotheca 8
  • hdfs accessor failed to list dir and read file contents

    hdfs accessor failed to list dir and read file contents

    ENV:

    1. doesn't config hdfs.root.
    2. hadoop file structure likes: / -- /aa -- --/aa/bc -- --/aa/cd -- /user

    problem:

    1. when I do list on "/aa/" , expect got "/aa/bc" and "/aa/cd" but in fact we got "user/" and "aa/"
    2. when read "/aa/bc", failed with exception file not exists: "/bc"
    opened by sandflee 8
  • Native compress support

    Native compress support

    Make opendal work well with zstd, zip and so on.

    For the first stage, we will focus on the read part of compress which means

    • decompress of a single compressed file like xxx.zstd, yyy.gz
    • unarchive of an archived file like aaa.tar, bbb.zip

    It's obvious that unarchive support will depend on decompress support.

    As requested by our community, we will support zip first to make sure our design is in the right direction.

    Promise

    • No API breakage (existing code will work as usual)
    • Zero cost (please sending PR if you can write better implementations)
    • Features gated (compress and archive will be gated by carge features)

    Future

    The specific API is subject to RFC

    After this feature supported, OpenDAL users can read compressed file like the following:

    let o = op.object("abc.zip");
    let meta = o.stat().await?;
    let r = if o.actions().decompress() {
       o.decompress()
    } else {
       o.reader()
    }
    // Read the data as usual.
    

    OpenDAL will help address all problems around gz, zstd, zip, xz, everything works!

    One more thing: docker image / CD / DVD are also archived files, so we can ...

    Unresolved Tasks

    • [ ] Introduce object actions (so that we can treat decompress/unarchive as an object action)
    • [ ] Support decompress action
    • [ ] Support unarchive action
    opened by Xuanwo 8
  • backend/s3: Anonymous access not supported

    backend/s3: Anonymous access not supported

    Background

    The s3://repo.databend.rs/dataset/stateful/ontime.csv ACL is Everyone (public access) on AWS S3.

    If the credentials not set for OpenDAL: image

    will get the error:

    displayText = Parse csv error at line 0, cause: unexpected: (op: read, path: /dataset/stateful/ontime.csv, source: failed to construct request: No credentials in the property bag
    

    Enhancement It would be better to add a test for minio bucket with public ACL.

    opened by BohuTANG 8
  • Feature request: atomic write on local fs

    Feature request: atomic write on local fs

    Because write on local fs is not atomic, when using local fs as backend, there is a chance a partially written file is left in local fs and can be seen by other client:

    https://github.com/datafuselabs/opendal/blob/9fdd2d6eecc4b927a489803f5728970543b50d1f/src/services/fs/backend.rs#L271-L280

    To use local fs as an object store, a write operation should be implemented as an atomic operation:

    • Write to a temp file in the same mount point as the destination file(rename requirements), e.g., write("my_file.tmp-12345");
    • Flush all possibly buffered data to disk.
    • Rename the temp file to its destination, .e.g., rename("my_file.tmp-12345", "my_file").

    One of the implementation concern is where to put these temp files: since a user file may have an arbitrary file name, it may have to introduce another config entry to define the temp file dir.

    enhancement 
    opened by drmingdrmer 7
  • feat(services/ftp): Setup integration tests

    feat(services/ftp): Setup integration tests

    Signed-off-by: Xuanwo [email protected]

    I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

    Summary

    Fix https://github.com/datafuselabs/opendal/issues/626

    opened by Xuanwo 7
  • oss: Allow specify presign endpoint

    oss: Allow specify presign endpoint

    OSS will have endpoints with and without internal. Users may use an internal endpoint for services but want to use a public endpoint to presign.

    We can add a new optional config called presign_endpoint and use it for presign if has been set.

    opened by Xuanwo 1
  • Implement parquet specific optimizations while reading from remote sources

    Implement parquet specific optimizations while reading from remote sources

    opendal is in general a generic way to access data from various storage backends. For datalake usecases where defacto standard being parquet there can be optimizations can be done specific to data reading.

    Some ideas were given in this blog post: https://developer.nvidia.com/blog/optimizing-access-to-parquet-data-with-fsspec/

    opened by kesavkolla 4
  • Auto ranged read support

    Auto ranged read support

    https://github.com/aws/aws-sdk-cpp/wiki/Improving-S3-Throughput-with-AWS-SDK-for-CPP-v1.9#working-with-the-new-s3-crt-client

    See https://github.com/awslabs/aws-c-s3/blob/374191a730faf7e040a776c7244b5b79c5eeed76/source/s3_auto_ranged_get.c for reference.

    opened by Xuanwo 0
Releases(v0.23.0)
  • v0.23.0(Dec 22, 2022)

    What's Changed

    • feat: Implement object handler so that we can do seek on file by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1091
    • feat: Implement blocking for hdfs by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1092
    • feat(services/hdfs): Implement open and blocking open by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1093
    • docs: Add mozilla/sccache into projects by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1094
    • Bump to version 0.23 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1095

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.22.6...v0.23.0

    Source code(tar.gz)
    Source code(zip)
  • v0.22.6(Dec 20, 2022)

    What's Changed

    • feat(io): make BlockingBytesRead Send + Sync by @sundy-li in https://github.com/datafuselabs/opendal/pull/1083
    • feat(fs): skip seek if offset is 0 by @yang-han in https://github.com/datafuselabs/opendal/pull/1082
    • RFC-1085: Object Handler by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1085
    • feat(services/s3,gcs): Allow accepting signer directly by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1087
    • Bump to version 0.22.6 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1089

    New Contributors

    • @yang-han made their first contribution in https://github.com/datafuselabs/opendal/pull/1082

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.22.5...v0.22.6

    Source code(tar.gz)
    Source code(zip)
  • v0.22.5(Dec 13, 2022)

    What's Changed

    • feat: Add service account support for gcs by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1076
    • Bump to version 0.22.5 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1077

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.22.4...v0.22.5

    Source code(tar.gz)
    Source code(zip)
  • v0.22.4(Dec 13, 2022)

    What's Changed

    • chore(deps): update base64 requirement from 0.13 to 0.20 by @dependabot in https://github.com/datafuselabs/opendal/pull/1067
    • fix: read a large range without error and add test by @Ranxy in https://github.com/datafuselabs/opendal/pull/1068
    • fix(services/oss): Enable standard behavior for oss range by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1070
    • improve blocking read use read_to_end by @sundy-li in https://github.com/datafuselabs/opendal/pull/1072
    • feat(services/gcs): Fully implement default credential support by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1073
    • Bump to version 0.22.4 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1074

    New Contributors

    • @Ranxy made their first contribution in https://github.com/datafuselabs/opendal/pull/1068

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.22.3...v0.22.4

    Source code(tar.gz)
    Source code(zip)
  • v0.22.3(Dec 11, 2022)

    What's Changed

    • fix(services/moka): Don't print all content in cache by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1057
    • feat(layers/metrics): Merge error and failure counters together by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1058
    • feat: Set MSRV to 1.60 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1060
    • feat: Add unwind safe flag for operator by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1061
    • feat(azblob): Add build from connection string support by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1064
    • Bump to version 0.22.3 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1065

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.22.2...v0.22.3

    Source code(tar.gz)
    Source code(zip)
  • v0.22.2(Dec 7, 2022)

    What's Changed

    • feat(presign): support presign head method for s3 and oss by @ZhiHanZ in https://github.com/datafuselabs/opendal/pull/1049
    • chore: use opendal method instead of write by hand by @ZhiHanZ in https://github.com/datafuselabs/opendal/pull/1050
    • Bump to version 0.22.2 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1051

    New Contributors

    • @ZhiHanZ made their first contribution in https://github.com/datafuselabs/opendal/pull/1049

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.22.1...v0.22.2

    Source code(tar.gz)
    Source code(zip)
  • v0.22.1(Dec 5, 2022)

    What's Changed

    • fix(services/s3): Allow disable loading from imds_v2 and assume_role by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1044
    • Bump to version 0.22.1 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1045

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.22.0...v0.22.1

    Source code(tar.gz)
    Source code(zip)
  • v0.22.0(Dec 4, 2022)

    What's Changed

    • chore(deps): update env_logger requirement from 0.9 to 0.10 by @dependabot in https://github.com/datafuselabs/opendal/pull/1015
    • docs: Fix broken links in docs by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1016
    • feat: improve temp file organization when enable atomic write in fs by @killme2008 in https://github.com/datafuselabs/opendal/pull/1017
    • refactor: Polish error handling of different services by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1018
    • refactor: Merge metadata and content cache together by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1020
    • feat: Allow configure LoggingLayer's level by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1021
    • feat: Enable users to specify the cache policy by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1024
    • feat: Implement presign for oss by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1035
    • fix(services/fs): Make sure writing file is truncated by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1036
    • refactor(layer/cache): Allow users implement cache by themselves by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1040
    • Bump to version 0.22 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1041

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.21.2...v0.22.0

    Source code(tar.gz)
    Source code(zip)
  • v0.21.2(Nov 27, 2022)

    What's Changed

    • feat: Add azdfs support by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1009
    • docs: Fix docs for azdfs service by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1010
    • feat: Set MSRV of opendal to 1.60 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1012
    • Bump to version 0.21.2 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1013

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.21.1...v0.21.2

    Source code(tar.gz)
    Source code(zip)
  • v0.21.1(Nov 26, 2022)

    What's Changed

    • deps: Remove not used thiserror and num-trait by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1005
    • feat: Export ObjectLister as public type by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1006
    • Bump to version 0.21.1 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1007

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.21.0...v0.21.1

    Source code(tar.gz)
    Source code(zip)
  • v0.21.0(Nov 25, 2022)

    Upgrade to v0.21

    v0.21 is an internal refactor version of OpenDAL. In this version, we refactored our error handling and our Accessor APIs. Thanks to those internal changes, we added an object-level metadata cache, making it nearly zero cost to reuse existing metadata continuously.

    Let's start with our errors.

    Error Handling

    As described in RFC-0977: Refactor Error, we refactor opendal error by a new error called opendal::Error.

    This change will affect all APIs that are used to return io::Error.

    To migrate this, please replace std::io::Error with opendal::Error:

    - use std::io::Result;
    + use opendal::Result;
    

    And the following error kinds should be updated:

    • std::io::ErrorKind::NotFound => opendal::ErrorKind::ObjectNotFound
    • std::io::ErrorKind::PermissionDenied => opendal::ErrorKind::ObjectPermissionDenied

    And since v0.21, we will return errors ObjectIsADirectory and ObjectNotADirectory instead of anyhow::Error.

    Accessor API

    In v0.21, we refactor the whole Accessor's API:

    - async fn write(&self, path: &str, args: OpWrite, r: BytesReader) -> Result<u64>
    + async fn write(&self, path: &str, args: OpWrite, r: BytesReader) -> Result<RpWrite>
    

    Since v0.21, we will return a reply struct for different operations called RpWrite instead of an exact type. We can split OpenDAL's public API and raw API with this change.

    ObjectList and ObjectPage

    Since v0.21, Accessor will return ObjectPager for List:

    - async fn list(&self, path: &str, args: OpList) -> Result<ObjectStreamer>
    + async fn list(&self, path: &str, args: OpList) -> Result<(RpList, ObjectPager)>
    

    And Object will return an ObjectLister which is built upon ObjectPage:

    pub async fn list(&self) -> Result<ObjectLister> { ... }
    

    ObjectLister can be used as an object stream as before. It also provides the function next_page to get the underlying pages directly:

    impl ObjectLister {
        pub async fn next_page(&mut self) -> Result<Option<Vec<Object>>>;
    }
    

    Code Layout

    Since v0.21, we have categorized all APIs into public and raw.

    Public APIs are exposed under opendal::Xxx; they are user-face APIs that are easy to use and understand.

    Raw APIs are exposed under opendal::raw::Xxx; they are implementation details for underlying services and layers.

    Please replace all usage of opendal::io_util::* and opendal::http_util::* to opendal::raw::* instead.

    With this change, new users of OpenDAL maybe be it easier to get started.

    Summary

    Sorry for introducing too much breaking change in a single version. This version can be a solid version for preparing OpenDAL v1.0.

    What's Changed

    • docs: Add greptimedb and mars into projects by @Xuanwo in https://github.com/datafuselabs/opendal/pull/975
    • RFC-0977: Refactor Error by @Xuanwo in https://github.com/datafuselabs/opendal/pull/977
    • refactor: Use seperate Error instead of std::io::Error to avoid confusing by @Xuanwo in https://github.com/datafuselabs/opendal/pull/976
    • fix: RetryAccessor is too verbose by @Xuanwo in https://github.com/datafuselabs/opendal/pull/980
    • refactor: Return ReplyCreate for create operation by @Xuanwo in https://github.com/datafuselabs/opendal/pull/981
    • refactor: Add ReplyRead for read operation by @Xuanwo in https://github.com/datafuselabs/opendal/pull/982
    • refactor: Add RpWrite for write operation by @Xuanwo in https://github.com/datafuselabs/opendal/pull/983
    • refactor: Add RpStat for stat operation by @Xuanwo in https://github.com/datafuselabs/opendal/pull/984
    • refactor: Add RpDelete for delete operations by @Xuanwo in https://github.com/datafuselabs/opendal/pull/985
    • refactor: Add RpPresign for presign operation by @Xuanwo in https://github.com/datafuselabs/opendal/pull/986
    • refactor: Add reply for all multipart operations by @Xuanwo in https://github.com/datafuselabs/opendal/pull/988
    • refactor: Add Reply for all blocking operations by @Xuanwo in https://github.com/datafuselabs/opendal/pull/989
    • feat: impl atomic write for fs service by @killme2008 in https://github.com/datafuselabs/opendal/pull/991
    • refactor: Avoid accessor in object entry by @Xuanwo in https://github.com/datafuselabs/opendal/pull/992
    • refactor: Move accessor into raw apis by @Xuanwo in https://github.com/datafuselabs/opendal/pull/994
    • refactor: Move io to raw by @Xuanwo in https://github.com/datafuselabs/opendal/pull/996
    • feat: Add OperatorMetadata to avoid expose AccessorMetadata by @Xuanwo in https://github.com/datafuselabs/opendal/pull/997
    • refactor: Move {path,wrapper,http_util,io_util} into raw modules by @Xuanwo in https://github.com/datafuselabs/opendal/pull/998
    • refactor: Move ObjectEntry and ObjectPage into raw by @Xuanwo in https://github.com/datafuselabs/opendal/pull/999
    • refactor: Accept Operator intead of Arc by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1001
    • feat: Improve display for error by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1002
    • Bump to version 0.21 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1003

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.20.1...v0.21.0

    Source code(tar.gz)
    Source code(zip)
  • v0.20.1(Nov 18, 2022)

    What's Changed

    • fix: Use std Duration as args instead by @Xuanwo in https://github.com/datafuselabs/opendal/pull/966
    • build: Make opendal buildable on 1.60 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/968
    • feat: Implement blocking operations for cache services by @Xuanwo in https://github.com/datafuselabs/opendal/pull/970
    • fix: Avoid cache missing after write by @Xuanwo in https://github.com/datafuselabs/opendal/pull/971
    • Bump to version 0.20.1 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/972

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.20.0...v0.20.1

    Source code(tar.gz)
    Source code(zip)
  • v0.20.0(Nov 17, 2022)

    Upgrade to v0.20

    v0.20 is a big release that we introduce a lot of performance related changes.

    To make the best of information from read operation, we propose and implemented RFC-0926: Object Reader. By this RFC, we can fetch content length from ObjectReader now!

    pub struct ObjectReader {
        inner: BytesReader
        meta: ObjectMetadata,
    }
    
    impl ObjectReader {
        pub fn content_length(&self) -> u64 {}
        pub fn last_modified(&self) -> Option<OffsetDateTime> {}
        pub fn etag(&self) -> Option<String> {}
    }
    

    To make this happen, we changed our Accessor API:

    - async fn read(&self, path: &str, args: OpRead) -> Result<BytesReader> {}
    + async fn read(&self, path: &str, args: OpRead) -> Result<ObjectReader> {}
    

    All layers should be updated to meet this change. Also, it's required to return content_length while building ObjectReader. Please make sure the returning ObjectMetadata is used correctly.

    What's Changed

    • RFC-0926: Object Reader by @Xuanwo in https://github.com/datafuselabs/opendal/pull/926
    • feat: Implement Object Reader by @Xuanwo in https://github.com/datafuselabs/opendal/pull/928
    • refactor: Return ObjectReader in Accessor::read by @Xuanwo in https://github.com/datafuselabs/opendal/pull/929
    • feat(services/s3): Return Object Meta for Read operation by @Xuanwo in https://github.com/datafuselabs/opendal/pull/932
    • feat: Implement Bytes Content Range by @Xuanwo in https://github.com/datafuselabs/opendal/pull/933
    • feat: Add Content Range support in ObjectMetadata by @Xuanwo in https://github.com/datafuselabs/opendal/pull/935
    • feat(layers/content_cache): Implement WholeCacheReader by @Xuanwo in https://github.com/datafuselabs/opendal/pull/936
    • feat: CompressAlgorithm derive serde. by @youngsofun in https://github.com/datafuselabs/opendal/pull/939
    • fix(ops): Fix suffix range behavior of bytes range by @Xuanwo in https://github.com/datafuselabs/opendal/pull/942
    • refactor(oay,oli): drop unnecessary patch.crates-io from Cargo.toml by @messense in https://github.com/datafuselabs/opendal/pull/944
    • feat: Allow using opendal without tls support by @messense in https://github.com/datafuselabs/opendal/pull/945
    • refactor: Refactor OpRead with BytesRange by @Xuanwo in https://github.com/datafuselabs/opendal/pull/946
    • feat: Allow using opendal with native tls support by @messense in https://github.com/datafuselabs/opendal/pull/949
    • docs: add docs for tls dependencies features by @messense in https://github.com/datafuselabs/opendal/pull/951
    • refactor: Polish bytes range by @Xuanwo in https://github.com/datafuselabs/opendal/pull/950
    • feat: Make ObjectReader content_length returned for all services by @Xuanwo in https://github.com/datafuselabs/opendal/pull/954
    • feat(layers): Implement fixed content cache by @Xuanwo in https://github.com/datafuselabs/opendal/pull/953
    • fix: Fix cache path not used correctly by @Xuanwo in https://github.com/datafuselabs/opendal/pull/958
    • refactor: Use simplifed kv adapter instead by @Xuanwo in https://github.com/datafuselabs/opendal/pull/959
    • feat: Enable default_ttl support for redis by @Xuanwo in https://github.com/datafuselabs/opendal/pull/960
    • Bump to version 0.20.0 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/961

    New Contributors

    • @messense made their first contribution in https://github.com/datafuselabs/opendal/pull/944

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.19.8...v0.20.0

    Source code(tar.gz)
    Source code(zip)
  • v0.19.8(Nov 13, 2022)

    What's Changed

    • ci: Disable ipfs's compress integration tests by @Xuanwo in https://github.com/datafuselabs/opendal/pull/899
    • deps(oay,oli): Update deps by @Xuanwo in https://github.com/datafuselabs/opendal/pull/906
    • refactor: Reduce backend builder log level to debug by @Xuanwo in https://github.com/datafuselabs/opendal/pull/907
    • ci: Make stable rust clippy happy by @Xuanwo in https://github.com/datafuselabs/opendal/pull/911
    • feat(services/moka): Use entry's bytes as capacity weigher by @Xuanwo in https://github.com/datafuselabs/opendal/pull/914
    • feat: Implement rocksdb service by @wfxr in https://github.com/datafuselabs/opendal/pull/913
    • docs: add docs for rocksdb service by @wfxr in https://github.com/datafuselabs/opendal/pull/915
    • fix(http): Check already read size before returning by @Xuanwo in https://github.com/datafuselabs/opendal/pull/919
    • refactor: Remove deprecated features by @Xuanwo in https://github.com/datafuselabs/opendal/pull/920
    • refactor: use moka::sync::SegmentedCache by @PsiACE in https://github.com/datafuselabs/opendal/pull/921
    • Bump to version 0.19.8 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/923

    New Contributors

    • @wfxr made their first contribution in https://github.com/datafuselabs/opendal/pull/913

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.19.7...v0.19.8

    Source code(tar.gz)
    Source code(zip)
  • v0.19.7(Oct 31, 2022)

    What's Changed

    • feat: Implement content type support for stat by @Xuanwo in https://github.com/datafuselabs/opendal/pull/891
    • refactor(layers/metrics): Holding all metrics handlers to avoid lock by @Xuanwo in https://github.com/datafuselabs/opendal/pull/894
    • refactor(layers/metrics): Only update metrics while dropping readers by @Xuanwo in https://github.com/datafuselabs/opendal/pull/896
    • Bump to version 0.19.7 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/897

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.19.6...v0.19.7

    Source code(tar.gz)
    Source code(zip)
  • v0.19.6(Oct 25, 2022)

    What's Changed

    • fix: Metrics blocking reader doesn't handle operation correctly by @Xuanwo in https://github.com/datafuselabs/opendal/pull/887
    • Bump to version 0.19.6 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/888

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.19.5...v0.19.6

    Source code(tar.gz)
    Source code(zip)
  • v0.19.5(Oct 24, 2022)

    What's Changed

    • chore(deps): update redis requirement from 0.21 to 0.22 by @dependabot in https://github.com/datafuselabs/opendal/pull/876
    • feat: add a feature named trust-dns by @PsiACE in https://github.com/datafuselabs/opendal/pull/879
    • feat: implement write_with by @ClSlaid in https://github.com/datafuselabs/opendal/pull/880
    • feat: content-type configuration by @ClSlaid in https://github.com/datafuselabs/opendal/pull/878
    • chore(deps): update quick-xml requirement from 0.25 to 0.26 by @dependabot in https://github.com/datafuselabs/opendal/pull/882
    • fix: Allow forward layers' acesser operations to inner by @Xuanwo in https://github.com/datafuselabs/opendal/pull/884
    • Bump to version 0.19.5 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/885

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.19.4...v0.19.5

    Source code(tar.gz)
    Source code(zip)
  • v0.19.4(Oct 15, 2022)

    What's Changed

    • refactor: replace md5 with md-5 by @PsiACE in https://github.com/datafuselabs/opendal/pull/862
    • feat: Improve into_stream by reduce zero byte fill by @Xuanwo in https://github.com/datafuselabs/opendal/pull/864
    • debug: Add log for sync http client by @Xuanwo in https://github.com/datafuselabs/opendal/pull/865
    • refactor: replace the hard code to X_AMZ_BUCKET_REGION constant by @WenyXu in https://github.com/datafuselabs/opendal/pull/866
    • feat: Add debug log for finishing read by @Xuanwo in https://github.com/datafuselabs/opendal/pull/867
    • feat: Try to use trust-dns-resolver by @Xuanwo in https://github.com/datafuselabs/opendal/pull/869
    • feat: Add log for dropping reader and streamer by @Xuanwo in https://github.com/datafuselabs/opendal/pull/870
    • Bump to version 0.19.4 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/871

    New Contributors

    • @WenyXu made their first contribution in https://github.com/datafuselabs/opendal/pull/866

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.19.3...v0.19.4

    Source code(tar.gz)
    Source code(zip)
  • v0.19.3(Oct 13, 2022)

    What's Changed

    • fix: Retry for wirte is not implemented correctly by @Xuanwo in https://github.com/datafuselabs/opendal/pull/860
    • Bump to version 0.19.3 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/861

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.19.2...v0.19.3

    Source code(tar.gz)
    Source code(zip)
  • v0.19.2(Oct 13, 2022)

    What's Changed

    • feat(experiment): Allow user to config http connection pool by @Xuanwo in https://github.com/datafuselabs/opendal/pull/843
    • feat: Add concurrent limit layer by @Xuanwo in https://github.com/datafuselabs/opendal/pull/848
    • feat: Allow kv services implemented without list support by @Xuanwo in https://github.com/datafuselabs/opendal/pull/850
    • feat(experiment): add a basic content data cache layer by @PsiACE in https://github.com/datafuselabs/opendal/pull/849
    • feat: Implement service for moka by @Xuanwo in https://github.com/datafuselabs/opendal/pull/852
    • docs: Add docs for moka service and concurrent limit layer by @Xuanwo in https://github.com/datafuselabs/opendal/pull/857
    • Bump to version 0.19.2 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/858

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.19.1...v0.19.2

    Source code(tar.gz)
    Source code(zip)
  • v0.19.1(Oct 11, 2022)

    What's Changed

    • feat: Allow retry read and write by @Xuanwo in https://github.com/datafuselabs/opendal/pull/826
    • feat: Convert interrupted error to permanent after retry by @Xuanwo in https://github.com/datafuselabs/opendal/pull/827
    • test: accelerate behaviour test test_list_rich_dir by @ClSlaid in https://github.com/datafuselabs/opendal/pull/828
    • feat(services/ftp): Add connection pool for FTP by @Xuanwo in https://github.com/datafuselabs/opendal/pull/832
    • feat: Implement retry for write operation by @Xuanwo in https://github.com/datafuselabs/opendal/pull/831
    • feat: Bump reqsign to latest version by @Xuanwo in https://github.com/datafuselabs/opendal/pull/837
    • feat(services/s3): Add role_arn and external_id for assume_role by @Xuanwo in https://github.com/datafuselabs/opendal/pull/838
    • fix: ObjectEntry returned in batch operator doesn't have corrent accessor by @Xuanwo in https://github.com/datafuselabs/opendal/pull/839
    • fix: Accessor in layers not set correctly by @Xuanwo in https://github.com/datafuselabs/opendal/pull/840
    • Bump to version 0.19.1 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/841

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.19.0...v0.19.1

    Source code(tar.gz)
    Source code(zip)
  • v0.19.0(Oct 8, 2022)

    Upgrade to v0.19

    OpenDAL deprecate some features:

    • serde: We will enable it by default.
    • layers-retry: We will enable retry support by default.
    • layers-metadata-cache: We will enable it by default.

    Deprecated types like DirEntry has been removed.


    What's Changed

    • refactor: Move object to mod by @Xuanwo in https://github.com/datafuselabs/opendal/pull/786
    • feat: Implement object page stream for services like s3 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/787
    • refactor: Implement azblob dir stream based on ObjectPageStream by @Xuanwo in https://github.com/datafuselabs/opendal/pull/790
    • RFC-0793: Generic KV Services by @Xuanwo in https://github.com/datafuselabs/opendal/pull/793
    • feat(services/kv): Implement Scoped Key by @Xuanwo in https://github.com/datafuselabs/opendal/pull/796
    • feat: Add scan in KeyValueAccessor by @Xuanwo in https://github.com/datafuselabs/opendal/pull/797
    • feat: Implement basic kv services support by @Xuanwo in https://github.com/datafuselabs/opendal/pull/799
    • refactor: Implement memory services by generic kv by @Xuanwo in https://github.com/datafuselabs/opendal/pull/800
    • feat: Introduce kv adapter for opendal by @Xuanwo in https://github.com/datafuselabs/opendal/pull/802
    • fix(services/redis): MATCH can't handle correctly by @Xuanwo in https://github.com/datafuselabs/opendal/pull/803
    • feat: Add integration test for redis by @Xuanwo in https://github.com/datafuselabs/opendal/pull/804
    • docs: Add docs for redis by @Xuanwo in https://github.com/datafuselabs/opendal/pull/807
    • fix: Disable ipfs redirection by @Xuanwo in https://github.com/datafuselabs/opendal/pull/809
    • fix(services/ipfs): Use ipfs files API to copy data by @Xuanwo in https://github.com/datafuselabs/opendal/pull/811
    • feat: Add OSS Service Support by @ClSlaid in https://github.com/datafuselabs/opendal/pull/801
    • feat: Add integration tests for OSS by @Xuanwo in https://github.com/datafuselabs/opendal/pull/814
    • fix(services/hdfs): Allow retrying would block by @Xuanwo in https://github.com/datafuselabs/opendal/pull/815
    • refactor: Don't expose backend to users by @Xuanwo in https://github.com/datafuselabs/opendal/pull/816
    • tests: allow running tests when env is true by @ClSlaid in https://github.com/datafuselabs/opendal/pull/818
    • refactor: Remove deprecated type aliases by @Xuanwo in https://github.com/datafuselabs/opendal/pull/819
    • test: list rich dir by @ClSlaid in https://github.com/datafuselabs/opendal/pull/820
    • Bump to version 0.19.0 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/821

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.18.2...v0.19.0

    Source code(tar.gz)
    Source code(zip)
  • v0.18.2(Oct 1, 2022)

    What's Changed

    • feat: Enable retry layer by default by @Xuanwo in https://github.com/datafuselabs/opendal/pull/781
    • ci: Enable IPFS NoFecth to avoid networking timeout by @Xuanwo in https://github.com/datafuselabs/opendal/pull/780
    • fix: Fix build error under release profile by @Xuanwo in https://github.com/datafuselabs/opendal/pull/782
    • ci: Build all feature in release to prevent build failure under release profile by @Xuanwo in https://github.com/datafuselabs/opendal/pull/783
    • Bump to version 0.18.2 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/784

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.18.1...v0.18.2

    Source code(tar.gz)
    Source code(zip)
  • v0.18.1(Oct 1, 2022)

    What's Changed

    • docs: Fix wrong link to object_store by @Xuanwo in https://github.com/datafuselabs/opendal/pull/773
    • fix(services/s3): Content MD5 not set during list by @Xuanwo in https://github.com/datafuselabs/opendal/pull/775
    • test: Add a test for ObjectEntry metadata cache by @Xuanwo in https://github.com/datafuselabs/opendal/pull/776
    • Bump to version 0.18.1 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/777

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.18.0...v0.18.1

    Source code(tar.gz)
    Source code(zip)
  • v0.18.0(Oct 1, 2022)

    Upgrade to v0.18

    OpenDAL v0.18 introduces the following breaking changes:

    • Deprecated feature flag services-http has been removed.
    • All DirXxx items have been renamed to ObjectXxx to make them more consistent.
      • DirEntry -> ObjectEntry
      • DirStream -> ObjectStream
      • DirStreamer -> ObjectStream
      • DirIterate -> ObjectIterate
      • DirIterator -> ObjectIterator

    Besides, we also make a big change to our ObjectEntry API. Since v0.18, we can fully reuse the metadata that fetched during list. Take entry.content_length() for example:

    • If content_lenght is already known, we will return directly.
    • If not, we will check if the object entry is complete:
      • If complete, the entry already fetched all metadata that it could have, return directly.
      • If not, we will send a stat call to get the metadata and refresh our cache.

    This change means:

    • All API like content_length will be changed into async functions.
    • metadata and blocking_metadata will not return errors anymore.
    • To retrieve the latest meta, please use entry.into_object().metadata() instead.

    What's Changed

    • fix(http_util): Disable auto compress and enable http proxy by @Xuanwo in https://github.com/datafuselabs/opendal/pull/731
    • refactor: replace error::other with new_other_object_error by @STRRL in https://github.com/datafuselabs/opendal/pull/738
    • feat: Add Metadata Cache Layer by @Xuanwo in https://github.com/datafuselabs/opendal/pull/739
    • feat: Bump reqsign version to 0.5 by @ClSlaid in https://github.com/datafuselabs/opendal/pull/741
    • docs: Add comparison with object_store by @Xuanwo in https://github.com/datafuselabs/opendal/pull/745
    • docs: Add docs for list with prefix by @Xuanwo in https://github.com/datafuselabs/opendal/pull/746
    • feat: Derive Hash, Eq, PartialEq for Operation by @Xuanwo in https://github.com/datafuselabs/opendal/pull/749
    • feat: Make AccessorMetadata public so outer users can use by @Xuanwo in https://github.com/datafuselabs/opendal/pull/750
    • feat: Expose AccessorCapability to users by @Xuanwo in https://github.com/datafuselabs/opendal/pull/751
    • chore(compress): log with trace level instead of debug. by @youngsofun in https://github.com/datafuselabs/opendal/pull/752
    • feat: Expose opendal's http util to users by @Xuanwo in https://github.com/datafuselabs/opendal/pull/753
    • feat: Implement convert from PresignedRequest by @Xuanwo in https://github.com/datafuselabs/opendal/pull/756
    • feat: Make ObjectMetadata setter public by @Xuanwo in https://github.com/datafuselabs/opendal/pull/758
    • refactor: Rename DirXxxx to ObjectXxxx instead by @Xuanwo in https://github.com/datafuselabs/opendal/pull/759
    • docs: rename dir entry to object entry by @PsiACE in https://github.com/datafuselabs/opendal/pull/760
    • feat: Implement cached metadata for ObjectEntry by @Xuanwo in https://github.com/datafuselabs/opendal/pull/761
    • deps: Fix build after bump deps of oli and oay by @Xuanwo in https://github.com/datafuselabs/opendal/pull/766
    • feat: Assign unique name for memory backend by @Xuanwo in https://github.com/datafuselabs/opendal/pull/769
    • Bump to version 0.18.0 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/768

    New Contributors

    • @STRRL made their first contribution in https://github.com/datafuselabs/opendal/pull/738
    • @youngsofun made their first contribution in https://github.com/datafuselabs/opendal/pull/752

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.17.4...v0.18.0

    Source code(tar.gz)
    Source code(zip)
  • v0.17.4(Sep 27, 2022)

    What's Changed

    • fix(http_util): Allow retry more errors by @Xuanwo in https://github.com/datafuselabs/opendal/pull/724
    • fix(services/ftp): Suffix endpoints with default port by @ClSlaid in https://github.com/datafuselabs/opendal/pull/726
    • Bump to version 0.17.4 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/727

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.17.3...v0.17.4

    Source code(tar.gz)
    Source code(zip)
  • v0.17.3(Sep 26, 2022)

    What's Changed

    • feat: Add SubdirLayer to allowing switch directory by @Xuanwo in https://github.com/datafuselabs/opendal/pull/718
    • fix: update metrics on result by @ClSlaid in https://github.com/datafuselabs/opendal/pull/716
    • fix: SubdirLayer should handle dir correctly by @Xuanwo in https://github.com/datafuselabs/opendal/pull/720
    • feat(layers/retry): Add warning log while retry happened by @Xuanwo in https://github.com/datafuselabs/opendal/pull/721
    • Bump to version 0.17.3 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/722

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.17.2...v0.17.3

    Source code(tar.gz)
    Source code(zip)
  • v0.17.2(Sep 26, 2022)

    What's Changed

    • fix(services/fs): Handle slash normalized false positives properly by @damooo in https://github.com/datafuselabs/opendal/pull/702
    • oli: Implement basic cp command by @eastfisher in https://github.com/datafuselabs/opendal/pull/688
    • chore: also parse 'FTPS' to Scheme::Ftp by @ClSlaid in https://github.com/datafuselabs/opendal/pull/704
    • chore: fix error message by @ClSlaid in https://github.com/datafuselabs/opendal/pull/705
    • fix: Tracing is too verbose by @Xuanwo in https://github.com/datafuselabs/opendal/pull/707
    • oli: Refactor copy implementation by @Xuanwo in https://github.com/datafuselabs/opendal/pull/710
    • refactor: remove enable_secure in FTP service. by @ClSlaid in https://github.com/datafuselabs/opendal/pull/709
    • Bump to version 0.17.2 by @ClSlaid in https://github.com/datafuselabs/opendal/pull/712

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.17.1...v0.17.2

    Source code(tar.gz)
    Source code(zip)
  • v0.17.1(Sep 19, 2022)

    What's Changed

    • feat: redis service implement by @ClSlaid in https://github.com/datafuselabs/opendal/pull/679
    • feat: Implement AsyncBufRead for IntoReader by @Xuanwo in https://github.com/datafuselabs/opendal/pull/690
    • refactor: avoid unnecessary parent creating in Redis service by @ClSlaid in https://github.com/datafuselabs/opendal/pull/692
    • feat: expose security token of s3 by @ClSlaid in https://github.com/datafuselabs/opendal/pull/693
    • fix: Handle write data in async way for IPMFS by @xprazak2 in https://github.com/datafuselabs/opendal/pull/694
    • refactor: Refactor HTTP Client to split sending and incoming logic by @Xuanwo in https://github.com/datafuselabs/opendal/pull/695
    • ci: Try make ipfs test stable by @Xuanwo in https://github.com/datafuselabs/opendal/pull/696
    • Bump to version 0.17.1 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/698

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.17.0...v0.17.1

    Source code(tar.gz)
    Source code(zip)
  • v0.17.0(Sep 15, 2022)

    Upgrade to v0.17

    OpenDAL v0.17 refactor the Accessor to make space for future features.

    We move path String out of the OpXxx to function args so that we don't need to clone twice.

    - async fn read(&self, args: OpRead) -> Result<BytesReader>
    + async fn read(&self, path: &str, args: OpRead) -> Result<BytesReader>
    

    For more information about this change, please refer to RFC-0661: Path In Accessor.

    And since OpenDAL v0.17, we will use rustls as default tls engine for our underlying http client. Since this release, we will not depend on openssl anymore.


    What's Changed

    • RFC: Path In Accessor by @Xuanwo in https://github.com/datafuselabs/opendal/pull/661
    • feat: Implement RFC-0661: Path In Accessor by @Xuanwo in https://github.com/datafuselabs/opendal/pull/664
    • docs: Add how to implement service docs by @Xuanwo in https://github.com/datafuselabs/opendal/pull/665
    • fix: Immutable Index Layer could return duplicated pathes by @Xuanwo in https://github.com/datafuselabs/opendal/pull/671
    • feat: Hide http client internal details from users by @Xuanwo in https://github.com/datafuselabs/opendal/pull/672
    • feat: make rustls the default tls implementation by @sunng87 in https://github.com/datafuselabs/opendal/pull/674
    • fix: Remove not needed type parameter for immutable_layer by @Xuanwo in https://github.com/datafuselabs/opendal/pull/677
    • refactor: update redis support rfc by @ClSlaid in https://github.com/datafuselabs/opendal/pull/676
    • feat: Implement benches for layers by @Xuanwo in https://github.com/datafuselabs/opendal/pull/681
    • fix: Don't trace buf field in poll_read by @Xuanwo in https://github.com/datafuselabs/opendal/pull/682
    • docs: update metrics documentation by @ClSlaid in https://github.com/datafuselabs/opendal/pull/684
    • fix: List non-exist dir should return empty by @Xuanwo in https://github.com/datafuselabs/opendal/pull/683
    • fix: Add path validation for fs backend by @Xuanwo in https://github.com/datafuselabs/opendal/pull/685
    • Bump to version 0.17 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/686

    Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.16.0...v0.17.0

    Source code(tar.gz)
    Source code(zip)
Owner
Datafuse Labs
The open-source runtime that powers the Modern Data Cloud
Datafuse Labs
rust_arango enables you to connect with ArangoDB server, access to database, execute AQL query, manage ArangoDB in an easy and intuitive way, both async and plain synchronous code with any HTTP ecosystem you love.

rust_arango enables you to connect with ArangoDB server, access to database, execute AQL query, manage ArangoDB in an easy and intuitive way, both async and plain synchronous code with any HTTP ecosystem you love.

Foretag 3 Mar 24, 2022
Making Postgres and Elasticsearch work together like it's 2021

Making Postgres and Elasticsearch work together like it's 2021 Readme ZomboDB brings powerful text-search and analytics features to Postgres by using

ZomboDB 4.2k Jan 2, 2023
Zenith substitutes PostgreSQL storage layer and redistributes data across a cluster of nodes

Zenith substitutes PostgreSQL storage layer and redistributes data across a cluster of nodes

null 5.7k Jan 6, 2023
A user crud written in Rust, designed to connect to a MySQL database with full integration test coverage.

SQLX User CRUD Purpose This application demonstrates the how to implement a common design for CRUDs in, potentially, a system of microservices. The de

null 78 Nov 27, 2022
This code features a viper-client, which can connect to a viper-server, a custom interface made for Comelit devices.

Viper Client ?? (WIP) This is code for my intercom; specifically for the Comelit Mini Wi-Fi MSFV. This features a ViperClient which can talk to the Co

Gerard 4 Feb 20, 2023
Scalable and fast data store optimised for time series data such as financial data, events, metrics for real time analysis

OnTimeDB Scalable and fast data store optimised for time series data such as financial data, events, metrics for real time analysis OnTimeDB is a time

Stuart 2 Apr 5, 2022
Databend aimed to be an open source elastic and reliable serverless data warehouse,

An elastic and reliable Serverless Data Warehouse, offers Blazing Fast Query and combines Elasticity, Simplicity, Low cost of the Cloud, built to make the Data Cloud easy

Datafuse Labs 5k Jan 3, 2023
a tokio-enabled data store for triple data

terminusdb-store, a tokio-enabled data store for triple data Overview This library implements a way to store triple data - data that consists of a sub

TerminusDB 307 Dec 18, 2022
A Modern Real-Time Data Processing & Analytics DBMS with Cloud-Native Architecture, built to make the Data Cloud easy

A Modern Real-Time Data Processing & Analytics DBMS with Cloud-Native Architecture, built to make the Data Cloud easy

Datafuse Labs 5k Jan 9, 2023
Implements the packet parser for Gran Turismo 7 telemetry data, allowing a developer to retrieve data from a running game.

gran-turismo-query Implements the packet parser for Gran Turismo 7 telemetry data, allowing a developer to retrieve data from a running game. Features

Carlos Menezes 3 Dec 11, 2023
Open Zignatures Database

The openZign project Zignatures and other binary identification database. For fun and to aid reverse-engineering tasks. Collected from various datasou

Cyrill Leutwiler 3 Sep 19, 2021
Owlyshield is an open-source AI-driven behaviour based antiransomware engine written in Rust.

Owlyshield (mailto:[email protected]) We at SitinCloud strongly believe that cybersecurity products should always be open-source: Critical decis

SitinCloud 255 Dec 25, 2022
open source training courses about distributed database and distributed systemes

Welcome to learn Talent Plan Courses! Talent Plan is an open source training program initiated by PingCAP. It aims to create or combine some open sour

PingCAP 8.3k Dec 30, 2022
XLite - query Excel (.xlsx, .xls) and Open Document spreadsheets (.ods) as SQLite virtual tables

XLite - query Excel (.xlsx, .xls) and Open Document spreadsheets (.ods) as SQLite virtual tables XLite is a SQLite extension written in Rust. The main

Sergey Khabibullin 1.1k Dec 28, 2022
LIMITS is yet another fully open source, interoperable, decentralised real-time communication protocol!

LIMITS: Limit-IM does not have ITS LIMITS We are undergoing a major refactoring and technology stack adjustment to better accommodate clustered deploy

Limit LAB 14 Feb 4, 2023
Materialize simplifies application development with streaming data. Incrementally-updated materialized views - in PostgreSQL and in real time. Materialize is powered by Timely Dataflow.

Materialize is a streaming database for real-time applications. Get started Check out our getting started guide. About Materialize lets you ask questi

Materialize, Inc. 4.7k Jan 8, 2023
🐸Slippi DB ingests Slippi replays and puts the data into a SQLite database for easier parsing.

The primary goal of this project is to make it easier to analyze large amounts of Slippi data. Its end goal is to create something similar to Ballchasing.com but for Melee.

Max Timkovich 20 Jan 2, 2023
A Key-Value data storage system. - dorea db

Dorea DB ?? Dorea is a key-value data storage system. It is based on the Bitcask storage model Documentation | Crates.io | API Doucment 简体中文 | English

ZhuoEr Liu 112 Dec 2, 2022
Hypergraph is data structure library to create a directed hypergraph in which a hyperedge can join any number of vertices.

Hypergraph is data structure library to create a directed hypergraph in which a hyperedge can join any number of vertices.

Davy Duperron 224 Dec 21, 2022