Open Data Access Layer that connect the whole world together

Datafuse Labs

Last update: Jan 4, 2023

Related tags

Overview

OpenDAL

Open Data Access Layer that connect the whole world together.

Status

OpenDAL is in alpha stage and has been early adopted by databend. Welcome any feedback at Discussions!

Quickstart

use opendal::Operator;
use opendal::services::fs;
use anyhow::Result;

#[tokio::main]
async fn main() -> Result<()>{
    let mut op = Operator::new(fs::Backend::build().finish().await?);
    
    // Write data into file.
    let n = op.write("/path/to/file.txt", 1024).run(buf).await?;
    
    // Read data from file.
    let mut r = op.read("/path/to/file.txt").run().await?;
    r.read_to_end(&mut buf).await?;
    
    // Get file metadata.
    let o = op.stat("/path/to/file.txt").run().await?;
    
    // Delete file.
    op.delete("/path/to/file.txt").run().await?;
}

License

OpenDAL is licensed under Apache 2.0.

Comments

refactor: replace error::other with new_other_object_error
Signed-off-by: STRRL [email protected]

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

close: https://github.com/datafuselabs/opendal/issues/563

I created 2 functions new_other_object_error and new_other_backend_error which could return a io:Error diectlly.
opened by STRRL 15
Tracking Issues of OpenDAL Observability
We have moved all our observability code into layers: https://opendal.databend.rs/opendal/layers/index.html

But we didn't track the structs returned by our APIs like reader and writer.

This issue is used to track this problem.

Tasks

Logging

[ ] Add log for Reader returned by read

[ ] Add log for Reader inputed in write

[ ] Add log for DirStreamer returned by list

Tracing

[ ] Add trace for Reader returned by read

[ ] Add trace for Reader inputed in write

[ ] Add trace for DirStreamer returned by list

Metrics

[ ] Add metrics for Reader returned by read

[ ] Add metrics for Reader inputed in write

[ ] Add metrics for DirStreamer returned by list

good first issue help wanted
opened by Xuanwo 10
ipfs should support init from existing CID/IPFS/IPNS

Our current ipfs implementation uses the MFS API.

To interact with existing CID, we need to cp /ipfs/<cid> / before starting:

Content added with "ipfs add" (which by default also becomes pinned) is not added to MFS. Any content can be lazily referenced from MFS with the command. "ipfs files cp /ipfs/ /some/path/" (see ipfs files cp --help).

Ref: https://docs.ipfs.tech/reference/kubo/cli/#ipfs-files-cp

opened by Xuanwo 9
Support concurrent requests in `memory` backend.

Currently memory backend use Arc<Mutex<HashMap<_, _>>> as it's inner store. Thus it doesn't allow concurrent requests at all (even on different inmem files). And blocks entire store while serving a single request on a single file.

Along with being non-performant, it doesn't allow to test concurrency behaviour of application using inmem backend (memory backend can be often used for such tests).

Much appropriate will be to use standard concurrent safe hashmap like dashmap, or evmap, which distributes locks at inner bucket level.

opened by damooo 8
oli: Implement basic cp command
I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Related to #675

Implement basic cp for oli:

oli cp <source_file> <target_file>

[x] Parse oli profiles from env, and then build accessor objects.

[x] Parse source and target file path, and then find the right accessor.

[x] Read from source and then write to target.

[ ] Add docs.
opened by eastfisher 8

feat: implemented ftp backend

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Summary about this PR I've implemented ftp backend. I tested it locally with vsftpd and it works fine. I didn't make too much tests on tls mode, but it should be working as supftp almost handle everything for us.

It also pass all the behavior tests:

czy@czy-Inspiron-5680:~/opendal$ cargo test ftp
   Compiling opendal v0.13.1 (/home/czy/opendal)
    Finished test [unoptimized + debuginfo] target(s) in 9.02s
     Running unittests src/lib.rs (target/debug/deps/opendal-5b46cd0e7a5789ec)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 29 filtered out; finished in 0.00s

     Running tests/behavior/main.rs (target/debug/deps/behavior-516c49dc2e664db6)

running 42 tests
test behavior::services_ftp::test_create_dir ... ok
test behavior::services_ftp::test_create_file_existing ... ok
test behavior::services_ftp::test_create_file_with_special_chars ... ok
test behavior::services_ftp::test_delete_empty_dir ... ok
test behavior::services_ftp::test_create_file ... ok
test behavior::services_ftp::test_list_dir_with_file_path ... ok
test behavior::services_ftp::test_delete ... ok
test behavior::services_ftp::test_list_dir ... ok
test behavior::services_ftp::test_delete_not_existing ... ok
test behavior::services_ftp::test_check ... ok
test behavior::services_ftp::test_create_dir_exising ... ok
test behavior::services_ftp::test_delete_with_special_chars ... ok
test behavior::services_ftp::test_list_nested_dir ... ok
test behavior::services_ftp::test_list_empty_dir ... ok
test behavior::services_ftp::test_list_sub_dir ... ok
test behavior::services_ftp::test_object_id ... ok
test behavior::services_ftp::test_object_name ... ok
test behavior::services_ftp::test_multipart_abort ... ok
test behavior::services_ftp::test_multipart_complete ... ok
test behavior::services_ftp::test_object_path ... ok
test behavior::services_ftp::test_presign_read ... ok
test behavior::services_ftp::test_presign_write ... ok
test behavior::services_ftp::test_presign_write_multipart ... ok
test behavior::services_ftp::test_metadata ... ok
test behavior::services_ftp::test_read_full ... ok
test behavior::services_ftp::test_read_not_exist ... ok
test behavior::services_ftp::test_read_range ... ok
test behavior::services_ftp::test_read_with_dir_path ... ok
test behavior::services_ftp::test_stat_dir ... ok
test behavior::services_ftp::test_stat_not_cleaned_path ... ok
test behavior::services_ftp::test_read_with_special_chars ... ok
test behavior::services_ftp::test_remove_all ... ok
test behavior::services_ftp::test_stat_not_exist ... ok
test behavior::services_ftp::test_stat_root ... ok
test behavior::services_ftp::test_stat_with_special_chars ... ok
test behavior::services_ftp::test_walk_bottom_up ... ok
test behavior::services_ftp::test_write ... ok
test behavior::services_ftp::test_write_with_special_chars ... ok
test behavior::services_ftp::test_write_with_dir_path ... ok
test behavior::services_ftp::test_stat ... ok
test behavior::services_ftp::test_walk_top_down ... ok
test behavior::services_ftp::test_walk_top_down_within_empty_dir ... ok

test result: ok. 42 passed; 0 failed; 0 ignored; 0 measured; 252 filtered out; finished in 0.01s

I created a example file in examples/ if you want to try. The error handling looks kind cluttered tho, and Im trying to improve it.

opened by ArberSephirotheca 8

hdfs accessor failed to list dir and read file contents
ENV:

doesn't config hdfs.root.

hadoop file structure likes: / -- /aa -- --/aa/bc -- --/aa/cd -- /user

problem:

when I do list on "/aa/" , expect got "/aa/bc" and "/aa/cd" but in fact we got "user/" and "aa/"

when read "/aa/bc", failed with exception file not exists: "/bc"
opened by sandflee 8
Native compress support
Make opendal work well with zstd, zip and so on.

For the first stage, we will focus on the read part of compress which means

decompress of a single compressed file like xxx.zstd, yyy.gz

unarchive of an archived file like aaa.tar, bbb.zip

It's obvious that unarchive support will depend on decompress support.

As requested by our community, we will support zip first to make sure our design is in the right direction.

Promise

No API breakage (existing code will work as usual)

Zero cost (please sending PR if you can write better implementations)

Features gated (compress and archive will be gated by carge features)

Future

The specific API is subject to RFC

After this feature supported, OpenDAL users can read compressed file like the following:

let o = op.object("abc.zip"); let meta = o.stat().await?; let r = if o.actions().decompress() { o.decompress() } else { o.reader() } // Read the data as usual.

OpenDAL will help address all problems around gz, zstd, zip, xz, everything works!

One more thing: docker image / CD / DVD are also archived files, so we can ...

Unresolved Tasks

[ ] Introduce object actions (so that we can treat decompress/unarchive as an object action)

[ ] Support decompress action

[ ] Support unarchive action
opened by Xuanwo 8
backend/s3: Anonymous access not supported
Background

The s3://repo.databend.rs/dataset/stateful/ontime.csv ACL is Everyone (public access) on AWS S3.

If the credentials not set for OpenDAL:

will get the error:

displayText = Parse csv error at line 0, cause: unexpected: (op: read, path: /dataset/stateful/ontime.csv, source: failed to construct request: No credentials in the property bag

Enhancement It would be better to add a test for minio bucket with public ACL.
opened by BohuTANG 8
Feature request: atomic write on local fs
Because write on local fs is not atomic, when using local fs as backend, there is a chance a partially written file is left in local fs and can be seen by other client:

https://github.com/datafuselabs/opendal/blob/9fdd2d6eecc4b927a489803f5728970543b50d1f/src/services/fs/backend.rs#L271-L280

To use local fs as an object store, a write operation should be implemented as an atomic operation:

Write to a temp file in the same mount point as the destination file(rename requirements), e.g., write("my_file.tmp-12345");

Flush all possibly buffered data to disk.

Rename the temp file to its destination, .e.g., rename("my_file.tmp-12345", "my_file").

One of the implementation concern is where to put these temp files: since a user file may have an arbitrary file name, it may have to introduce another config entry to define the temp file dir.
enhancement
opened by drmingdrmer 7
feat(services/ftp): Setup integration tests

Signed-off-by: Xuanwo [email protected]

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

Fix https://github.com/datafuselabs/opendal/issues/626

opened by Xuanwo 7
oss: Allow specify presign endpoint

OSS will have endpoints with and without internal. Users may use an internal endpoint for services but want to use a public endpoint to presign.

We can add a new optional config called presign_endpoint and use it for presign if has been set.

opened by Xuanwo 1
Implement parquet specific optimizations while reading from remote sources

opendal is in general a generic way to access data from various storage backends. For datalake usecases where defacto standard being parquet there can be optimizations can be done specific to data reading.

Some ideas were given in this blog post: https://developer.nvidia.com/blog/optimizing-access-to-parquet-data-with-fsspec/

opened by kesavkolla 4
Auto ranged read support

https://github.com/aws/aws-sdk-cpp/wiki/Improving-S3-Throughput-with-AWS-SDK-for-CPP-v1.9#working-with-the-new-s3-crt-client

See https://github.com/awslabs/aws-c-s3/blob/374191a730faf7e040a776c7244b5b79c5eeed76/source/s3_auto_ranged_get.c for reference.

opened by Xuanwo 0

Releases(v0.23.0)

v0.23.0(Dec 22, 2022)
What's Changed

feat: Implement object handler so that we can do seek on file by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1091

feat: Implement blocking for hdfs by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1092

feat(services/hdfs): Implement open and blocking open by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1093

docs: Add mozilla/sccache into projects by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1094

Bump to version 0.23 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1095

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.22.6...v0.23.0
Source code(tar.gz)
Source code(zip)
v0.22.6(Dec 20, 2022)
What's Changed

feat(io): make BlockingBytesRead Send + Sync by @sundy-li in https://github.com/datafuselabs/opendal/pull/1083

feat(fs): skip seek if offset is 0 by @yang-han in https://github.com/datafuselabs/opendal/pull/1082

RFC-1085: Object Handler by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1085

feat(services/s3,gcs): Allow accepting signer directly by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1087

Bump to version 0.22.6 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1089

New Contributors

@yang-han made their first contribution in https://github.com/datafuselabs/opendal/pull/1082

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.22.5...v0.22.6
Source code(tar.gz)
Source code(zip)
v0.22.5(Dec 13, 2022)
What's Changed

feat: Add service account support for gcs by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1076

Bump to version 0.22.5 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1077

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.22.4...v0.22.5
Source code(tar.gz)
Source code(zip)
v0.22.4(Dec 13, 2022)
What's Changed

chore(deps): update base64 requirement from 0.13 to 0.20 by @dependabot in https://github.com/datafuselabs/opendal/pull/1067

fix: read a large range without error and add test by @Ranxy in https://github.com/datafuselabs/opendal/pull/1068

fix(services/oss): Enable standard behavior for oss range by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1070

improve blocking read use read_to_end by @sundy-li in https://github.com/datafuselabs/opendal/pull/1072

feat(services/gcs): Fully implement default credential support by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1073

Bump to version 0.22.4 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1074

New Contributors

@Ranxy made their first contribution in https://github.com/datafuselabs/opendal/pull/1068

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.22.3...v0.22.4
Source code(tar.gz)
Source code(zip)
v0.22.3(Dec 11, 2022)
What's Changed

fix(services/moka): Don't print all content in cache by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1057

feat(layers/metrics): Merge error and failure counters together by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1058

feat: Set MSRV to 1.60 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1060

feat: Add unwind safe flag for operator by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1061

feat(azblob): Add build from connection string support by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1064

Bump to version 0.22.3 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1065

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.22.2...v0.22.3
Source code(tar.gz)
Source code(zip)
v0.22.2(Dec 7, 2022)
What's Changed

feat(presign): support presign head method for s3 and oss by @ZhiHanZ in https://github.com/datafuselabs/opendal/pull/1049

chore: use opendal method instead of write by hand by @ZhiHanZ in https://github.com/datafuselabs/opendal/pull/1050

Bump to version 0.22.2 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1051

New Contributors

@ZhiHanZ made their first contribution in https://github.com/datafuselabs/opendal/pull/1049

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.22.1...v0.22.2
Source code(tar.gz)
Source code(zip)
v0.22.1(Dec 5, 2022)
What's Changed

fix(services/s3): Allow disable loading from imds_v2 and assume_role by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1044

Bump to version 0.22.1 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1045

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.22.0...v0.22.1
Source code(tar.gz)
Source code(zip)
v0.22.0(Dec 4, 2022)
What's Changed

chore(deps): update env_logger requirement from 0.9 to 0.10 by @dependabot in https://github.com/datafuselabs/opendal/pull/1015

docs: Fix broken links in docs by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1016

feat: improve temp file organization when enable atomic write in fs by @killme2008 in https://github.com/datafuselabs/opendal/pull/1017

refactor: Polish error handling of different services by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1018

refactor: Merge metadata and content cache together by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1020

feat: Allow configure LoggingLayer's level by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1021

feat: Enable users to specify the cache policy by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1024

feat: Implement presign for oss by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1035

fix(services/fs): Make sure writing file is truncated by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1036

refactor(layer/cache): Allow users implement cache by themselves by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1040

Bump to version 0.22 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1041

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.21.2...v0.22.0
Source code(tar.gz)
Source code(zip)
v0.21.2(Nov 27, 2022)
What's Changed

feat: Add azdfs support by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1009

docs: Fix docs for azdfs service by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1010

feat: Set MSRV of opendal to 1.60 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1012

Bump to version 0.21.2 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1013

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.21.1...v0.21.2
Source code(tar.gz)
Source code(zip)
v0.21.1(Nov 26, 2022)
What's Changed

deps: Remove not used thiserror and num-trait by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1005

feat: Export ObjectLister as public type by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1006

Bump to version 0.21.1 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1007

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.21.0...v0.21.1
Source code(tar.gz)
Source code(zip)
v0.21.0(Nov 25, 2022)
Upgrade to v0.21

v0.21 is an internal refactor version of OpenDAL. In this version, we refactored our error handling and our Accessor APIs. Thanks to those internal changes, we added an object-level metadata cache, making it nearly zero cost to reuse existing metadata continuously.

Let's start with our errors.

Error Handling

As described in RFC-0977: Refactor Error, we refactor opendal error by a new error called opendal::Error.

This change will affect all APIs that are used to return io::Error.

To migrate this, please replace std::io::Error with opendal::Error:

- use std::io::Result; + use opendal::Result;

And the following error kinds should be updated:

std::io::ErrorKind::NotFound => opendal::ErrorKind::ObjectNotFound

std::io::ErrorKind::PermissionDenied => opendal::ErrorKind::ObjectPermissionDenied

And since v0.21, we will return errors ObjectIsADirectory and ObjectNotADirectory instead of anyhow::Error.

Accessor API

In v0.21, we refactor the whole Accessor's API:

- async fn write(&self, path: &str, args: OpWrite, r: BytesReader) -> Result<u64> + async fn write(&self, path: &str, args: OpWrite, r: BytesReader) -> Result<RpWrite>

Since v0.21, we will return a reply struct for different operations called RpWrite instead of an exact type. We can split OpenDAL's public API and raw API with this change.

ObjectList and ObjectPage

Since v0.21, Accessor will return ObjectPager for List:

- async fn list(&self, path: &str, args: OpList) -> Result<ObjectStreamer> + async fn list(&self, path: &str, args: OpList) -> Result<(RpList, ObjectPager)>

And Object will return an ObjectLister which is built upon ObjectPage:

pub async fn list(&self) -> Result<ObjectLister> { ... }

ObjectLister can be used as an object stream as before. It also provides the function next_page to get the underlying pages directly:

impl ObjectLister { pub async fn next_page(&mut self) -> Result<Option<Vec<Object>>>; }

Code Layout

Since v0.21, we have categorized all APIs into public and raw.

Public APIs are exposed under opendal::Xxx; they are user-face APIs that are easy to use and understand.

Raw APIs are exposed under opendal::raw::Xxx; they are implementation details for underlying services and layers.

Please replace all usage of opendal::io_util::* and opendal::http_util::* to opendal::raw::* instead.

With this change, new users of OpenDAL maybe be it easier to get started.

Summary

Sorry for introducing too much breaking change in a single version. This version can be a solid version for preparing OpenDAL v1.0.

What's Changed

docs: Add greptimedb and mars into projects by @Xuanwo in https://github.com/datafuselabs/opendal/pull/975

RFC-0977: Refactor Error by @Xuanwo in https://github.com/datafuselabs/opendal/pull/977

refactor: Use seperate Error instead of std::io::Error to avoid confusing by @Xuanwo in https://github.com/datafuselabs/opendal/pull/976

fix: RetryAccessor is too verbose by @Xuanwo in https://github.com/datafuselabs/opendal/pull/980

refactor: Return ReplyCreate for create operation by @Xuanwo in https://github.com/datafuselabs/opendal/pull/981

refactor: Add ReplyRead for read operation by @Xuanwo in https://github.com/datafuselabs/opendal/pull/982

refactor: Add RpWrite for write operation by @Xuanwo in https://github.com/datafuselabs/opendal/pull/983

refactor: Add RpStat for stat operation by @Xuanwo in https://github.com/datafuselabs/opendal/pull/984

refactor: Add RpDelete for delete operations by @Xuanwo in https://github.com/datafuselabs/opendal/pull/985

refactor: Add RpPresign for presign operation by @Xuanwo in https://github.com/datafuselabs/opendal/pull/986

refactor: Add reply for all multipart operations by @Xuanwo in https://github.com/datafuselabs/opendal/pull/988

refactor: Add Reply for all blocking operations by @Xuanwo in https://github.com/datafuselabs/opendal/pull/989

feat: impl atomic write for fs service by @killme2008 in https://github.com/datafuselabs/opendal/pull/991

refactor: Avoid accessor in object entry by @Xuanwo in https://github.com/datafuselabs/opendal/pull/992

refactor: Move accessor into raw apis by @Xuanwo in https://github.com/datafuselabs/opendal/pull/994

refactor: Move io to raw by @Xuanwo in https://github.com/datafuselabs/opendal/pull/996

feat: Add OperatorMetadata to avoid expose AccessorMetadata by @Xuanwo in https://github.com/datafuselabs/opendal/pull/997

refactor: Move {path,wrapper,http_util,io_util} into raw modules by @Xuanwo in https://github.com/datafuselabs/opendal/pull/998

refactor: Move ObjectEntry and ObjectPage into raw by @Xuanwo in https://github.com/datafuselabs/opendal/pull/999

refactor: Accept Operator intead of Arc by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1001

feat: Improve display for error by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1002

Bump to version 0.21 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/1003

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.20.1...v0.21.0
Source code(tar.gz)
Source code(zip)
v0.20.1(Nov 18, 2022)
What's Changed

fix: Use std Duration as args instead by @Xuanwo in https://github.com/datafuselabs/opendal/pull/966

build: Make opendal buildable on 1.60 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/968

feat: Implement blocking operations for cache services by @Xuanwo in https://github.com/datafuselabs/opendal/pull/970

fix: Avoid cache missing after write by @Xuanwo in https://github.com/datafuselabs/opendal/pull/971

Bump to version 0.20.1 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/972

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.20.0...v0.20.1
Source code(tar.gz)
Source code(zip)
v0.20.0(Nov 17, 2022)
Upgrade to v0.20

v0.20 is a big release that we introduce a lot of performance related changes.

To make the best of information from read operation, we propose and implemented RFC-0926: Object Reader. By this RFC, we can fetch content length from ObjectReader now!

pub struct ObjectReader { inner: BytesReader meta: ObjectMetadata, } impl ObjectReader { pub fn content_length(&self) -> u64 {} pub fn last_modified(&self) -> Option<OffsetDateTime> {} pub fn etag(&self) -> Option<String> {} }

To make this happen, we changed our Accessor API:

- async fn read(&self, path: &str, args: OpRead) -> Result<BytesReader> {} + async fn read(&self, path: &str, args: OpRead) -> Result<ObjectReader> {}

All layers should be updated to meet this change. Also, it's required to return content_length while building ObjectReader. Please make sure the returning ObjectMetadata is used correctly.

What's Changed

RFC-0926: Object Reader by @Xuanwo in https://github.com/datafuselabs/opendal/pull/926

feat: Implement Object Reader by @Xuanwo in https://github.com/datafuselabs/opendal/pull/928

refactor: Return ObjectReader in Accessor::read by @Xuanwo in https://github.com/datafuselabs/opendal/pull/929

feat(services/s3): Return Object Meta for Read operation by @Xuanwo in https://github.com/datafuselabs/opendal/pull/932

feat: Implement Bytes Content Range by @Xuanwo in https://github.com/datafuselabs/opendal/pull/933

feat: Add Content Range support in ObjectMetadata by @Xuanwo in https://github.com/datafuselabs/opendal/pull/935

feat(layers/content_cache): Implement WholeCacheReader by @Xuanwo in https://github.com/datafuselabs/opendal/pull/936

feat: CompressAlgorithm derive serde. by @youngsofun in https://github.com/datafuselabs/opendal/pull/939

fix(ops): Fix suffix range behavior of bytes range by @Xuanwo in https://github.com/datafuselabs/opendal/pull/942

refactor(oay,oli): drop unnecessary patch.crates-io from Cargo.toml by @messense in https://github.com/datafuselabs/opendal/pull/944

feat: Allow using opendal without tls support by @messense in https://github.com/datafuselabs/opendal/pull/945

refactor: Refactor OpRead with BytesRange by @Xuanwo in https://github.com/datafuselabs/opendal/pull/946

feat: Allow using opendal with native tls support by @messense in https://github.com/datafuselabs/opendal/pull/949

docs: add docs for tls dependencies features by @messense in https://github.com/datafuselabs/opendal/pull/951

refactor: Polish bytes range by @Xuanwo in https://github.com/datafuselabs/opendal/pull/950

feat: Make ObjectReader content_length returned for all services by @Xuanwo in https://github.com/datafuselabs/opendal/pull/954

feat(layers): Implement fixed content cache by @Xuanwo in https://github.com/datafuselabs/opendal/pull/953

fix: Fix cache path not used correctly by @Xuanwo in https://github.com/datafuselabs/opendal/pull/958

refactor: Use simplifed kv adapter instead by @Xuanwo in https://github.com/datafuselabs/opendal/pull/959

feat: Enable default_ttl support for redis by @Xuanwo in https://github.com/datafuselabs/opendal/pull/960

Bump to version 0.20.0 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/961

New Contributors

@messense made their first contribution in https://github.com/datafuselabs/opendal/pull/944

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.19.8...v0.20.0
Source code(tar.gz)
Source code(zip)
v0.19.8(Nov 13, 2022)
What's Changed

ci: Disable ipfs's compress integration tests by @Xuanwo in https://github.com/datafuselabs/opendal/pull/899

deps(oay,oli): Update deps by @Xuanwo in https://github.com/datafuselabs/opendal/pull/906

refactor: Reduce backend builder log level to debug by @Xuanwo in https://github.com/datafuselabs/opendal/pull/907

ci: Make stable rust clippy happy by @Xuanwo in https://github.com/datafuselabs/opendal/pull/911

feat(services/moka): Use entry's bytes as capacity weigher by @Xuanwo in https://github.com/datafuselabs/opendal/pull/914

feat: Implement rocksdb service by @wfxr in https://github.com/datafuselabs/opendal/pull/913

docs: add docs for rocksdb service by @wfxr in https://github.com/datafuselabs/opendal/pull/915

fix(http): Check already read size before returning by @Xuanwo in https://github.com/datafuselabs/opendal/pull/919

refactor: Remove deprecated features by @Xuanwo in https://github.com/datafuselabs/opendal/pull/920

refactor: use moka::sync::SegmentedCache by @PsiACE in https://github.com/datafuselabs/opendal/pull/921

Bump to version 0.19.8 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/923

New Contributors

@wfxr made their first contribution in https://github.com/datafuselabs/opendal/pull/913

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.19.7...v0.19.8
Source code(tar.gz)
Source code(zip)
v0.19.7(Oct 31, 2022)
What's Changed

feat: Implement content type support for stat by @Xuanwo in https://github.com/datafuselabs/opendal/pull/891

refactor(layers/metrics): Holding all metrics handlers to avoid lock by @Xuanwo in https://github.com/datafuselabs/opendal/pull/894

refactor(layers/metrics): Only update metrics while dropping readers by @Xuanwo in https://github.com/datafuselabs/opendal/pull/896

Bump to version 0.19.7 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/897

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.19.6...v0.19.7
Source code(tar.gz)
Source code(zip)
v0.19.6(Oct 25, 2022)
What's Changed

fix: Metrics blocking reader doesn't handle operation correctly by @Xuanwo in https://github.com/datafuselabs/opendal/pull/887

Bump to version 0.19.6 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/888

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.19.5...v0.19.6
Source code(tar.gz)
Source code(zip)
v0.19.5(Oct 24, 2022)
What's Changed

chore(deps): update redis requirement from 0.21 to 0.22 by @dependabot in https://github.com/datafuselabs/opendal/pull/876

feat: add a feature named trust-dns by @PsiACE in https://github.com/datafuselabs/opendal/pull/879

feat: implement write_with by @ClSlaid in https://github.com/datafuselabs/opendal/pull/880

feat: content-type configuration by @ClSlaid in https://github.com/datafuselabs/opendal/pull/878

chore(deps): update quick-xml requirement from 0.25 to 0.26 by @dependabot in https://github.com/datafuselabs/opendal/pull/882

fix: Allow forward layers' acesser operations to inner by @Xuanwo in https://github.com/datafuselabs/opendal/pull/884

Bump to version 0.19.5 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/885

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.19.4...v0.19.5
Source code(tar.gz)
Source code(zip)
v0.19.4(Oct 15, 2022)
What's Changed

refactor: replace md5 with md-5 by @PsiACE in https://github.com/datafuselabs/opendal/pull/862

feat: Improve into_stream by reduce zero byte fill by @Xuanwo in https://github.com/datafuselabs/opendal/pull/864

debug: Add log for sync http client by @Xuanwo in https://github.com/datafuselabs/opendal/pull/865

refactor: replace the hard code to X_AMZ_BUCKET_REGION constant by @WenyXu in https://github.com/datafuselabs/opendal/pull/866

feat: Add debug log for finishing read by @Xuanwo in https://github.com/datafuselabs/opendal/pull/867

feat: Try to use trust-dns-resolver by @Xuanwo in https://github.com/datafuselabs/opendal/pull/869

feat: Add log for dropping reader and streamer by @Xuanwo in https://github.com/datafuselabs/opendal/pull/870

Bump to version 0.19.4 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/871

New Contributors

@WenyXu made their first contribution in https://github.com/datafuselabs/opendal/pull/866

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.19.3...v0.19.4
Source code(tar.gz)
Source code(zip)
v0.19.3(Oct 13, 2022)
What's Changed

fix: Retry for wirte is not implemented correctly by @Xuanwo in https://github.com/datafuselabs/opendal/pull/860

Bump to version 0.19.3 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/861

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.19.2...v0.19.3
Source code(tar.gz)
Source code(zip)
v0.19.2(Oct 13, 2022)
What's Changed

feat(experiment): Allow user to config http connection pool by @Xuanwo in https://github.com/datafuselabs/opendal/pull/843

feat: Add concurrent limit layer by @Xuanwo in https://github.com/datafuselabs/opendal/pull/848

feat: Allow kv services implemented without list support by @Xuanwo in https://github.com/datafuselabs/opendal/pull/850

feat(experiment): add a basic content data cache layer by @PsiACE in https://github.com/datafuselabs/opendal/pull/849

feat: Implement service for moka by @Xuanwo in https://github.com/datafuselabs/opendal/pull/852

docs: Add docs for moka service and concurrent limit layer by @Xuanwo in https://github.com/datafuselabs/opendal/pull/857

Bump to version 0.19.2 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/858

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.19.1...v0.19.2
Source code(tar.gz)
Source code(zip)
v0.19.1(Oct 11, 2022)
What's Changed

feat: Allow retry read and write by @Xuanwo in https://github.com/datafuselabs/opendal/pull/826

feat: Convert interrupted error to permanent after retry by @Xuanwo in https://github.com/datafuselabs/opendal/pull/827

test: accelerate behaviour test test_list_rich_dir by @ClSlaid in https://github.com/datafuselabs/opendal/pull/828

feat(services/ftp): Add connection pool for FTP by @Xuanwo in https://github.com/datafuselabs/opendal/pull/832

feat: Implement retry for write operation by @Xuanwo in https://github.com/datafuselabs/opendal/pull/831

feat: Bump reqsign to latest version by @Xuanwo in https://github.com/datafuselabs/opendal/pull/837

feat(services/s3): Add role_arn and external_id for assume_role by @Xuanwo in https://github.com/datafuselabs/opendal/pull/838

fix: ObjectEntry returned in batch operator doesn't have corrent accessor by @Xuanwo in https://github.com/datafuselabs/opendal/pull/839

fix: Accessor in layers not set correctly by @Xuanwo in https://github.com/datafuselabs/opendal/pull/840

Bump to version 0.19.1 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/841

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.19.0...v0.19.1
Source code(tar.gz)
Source code(zip)
v0.19.0(Oct 8, 2022)
Upgrade to v0.19

OpenDAL deprecate some features:

serde: We will enable it by default.

layers-retry: We will enable retry support by default.

layers-metadata-cache: We will enable it by default.

Deprecated types like DirEntry has been removed.

What's Changed

refactor: Move object to mod by @Xuanwo in https://github.com/datafuselabs/opendal/pull/786

feat: Implement object page stream for services like s3 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/787

refactor: Implement azblob dir stream based on ObjectPageStream by @Xuanwo in https://github.com/datafuselabs/opendal/pull/790

RFC-0793: Generic KV Services by @Xuanwo in https://github.com/datafuselabs/opendal/pull/793

feat(services/kv): Implement Scoped Key by @Xuanwo in https://github.com/datafuselabs/opendal/pull/796

feat: Add scan in KeyValueAccessor by @Xuanwo in https://github.com/datafuselabs/opendal/pull/797

feat: Implement basic kv services support by @Xuanwo in https://github.com/datafuselabs/opendal/pull/799

refactor: Implement memory services by generic kv by @Xuanwo in https://github.com/datafuselabs/opendal/pull/800

feat: Introduce kv adapter for opendal by @Xuanwo in https://github.com/datafuselabs/opendal/pull/802

fix(services/redis): MATCH can't handle correctly by @Xuanwo in https://github.com/datafuselabs/opendal/pull/803

feat: Add integration test for redis by @Xuanwo in https://github.com/datafuselabs/opendal/pull/804

docs: Add docs for redis by @Xuanwo in https://github.com/datafuselabs/opendal/pull/807

fix: Disable ipfs redirection by @Xuanwo in https://github.com/datafuselabs/opendal/pull/809

fix(services/ipfs): Use ipfs files API to copy data by @Xuanwo in https://github.com/datafuselabs/opendal/pull/811

feat: Add OSS Service Support by @ClSlaid in https://github.com/datafuselabs/opendal/pull/801

feat: Add integration tests for OSS by @Xuanwo in https://github.com/datafuselabs/opendal/pull/814

fix(services/hdfs): Allow retrying would block by @Xuanwo in https://github.com/datafuselabs/opendal/pull/815

refactor: Don't expose backend to users by @Xuanwo in https://github.com/datafuselabs/opendal/pull/816

tests: allow running tests when env is true by @ClSlaid in https://github.com/datafuselabs/opendal/pull/818

refactor: Remove deprecated type aliases by @Xuanwo in https://github.com/datafuselabs/opendal/pull/819

test: list rich dir by @ClSlaid in https://github.com/datafuselabs/opendal/pull/820

Bump to version 0.19.0 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/821

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.18.2...v0.19.0
Source code(tar.gz)
Source code(zip)
v0.18.2(Oct 1, 2022)
What's Changed

feat: Enable retry layer by default by @Xuanwo in https://github.com/datafuselabs/opendal/pull/781

ci: Enable IPFS NoFecth to avoid networking timeout by @Xuanwo in https://github.com/datafuselabs/opendal/pull/780

fix: Fix build error under release profile by @Xuanwo in https://github.com/datafuselabs/opendal/pull/782

ci: Build all feature in release to prevent build failure under release profile by @Xuanwo in https://github.com/datafuselabs/opendal/pull/783

Bump to version 0.18.2 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/784

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.18.1...v0.18.2
Source code(tar.gz)
Source code(zip)
v0.18.1(Oct 1, 2022)
What's Changed

docs: Fix wrong link to object_store by @Xuanwo in https://github.com/datafuselabs/opendal/pull/773

fix(services/s3): Content MD5 not set during list by @Xuanwo in https://github.com/datafuselabs/opendal/pull/775

test: Add a test for ObjectEntry metadata cache by @Xuanwo in https://github.com/datafuselabs/opendal/pull/776

Bump to version 0.18.1 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/777

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.18.0...v0.18.1
Source code(tar.gz)
Source code(zip)
v0.18.0(Oct 1, 2022)
Upgrade to v0.18

OpenDAL v0.18 introduces the following breaking changes:

Deprecated feature flag services-http has been removed.

All DirXxx items have been renamed to ObjectXxx to make them more consistent.

DirEntry -> ObjectEntry

DirStream -> ObjectStream

DirStreamer -> ObjectStream

DirIterate -> ObjectIterate

DirIterator -> ObjectIterator

Besides, we also make a big change to our ObjectEntry API. Since v0.18, we can fully reuse the metadata that fetched during list. Take entry.content_length() for example:

If content_lenght is already known, we will return directly.

If not, we will check if the object entry is complete:

If complete, the entry already fetched all metadata that it could have, return directly.

If not, we will send a stat call to get the metadata and refresh our cache.

This change means:

All API like content_length will be changed into async functions.

metadata and blocking_metadata will not return errors anymore.

To retrieve the latest meta, please use entry.into_object().metadata() instead.

What's Changed

fix(http_util): Disable auto compress and enable http proxy by @Xuanwo in https://github.com/datafuselabs/opendal/pull/731

refactor: replace error::other with new_other_object_error by @STRRL in https://github.com/datafuselabs/opendal/pull/738

feat: Add Metadata Cache Layer by @Xuanwo in https://github.com/datafuselabs/opendal/pull/739

feat: Bump reqsign version to 0.5 by @ClSlaid in https://github.com/datafuselabs/opendal/pull/741

docs: Add comparison with object_store by @Xuanwo in https://github.com/datafuselabs/opendal/pull/745

docs: Add docs for list with prefix by @Xuanwo in https://github.com/datafuselabs/opendal/pull/746

feat: Derive Hash, Eq, PartialEq for Operation by @Xuanwo in https://github.com/datafuselabs/opendal/pull/749

feat: Make AccessorMetadata public so outer users can use by @Xuanwo in https://github.com/datafuselabs/opendal/pull/750

feat: Expose AccessorCapability to users by @Xuanwo in https://github.com/datafuselabs/opendal/pull/751

chore(compress): log with trace level instead of debug. by @youngsofun in https://github.com/datafuselabs/opendal/pull/752

feat: Expose opendal's http util to users by @Xuanwo in https://github.com/datafuselabs/opendal/pull/753

feat: Implement convert from PresignedRequest by @Xuanwo in https://github.com/datafuselabs/opendal/pull/756

feat: Make ObjectMetadata setter public by @Xuanwo in https://github.com/datafuselabs/opendal/pull/758

refactor: Rename DirXxxx to ObjectXxxx instead by @Xuanwo in https://github.com/datafuselabs/opendal/pull/759

docs: rename dir entry to object entry by @PsiACE in https://github.com/datafuselabs/opendal/pull/760

feat: Implement cached metadata for ObjectEntry by @Xuanwo in https://github.com/datafuselabs/opendal/pull/761

deps: Fix build after bump deps of oli and oay by @Xuanwo in https://github.com/datafuselabs/opendal/pull/766

feat: Assign unique name for memory backend by @Xuanwo in https://github.com/datafuselabs/opendal/pull/769

Bump to version 0.18.0 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/768

New Contributors

@STRRL made their first contribution in https://github.com/datafuselabs/opendal/pull/738

@youngsofun made their first contribution in https://github.com/datafuselabs/opendal/pull/752

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.17.4...v0.18.0
Source code(tar.gz)
Source code(zip)
v0.17.4(Sep 27, 2022)
What's Changed

fix(http_util): Allow retry more errors by @Xuanwo in https://github.com/datafuselabs/opendal/pull/724

fix(services/ftp): Suffix endpoints with default port by @ClSlaid in https://github.com/datafuselabs/opendal/pull/726

Bump to version 0.17.4 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/727

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.17.3...v0.17.4
Source code(tar.gz)
Source code(zip)
v0.17.3(Sep 26, 2022)
What's Changed

feat: Add SubdirLayer to allowing switch directory by @Xuanwo in https://github.com/datafuselabs/opendal/pull/718

fix: update metrics on result by @ClSlaid in https://github.com/datafuselabs/opendal/pull/716

fix: SubdirLayer should handle dir correctly by @Xuanwo in https://github.com/datafuselabs/opendal/pull/720

feat(layers/retry): Add warning log while retry happened by @Xuanwo in https://github.com/datafuselabs/opendal/pull/721

Bump to version 0.17.3 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/722

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.17.2...v0.17.3
Source code(tar.gz)
Source code(zip)
v0.17.2(Sep 26, 2022)
What's Changed

fix(services/fs): Handle slash normalized false positives properly by @damooo in https://github.com/datafuselabs/opendal/pull/702

oli: Implement basic cp command by @eastfisher in https://github.com/datafuselabs/opendal/pull/688

chore: also parse 'FTPS' to Scheme::Ftp by @ClSlaid in https://github.com/datafuselabs/opendal/pull/704

chore: fix error message by @ClSlaid in https://github.com/datafuselabs/opendal/pull/705

fix: Tracing is too verbose by @Xuanwo in https://github.com/datafuselabs/opendal/pull/707

oli: Refactor copy implementation by @Xuanwo in https://github.com/datafuselabs/opendal/pull/710

refactor: remove enable_secure in FTP service. by @ClSlaid in https://github.com/datafuselabs/opendal/pull/709

Bump to version 0.17.2 by @ClSlaid in https://github.com/datafuselabs/opendal/pull/712

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.17.1...v0.17.2
Source code(tar.gz)
Source code(zip)
v0.17.1(Sep 19, 2022)
What's Changed

feat: redis service implement by @ClSlaid in https://github.com/datafuselabs/opendal/pull/679

feat: Implement AsyncBufRead for IntoReader by @Xuanwo in https://github.com/datafuselabs/opendal/pull/690

refactor: avoid unnecessary parent creating in Redis service by @ClSlaid in https://github.com/datafuselabs/opendal/pull/692

feat: expose security token of s3 by @ClSlaid in https://github.com/datafuselabs/opendal/pull/693

fix: Handle write data in async way for IPMFS by @xprazak2 in https://github.com/datafuselabs/opendal/pull/694

refactor: Refactor HTTP Client to split sending and incoming logic by @Xuanwo in https://github.com/datafuselabs/opendal/pull/695

ci: Try make ipfs test stable by @Xuanwo in https://github.com/datafuselabs/opendal/pull/696

Bump to version 0.17.1 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/698

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.17.0...v0.17.1
Source code(tar.gz)
Source code(zip)
v0.17.0(Sep 15, 2022)
Upgrade to v0.17

OpenDAL v0.17 refactor the Accessor to make space for future features.

We move path String out of the OpXxx to function args so that we don't need to clone twice.

- async fn read(&self, args: OpRead) -> Result<BytesReader> + async fn read(&self, path: &str, args: OpRead) -> Result<BytesReader>

For more information about this change, please refer to RFC-0661: Path In Accessor.

And since OpenDAL v0.17, we will use rustls as default tls engine for our underlying http client. Since this release, we will not depend on openssl anymore.

What's Changed

RFC: Path In Accessor by @Xuanwo in https://github.com/datafuselabs/opendal/pull/661

feat: Implement RFC-0661: Path In Accessor by @Xuanwo in https://github.com/datafuselabs/opendal/pull/664

docs: Add how to implement service docs by @Xuanwo in https://github.com/datafuselabs/opendal/pull/665

fix: Immutable Index Layer could return duplicated pathes by @Xuanwo in https://github.com/datafuselabs/opendal/pull/671

feat: Hide http client internal details from users by @Xuanwo in https://github.com/datafuselabs/opendal/pull/672

feat: make rustls the default tls implementation by @sunng87 in https://github.com/datafuselabs/opendal/pull/674

fix: Remove not needed type parameter for immutable_layer by @Xuanwo in https://github.com/datafuselabs/opendal/pull/677

refactor: update redis support rfc by @ClSlaid in https://github.com/datafuselabs/opendal/pull/676

feat: Implement benches for layers by @Xuanwo in https://github.com/datafuselabs/opendal/pull/681

fix: Don't trace buf field in poll_read by @Xuanwo in https://github.com/datafuselabs/opendal/pull/682

docs: update metrics documentation by @ClSlaid in https://github.com/datafuselabs/opendal/pull/684

fix: List non-exist dir should return empty by @Xuanwo in https://github.com/datafuselabs/opendal/pull/683

fix: Add path validation for fs backend by @Xuanwo in https://github.com/datafuselabs/opendal/pull/685

Bump to version 0.17 by @Xuanwo in https://github.com/datafuselabs/opendal/pull/686

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.16.0...v0.17.0
Source code(tar.gz)
Source code(zip)