Hi @nevi-me , I noticed the GH action was failing to build so I added the component for rustfmt. I then noticed tests failing and #29 so here is the start of integrating Postgres into GH. It is still failing with relation \"arrow_data\" does not exist" though but I figured you would know more about how you want to run setup for these tests.

I am interested in working on adding support for loading a partitioned parquet dataset (multiple files) as datasource in a read transformation.

What do you think is the best way to implement this? Here are couple options that come to my mind:

• add a new variant DataSrouceType::ParquetDataset
• extend DataSrouceType::Parquet to also take a list of file paths
• introduce to higher level abstraction that represents a sequence of data sources so other formats like CSV can reuse this feature as well

Add the ability to sort dataframe by one or multiple criteria. Before implementing this, look at what Arrow's doing, and whether it's possible to implement this functionality there.

Hi - I appreciate the good work you guys are doing here and I'd like to contribute to this project. Is there a way to get in contact? My e-mail address is [email protected].

Adds inital sorting support, currently limited to Arrow's ArrowNumericType. More sort types are pending upstream work (floats, lists, structs, etc).

Closes #16

Add the ability to read a data source and create a deferred dataframe with the source's schema.

There should be a common interface for:

• [x] CSV
• [x] JSON
• [x] SQL
• [x] Arrow File

ref #4

This PR implements a ChunkedArray, Column and Table, which although not part of the Arrow format, are used in CPP, Python, Go and potentially other implementations.

This is currently incomplete, and I need to think of:

• How to make the functions module work with Column instead of PrimitiveArray<T>.
• Where to use references to avoid copies and moves
• How to zero-copy arrays and chunks thereof

A Table that works similarly to other Arrow impls (cpp, python, js, Java)

The aim is to move this to the Arrow repo once I'm happy with it. There is a dependency on:

• [x] https://issues.apache.org/jira/browse/ARROW-3954 (slice to array)
• [x] https://issues.apache.org/jira/browse/ARROW-3706 (record batch reader trait)
• [x] https://issues.apache.org/jira/browse/ARROW-3688 (append_values)

For some computations, we need the ability to convert primitive types to native types. I thought a simple std::convert::From would work, but it's not working.

Should be able to:

• [x] Create a dataframe from data [#3]
• [x] Add columns to dataframe
• [x] Remove columns from dataframe
• [x] Create new columns as computations
• [x] Select a subset of the dataframe by column names

From table.rs:195-207 there seem to be the possibility of multiple branch on variable bounded_len

    pub fn take(&self, indices: &UInt32Array, chunk_size: usize) -> Result<Self> {
        let mut consumed_len = 0;
        let total_len = indices.len();
        let values = self.to_array()?;
        let mut outputs = vec![];
        while consumed_len < total_len {
            let bounded_len = if total_len < chunk_size {
                total_len
            } else if consumed_len + chunk_size > total_len {
                chunk_size
            } else {
                total_len - consumed_len
            };

I was trying to keep things simple, avoiding a context, but the typesystem won't allow me to have custom sources without some stateful place to register them.

Nothing special here, so I'll take a cue from DataFusion. What I'm interested in bikeshedding here is how to create an expressive API that allows data sources to declare their capabilities (e.g. I can pushdown sorts, filters, projections).

If I can get it to work, I'd want to contribute it to DataFusion, as that's where I think a de-facto Rust data analysis library should be.

I am experimenting with evaluating lazy frame with a custom data source. However, looks like Reader being declared as a struct makes it hard to add support for custom data source that shouldn't be part of the dataframe core code base.

Would it make sense to change Reader and Writer into traits so that custom data source implementations can be fully decoupled from the core code base?

Consider adding support for reading data from an Apache Arrow Flight server. Such support could be in the form of a flight client, that gets Arrow data and converts it into a table

The lazy evaluation model seems to be fine for most operations, and can be usable when the existing holes are plugged. Next step is to plot out how optimisations on Vec<Computation> would work. I can try out simple optimisations such as reordering a filter and calculate to filter before calculating.

In order to implement aggregations, we need to be able to group data. Like joins, the task of grouping probably belongs upstream, but we should be able to define how to group data.

The LazyFrame might need some state (whether it's grouped or not) to prevent 'normal' calculations when it's in a grouped state. I don't want to implement a GroupedLazyFrame because we rely on mutating the &mut LazyFrame to add on computations.

An aggregation should ideally take in multiple aggregations. A grouping should take in multiple columns, with columns that aren't grouped or aggregated, getting dropped.