Cloud native log storage and management for Kubernetes, containerised workloads

Overview

Parseable Parseable

website status contributors commits activity monthly Github stars License Twitter

Live Demo | Website | API Workspace on Postman

Parseable is an open source, cloud native, log storage and management platform.

Parseable helps you ingest high volumes of log data from various sources (Fluent Bit, Logstash etc or directly from applications). Parseable stores log data into highly compressed Parquet file format. With object storage as primary storage for Parseable, you get seamless scale and flexibility for ever growing log data.

Parseable Overview Parseable Overview

Live Demo

Try out Parseable server with our demo instance.

  1. Post log data to default log stream frontend
curl --location --request POST 'https://demo.parseable.io/api/v1/logstream/frontend' \
--header 'X-P-META-meta1: value1' \
--header 'X-P-META-meta2: value2' \
--header 'X-P-TAGS-tag1: value1' \
--header 'X-P-TAGS-tag2: value2' \
--header 'Authorization: Basic cGFyc2VhYmxlOnBhcnNlYWJsZQ==' \
--header 'Content-Type: application/json' \
--data-raw '[
    {
        "id": 4,
        "host":"153.10.110.81", 
        "user-identifier":"-", 
        "datetime":"24/Jun/2022:14:12:15 +0000", 
        "method": "DELETE", 
        "request": "/virtual/drive", 
        "protocol":"HTTP/2.0", 
        "status":500, 
        "bytes":21969, 
        "referer": "http://www.google.com/"
    }
]'
  1. Then access the Parseable dashboard to verify the log data is present
URL https://demo.parseable.io
Username parseable
Password parseable

For complete Parseable API documentation, refer to Parseable API workspace on Postman.

📝 Parseable is a work in progress. Features will evolve over time and there may be breaking changes between releases. Please give us your feedback in Slack, or Issues.
Please do not store any sensitive data on this server as the data is openly accessible. We'll delete the data on this server periodically.

Why Parseable

  • Log data compressed and stored in Parquet - columnar, open data format, designed for analytics.
  • Query logs via Parseable or bring your own analytics platform.
  • Stateless design with object storage (S3, MinIO) as primary storage. Allowing cost effective scale and flexibility.
  • Own your log data on your object storage buckets.
  • SDK less, simple REST API calls for log ingestion.
  • Indexing free design.

Features

  • Cloud native design, get started with Pod logs in minutes.
  • Filter log data on labels.
  • SQL compatible API for querying log data.
  • Intuitive dashboard to parse and query the log data.
  • Bring your own analytics platform for deeper analysis of log data.

Contributing

Refer to the contributing guide here.

License

Copyright (c) 2022 Parseable, Inc.

Licensed under the GNU Affero General Public License, Version 3 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

https://www.gnu.org/licenses/agpl-3.0.txt

Comments
  • Document FluentBit with Parseable

    Document FluentBit with Parseable

    Parseable is compatible with FluentBit HTTP Output: https://docs.fluentbit.io/manual/pipeline/outputs/http

    We need to document step by step guide on setting up FluentBit with Parseable

    documentation good first issue 
    opened by nitisht 8
  • Improve error message

    Improve error message

    In case a stream is empty, i.e. has no log data yet, the Query API responds with Failed to execute query due to err: missing record batch.

    This is confusing and might not make sense to end user. This error should be changed to say No log entries found in log stream xxx. Also the status code should be 404.

    bug good first issue 
    opened by nitisht 6
  • New log streams not showing up in log streams list

    New log streams not showing up in log streams list

    I was able to create a log stream but it wont show up when I list the streams, even when I have successfully pushed logs into the stream.

    How to reproduce?

    1. Delete log stream (in-case it exists already)
    curl --request DELETE "http://0.0.0.0:8000/api/v1/logstream/testnewlogbug" --header 'Authorization: Basic cGFyc2VhYmxlOnBhcnNlYWJsZQ=='
    
    1. Create a new log stream testnewlogbug
    curl --request PUT "http://0.0.0.0:8000/api/v1/logstream/testnewlogbug" --header 'Authorization: Basic cGFyc2VhYmxlOnBhcnNlYWJsZQ=='
    
    1. List log stream
    curl --request GET "http://0.0.0.0:8000/api/v1/logstream" --header 'Authorization: Basic cGFyc2VhYmxlOnBhcnNlYWJsZQ=='
    

    The stream testnewlogbug does not show up. Double check if testnewlogbug folder is present in s3

    aws --endpoint=https://sgp1.digitaloceanspaces.com --region="sgp1" s3 ls s3://parseable/
    
    opened by 123vivekr 5
  • Show git semver in banner

    Show git semver in banner

    Closes https://github.com/parseablehq/parseable/issues/146

    Description

    Show git semver in banner


    This PR has:

    • [ ] been tested to ensure log ingestion and log query works.
    • [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
    • [ ] added documentation for new or modified features or behaviors.
    opened by 123vivekr 3
  • Refactor alerts

    Refactor alerts

    Description

    Alerts are stored on metadata map which is behind and rwlock. This is fine but having to hold writer lock for alert check is not ideal. This PR refactors code around alerts implementation. Each alerts rule should be self contained with its own interior mutable state if it requires any. Interior mutablity pattern allows for alert to be resolved with shared reference. An alert can now contain different variant of rules which can be state or stateless.


    This PR has:

    • [ ] been tested to ensure log ingestion and log query works.
    • [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
    • [ ] added documentation for new or modified features or behaviors.
    opened by trueleo 3
  • Set zip version with only deflate feature enabled

    Set zip version with only deflate feature enabled

    Description

    Set zip crate back to stable crates.io version with minimal features enabled. This gets rid of bzip2 as indirect dependency and transitively on bzip2-sys crate. Console releases on GitHub use deflate method for zip, so Parseable build step should only requires this feature.

    opened by trueleo 3
  • [Document] How to send logs to Parseable from a Rust application

    [Document] How to send logs to Parseable from a Rust application

    Parseable website has docs on how to directly send logs from Go, Python, Java, JS etc. Ref: https://www.parseable.io/docs/log-ingestion/applications/go

    We need to add docs on how to directly sends logs from a Rust application.

    Please send the PR to parseablehq/parseable repo and we'll add it to the website.

    documentation good first issue 
    opened by nitisht 3
  • Populate the demo instance with large amounts of data

    Populate the demo instance with large amounts of data

    Hi. It'd be nice for the demo (demo.parseable.io) to be preloaded with large amounts of data to get a good sense of how it functions. I was playing around with it and could only see two events logged in the last 6 months.

    If it helps, here, I've a cron job that generates and populates dummy, but current data every day. A simple approach like that would do.

    bug 
    opened by knadh 3
  • Metadata management flow

    Metadata management flow

    Current STREAM_INFO map may have an issue with parallel writes to the map for the same stream.

    1. We need to implement a locking mechanism that allows exclusive writes to the map.
    2. Additionally Metadata map get and get methods should handle both local and S3. This way we don’t duplicate all the code to fallback on object store everywhere. a. Get Flow: Fetch metadata from local and fallback to object store if not found. b. Set Flow: Set to object store and then try setting it locally.
    enhancement 
    opened by nitisht 3
  • Show git sermver in banner

    Show git sermver in banner

    vergen crate sets environment variable VERGEN_GIT_SEMVER at build time which looks like this 5.0.0-2-gf49246c. Banner should show this version and possibly some other information if Parseable is ran from a git environment where HEAD is not on version tagged commit. Otherwise it should show semver like 5.0.0 for the release build done on tagged commit

    enhancement good first issue 
    opened by trueleo 2
  • Document Syslog-ng log forwarding to Parseable

    Document Syslog-ng log forwarding to Parseable

    Syslog-ng supports sending logs to http endpoints as per the docs here: https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.24/administration-guide/37#TOPIC-1298006

    We need to document how to fetch logs in json format and forward logs to Parseable

    documentation good first issue 
    opened by nitisht 2
  • [feature] Support schema evolution in a given stream

    [feature] Support schema evolution in a given stream

    As applications evolve, the log events they generate evolve as well. It would be very useful for Parseable server allow schema evolution of a given stream as source applications evolve. This is also reported a as a common issue with ES.

    This change would

    • further reduce the burden on the user/operator to manage schema
    • ensure application's entire life cycle log event data resides on a single Parseable stream, allowing easier analysis
    • remove fragmentation of log data across streams
    enhancement 
    opened by nitisht 0
  • Remove redundant ListObjectsV2 call when fetching from S3

    Remove redundant ListObjectsV2 call when fetching from S3

    In S3 query flow, datafusion ListingOptions is configured with extension to look for, this internally makes datafusion list files using ListObjectsV2 before actually fetching the file. This call maybe removed if filename is known beforehand.

    Todo:

    • [ ] What to do in case of multiple parseable instances with different hostnames.
    • [ ] Figure out how to provide table without doing list object calls.
    enhancement 
    opened by trueleo 1
  • [Document] LogStash with Parseable

    [Document] LogStash with Parseable

    Parseable is compatible with LogStash HTTP Output: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-http.html

    We need to document step by step guide on setting up LogStash with Parseable

    documentation good first issue 
    opened by nitisht 2
  • Improve query execution

    Improve query execution

    When query occurs and end time of query is greater than ( current time - upload interval ) then there is need to execute query on local data as well ( data that is yet to be synced ). This is handled by execute_on_cache but the datafusion context used is not aware about localsync process. There might be chances that file is moved after listing happened and it is ignored for the query.

    Todo

    • [ ] Custom table provider which maps local data without any issues caused by sync cycle.
    enhancement high prio 
    opened by trueleo 0
Releases(v0.0.6)
  • v0.0.6(Nov 10, 2022)

    What's Changed

    • Bump crates to current latest release by @trueleo in https://github.com/parseablehq/parseable/pull/170
    • Update Dockerfile to work with the new changes in cli format by @nitisht in https://github.com/parseablehq/parseable/pull/171
    • docs: Add syslog-ng log forwarding documentation by @sakti in https://github.com/parseablehq/parseable/pull/173
    • Update helm to latest release, also format & fix Dockerfile by @nitisht in https://github.com/parseablehq/parseable/pull/174
    • Set zip version with only deflate feature enabled by @trueleo in https://github.com/parseablehq/parseable/pull/175
    • Update helm chart for better handling of secrets by @nitisht in https://github.com/parseablehq/parseable/pull/176
    • Fix missing fields in the secret / values.yaml by @nitisht in https://github.com/parseablehq/parseable/pull/177
    • [Fix:DockerBuild] Added musl-dev in compiler stage by @Mayon-Francis in https://github.com/parseablehq/parseable/pull/180
    • Add path validation for cert and key by @trueleo in https://github.com/parseablehq/parseable/pull/181
    • Fix system memory calculation and formatting by @trueleo in https://github.com/parseablehq/parseable/pull/182
    • Refactor alerts by @trueleo in https://github.com/parseablehq/parseable/pull/179
    • Divide alert into submodules by @trueleo in https://github.com/parseablehq/parseable/pull/185
    • Add StringRule for alerts by @trueleo in https://github.com/parseablehq/parseable/pull/186
    • Readme cleanup by @nitisht in https://github.com/parseablehq/parseable/pull/187
    • Readme add links by @nitisht in https://github.com/parseablehq/parseable/pull/188
    • Introduce alert state and timeouts by @trueleo in https://github.com/parseablehq/parseable/pull/189
    • Create a Dockerfile that outputs a static binary during build by @Mayon-Francis in https://github.com/parseablehq/parseable/pull/190
    • Introduce Stats API by @trueleo in https://github.com/parseablehq/parseable/pull/191
    • prepare for new release v0.0.6 by @nitisht in https://github.com/parseablehq/parseable/pull/192

    New Contributors

    • @Mayon-Francis made their first contribution in https://github.com/parseablehq/parseable/pull/180

    Full Changelog: https://github.com/parseablehq/parseable/compare/v0.0.5...v0.0.6

    Source code(tar.gz)
    Source code(zip)
    checksums.txt(129 bytes)
    parseable_linux_x86_64(57.60 MB)
    parseable_macos_x86_64(46.74 MB)
  • v0.0.5(Oct 14, 2022)

    What's Changed

    • refactor: improve dockerfile using latest binary by @adripo in https://github.com/parseablehq/parseable/pull/157
    • Add Dockerfile for main branch build and another for release builds by @nitisht in https://github.com/parseablehq/parseable/pull/158
    • Use kebab-case value placeholders by @trueleo in https://github.com/parseablehq/parseable/pull/159
    • docs : add contributors list by @ProgrammerShri in https://github.com/parseablehq/parseable/pull/161
    • Add demo flag documentation in error message by @sakti in https://github.com/parseablehq/parseable/pull/162
    • Use vergen crate by @trueleo in https://github.com/parseablehq/parseable/pull/164
    • Add proper user agent while checking for update by @nitisht in https://github.com/parseablehq/parseable/pull/166
    • Add a top level server command to Parseable by @nitisht in https://github.com/parseablehq/parseable/pull/168
    • update to next release by @nitisht in https://github.com/parseablehq/parseable/pull/169

    New Contributors

    • @adripo made their first contribution in https://github.com/parseablehq/parseable/pull/157
    • @ProgrammerShri made their first contribution in https://github.com/parseablehq/parseable/pull/161
    • @sakti made their first contribution in https://github.com/parseablehq/parseable/pull/162

    Full Changelog: https://github.com/parseablehq/parseable/compare/v0.0.4...v0.0.5

    Source code(tar.gz)
    Source code(zip)
    checksums.txt(129 bytes)
    parseable_linux_x86_64(57.42 MB)
    parseable_macos_x86_64(49.89 MB)
  • v0.0.4(Oct 7, 2022)

    What's Changed

    • Use register_table for execute on cache by @trueleo in https://github.com/parseablehq/parseable/pull/128
    • Update banner to show latest available version by @trueleo in https://github.com/parseablehq/parseable/pull/131
    • Use rustls instead of openssl by @trueleo in https://github.com/parseablehq/parseable/pull/133
    • Fix search issue in console with new release by @nitisht in https://github.com/parseablehq/parseable/pull/135
    • Error type refactor by @trueleo in https://github.com/parseablehq/parseable/pull/137
    • Bump aws_sdk_s3 version and provide sleep and retry impl for S3 client by @trueleo in https://github.com/parseablehq/parseable/pull/138
    • Switch from structopt to clap by @Celeo in https://github.com/parseablehq/parseable/pull/141
    • Add version number for the current object storage format by @nitisht in https://github.com/parseablehq/parseable/pull/143
    • Lock metadata on first event. by @trueleo in https://github.com/parseablehq/parseable/pull/142
    • Add demo flag for config by @trueleo in https://github.com/parseablehq/parseable/pull/149
    • Create all temporary directories on startup by @trueleo in https://github.com/parseablehq/parseable/pull/153
    • Update to latest console release by @nitisht in https://github.com/parseablehq/parseable/pull/155

    New Contributors

    • @Celeo made their first contribution in https://github.com/parseablehq/parseable/pull/141

    Full Changelog: https://github.com/parseablehq/parseable/compare/v0.0.3...v0.0.4

    Source code(tar.gz)
    Source code(zip)
    checksums.txt(129 bytes)
    parseable_linux_x86_64(61.22 MB)
    parseable_macos_x86_64(49.91 MB)
  • v0.0.3(Sep 20, 2022)

    What's Changed

    • Update Parseable helm chart for next release by @nitisht in https://github.com/parseablehq/parseable/pull/107
    • Change x-p-tags to x-p-tag by @trueleo in https://github.com/parseablehq/parseable/pull/111
    • update readme by @nitisht in https://github.com/parseablehq/parseable/pull/112
    • Update description by @nitisht in https://github.com/parseablehq/parseable/pull/113
    • Update .deepsource.toml by @deepsource-autofix in https://github.com/parseablehq/parseable/pull/119
    • Fix code style and anti patterns reported by DeepSource by @nitisht in https://github.com/parseablehq/parseable/pull/118
    • Update .deepsource.toml by @deepsource-autofix in https://github.com/parseablehq/parseable/pull/120
    • Delegate parquet creation to s3sync instead by @nitisht in https://github.com/parseablehq/parseable/pull/117

    Full Changelog: https://github.com/parseablehq/parseable/compare/v0.0.2...v0.0.3

    Source code(tar.gz)
    Source code(zip)
    checksums.txt(129 bytes)
    parseable_linux_x86_64(60.16 MB)
    parseable_macos_x86_64(49.07 MB)
  • v0.0.2(Sep 11, 2022)

    What's Changed

    • Update Parseable helm chart to v0.0.1 by @nitisht in https://github.com/parseablehq/parseable/pull/89
    • Use Datafusion's re-exported arrow and parquet by @trueleo in https://github.com/parseablehq/parseable/pull/93
    • Use Arrow Schema instead of custom schema type. by @trueleo in https://github.com/parseablehq/parseable/pull/94
    • Update readme with get started section by @nitisht in https://github.com/parseablehq/parseable/pull/95
    • Reorg readme for readability by @nitisht in https://github.com/parseablehq/parseable/pull/96
    • Extend readme section for docker by @nitisht in https://github.com/parseablehq/parseable/pull/97
    • Add k8s installation in readme by @nitisht in https://github.com/parseablehq/parseable/pull/98
    • Config Datafusion query with schema by @trueleo in https://github.com/parseablehq/parseable/pull/99
    • Append arrow directly to local data by @trueleo in https://github.com/parseablehq/parseable/pull/100
    • Delay s3sync polling using tokio sleep by @trueleo in https://github.com/parseablehq/parseable/pull/101
    • Fix cases where num cpu is 1 by @nitisht in https://github.com/parseablehq/parseable/pull/102
    • Set max thread priority for local-sync by @trueleo in https://github.com/parseablehq/parseable/pull/104
    • Add .deepsource.toml by @deepsource-autofix in https://github.com/parseablehq/parseable/pull/105
    • Improve LocalStreamWriter along with error types by @trueleo in https://github.com/parseablehq/parseable/pull/106

    Full Changelog: https://github.com/parseablehq/parseable/compare/v0.0.1...v0.0.2

    Source code(tar.gz)
    Source code(zip)
    checksums.txt(129 bytes)
    parseable_linux_x86_64(69.62 MB)
    parseable_macos_x86_64(49.07 MB)
  • v0.0.1(Sep 2, 2022)

    Changelog

    • add badges to readme by @nitisht in https://github.com/parseablehq/parseable/pull/7
    • disable docker ci workflow temporarily by @nitisht in https://github.com/parseablehq/parseable/pull/8
    • update readme by @nitisht in https://github.com/parseablehq/parseable/pull/9
    • update ui build directory by @nitisht in https://github.com/parseablehq/parseable/pull/10
    • Fix syntax error in logstream put handler by @trueleo in https://github.com/parseablehq/parseable/pull/11
    • Update build directory by @ameen4455 in https://github.com/parseablehq/parseable/pull/12
    • Update build directory by @ameen4455 in https://github.com/parseablehq/parseable/pull/13
    • Move authentication middleware to new scope by @trueleo in https://github.com/parseablehq/parseable/pull/14
    • Move logo and images to docs repo and readme update by @nitisht in https://github.com/parseablehq/parseable/pull/15
    • remove localhost calls from ui by @nitisht in https://github.com/parseablehq/parseable/pull/16
    • Fix auth issue by @ameen4455 in https://github.com/parseablehq/parseable/pull/19
    • update link by @nitisht in https://github.com/parseablehq/parseable/pull/21
    • helm chart for parseable by @nitisht in https://github.com/parseablehq/parseable/pull/23
    • Fix build process for unified binary by @trueleo in https://github.com/parseablehq/parseable/pull/24
    • UI code is now moved to a separate repo by @nitisht in https://github.com/parseablehq/parseable/pull/25
    • Use correct path for frontend resource. by @trueleo in https://github.com/parseablehq/parseable/pull/26
    • Apply clippy suggestions by @trueleo in https://github.com/parseablehq/parseable/pull/27
    • Fix helm chart issues by @nitisht in https://github.com/parseablehq/parseable/pull/30
    • Change stream_name type by @trueleo in https://github.com/parseablehq/parseable/pull/31
    • fix diagram links by @nitisht in https://github.com/parseablehq/parseable/pull/33
    • Add rstest for testing by @trueleo in https://github.com/parseablehq/parseable/pull/32
    • Add serial-test by @trueleo in https://github.com/parseablehq/parseable/pull/34
    • Refactor events by @trueleo in https://github.com/parseablehq/parseable/pull/37
    • Add helm chart initial release by @nitisht in https://github.com/parseablehq/parseable/pull/38
    • Add helm index.yaml and include collector charts here by @nitisht in https://github.com/parseablehq/parseable/pull/39
    • Fix granularity subtraction by @trueleo in https://github.com/parseablehq/parseable/pull/41
    • Add dark mode image link by @nitisht in https://github.com/parseablehq/parseable/pull/44
    • Introduce a new error type for ObjectStorage api by @trueleo in https://github.com/parseablehq/parseable/pull/45
    • Update console release version by @nitisht in https://github.com/parseablehq/parseable/pull/46
    • Fix query validation to ensure we use actual query value by @nitisht in https://github.com/parseablehq/parseable/pull/50
    • Manage sync threads failures by @trueleo in https://github.com/parseablehq/parseable/pull/49
    • feat(alerts): foundational implementation by @de-sh in https://github.com/parseablehq/parseable/pull/28
    • Ignore empty alerts by @trueleo in https://github.com/parseablehq/parseable/pull/59
    • Update Parseable Helm Chart by @AdheipSingh in https://github.com/parseablehq/parseable/pull/57
    • k8s: helm fix mount paths by @AdheipSingh in https://github.com/parseablehq/parseable/pull/60
    • Refactor StorageSync by @trueleo in https://github.com/parseablehq/parseable/pull/55
    • feat: update dependencies by @de-sh in https://github.com/parseablehq/parseable/pull/58
    • Use correct environment variable for validation by @nitisht in https://github.com/parseablehq/parseable/pull/62
    • Update readme by @nitisht in https://github.com/parseablehq/parseable/pull/64
    • Update DO default creds by @nitisht in https://github.com/parseablehq/parseable/pull/66
    • Update logo by @ameen4455 in https://github.com/parseablehq/parseable/pull/67
    • Exact match when deleting stream by @trueleo in https://github.com/parseablehq/parseable/pull/68
    • Migrate to datafusion 11.0 by @nitisht in https://github.com/parseablehq/parseable/pull/65
    • Use common prefixes for logstream listing by @trueleo in https://github.com/parseablehq/parseable/pull/69
    • update readme by @nitisht in https://github.com/parseablehq/parseable/pull/71
    • Fix logstream listing and deletion by @trueleo in https://github.com/parseablehq/parseable/pull/72
    • Use / for listing by @trueleo in https://github.com/parseablehq/parseable/pull/77
    • update .gitignore by @nitisht in https://github.com/parseablehq/parseable/pull/78
    • Move leftover data.parquet to tmp with valid prefix on server startup by @trueleo in https://github.com/parseablehq/parseable/pull/73
    • ObjectStorage: add AuthenticationError type by @123vivekr in https://github.com/parseablehq/parseable/pull/80
    • Collect label and metadata from header by @trueleo in https://github.com/parseablehq/parseable/pull/79
    • Change separator for metadata and tags by @trueleo in https://github.com/parseablehq/parseable/pull/82
    • Update the max number of headers for meta and tags each to 10 by @nitisht in https://github.com/parseablehq/parseable/pull/83
    • Update to latest console release by @nitisht in https://github.com/parseablehq/parseable/pull/84
    • Update to console release 0.0.4 by @nitisht in https://github.com/parseablehq/parseable/pull/85
    • Use existing schema when querying local data. by @trueleo in https://github.com/parseablehq/parseable/pull/86
    • Update readme and upgrade console to 0.0.5 by @nitisht in https://github.com/parseablehq/parseable/pull/88

    New Contributors

    • @trueleo made their first contribution in https://github.com/parseablehq/parseable/pull/11
    • @ameen4455 made their first contribution in https://github.com/parseablehq/parseable/pull/12
    • @de-sh made their first contribution in https://github.com/parseablehq/parseable/pull/28
    • @AdheipSingh made their first contribution in https://github.com/parseablehq/parseable/pull/57
    • @123vivekr made their first contribution in https://github.com/parseablehq/parseable/pull/80

    Full Changelog: https://github.com/parseablehq/parseable/commits/v0.0.1

    Source code(tar.gz)
    Source code(zip)
    checksums.txt(141 bytes)
    parseable_0.0.1_Linux_x86_64(74.37 MB)
    parseable_0.0.1_macOS_x86_64(53.12 MB)
Owner
Parseable, Inc.
Parseable, Inc.
A Modern Real-Time Data Processing & Analytics DBMS with Cloud-Native Architecture, written in Rust

Datafuse Modern Real-Time Data Processing & Analytics DBMS with Cloud-Native Architecture Datafuse is a Real-Time Data Processing & Analytics DBMS wit

Datafuse Labs 4.8k Nov 30, 2022
Provides multiple-dtype columner storage, known as DataFrame in pandas/R

brassfibre Provides multiple-dtype columner storage, known as DataFrame in pandas/R. Series Single-dtype 1-dimentional vector with label (index). Crea

Sinhrks 21 Nov 28, 2022
Perhaps the fastest and most memory efficient way to pull data from PostgreSQL into pandas and numpy. 🚀

flaco Perhaps the fastest and most memory efficient way to pull data from PostgreSQL into pandas and numpy. ?? Have a gander at the initial benchmarks

Miles Granger 14 Oct 31, 2022
ndarray: an N-dimensional array with array views, multidimensional slicing, and efficient operations

ndarray The ndarray crate provides an n-dimensional container for general elements and for numerics. Please read the API documentation on docs.rs or t

null 2.5k Dec 3, 2022
Dataframe structure and operations in Rust

Utah Utah is a Rust crate backed by ndarray for type-conscious, tabular data manipulation with an expressive, functional interface. Note: This crate w

Suchin 139 Sep 26, 2022
A Rust crate that reads and writes tfrecord files

tfrecord-rust The crate provides the functionality to serialize and deserialize TFRecord data format from TensorFlow. Features Provide both high level

null 22 Nov 3, 2022
Apache Arrow DataFusion and Ballista query engines

DataFusion is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.

The Apache Software Foundation 2.8k Dec 2, 2022
Fastest and safest Rust implementation of parquet. `unsafe` free. Integration-tested against pyarrow

Parquet2 This is a re-write of the official parquet crate with performance, parallelism and safety in mind. The five main differentiators in compariso

Jorge Leitao 221 Nov 25, 2022
Orkhon: ML Inference Framework and Server Runtime

Orkhon: ML Inference Framework and Server Runtime Latest Release License Build Status Downloads Gitter What is it? Orkhon is Rust framework for Machin

Theo M. Bulut 127 Nov 20, 2022
Tiny, no-nonsense, self-contained, Tensorflow and ONNX inference

Sonos' Neural Network inference engine. This project used to be called tfdeploy, or Tensorflow-deploy-rust. What ? tract is a Neural Network inference

Sonos, Inc. 1.4k Nov 29, 2022
ConnectorX - Fastest library to load data from DB to DataFrames in Rust and Python

ConnectorX enables you to load data from databases into Python in the fastest and most memory efficient way.

SFU Database Group 888 Nov 25, 2022
Provides a way to use enums to describe and execute ordered data pipelines. 🦀🐾

enum_pipline Provides a way to use enums to describe and execute ordered data pipelines. ?? ?? I needed a succinct way to describe 2d pixel map operat

Ben Greenier 0 Oct 29, 2021
AppFlowy is an open-source alternative to Notion. You are in charge of your data and customizations

AppFlowy is an open-source alternative to Notion. You are in charge of your data and customizations. Built with Flutter and Rust.

null 29.6k Nov 26, 2022
New generation decentralized data warehouse and streaming data pipeline

World's first decentralized real-time data warehouse, on your laptop Docs | Demo | Tutorials | Examples | FAQ | Chat Get Started Watch this introducto

kamu 181 Nov 26, 2022
An example repository on how to start building graph applications on streaming data. Just clone and start building 💻 💪

An example repository on how to start building graph applications on streaming data. Just clone and start building ?? ??

Memgraph 39 Nov 10, 2022
Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing

Apache Arrow Powering In-Memory Analytics Apache Arrow is a development platform for in-memory analytics. It contains a set of technologies that enabl

The Apache Software Foundation 10.7k Dec 2, 2022
This library provides a data view for reading and writing data in a byte array.

Docs This library provides a data view for reading and writing data in a byte array. This library requires feature(generic_const_exprs) to be enabled.

null 2 Nov 2, 2022
The rust client for CeresDB. CeresDB is a high-performance, distributed, schema-less, cloud native time-series database that can handle both time-series and analytics workloads.

The rust client for CeresDB. CeresDB is a high-performance, distributed, schema-less, cloud native time-series database that can handle both time-series and analytics workloads.

null 12 Nov 18, 2022
Log for concurrent workloads, with support for atomic batches and in-order recovery

sharded-log A batch-oriented multi-threaded sharded log for workloads that occasionally flush logs into some other system. All batches have a 32-bit C

Komora 16 Nov 20, 2022
Easy c̵̰͠r̵̛̠ö̴̪s̶̩̒s̵̭̀-t̶̲͝h̶̯̚r̵̺͐e̷̖̽ḁ̴̍d̶̖̔ ȓ̵͙ė̶͎ḟ̴͙e̸̖͛r̶̖͗ë̶̱́ṉ̵̒ĉ̷̥e̷͚̍ s̷̹͌h̷̲̉a̵̭͋r̷̫̊ḭ̵̊n̷̬͂g̵̦̃ f̶̻̊ơ̵̜ṟ̸̈́ R̵̞̋ù̵̺s̷̖̅ţ̸͗!̸̼͋

Rust S̵̓i̸̓n̵̉ I̴n̴f̶e̸r̵n̷a̴l mutability! Howdy, friendly Rust developer! Ever had a value get m̵̯̅ð̶͊v̴̮̾ê̴̼͘d away right under your nose just when

null 293 Sep 22, 2022