⚡🦀 🧨 make your rust types fit DynamoDB and visa versa

Overview
🦀 🧨

dynomite

dynomite makes DynamoDB fit your types (and visa versa)


Overview

Goals

  • make writing dynamodb applications in rust a productive experience
  • 🦀 exploit rust's type safety features
  • 👩‍💻 leverage existing work of the rusoto rust project
  • commitment to supporting applications built using stable rust
  • 📚 commitment to documentation

Features

  • 💌 less boilerplate
  • ♻️ automatic async pagination
  • 🕶️ client level retry interfaces for robust error handling

From this

use std::collections::HashMap;
use rusoto_dynamodb::AttributeValue;
use uuid::Uuid;

let mut item = HashMap.new();
item.insert(
  "pk".to_string(), AttributeValue {
    s: Some(Uuid::new_v4().to_hyphenated().to_string()),
    ..AttributeValue::default()
  }
);
item.insert(
  // 🤬typos anyone?
  "quanity".to_string(), AttributeValue {
    n: Some("whoops".to_string()),
    ..AttributeValue::default()
  }
);

To this

use dynomite::Item;
use uuid::Uuid;

#[derive(Item)]
struct Order {
  #[dynomite(partition_key)]
  pk: Uuid,
  quantity: u16
}

let item = Order {
  pk: Uuid::new_v4(),
  quantity: 4
}.into();

Please see the API documentation for how to get started. Enjoy.

📦 Install

In your Cargo.toml file, add the following under the [dependencies] heading

dynomite = "0.10"

🤸 Examples

You can find some example application code under dynomite/examples

DynamoDB local

AWS provides a convenient way to host a local instance of DynamoDB for testing.

Here is a short example of how to get up a testing locally quickly with both dynomite as well as rusoto_dynamodb.

In one terminal spin up a Docker container for DynamoDB local listening on port 8000

$ docker run --rm -p 8000:8000 amazon/dynamodb-local

In another, run a rust binary with a client initialized like you see the the local.rs example

Resources

Doug Tangren (softprops) 2018-2020

Comments
  • Introduce #[dynomite(flatten)] attribute

    Introduce #[dynomite(flatten)] attribute

    What did you implement:

    This should be useful when we have fat enums support in dynomite (https://github.com/softprops/dynomite/issues/131), for enum variants to be able to be used as partition/sort keys.

    This works like #[serde(flatten)], the fields of the nested struct are flattened to the attributes of the surrounding struct where the flattened field is declared. The reverse mapping is done when parsing the struct from raw Attributes.

    How did you verify your change:

    Added a unit test, compile_fail test, and documentation (doc test)

    What (if anything) would need to be called out in the CHANGELOG for the next release:

    • Introduce #[dynomite(flatten)] field attribute UPD
    • BREAKING CHANGE FromAttributes now requires the impl of fn from_mut_attrs(attrs: &mut Attributes) instead of from_attrs(attrs: Attributes), the latter now has the default impl based on from_mut_attrs impl
    opened by Veetaha 10
  • How to deal with optional attributes?

    How to deal with optional attributes?

    New to rust and dynomite here, so sorry if it's a dumb question, but I couldn't find any example or documentation on how to deal with optional attributes. I am used to other libraries/languages where we can specified certain attributes as "nullable" meaning that if they are not set the value is set to null/None.

    In other words if I have something like this:

    
    [derive(Item, Debug, Clone, GraphQLObject)]
    struct Event {
        #[dynomite(partition_key)]
        id: String,
        status: String,
    }
    
    

    Some events won't have the attribute status set, for example. When calling Event::from_attrs(result) I am getting a MissingField error, instead of getting a None value.

    I tried this too but didn't seem to make a difference:

    
    [derive(Item, Debug, Clone, GraphQLObject)]
    struct Event {
        #[dynomite(partition_key)]
        id: String,
        status: Option<String>,
    }
    

    Is there a way to deal with this?

    Thanks for the help.

    opened by samlll42-github 10
  • Rename attributes `hash` to `partition_key` and `range` to `sort_key`

    Rename attributes `hash` to `partition_key` and `range` to `sort_key`

    What did you implement:

    Renamed attributes hash to partition_key and range to sort_key.

    See reasoning in #69.

    Closes: https://github.com/softprops/dynomite/issues/69

    How did you verify your change:

    $ cargo test # all non-ignored tests pass
    
    $ rg -t rust -w hash | wc -l
           0
    
    $ rg -t rust -w range | wc -l
           0
    
    $ rg -t rust -w partition_key | wc -l
          10
    
    $ rg -t rust -w sort_key | wc -l     
           4
    

    What (if anything) would need to be called out in the CHANGELOG for the next release:

    This is a breaking change so we should call out the rename and probably bump to 0.6.

    opened by phrohdoh 9
  • Added support for rfc3339 timestamps through the humantime crate and feature

    Added support for rfc3339 timestamps through the humantime crate and feature

    What did you implement:

    Added a cargo feature to the crate which allows end users to easily serialize std::time::SystemTimes into human readable rfc3339 timestamps, by implementing Attribute for SystemTime and using humantime's implementation of rfc3339 timestamps internally.

    How did you verify your change:

    Added 2 tests to lib.rs

    What (if anything) would need to be called out in the CHANGELOG for the next release:

    "Added support for rfc3339 timestamps through the humantime crate and feature"

    opened by cedric-h 7
  • Support renaming fields on types deriving `Item`

    Support renaming fields on types deriving `Item`

    What did you implement:

    Implemented field renaming on Item-derived types via an attribute similar to serde's rename.

    Example

    #[derive(Item, PartialEq, Debug, Clone)]
    struct Recipe {
        #[hash]
        #[dynomite(rename = "recipe_id")]
        id: String,
        servings: u64,
    }
    
    let value = Recipe {
        id: "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee".into(),
        servings: 2,
    };
    
    let attrs: Attributes = value.into();
    
    println!("{:#?}", attrs); // see next code block
    
    {
        "recipe_id": AttributeValue {
            b: None,
            bool: None,
            bs: None,
            l: None,
            m: None,
            n: None,
            ns: None,
            null: None,
            s: Some(
                "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee",
            ),
            ss: None,
        },
        "servings": AttributeValue {
            b: None,
            bool: None,
            bs: None,
            l: None,
            m: None,
            n: Some(
                "2",
            ),
            ns: None,
            null: None,
            s: None,
            ss: None,
        },
    }
    

    Closes: https://github.com/softprops/dynomite/issues/53

    How did you verify your change:

    Added and ran a new test, field_rename, in derived.rs
    $ cargo t field_rename
        Finished dev [unoptimized + debuginfo] target(s) in 0.09s
         Running target/debug/deps/dynomite-ae35a6724a5f60c6
    
    running 0 tests
    
    test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 36 filtered out
    
         Running target/debug/deps/derived-b5a5da7206a8af0c
    
    running 1 test
    test tests::field_rename ... ok
    
    test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out
    
         Running target/debug/deps/dynomite_derive-050c7f0267e4327d
    
    running 0 tests
    
    test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
    
    Used `cargo expand` to verify derive output
    struct Recipe {
        #[hash]
        #[dynomite(rename = "recipe_id")]
        id: String,
        servings: u64,
    }
    impl ::dynomite::FromAttributes for Recipe {
        fn from_attrs(mut attrs: ::dynomite::Attributes) -> Result<Self, ::dynomite::AttributeError> {
            Ok(Self {
                id: ::dynomite::Attribute::from_attr(attrs.remove("recipe_id").ok_or(
                    ::dynomite::AttributeError::MissingField {
                        name: "recipe_id".to_string(),
                    },
                )?)?,
                servings: ::dynomite::Attribute::from_attr(attrs.remove("servings").ok_or(
                    ::dynomite::AttributeError::MissingField {
                        name: "servings".to_string(),
                    },
                )?)?,
            })
        }
    }
    impl ::std::convert::From<Recipe> for ::dynomite::Attributes {
        fn from(item: Recipe) -> Self {
            let mut values = Self::new();
            values.insert(
                "recipe_id".to_string(),
                ::dynomite::Attribute::into_attr(item.id),
            );
            values.insert(
                "servings".to_string(),
                ::dynomite::Attribute::into_attr(item.servings),
            );
            values
        }
    }
    impl ::dynomite::Item for Recipe {
        fn key(&self) -> ::std::collections::HashMap<String, ::dynomite::dynamodb::AttributeValue> {
            let mut keys = ::std::collections::HashMap::new();
            keys.insert(
                "recipe_id".to_string(),
                ::dynomite::Attribute::into_attr(self.id.clone()),
            );
            keys
        }
    }
    struct RecipeKey {
        recipe_id: String,
    }
    impl ::dynomite::FromAttributes for RecipeKey {
        fn from_attrs(mut attrs: ::dynomite::Attributes) -> Result<Self, ::dynomite::AttributeError> {
            Ok(Self {
                recipe_id: ::dynomite::Attribute::from_attr(attrs.remove("recipe_id").ok_or(
                    ::dynomite::AttributeError::MissingField {
                        name: "recipe_id".to_string(),
                    },
                )?)?,
            })
        }
    }
    impl ::std::convert::From<RecipeKey> for ::dynomite::Attributes {
        fn from(item: RecipeKey) -> Self {
            let mut values = Self::new();
            values.insert(
                "recipe_id".to_string(),
                ::dynomite::Attribute::into_attr(item.recipe_id),
            );
            values
        }
    }
    

    What (if anything) would need to be called out in the CHANGELOG for the next release:

    Mention the new functionality!

    Potential Follow-Up Work

    #[dynomite(rename_all = "...")] similar to serde's rename_all.

    Other Notes

    ~~This depends on #66 (and as such includes the commit from it).~~ #66 has been merged and this was rebased on top of it.

    opened by phrohdoh 7
  • Add initial support for fat enums (internally-tagged representation)

    Add initial support for fat enums (internally-tagged representation)

    ~~Blocked by (these PRS have to reviewed and merged first before this one):~~

    • [x] ~~#134~~
    • [x] ~~#132~~

    Internally tagged enums

    The docs are pretty self-explanatory. This PR adds support for simple fat enums that have either one-element-tuple variants of the nested value of type T such that T: Attributes, or unit variants.

    use dynomite::Attributes;
    
    #[derive(Attributes)]
    #[dynomite(tag = "kind")] // required to specify for fat enums
    enum Shape {
        Rectangle(Rectangle),
        // Use `rename` to change the **value** of the tag for a particular variant
        // by default the tag for a particular variant is the name of the variant verbatim
        #[dynomite(rename = "my_circle")]
        Circle(Circle),
        Unknown,
    }
    
    #[derive(Attributes)]
    struct Circle {
        radius: u32,
    }
    
    #[derive(Attributes)]
    struct Rectangle {
        width: u32,
        height: u32,
    }
    

    These Rust definition correspond to the following representation in DynamoDB for each enum variant:

    • Rectangle:
      {
          "kind": "Rectangle",
          "width": 42,
          "height": 64
      }
      
    • Circle:
      {
          "kind": "my_circle",
          "radius": 54
      }
      
    • Unknown:
      {
          "kind": "Unknown"
      }
      

    Closes: #131

    How did you verify your change:

    Added a unit test, trybuild tests and doc tests

    What (if anything) would need to be called out in the CHANGELOG for the next release:

    • Add initial support for fat enums (only one-element tuple variants of T: Attributes values and unit variants)
    opened by Veetaha 6
  • Introduce `#[dynomite(skip_serializing_if =

    Introduce `#[dynomite(skip_serializing_if = "..")]` attribute

    Tested:

    • There are no breaking changes
    • The changes are tested in integration tests for both successful and error (trybuild) cases

    What did you implement:

    This basically reflects serde::skip_serializing_if attribute. This will be very useful for us to get rid of manual workarounds where we need to store empty binary sets in a db record.

    Also had to fix a clippy lint of implementing Into trait instead of From to let CI pass

    How did you verify your change:

    Ran tests. I've also moved integration tests into a single test binary as per this article

    What (if anything) would need to be called out in the CHANGELOG for the next release:

    The new feature is specified in changelog in this diff

    opened by Veetaha 4
  • Bump rusoto dependencies

    Bump rusoto dependencies

    What did you implement:

    Updated rusoto* dependencies to 0.45 version I would appreciate if you could release this as a patch version to cartes.io, @softprops

    How did you verify your change:

    cargo test # worked fine for me
    

    What (if anything) would need to be called out in the CHANGELOG for the next release:

    Bump rusoto dependencies to version 0.45

    opened by Veetaha 4
  • How to deal with nested sructures?

    How to deal with nested sructures?

    I'm not sure if this is a bug, my mistake or this has not been implemented yet ...

    🐛 Bug description

    Nested structure

    #[derive(Item, Serialize, Deserialize, Debug)]
    struct t_list {
        #[dynomite(partition_key)]
        lid: Uuid,
        items: Option<Vec<t_list_item>>,
    }
    

    fails over items: Option<Vec<t_list_item>> with this error message

    the trait bound `std::option::Option<std::vec::Vec<t_list_item>>: dynomite::Attribute` is not satisfied
    the following implementations were found:
      <std::option::Option<T> as dynomite::Attribute>
    required by `dynomite::Attribute::from_attr`rustc(E0277)
    main.rs(134, 10): Error originated from macro here
    

    🤔 Expected Behavior

    I was hoping to get the structure similar to this

    {
      "lid": "ff0540dd-3e1c-4977-8d54-5e044688dba1",
      "items": [
        {
          "liid": "30000000-012e-4dd7-9697-dcc06d148a64",
          "title": "... some text ..."
        },
        {
          "liid": "30000001-012e-4dd7-9697-dcc06d148a64",
          "title": "... some text ..."
        }]
    }
    

    👟 Steps to reproduce

    Something as simple as

    #[derive(Item, Serialize, Deserialize, Debug)]
    struct t_list {
        #[dynomite(partition_key)]
        lid: Uuid,
        items: Option<t_list_item>,
    }
    

    where t_list_item is another struct should produce the same error.

    🌍 Your environment

    dynomite = {version = "0.8.2", default-features = false, features = ["rustls"]}

    opened by rimutaka 4
  • Derive item subset

    Derive item subset

    💡 Feature description

    In some cases you don't always want all fields returned in queries when a simple projection will do. Make it easy to derive subset of item attributes

    💻 Basic example

    #[derive(Attributes)]
    struct Attrs {
      field: String
    }
    

    vs

    #[derive(Item)]
    struct Foo {
      #[dynomite(partition_key)]
      id: String,
      field: String
    }
    
    enhancement 
    opened by softprops 4
  • Improved support for optional attribute values.

    Improved support for optional attribute values.

    What did you implement:

    Closes: #83

    How did you verify your change:

    1. cargo build and cargo test -- option_.
    2. Integration in separate project is working correctly.

    What (if anything) would need to be called out in the CHANGELOG for the next release:

    This PR fixes an issue with using None when sending PutItem request to the AWS API.

    opened by elslooo 4
  • Upgrade `uuid` to version 1.1.2

    Upgrade `uuid` to version 1.1.2

    uuid has made a major 1.0.0 release and subsequent releases up to 1.1.2. Depending on dynomite while trying to upgrade uuid results in errors due to multiple crate versions.

    What did you implement:

    • Upgrade lambda_http to version 0.5.2
    • Upgrade uuid to version 1.1.2
    • Update failing trybuild test cases

    How did you verify your change:

    By running:

    cargo build --all-targets
    cargo test
    

    What (if anything) would need to be called out in the CHANGELOG for the next release:

    The new uuid version should probably be called out.

    opened by jreyes33 0
  • Fix apostrophe typo in error message

    Fix apostrophe typo in error message

    👋 Super excited that this project exists!

    This PR fixes a small typo in an error message returned by make_dynomite_item. There shouldn't be an apostrophe here because, in this phrase, the item isn't being used as as a possessive.

    opened by jameslittle230 0
  • Make the derivation of attribute mapping handle raw identifiers (`r#…`) properly

    Make the derivation of attribute mapping handle raw identifiers (`r#…`) properly

    This fixes an issue, whereby dynomite would not treat raw identifiers (r#…) properly as it would include the syntactical r# prefix in the serialized attribute values.

    For example, the value of the following enum:

    #[derive(Attribute, Clone, Copy, Debug, PartialEq)]
    pub enum EnumWithRawFieldNameIdentifier {
        r#VariantName,
    }
    

    would be serialized as "r#VariantName".

    With this fix it will be serialized as "VariantName", as, I think, expected.

    opened by jakubadamw 0
  • Single Table Design

    Single Table Design

    It would be great to see support for using dynomite with Single Table Design.

    https://github.com/jeremydaly/dynamodb-toolbox is a great option for TypeScript developers that could serve as inspiration.

    opened by fbjork 0
  • Update env_logger requirement from 0.8 to 0.9

    Update env_logger requirement from 0.8 to 0.9

    Updates the requirements on env_logger to permit the latest version.

    Release notes

    Sourced from env_logger's releases.

    v0.9.0

    Breaking Changes:

    • Default message format now prints the target instead of the module

    Improvements:

    • Added a method to print the module instead of the target
    Commits
    • 04856ac bump version to 0.9.0
    • e4744ff Merge pull request #209 from gtsiam/main
    • c5fa7a2 refactor: fix clippy warnings
    • 34574df Update link to examples
    • 1888497 Clarified documentation about log filtering
    • 365ffaf Show target instead of module path by default
    • d2998a6 Add option to print log target
    • 13cafce Bump version to 0.8.4
    • 0900811 Ensure unique directive names when building filters
    • 1a8379a Allow writing logs to a custom output target (Target::Pipe)
    • Additional commits viewable in compare view

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language
    • @dependabot badge me will comment on this PR with code to add a "Dependabot enabled" badge to your readme

    Additionally, you can set the following in your Dependabot dashboard:

    • Update frequency (including time of day and day of week)
    • Pull request limits (per update run and/or open at any time)
    • Automerge options (never/patch/minor, and dev/runtime dependencies)
    • Out-of-range updates (receive only lockfile updates, if desired)
    • Security updates (receive only security updates, if desired)
    dependencies 
    opened by dependabot-preview[bot] 0
Owner
Doug Tangren
rusting at sea
Doug Tangren
Learn Rust black magics by implementing basic types in database systems

Type Exercise in Rust (In Chinese) 数据库表达式执行的黑魔法:用 Rust 做类型体操 This is a short lecture on how to use the Rust type system to build necessary components

Alex Chi 996 Jan 3, 2023
Command-line tool to make Rust source code entities from Postgres tables.

pg2rs Command-line tool to make Rust source code entities from Postgres tables. Generates: enums structs which can be then used like mod structs; use

Stanislav 10 May 20, 2022
A Modern Real-Time Data Processing & Analytics DBMS with Cloud-Native Architecture, built to make the Data Cloud easy

A Modern Real-Time Data Processing & Analytics DBMS with Cloud-Native Architecture, built to make the Data Cloud easy

Datafuse Labs 5k Jan 9, 2023
Thin wrapper around [`tokio::process`] to make it streamable

process-stream Wraps tokio::process::Command to future::stream. Install process-stream = "0.2.2" Example usage: From Vec<String> or Vec<&str> use proc

null 4 Jun 25, 2022
Make a DAO Drop!

DAO drop tool This tool parses a Cosmos SDK chain export JSON file, to produce a CSV list of addresses and amounts. It can handle extremely large file

DAO DAO 5 Jan 15, 2023
asynchronous and synchronous interfaces and persistence implementations for your OOD architecture

OOD Persistence Asynchronous and synchronous interfaces and persistence implementations for your OOD architecture Installation Add ood_persistence = {

Dmitriy Pleshevskiy 1 Feb 15, 2022
Query is a Rust server for your remote SQLite databases and a CLI to manage them.

Query Query is a Rust server for your remote SQLite databases and a CLI to manage them. Table Of Contents Run A Query Server CLI Install Use The Insta

Víctor García 6 Oct 6, 2023
Grsql is a great tool to allow you set up your remote sqlite database as service and CRUD(create/read/update/delete) it using gRPC.

Grsql is a great tool to allow you set up your remote sqlite database as service and CRUD (create/ read/ update/ delete) it using gRPC. Why Create Thi

Bruce Yuan 33 Dec 16, 2022
Document your SQLite tables and columns with in-line comments

sqlite-docs A SQLite extension, CLI, and library for documentating SQLite tables, columns, and extensions. Warning sqlite-docs is still young and not

Alex Garcia 20 Jul 2, 2023
Provides a Rust-based SQLite extension for using Hypercore as the VFS for your databases.

SQLite and Hypercore A Rust library providing SQLite with an virtual file system to enable Hypercore as a means of storage. Contributing The primary r

Jacky Alciné 14 Dec 5, 2022
Generate type-checked Rust from your PostgreSQL.

Cornucopia Generate type checked Rust from your SQL Install | Example Cornucopia is a small CLI utility resting on postgres designed to facilitate Pos

null 206 Dec 25, 2022
Visualize your database schema

dbviz Visualize your database schema. The tool loads database schema and draws it as a graph. Usage $ dbviz -d database_name | dot -Tpng > schema.png

yunmikun2 2 Sep 4, 2022
Macros that allow for implicit await in your async code.

suspend fn Disclaimer: this was mostly made as a proof of concept for the proposal below. I haven't tested if there is a performance cost to this macr

null 6 Dec 22, 2021
SubZero - a standalone web server that turns your database directly into a REST/GraphQL api

What is this? This is a demo repository for the new subzero codebase implemented in Rust. subZero is a standalone web server that turns your database

subZero 82 Jan 1, 2023
Replibyte - a powerful tool to seed your databases

Seed Your Development Database With Real Data ⚡️ Replibyte is a powerful tool to seed your databases with real data and other cool features ?? Feature

Qovery 3.4k Jan 9, 2023
Seed your development database with real data ⚡️

Seed Your Development Database With Real Data ⚡️ Replibyte is a blazingly fast tool to seed your databases with your production data while keeping sen

Qovery 3.4k Jan 2, 2023
Teach your PostgreSQL database how to speak MongoDB Wire Protocol

“If it looks like MongoDB, swims like MongoDB, and quacks like MongoDB, then it probably is PostgreSQL.” ?? Discord | Online Demo | Intro Video | Quic

Felipe Coury 261 Jun 18, 2023
ReefDB is a minimalistic, in-memory and on-disk database management system written in Rust, implementing basic SQL query capabilities and full-text search.

ReefDB ReefDB is a minimalistic, in-memory and on-disk database management system written in Rust, implementing basic SQL query capabilities and full-

Sacha Arbonel 75 Jun 12, 2023
Skybase is an extremely fast, secure and reliable real-time NoSQL database with automated snapshots and SSL

Skybase The next-generation NoSQL database What is Skybase? Skybase (or SkybaseDB/SDB) is an effort to provide the best of key/value stores, document

Skybase 1.4k Dec 29, 2022