An HCL serializer/deserializer for rust

Related tags

Encoding JSON rust serde hcl
Overview

hcl-rs

Build Status docs.rs MIT License

This crate provides functionality to deserialize, serialize and manipulate HCL data.

The main types are Deserializer for deserializing data, Serializer for serialization. Furthermore the provided Body and Value types can be used to construct HCL data or as a deserialization target.

Some supporting macros like body! to build HCL data structures are provided as well.

Deserialization examples

Deserialize arbitrary HCL according to the HCL JSON Specification:

use serde_json::{json, Value};

let input = r#"
    some_attr = {
      foo = [1, 2]
      bar = true
    }

    some_block "some_block_label" {
      attr = "value"
    }
"#;

let expected = json!({
    "some_attr": {
        "foo": [1, 2],
        "bar": true
    },
    "some_block": {
        "some_block_label": {
            "attr": "value"
        }
    }
});

let value: Value = hcl::from_str(input).unwrap();

assert_eq!(value, expected);

If you need to preserve context about the HCL structure, deserialize into hcl::Body instead:

use hcl::{Block, Body, Expression};

let input = r#"
    some_attr = {
      "foo" = [1, 2]
      "bar" = true
    }

    some_block "some_block_label" {
      attr = "value"
    }
"#;

let expected = Body::builder()
    .add_attribute((
        "some_attr",
        Expression::from_iter([
            ("foo", Expression::from(vec![1, 2])),
            ("bar", Expression::Bool(true)),
        ]),
    ))
    .add_block(
        Block::builder("some_block")
            .add_label("some_block_label")
            .add_attribute(("attr", "value"))
            .build(),
    )
    .build();

let body: Body = hcl::from_str(input).unwrap();

assert_eq!(body, expected);

Serialization examples

A simple example to serialize some terraform configuration:

use hcl::{Block, Body, RawExpression};

let body = Body::builder()
    .add_block(
        Block::builder("resource")
            .add_label("aws_sns_topic_subscription")
            .add_label("topic")
            .add_attribute(("topic_arn", RawExpression::new("aws_sns_topic.queue.arn")))
            .add_attribute(("protocol", "sqs"))
            .add_attribute(("endpoint", RawExpression::new("aws_sqs_queue.queue.arn")))
            .build(),
    )
    .build();

let expected = r#"
resource "aws_sns_topic_subscription" "topic" {
  topic_arn = aws_sns_topic.queue.arn
  protocol = "sqs"
  endpoint = aws_sqs_queue.queue.arn
}
"#.trim_start();

let serialized = hcl::to_string(&body).unwrap();

assert_eq!(serialized, expected);

Also have a look at the other examples provided in the documentation of the ser module.

Macros

This crate provides a couple of macros to ease building HCL data structures. Have a look at their documentation for usage examples.

License

The source code of hcl-rs is released under the MIT License. See the bundled LICENSE file for details.

Comments
  • Nested function calls leads to extreme parsing times

    Nested function calls leads to extreme parsing times

    Here's a heavily reduced version of a terraform variable definition:

    variable "test" {
  level1 = map(object({
    level2 = map(object({
      level3 = map(object({
        level4 = map(object({
        }))
      }))
    }))
  }))
}

    On my system the above takes ~24 seconds to parse.

    Removing a level by commenting it out:

    variable "test" {
  level1 = map(object({
    level2 = map(object({
      level3 = map(object({
        # level4 = map(object({
        # }))
      }))
    }))
  }))
}

    Takes about ~1.77 seconds to parse

    Removing one additional level (level3) and parsing this file drops to ~0.455 seconds to parse.

    It seems that there is something that goes quadratic when having multiple function calls nested. We have a bunch of terraform that heavily uses this to make sure the type of the variable is correct:

    variable "network_integration" {
  description = "Map of networking integrations between accounts"
  type = map(object({
    friendly_name = string,
    vpcs = map(object({
      id           = string
      cidr         = string
      region       = string
      description  = string
      subnets      = map(string)
      route_tables = map(string)
      security_groups = map(object({
        id = string
        rules = map(object({
          direction   = string
          protocol    = string
          from_port   = string
          to_port     = string
          description = string
        }))
      }))
    }))
    additional_propagated_vpcs   = list(string)
    additional_static_vpc_routes = list(string)
  }))
  default = {}
}

    As a full example (which this takes ~130 seconds to parse).

    performance 
    opened by archoversight 13
  • feat: Add specsuite integration harness

    feat: Add specsuite integration harness

    Integration tests as mentioned in #8

    Any HCL construct should have the extension .hcl under the folder specsuite. The JSON equivalent of the parsed HCLis the same name with the extension .hcl.json . If a .hcl.json file does not exist the test is considered ignored. For convenience, this also dumps the JSON output suitable for creating .hcl.json files.

    template {
  source = <<-EOF
  This is valid
  EOF
}

    {
  "template": {
    "source": "This is valid"
  }
}

    Errors are JSON files with the form, and are also automatically dumped if missing json files.

    {
  "Message": "error string"
}
    opened by venix1 5
  • Splat operators lead to invalid type

    Splat operators lead to invalid type

    Using an input file such as:

    output "sometest" { value = {
  name = module.instance.*.tags.Name
  id   = module.instance.*.id
} }

    When attempting to read it, will lead to this error:

    Error: invalid type: string ".*", expected unit

    The splat operator is used heavily in our terraform configuration for outputs.

    opened by archoversight 4
  • Panic attempting to parse an integer

    Panic attempting to parse an integer

    Given the following HCL:

    resource "test" "testing" {
  ids      = [module.instance.0.id]
}

    It leads to this panic:

    thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ParseIntError { kind: InvalidDigit }', ~/.cargo/registry/src/github.com-1ecc6299db9ec823/hcl-rs-0.8.2/src/parser/mod.rs:342:32
    opened by archoversight 3
  • Is it possible to use raw expressions in the block macro?

    Is it possible to use raw expressions in the block macro?

    Using hcl-rs = "0.6.5" I am trying to produce the output

    module "foo" {
    name = var.name
}

    with the block macro. But nothing I have tried can produce this output, at most it always quotes the var.name which is not what I need. I have tried:

        hcl::block!(
        module "foo" {
            // name = var.name // error: recursion limit reached
            name = "var.name" // Produces `name = "var.name"` (as expected)
            name = (RawExpression::new("var.name")) // Produces `name = "var.name"`
            // name = RawExpression::new("var.name") // error: recursion limit reached
        }
    )

    And cannot see anything other way to get this to work. Is this currently possible with the macros?

    Using the builder does work as expected:

        Block::builder("module")
            .add_label("foo")
            .add_attribute(("name", RawExpression::new("var.name")))
            .build()

    But ends up being a lot noisier than the macros.

    opened by mdaffin 3
  • Is evalulating expressions supported?

    Is evalulating expressions supported?

    Hi! I'm just wondering if evaulating expressions are supported or even in scope for hcl-rs?

    Currently, I want to serialize a HCL config with expressions into a file, but seems like the value just returns a literal string of the raw expression.

    Is expression support going to be in hcl-rs?

    opened by korewaChino 3
  • identifiers should be validated upon creation

    identifiers should be validated upon creation

    Right now, Identifiers can be created from any string value and will fail to format if they contain invalid characters.

    The type constructors should be charged to make it impossible to create invalid Identifier instances. This would remove the need to validate identifiers during formatting. The identifier validation is the only fallible operation (apart from write errors) that can happen during formatting. If it is removed, we can ensure that to_string really cannot fail.

    • Deprecate Identifier::new and Identifier::from in order to eventually remove them in a future release
    • Add Identifier::new_unchecked to allow deliberate bypassing of the validation for users that ensure identifier validity by other means. It should be marked with a warning to indicate that invalid identifiers may produce invalid HCL.
    • Implement FromStr and TryFrom for `Identifier
    • Move validation from format module into the FromStr impl
    enhancement 
    opened by martinohmann 2
  • Expected option when passing an option type, instead of type inside of option

    Expected option when passing an option type, instead of type inside of option 

    // ser.rs
#[derive(Deserialize)]
pub struct Project {
    pub proj_type: String,
    pub spec: Option<PathBuf>,
    pub dockerfile: Option<PathBuf>,
    pub scripts: Option<Vec<Script>>,
}

#[derive(Deserialize)]
pub struct Script {
    pub name: String,
    pub command: String,
}

// function to load HCL config here...

    // conf.hcl
project "a" {
    proj_type = "generic"
    spec = "./test.spec"
}

    returns error

    Message { msg: "invalid type: string \"./test.spec\", expected option", location: None }
    opened by korewaChino 2
  • chore(main): release 0.11.2

    chore(main): release 0.11.2

    :robot: I have created a release beep boop

    0.11.2 (2022-12-31)

    Features

    • structure: add Structure::into_{attribute, block} (e927806)

    This PR was generated with Release Please. See documentation.

    autorelease: tagged 
    opened by github-actions[bot] 1
  • chore(main): release 0.11.1

    chore(main): release 0.11.1

    opened by github-actions[bot] 1
  • chore(main): release 0.11.0

    chore(main): release 0.11.0

    :robot: I have created a release beep boop

    0.11.0 (2022-12-25)

    ⚠ BREAKING CHANGES

    • serde: hcl::ser::Serializer does not implement serde::Serializer anymore. Use serializer.serialize(&value) in places where value.serialize(&mut serializer) was used before.

    Features

    • de: support expressions in custom types (#139) (3fd68ec), closes #137
    • serde: use internal serialization to roundtrip crate types (#135) (fbd555b)
    • ser: support serializing blocks from custom types (#140) (d97bceb), closes #138

    Miscellaneous

    • clippy: fix newly added lints (e95e4d0)
    • deps: bump dependencies to latest versions (#141) (509e62e)

    This PR was generated with Release Please. See documentation.

    autorelease: tagged 
    opened by github-actions[bot] 1
  • chore(main): release 0.12.0

    chore(main): release 0.12.0

    :robot: I have created a release beep boop

    0.12.0 (2023-01-01)

    ⚠ BREAKING CHANGES

    • structure: The signature of Block::new was changed from Block::new(ident, labels, body) to Block::new(ident). If you depend on the previos behaviour, use Body::from((ident, labels, body)) instead.

    Features

    • structure: implement all structure iterators for Body (#149) (ee752fa)

    Bug Fixes

    • structure: update signature of Block::new (2622f4d)

    Miscellaneous

    • deps: bump actions/cache from 3.0.11 to 3.2.2 (#148) (10c550a)
    • license: update year (faee584)

    This PR was generated with Release Please. See documentation.

    autorelease: pending 
    opened by github-actions[bot] 0
  • Feature request: keeping comments in tact

    Feature request: keeping comments in tact

    I am parsing HCL files that contain comments to be able to update some blocks/attributes, but I would like to keep comments in tact after parsing -> serializing back to text.

    Maybe I have missed it, if so please feel free to link me to the documentation.

    Thank you!

    enhancement 
    opened by archoversight 3
Releases(v0.11.2)

  • v0.11.2(Dec 31, 2022)

  • v0.11.1(Dec 29, 2022)

  • v0.11.0(Dec 25, 2022)

    0.11.0 (2022-12-25)

    ⚠ BREAKING CHANGES

    • serde: hcl::ser::Serializer does not implement serde::Serializer anymore. Use serializer.serialize(&value) in places where value.serialize(&mut serializer) was used before.

    Features

    • de: support expressions in custom types (#139) (3fd68ec), closes #137
    • serde: use internal serialization to roundtrip crate types (#135) (fbd555b)
    • ser: support serializing blocks from custom types (#140) (d97bceb), closes #138

    Miscellaneous

    • clippy: fix newly added lints (e95e4d0)
    • deps: bump dependencies to latest versions (#141) (509e62e)
    Source code(tar.gz)
    Source code(zip)

  • v0.10.0(Nov 17, 2022)

    0.10.0 (2022-11-17)

    ⚠ BREAKING CHANGES

    • eval: the Evaluate implementation of TemplateExpr returns a Value instead of a String now to support interpolation unwrapping.

    Features

    • format: add prefer_ident_keys to FormatterBuilder (#134) (de48f5c), closes #132
    • template: implement Serialize/Deserialize for Template (49e3bdd)

    Bug Fixes

    • eval: correctly handle interpolation unwrapping (85bed59)
    • format: prevent double-escaping of template strings (#133) (9d0d6b4), closes #131
    • parser: improve grammar for heredoc (#130) (d366a32)

    Miscellaneous

    Source code(tar.gz)
    Source code(zip)

  • v0.9.3(Nov 7, 2022)

  • v0.9.2(Nov 6, 2022)

  • v0.9.1(Nov 2, 2022)

    0.9.1 (2022-11-02)

    Features

    • re-export Template type at the crate root (3fac02b)

    Bug Fixes

    Miscellaneous

    • deps: bump actions/cache from 3.0.9 to 3.0.11 (#113) (4706b46)
    • deps: update textwrap requirement from 0.15.0 to 0.16.0 (#114) (9b4f5a4)
    • format: remove ident validation (#116) (d21083e)
    • move Identifier type to crate root (#110) (c8e6af0)
    • move expression types into expr module (#115) (6acba6c), closes #100
    • update description in Cargo.toml (8fc4d9b)
    Source code(tar.gz)
    Source code(zip)

  • v0.9.0(Oct 27, 2022)

    0.9.0 (2022-10-27)

    ⚠ BREAKING CHANGES

    • The type of Expression's Variable variant changed from Identifier to Variable. You can create a variable from an identifier via Variable::from(identifier) or by using Variable::new and Variable::sanitized.
    • The From<Identifier> implementation for String has been removed because it causes problems with trait bounds in From implementations for other types. Use Identifier::into_inner instead.
    • The Block struct's identifier field type was changed from String to Identifier. Furthermore, the trait bound for the block identifier on Block::new changed from Into<String> to Into<Identifier>.
    • The Attribute struct's key field type was changed from String to Identifier. Furthermore, the trait bound for the attribute identifier on Attribute::new changed from Into<String> to Into<Identifier>.
    • Identifier::new is fallible now and the return value changed from Identifier to Result<Identifier, Error>. An infallible alternative is provided with Identifier::sanitized. The inner String field of Identifier is now private to prevent direct modification.

    Features

    • add Variable type (#108) (ee5a5c8)
    • expression and template evaluation (#99) (ce0d229)
    • implement AsRef&lt;str&gt; and Borrow<str> for Identifier (0a616e1)

    Bug Fixes

    • change Attribute key field type to Identifier (#106) (84e1538)
    • change Block identifier field type to Identifier (#107) (badce8a)
    • change type of Expression::Variable variant (#109) (5e1501a)
    • remove From&lt;Identifier&gt; for String (32a94ff)
    • sanitize identifiers upon creation (#105) (7b085d7)

    Miscellaneous

    • mark BlockLabel::{string,identifier} functions as deprecated (86dda75)
    • mark ObjectKey::identifier function as deprecated (d2f5f94)
    Source code(tar.gz)
    Source code(zip)

  • v0.8.8(Oct 22, 2022)

  • v0.8.7(Oct 22, 2022)

  • v0.8.6(Oct 14, 2022)

  • v0.8.5(Oct 12, 2022)

  • v0.8.4(Oct 7, 2022)

  • v0.8.3(Oct 6, 2022)

  • v0.8.2(Sep 30, 2022)

  • v0.8.1(Sep 30, 2022)

  • v0.8.0(Sep 29, 2022)

    0.8.0 (2022-09-29)

    ⚠ BREAKING CHANGES

    • The Number type was changed from an enum to an opaque struct. Use Number::from to create a number from an integer. Furthermore, the From implementations for f32 and f64 were removed. Use the newly added Number::from_f64 instead.
    • The RawExpression and String variants of the ObjectKey enum were removed in favor of the newly added Expression variant. Furthermore the methods Object::raw_expression and ObjectKey::string were removed. Use ObjectKey::from instead.
    • The underlying map implementation for the Object<K, V> type changed from IndexMap<K, V> to VecMap<K, V>. For the most common operations this is a drop-in replacement, but VecMap lacks some of the more exotic APIs the IndexMap provides.
    • Heredocs and quoted strings containing template interpolations and/or template directives are not parsed as Expression::String anymore, but end up as Expression::TemplateExpr (which can be further parsed into the template elements via Template::from_expr) instead. Expressions of kind Expression::String are guaranteed to not include any templating anymore.

    Features

    Bug Fixes

    • allow - in identifiers (be06f53)
    • always do f64 division (5591e22)
    • correctly handle Expression::Null in serializer (ae74def)
    • correctly handle Expression variants in ExpressionSerializer (#71) (8d89437)
    • prevent creation of infinite and NaN Number (#74) (f751fc0)
    • use correct deserializer for index element access (#67) (f7bdd5c)

    Miscellaneous

    • deps: bump vecmap-rs to 0.1.3 (5670415)
    Source code(tar.gz)
    Source code(zip)

  • v0.7.0(Sep 6, 2022)

  • v0.6.5(Aug 19, 2022)

  • v0.6.4(Jul 30, 2022)

  • v0.6.3(Jul 30, 2022)

  • v0.6.2(Jul 13, 2022)

  • v0.6.1(Jun 12, 2022)

  • v0.6.0(Jun 10, 2022)

  • v0.5.2(Jun 6, 2022)

  • v0.5.1(Jun 5, 2022)

  • v0.5.0(Jun 3, 2022)

  • v0.4.0(Jun 3, 2022)

  • v0.3.3(Mar 26, 2022)

  • v0.3.2(Mar 26, 2022)

Owner
null
GitHub
Strongly typed JSON library for Rust

Serde JSON   Serde is a framework for serializing and deserializing Rust data structures efficiently and generically. [dependencies] serde_json = "1.0

null 3.6k Jan 5, 2023
Rust port of simdjson

SIMD Json for Rust   Rust port of extremely fast simdjson JSON parser with serde compatibility. readme (for real!) simdjson version Currently tracking

null 737 Dec 30, 2022
JSON implementation in Rust

json-rust Parse and serialize JSON with ease. Changelog - Complete Documentation - Cargo - Repository Why? JSON is a very loose format where anything

Maciej Hirsz 500 Dec 21, 2022
Rust port of gjson，get JSON value by dotpath syntax

A-JSON Read JSON values quickly - Rust JSON Parser change name to AJSON, see issue Inspiration comes from gjson in golang Installation Add it to your

Chen Jiaju 90 Dec 6, 2022
Get JSON values quickly - JSON parser for Rust

get json values quickly GJSON is a Rust crate that provides a fast and simple way to get values from a json document. It has features such as one line

Josh Baker 160 Dec 29, 2022
This library is a pull parser for CommonMark, written in Rust

This library is a pull parser for CommonMark, written in Rust. It comes with a simple command-line tool, useful for rendering to HTML, and is also designed to be easy to use from as a library.

Raph Levien 1.5k Jan 1, 2023
A rust script to convert a better bibtex json file from Zotero into nice organised notes in Obsidian

Zotero to Obsidian script This is a script that takes a better bibtex JSON file exported by Zotero and generates an organised collection of reference

Sashin Exists 3 Oct 9, 2022
Fontdue - The fastest font renderer in the world, written in pure rust.

Fontdue is a simple, no_std (does not use the standard library for portability), pure Rust, TrueType (.ttf/.ttc) & OpenType (.otf) font rasterizer and layout tool. It strives to make interacting with fonts as fast as possible, and currently has the lowest end to end latency for a font rasterizer.

Joe C 1k Jan 2, 2023
CLI tool to convert HOCON into valid JSON or YAML written in Rust.

{hocon:vert} CLI Tool to convert HOCON into valid JSON or YAML. Under normal circumstances this is mostly not needed because hocon configs are parsed

Mathias Oertel 23 Jan 6, 2023
Typify - Compile JSON Schema documents into Rust types.

Typify Compile JSON Schema documents into Rust types. This can be used ... via the macro import_types!("types.json") to generate Rust types directly i

Oxide Computer Company 73 Dec 27, 2022
A easy and declarative way to test JSON input in Rust.

assert_json A easy and declarative way to test JSON input in Rust. assert_json is a Rust macro heavily inspired by serde json macro. Instead of creati

Charles Vandevoorde 8 Dec 5, 2022
Hjson for Rust

hjson-rust for serde { # specify rate in requests/second (because comments are helpful!) rate: 1000 // prefer c-style comments? /* feeling ol

Hjson 83 Oct 5, 2022
A small rust database that uses json in memory.

Tiny Query Database (TQDB) TQDB is a small library for creating a query-able database that is encoded with json. The library is well tested (~96.30% c

Kace Cottam 2 Jan 4, 2022
A JSON Query Language CLI tool built with Rust 🦀

JQL A JSON Query Language CLI tool built with Rust ?? ?? Core philosophy ?? Stay lightweight ?? Keep its features as simple as possible ?? Avoid redun

Davy Duperron 872 Jan 1, 2023
An implementation of the JSONPath A spec in Rust, with several extensions added on

Rust JSONPath Plus An implementation of the JSONPath A spec in Rust, with several extensions added on. This library also supports retrieving AST analy

Rune Tynan 4 Jul 13, 2022
Rust libraries and tools to help with interoperability and testing of serialization formats based on Serde.

The repository zefchain/serde-reflection is based on Facebook's repository novifinancial/serde-reflection. We are now maintaining the project here and

Zefchain Labs 46 Dec 22, 2022
Blazing fast Rust JSONPath query engine.

rsonpath – SIMD-powered JSONPath ?? Experimental JSONPath engine for querying massive streamed datasets. Features The rsonpath crate provides a JSONPa

V0ldek 21 Apr 11, 2023
A Rust program that analyzes your TikTok data.

The TikTok JSON analyzer This is a program to analyze your TikTok data and calculate these statistics : Number of logins (in the last 6 months) and lo

Elazrod56 3 May 2, 2023
Yet Another Serializer/Deserializer

yaserde   Yet Another Serializer/Deserializer specialized for XML Goal This library will support XML de/ser-ializing with all specific features. Suppo

null 137 Jan 5, 2023
Serializer and deserializer for the VCR Cassette format

vcr-cassette Serializer and deserializer for the VCR Cassette format API Docs | Releases | Contributing Examples Given the following .json VCR Cassett

http-rs 12 Sep 15, 2021