Summary

The C# implementation of enum generation looks at all variants of an enum and, if all are of type Unit, it then generates a native enum and otherwise generates a class. The current Dart implementation simply generates all classes if c_style_enums = false or all enums if c_style_enums = true.

This PR changes the Dart logic to match C# (the only other language that currently implements the c_style_enums config. dart.rs will now generate all classes if c_style_enums = false and if it is true it will generate native Dart enums unless the enum contains data, then it will generate a class.

Test Plan

A test was added that verifies that when c_style_enums is enabled the generator will generate native Dart enums for the simple case but generate classes for data-containing enums.

Summary

The 128 bit type implementations used a 64 bit number which isn't allowed on the Dart web platform (32 bits is the max supported). This switches from the BigInt from which takes a number to the parse function which uses a string to create a BigInt instance.

Test Plan

from and parse are identical except that parse can throw an exception, in our usage it can never throw because the value is static.

🚀 Feature Request

Motivation

Code generation is inefficient for static size byte arrays

Hash:
  NEWTYPESTRUCT:
    TUPLEARRAY:
      CONTENT: U8
      SIZE: 32

generates

type Hash [32]uint8

func (obj *Hash) Serialize(serializer serde.Serializer) error {
	if err := serializer.IncreaseContainerDepth(); err != nil {
		return err
	}
	if err := serialize_array32_u8_array((([32]uint8)(*obj)), serializer); err != nil {
		return err
	}
	serializer.DecreaseContainerDepth()
	return nil
}

func serialize_array32_u8_array(value [32]uint8, serializer serde.Serializer) error {
	for _, item := range value {
		if err := serializer.SerializeU8(item); err != nil {
			return err
		}
	}
	return nil
}

Pitch

Describe the solution you'd like

• Extend serde Go interface to be aware of byte arrays
• Implement bincode serde for byte arrays with a single function call

Describe alternatives you've considered

none

Are you willing to open a pull request? (See CONTRIBUTING)

Maybe later

Additional context

Summary

Initial implementation of the Dart runtime handled string ser/de with [utf-16] code units. This had the appearance of working as long as the given character was < 128. This PR switches to utf-8 encode/decode to fix this bug.

Test Plan

A test is included with Japanese text which causes the current implementation to fail.

Summary

Add the common copyWith utility method to generated Dart classes.

Uses this method for allowing nullable properties to be set to null.

Additionally, I noticed some incorrect formatting of the operator == method output while inspecting the code. I added a fix for that, but I can split into a separate PR if that is preferable.

  part of example_types;

@immutable
class OtherTypes {
  const OtherTypes({
    required this.fString,
    required this.fBytes,
    this.fOption,
    required this.fUnit,
    required this.fSeq,
    required this.fTuple,
    required this.fStringmap,
    required this.fIntset,
    required this.fNestedSeq,
  });

  OtherTypes.deserialize(BinaryDeserializer deserializer) :
    fString = deserializer.deserializeString(),
    fBytes = deserializer.deserializeBytes(),
    fOption = TraitHelpers.deserializeOptionStruct(deserializer),
    fUnit = deserializer.deserializeUnit(),
    fSeq = TraitHelpers.deserializeVectorStruct(deserializer),
    fTuple = TraitHelpers.deserializeTuple2U8U16(deserializer),
    fStringmap = TraitHelpers.deserializeMapStrToU32(deserializer),
    fIntset = TraitHelpers.deserializeMapU64ToUnit(deserializer),
    fNestedSeq = TraitHelpers.deserializeVectorVectorStruct(deserializer);

  static OtherTypes bincodeDeserialize(Uint8List input) {
    final deserializer = BincodeDeserializer(input);
    final value = OtherTypes.deserialize(deserializer);
    if (deserializer.offset < input.length) {
      throw Exception('Some input bytes were not read');
    }
    return value;
  }

  static OtherTypes bcsDeserialize(Uint8List input) {
    final deserializer = BcsDeserializer(input);
    final value = OtherTypes.deserialize(deserializer);
    if (deserializer.offset < input.length) {
      throw Exception('Some input bytes were not read');
    }
    return value;
  }

  final String fString;
  final Bytes fBytes;
  final Struct? fOption;
  final Unit fUnit;
  final List<Struct> fSeq;
  final Tuple2<int, int> fTuple;
  final Map<String, int> fStringmap;
  final Map<Uint64, Unit> fIntset;
  final List<List<Struct>> fNestedSeq;

  OtherTypes copyWith({
    String? fString,
    Bytes? fBytes,
    Struct? Function()? fOption,
    Unit? fUnit,
    List<Struct>? fSeq,
    Tuple2<int, int>? fTuple,
    Map<String, int>? fStringmap,
    Map<Uint64, Unit>? fIntset,
    List<List<Struct>>? fNestedSeq,
  }) {
    return OtherTypes(
      fString: fString ?? this.fString,
      fBytes: fBytes ?? this.fBytes,
      fOption: fOption == null ? this.fOption : fOption(),
      fUnit: fUnit ?? this.fUnit,
      fSeq: fSeq ?? this.fSeq,
      fTuple: fTuple ?? this.fTuple,
      fStringmap: fStringmap ?? this.fStringmap,
      fIntset: fIntset ?? this.fIntset,
      fNestedSeq: fNestedSeq ?? this.fNestedSeq,
    );
  }

  void serialize(BinarySerializer serializer) {
    serializer.serializeString(fString);
    serializer.serializeBytes(fBytes);
    TraitHelpers.serializeOptionStruct(fOption, serializer);
    serializer.serializeUnit(fUnit);
    TraitHelpers.serializeVectorStruct(fSeq, serializer);
    TraitHelpers.serializeTuple2U8U16(fTuple, serializer);
    TraitHelpers.serializeMapStrToU32(fStringmap, serializer);
    TraitHelpers.serializeMapU64ToUnit(fIntset, serializer);
    TraitHelpers.serializeVectorVectorStruct(fNestedSeq, serializer);
  }

  Uint8List bincodeSerialize() {
      final serializer = BincodeSerializer();
      serialize(serializer);
      return serializer.bytes;
  }

  Uint8List bcsSerialize() {
      final serializer = BcsSerializer();
      serialize(serializer);
      return serializer.bytes;
  }

  @override
  bool operator ==(Object other) {
    if (identical(this, other)) return true;
    if (other.runtimeType != runtimeType) return false;

    return other is OtherTypes
      && fString == other.fString
      && fBytes == other.fBytes
      && fOption == other.fOption
      && fUnit == other.fUnit
      && listEquals(fSeq, other.fSeq)
      && fTuple == other.fTuple
      && mapEquals(fStringmap, other.fStringmap)
      && mapEquals(fIntset, other.fIntset)
      && listEquals(fNestedSeq, other.fNestedSeq);
  }

  @override
  int get hashCode => Object.hash(
        fString,
        fBytes,
        fOption,
        fUnit,
        fSeq,
        fTuple,
        fStringmap,
        fIntset,
        fNestedSeq,
      );

  @override
  String toString() {
    String? fullString;

    assert(() {
      fullString = '$runtimeType('
        'fString: $fString, '
        'fBytes: $fBytes, '
        'fOption: $fOption, '
        'fUnit: $fUnit, '
        'fSeq: $fSeq, '
        'fTuple: $fTuple, '
        'fStringmap: $fStringmap, '
        'fIntset: $fIntset, '
        'fNestedSeq: $fNestedSeq'
        ')';
      return true;
    }());

    return fullString ?? 'OtherTypes';
  }
}

Test Plan

I'm not sure if this needs separate tests outside of analyze still passing. To verify my code worked, I logged out the path for one of the test projects and checked the output. I'm not sure if there is a better way to inspect generated code.

cc: @jerel

🚀 Feature Request

When generating code, enum variants should be expressed in the native preferred format. In Swift, that is camelCased, as opposed to Rust's UpperCamelCased.

Motivation

Swift types generated with serde-generate feel out of place since they start with a capital letter. It's minor, but the code would be closer to a first class citizen if they were properly camelCased.

Bumps regex from 1.5.4 to 1.5.5.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Summary

Changes Swift enum variants to be expressed in camelCase, rather than Rust's UpperCamelCase. Addresses #10

Test Plan

Swift tests are passing. It would be nice to have better testing in general on the generated code, as the coverage for a change like this is very small (as you can see in the test structure, it was a single character change), but I didn't want to try to write new tests unprompted, given how complicated the current testing setup appears to be.

Summary

• Add missing to_caml_case()
• Also test if field names are Ocaml keywords
• Prefix enum variants with the name of the enum (this is a bit sad in Ocaml but hard to avoid because Rust works like this)

Test Plan

CI

curl https://raw.githubusercontent.com/aptos-labs/aptos-core/master/testsuite/generate-format/tests/staged/aptos.yaml > aptos.yaml

rm -rf test

cargo run --bin serdegen -- --language ocaml --with-runtimes serde bcs --target-source-dir test -- aptos.yaml

(cd test && dune build)

🚀 Feature Request

Motivation

cargo tree currently reports the following dependencies for serde-generate:

serde-generate v0.24.0
├── bcs v0.1.4
│   ├── serde v1.0.148
│   │   └── serde_derive v1.0.148 (proc-macro)
│   │       ├── proc-macro2 v1.0.47
│   │       │   └── unicode-ident v1.0.5
│   │       ├── quote v1.0.21
│   │       │   └── proc-macro2 v1.0.47 (*)
│   │       └── syn v1.0.104
│   │           ├── proc-macro2 v1.0.47 (*)
│   │           ├── quote v1.0.21 (*)
│   │           └── unicode-ident v1.0.5
│   └── thiserror v1.0.37
│       └── thiserror-impl v1.0.37 (proc-macro)
│           ├── proc-macro2 v1.0.47 (*)
│           ├── quote v1.0.21 (*)
│           └── syn v1.0.104 (*)
├── bincode v1.3.3
│   └── serde v1.0.148 (*)
├── heck v0.3.3
│   └── unicode-segmentation v1.10.0
├── include_dir v0.6.2
│   ├── glob v0.3.0
│   ├── include_dir_impl v0.6.2 (proc-macro)
│   │   ├── anyhow v1.0.66
│   │   ├── proc-macro-hack v0.5.19 (proc-macro)
│   │   ├── proc-macro2 v1.0.47 (*)
│   │   ├── quote v1.0.21 (*)
│   │   └── syn v1.0.104 (*)
│   └── proc-macro-hack v0.5.19 (proc-macro)
├── maplit v1.0.2
├── phf v0.10.1
│   ├── phf_macros v0.10.0 (proc-macro)
│   │   ├── phf_generator v0.10.0
│   │   │   ├── phf_shared v0.10.0
│   │   │   │   └── siphasher v0.3.10
│   │   │   └── rand v0.8.5
│   │   │       ├── libc v0.2.137
│   │   │       ├── rand_chacha v0.3.1
│   │   │       │   ├── ppv-lite86 v0.2.17
│   │   │       │   └── rand_core v0.6.4
│   │   │       │       └── getrandom v0.2.8
│   │   │       │           ├── cfg-if v1.0.0
│   │   │       │           └── libc v0.2.137
│   │   │       └── rand_core v0.6.4 (*)
│   │   ├── phf_shared v0.10.0 (*)
│   │   ├── proc-macro-hack v0.5.19 (proc-macro)
│   │   ├── proc-macro2 v1.0.47 (*)
│   │   ├── quote v1.0.21 (*)
│   │   └── syn v1.0.104 (*)
│   ├── phf_shared v0.10.0 (*)
│   └── proc-macro-hack v0.5.19 (proc-macro)
├── serde v1.0.148 (*)
├── serde-reflection v0.3.6
│   ├── once_cell v1.16.0
│   ├── serde v1.0.148 (*)
│   └── thiserror v1.0.37 (*)
├── serde_bytes v0.11.8
│   └── serde v1.0.148 (*)
├── serde_yaml v0.8.26
│   ├── indexmap v1.9.2
│   │   └── hashbrown v0.12.3
│   │   [build-dependencies]
│   │   └── autocfg v1.1.0
│   ├── ryu v1.0.11
│   ├── serde v1.0.148 (*)
│   └── yaml-rust v0.4.5
│       └── linked-hash-map v0.5.6
├── structopt v0.3.26
│   ├── clap v2.34.0
│   │   ├── ansi_term v0.12.1
│   │   ├── atty v0.2.14
│   │   │   └── libc v0.2.137
│   │   ├── bitflags v1.3.2
│   │   ├── strsim v0.8.0
│   │   ├── textwrap v0.11.0
│   │   │   └── unicode-width v0.1.10
│   │   ├── unicode-width v0.1.10
│   │   └── vec_map v0.8.2
│   ├── lazy_static v1.4.0
│   └── structopt-derive v0.4.18 (proc-macro)
│       ├── heck v0.3.3 (*)
│       ├── proc-macro-error v1.0.4
│       │   ├── proc-macro-error-attr v1.0.4 (proc-macro)
│       │   │   ├── proc-macro2 v1.0.47 (*)
│       │   │   └── quote v1.0.21 (*)
│       │   │   [build-dependencies]
│       │   │   └── version_check v0.9.4
│       │   ├── proc-macro2 v1.0.47 (*)
│       │   ├── quote v1.0.21 (*)
│       │   └── syn v1.0.104 (*)
│       │   [build-dependencies]
│       │   └── version_check v0.9.4
│       ├── proc-macro2 v1.0.47 (*)
│       ├── quote v1.0.21 (*)
│       └── syn v1.0.104 (*)
└── textwrap v0.13.4
    ├── smawk v0.3.1
    └── unicode-width v0.1.10

Ideally serde-generate would provide an API that put dependencies behind different feature flags, so that not everything is required if only a small part of the crate is used.

Especially that includes the following dependencies:

• structopt as that's not needed if serde-generate is used as dependency
• bcs and bincode as sometimes you want to use one or the other
• serde_yaml as it's not always required to build a yaml representation of the corresponding types

I'm not sure about other dependencies like textwrap, heck, serde_bytes, include_dir, maplit and phf.

Pitch

Describe the solution you'd like Put parts of the API behind feature flags, so that users can choose what to use + what gate corresponding dependencies on these flags.

Describe alternatives you've considered

Do not change anything and pull in potentially unneeded dependencies.

Are you willing to open a pull request? (See CONTRIBUTING)

I would be willing to contribute a PR for that

Additional context

🚀 Feature Request

Motivation

cargo tree currently reports the following dependencies for serde-generate:

serde-generate v0.24.0
├── bcs v0.1.4
│   ├── serde v1.0.148
│   │   └── serde_derive v1.0.148 (proc-macro)
│   │       ├── proc-macro2 v1.0.47
│   │       │   └── unicode-ident v1.0.5
│   │       ├── quote v1.0.21
│   │       │   └── proc-macro2 v1.0.47 (*)
│   │       └── syn v1.0.104
│   │           ├── proc-macro2 v1.0.47 (*)
│   │           ├── quote v1.0.21 (*)
│   │           └── unicode-ident v1.0.5
│   └── thiserror v1.0.37
│       └── thiserror-impl v1.0.37 (proc-macro)
│           ├── proc-macro2 v1.0.47 (*)
│           ├── quote v1.0.21 (*)
│           └── syn v1.0.104 (*)
├── bincode v1.3.3
│   └── serde v1.0.148 (*)
├── heck v0.3.3
│   └── unicode-segmentation v1.10.0
├── include_dir v0.6.2
│   ├── glob v0.3.0
│   ├── include_dir_impl v0.6.2 (proc-macro)
│   │   ├── anyhow v1.0.66
│   │   ├── proc-macro-hack v0.5.19 (proc-macro)
│   │   ├── proc-macro2 v1.0.47 (*)
│   │   ├── quote v1.0.21 (*)
│   │   └── syn v1.0.104 (*)
│   └── proc-macro-hack v0.5.19 (proc-macro)
├── maplit v1.0.2
├── phf v0.10.1
│   ├── phf_macros v0.10.0 (proc-macro)
│   │   ├── phf_generator v0.10.0
│   │   │   ├── phf_shared v0.10.0
│   │   │   │   └── siphasher v0.3.10
│   │   │   └── rand v0.8.5
│   │   │       ├── libc v0.2.137
│   │   │       ├── rand_chacha v0.3.1
│   │   │       │   ├── ppv-lite86 v0.2.17
│   │   │       │   └── rand_core v0.6.4
│   │   │       │       └── getrandom v0.2.8
│   │   │       │           ├── cfg-if v1.0.0
│   │   │       │           └── libc v0.2.137
│   │   │       └── rand_core v0.6.4 (*)
│   │   ├── phf_shared v0.10.0 (*)
│   │   ├── proc-macro-hack v0.5.19 (proc-macro)
│   │   ├── proc-macro2 v1.0.47 (*)
│   │   ├── quote v1.0.21 (*)
│   │   └── syn v1.0.104 (*)
│   ├── phf_shared v0.10.0 (*)
│   └── proc-macro-hack v0.5.19 (proc-macro)
├── serde v1.0.148 (*)
├── serde-reflection v0.3.6
│   ├── once_cell v1.16.0
│   ├── serde v1.0.148 (*)
│   └── thiserror v1.0.37 (*)
├── serde_bytes v0.11.8
│   └── serde v1.0.148 (*)
├── serde_yaml v0.8.26
│   ├── indexmap v1.9.2
│   │   └── hashbrown v0.12.3
│   │   [build-dependencies]
│   │   └── autocfg v1.1.0
│   ├── ryu v1.0.11
│   ├── serde v1.0.148 (*)
│   └── yaml-rust v0.4.5
│       └── linked-hash-map v0.5.6
├── structopt v0.3.26
│   ├── clap v2.34.0
│   │   ├── ansi_term v0.12.1
│   │   ├── atty v0.2.14
│   │   │   └── libc v0.2.137
│   │   ├── bitflags v1.3.2
│   │   ├── strsim v0.8.0
│   │   ├── textwrap v0.11.0
│   │   │   └── unicode-width v0.1.10
│   │   ├── unicode-width v0.1.10
│   │   └── vec_map v0.8.2
│   ├── lazy_static v1.4.0
│   └── structopt-derive v0.4.18 (proc-macro)
│       ├── heck v0.3.3 (*)
│       ├── proc-macro-error v1.0.4
│       │   ├── proc-macro-error-attr v1.0.4 (proc-macro)
│       │   │   ├── proc-macro2 v1.0.47 (*)
│       │   │   └── quote v1.0.21 (*)
│       │   │   [build-dependencies]
│       │   │   └── version_check v0.9.4
│       │   ├── proc-macro2 v1.0.47 (*)
│       │   ├── quote v1.0.21 (*)
│       │   └── syn v1.0.104 (*)
│       │   [build-dependencies]
│       │   └── version_check v0.9.4
│       ├── proc-macro2 v1.0.47 (*)
│       ├── quote v1.0.21 (*)
│       └── syn v1.0.104 (*)
└── textwrap v0.13.4
    ├── smawk v0.3.1
    └── unicode-width v0.1.10

Ideally serde-generate would provide an API that put dependencies behind different feature flags, so that not everything is required if only a small part of the crate is used.

Especially that includes the following dependencies:

• structopt as that's not needed if serde-generate is used as dependency
• bcs and bincode as sometimes you want to use one or the other
• serde_yaml as it's not always required to build a yaml representation of the corresponding types

I'm not sure about other dependencies like textwrap, heck, serde_bytes, include_dir, maplit and phf.

Pitch

Describe the solution you'd like Put parts of the API behind feature flags, so that users can choose what to use + what gate corresponding dependencies on these flags.

Describe alternatives you've considered

Do not change anything and pull in potentially unneeded dependencies.

Are you willing to open a pull request? (See CONTRIBUTING)

I would be willing to contribute a PR for that

Additional context

🐛 Bug

#[derive(Serialize, Deserialize, Default)] pub struct A { pub created_at: chrono::NaiveDateTime, pub xxx: i64, }

let mut tracer = serde_reflection::Tracer::new(serde_reflection::TracerConfig::default()); let mut samples = serde_reflection::Samples::new(); tracer.trace_type::(&mut samples).unwrap(); let registry = tracer.registry().unwrap(); serde_json::to_string_pretty(&registry).unwrap_or_default();

🐛 Bug

Generated swift code, which includes types that reference each other, prefixes local references with module name provided to CodeGeneratorConfig. The code fails to compile with an error "Cannot find type '' in scope".

The problem can be worked around by searching and replacing the module name followed by "." (e.g. "module.") with an empty string, effectively stripping the namespacing out of the generated code.

To reproduce

Types to be traced

#[derive(Serialize, Deserialize)]
pub struct Request {
    pub uuid: Vec<u8>,
    pub body: RequestBody,
}

#[derive(Serialize, Deserialize)]
pub enum RequestBody {
   // ... variants
}

Tracing and generating build.rs script

use rmm::{Request, RequestBody};
use serde_reflection::{Tracer, TracerConfig};
use std::{fs::File, io::Write, path::PathBuf};

fn main() {
    let mut tracer = Tracer::new(TracerConfig::default());
    tracer.trace_simple_type::<Request>().unwrap();
    tracer.trace_simple_type::<RequestBody>().unwrap();
    let registry = tracer.registry().unwrap();

    // Create Swift definitions.
    let mut source = Vec::new();
    let config = serde_generate::CodeGeneratorConfig::new("shared".to_string())
        .with_encodings(vec![serde_generate::Encoding::Bincode]);

    let generator = serde_generate::swift::CodeGenerator::new(&config);
    generator.output(&mut source, &registry).unwrap();

    let path = "./generated/shared_types.swift";
    let mut output = File::create(path).unwrap();
    write!(output, "{}", out).unwrap();
}

Relevant part of the generated Swift code

public struct Response: Hashable {
    @Indirect public var uuid: [UInt8]
    @Indirect public var body: shared.ResponseBody // "Error: Cannot find type 'shared' in scope"

    (...)

Removing shared. from the above resolves the problem.

Expected Behavior

Generated Swift code should compile without errors.

System information

Serde reflection v0.3.6 Serde generate v0.24.0

🚀 Feature Request

serde_reflection::trace_type relies on a few ideas and heuristics that it might be time to revisit.

Notably, we use the current table of formats tracer.registry to guide the tracing in most decision points -- with the exception of tracer.incomplete_enums that is also used to decide how to trace enum variants.

This approach was a natural starting point but it is not ideal in the long run for a few reasons:

Calling trace_value before trace_type may update the registry in an incomplete way and defeat further tracing with trace_type. We don't have an easy way to detect non-termination if the base case of enums is not the first variant. We cannot call trace_type_once<T> repeatedly and hope to make progress in finding variants that are deeper that T. (For instance, after one tracing call on struct T(option<S>) we are not going to look into S ever again because it has a format: https://github.com/novifinancial/serde-reflection/blob/master/serde-reflection/src/de.rs#L199)

Moving this issue across from https://github.com/novifinancial/serde-reflection/issues/104

Summary

My original implementation of Dart followed one of the other language's enum implementations in that it appended Item to the name of each variant. After using it for a while it turns out it's rather annoying and seems to be wholly unnecessary as the enum name is prepended to each variant when generating class based enums and this seems to avoid any chance of collision. Considering that Dart is still considered "under development" I figured it was a good time to walk this back.

For Rust enum:

enum Choice {
  A,
  B,
  C
}

Current output:

abstract class Choice {}

class ChoiceAItem extends Choice {}
class ChoiceBItem extends Choice {}
class ChoiceCItem extends Choice {} 

This PR's output:

abstract class Choice {}

class ChoiceA extends Choice {}
class ChoiceB extends Choice {}
class ChoiceC extends Choice {} 

Test Plan

Uses existing test.