would you be so kind as to review the serde_macros and Deserializer changes?

current state:

• [x] xml to struct deserialization
• [x] deserialize bool, int, string from <anytagname>value</anythingelse>
• [x] deserialize sequences (tuple, array, vector) as struct member
• [x] deserialize escaped chars (&abcd;)
• [x] deserialize CDATA
• [ ] deserialize enumerations
• [ ] deserialize arrays of enumerations
• [ ] deserialize errors instead of assertions
• [ ] more deserialize tests
• [x] parse to dom tree
• [ ] remove debug print! messages
• [ ] struct to xml serialization

anti-features (just ignoring stuff I don't like):

• [x] ignore namespaces
• [x] skip xml comments
• [x] skip xml version tag
• [x] ignoring xml-attributes

things that would be nice to have but might not be possible

• [ ] xsd verification
• [ ] sequences of sequences (how would these even look like in xml?)
• [ ] attributes to collapse xml elements that only contain a single type of element.

I am interacting with an API which returns pagination information on most of their requests. Here is an example:

{
    "limit": 25,
    "offset": 0,
    "total": 1,
    "users": [
        {...}
    ]
}

limit, offset, and total are present for many of the responses received. Here is the struct I deserialize the JSON response into:

#[derive(Debug, Serialize, Deserialize)]
pub struct Users {
    limit: i32,
    offset: i32,
    total: i32,
    users: Vec<User>
}

The problem that I'm having is that I don't know how I can avoid repeating those same 3 fields in other structs.

I would like to have something similar to this:

#[derive(Debug, Serialize, Deserialize)]
pub struct Pagination {
    offset: i32,
    limit: i32,
    total: i32
}

#[derive(Debug, Serialize, Deserialize)]
pub struct Users {
    pagination: Pagination,
    users: Vec<User>
}

Here serde would see that no field pagination was present but the type of the struct field is Pagination which contains fields which are present in the response.

I guess this could be solved with some kind of struct inheritance but Rust doesn't have that at the moment which restricts us to use composition.

Any ideas?

For example the ones from https://github.com/serde-rs/serde/issues/550 and https://github.com/serde-rs/serde-rs.github.io/issues/22. These don't need to be in serde itself but if there are ones that people ask for over and over, we can stick those in a helper crate and provide a better out-of-the-box experience compared to "implement your own serialize_with" or "paste this code into your project."

I have worked only with serde_json, to I may be missing something that applies to other (de)serializers.

Serialization

Serialization of enums seems to always produce objects, even for trivial constructors

#[derive(Serialize, Deserialize)]
pub enum Foo {
    A, B, C
}

fn main() {
  println!("{}", serde_json::to_string(&Foo::A).unwrap());
  // prints: `{"A":[]}`
}

This produces {"A":[]}. I would prefer if it produced just the string "A".

Deserialization

More importantly, deserialization of enums seems to always require objects, which is annoying when one attempts to implement a protocol.

#[derive(Serialize, Deserialize)]
pub enum Foo {
    A, B, C
}

fn main() {
  let a : Foo = self::serde_json::from_str("\"A\"").unwrap();
  // panics with a syntax error
}

I would have preferred if it managed to deserialize the string "A" into enum constructor A.

edit Fixed typo in the second example.

The way serde currently treats ref-counted data structures means that it creates copies of the references objects. This is bad for several reasons:

• Potentially increased memory usage
• Equality comparison that relies on comparison of address breaks
• Interior mutability is not reflected in copies and maybe more that I can't think of right away. I think there should at least be big warnings in the documentation about this behaviour, but the support should be either fixed or deprecated, imo.

@alexcrichton https://github.com/serde-rs/serde/pull/346#issuecomment-223723712

Changes like this which update syntex, a public dependency of serde_codegen (e.g. it reexports types from syntex) are actually a breaking change for the serde crate. Could this be signaled with a major version update or holding off the upgrade? This unfortunately breaks any other crate which is depending on serde_codegen as it may get a mismatch of syntex versions otherwise.

We're probably damned either way. If Serde were on version 0.33, the 275 crates that depend on Serde would be fragmented across all of them (about half are still on ^0.6) and would not work together.

On the one hand (no major version change), we occasionally break crates that use a ^ dependency on Serde and also have another dependency on Syntex. Where this is unacceptable, you could use an = dependency.

On the other hand, we would never break ^ dependencies but practically every time you do cargo update you would end up in a broken state until 275 crates synchronize their Serde dependency.

#[derive(Serialize, Deserialize)]
struct S {
    a: u32,
    b: String,
    #[serde(other)]
    other: Map<String, Value>,
}

{"a":0,"b":"","c":true}

This would deserialize into other containing "c" => true.

Serialization for machines

When structures are serialized to JSON to be transferred over the wire, there may be servers that expect null values to be omitted, possibly to save bandwidth. Right now, there is no way in serde to omit null values.

As a practical example, imagine someone trying to upload a video to youtube:

$ youtube3 --debug videos insert \
-r snippet \
title="Google APIs for Rust: Developer Diary #2 [Making CLIs]" \
description="TBD" \
tags="Google APIs, Google, rust-lang, Diary, OSS" \
category-id=22 \
..status privacy-status=private \
embeddable=true \
license=youtube \
-u resumable ~/Movies/youtube-originals/Google\ APIs\ for\ Rust\ -\ Using\ youtube3\ to\ upload\ a\ video.mov application/octet-stream

Which yields the following error:

Bad Requst (400): Invalid value for: null is not a valid value

As well as the following dialogue between client and server:

POST /resumable/upload/youtube/v3/videos?part=status%2Csnippet&alt=json&uploadType=resumable HTTP/1.1
User-Agent: google-api-rust-client/0.1.6
Host: www.googleapis.com
Transfer-Encoding: chunked
Content-Type: application/json
X-Upload-Content-Type: application/octet-stream
Authorization: Bearer ya29.YwHqwwVjMrn7y_qO7d6RR5KeowbDJFO_2mLk5pTPs9iJZP0k3DEHUm6E4xkOv3pw5oEhX3GBjI-H4A

33C
{"status":{"license":"youtube","embeddable":true,"privacyStatus":"private","publishAt":null,"publicStatsViewable":null,"uploadStatus":null,"rejectionReason":null,"failureReason":null},"topicDetails":null,"monetizationDetails":null,"suggestions":null,"ageGating":null,"fileDetails":null,"player":null,"id":null,"localizations":null,"liveStreamingDetails":null,"snippet":{"description":"TBD","tags":["Google APIs, Google, rust-lang, Diary, OSS"],"channelId":null,"defaultLanguage":null,"liveBroadcastContent":null,"publishedAt":null,"thumbnails":null,"title":"Google APIs for Rust: Developer Diary #2 [Making CLIs]","categoryId":"22","localized":null,"channelTitle":null},"kind":null,"statistics":null,"projectDetails":null,"conversionPings":null,"processingDetails":null,"etag":null,"contentDetails":null,"recordingDetails":null}
0

HTTP/1.1 400 Bad Request
Vary: Origin
Vary: X-Origin
Content-Type: application/json; charset=UTF-8
Content-Length: 234
Date: Tue, 28 Apr 2015 06:31:59 GMT
Server: UploadServer ("Built on Apr 20 2015 22:37:13 (1429594633)")
Alternate-Protocol: 443:quic,p=1

{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "invalid",
    "message": "Invalid value for: null is not a valid value"
   }
  ],
  "code": 400,
  "message": "Invalid value for: null is not a valid value"
 }
}

As you can see, the request contains null values which are not allowed.

To further stress the importance of this feature, have a look at the respective Go implementation ...

type AccessPolicy struct {
    // Allowed: The value of allowed indicates whether the access to the
    // policy is allowed or denied by default.
    Allowed bool `json:"allowed,omitempty"`

    // Exception: A list of region codes that identify countries where the
    // default policy do not apply.
    Exception []string `json:"exception,omitempty"`
}

... where the marker omitempty will prevent it to be serialized if unset.

You can try it yourself using the youtube3 program, which can be downloaded here.

Serialization for human consumption

Right now there is exactly one method to get 'pretty', i.e. more human-friendly json output. It provides no option to specify how exactly that could be done.

The most prominent one to me would be a setting for whether or not to ignore null values. Other options could be the indentation string to use, e.g. \t or .

Motivation

When printing the server response of any of the various google apis using a generated command-line interface, simple invocation yield results like this:

$ discovery1 apis get-rest discovery v1
{
  "protocol": "rest",
  "methods": null,
  "labels": null,
  "kind": "discovery#restDescription",
  "canonicalName": null,
  "ownerName": "Google",
  "documentationLink": "https://developers.google.com/discovery/",
  "auth": null,
  "packagePath": null,
  "batchPath": "batch",
  "id": "discovery:v1",
  "features": null,
  "ownerDomain": "google.com",
  "rootUrl": "https://www.googleapis.com/",
  "name": "discovery",
  "parameters": {
    "key": {
      "description": "API key. Your API key identifies your project and provides you with API access, quota, and reports. Required unless you provide an OAuth 2.0 token.",
      "format": null,
      "enum": null,
      "variant": null,
      "enumDescriptions": null,
      "readOnly": null,
      "minimum": null,
      "repeated": null,
      "id": null,
      "$ref": null,
      "default": null,
      "items": null,
      "required": null,
      "maximum": null,
      "properties": null,
      "location": "query",
      "pattern": null,
      "additionalProperties": null,
      "type": "string",
      "annotations": null
    },
    "userIp": {
      "description": "IP address of the site where the request originates. Use this if you want to enforce per-user limits.",
      "format": null,
      "enum": null,
      "variant": null,
      "enumDescriptions": null,
      "readOnly": null,
      "minimum": null,
      "repeated": null,
      "id": null,
      "$ref": null,
      "default": null,
      "items": null,
      "required": null,
      "maximum": null,
      "properties": null,
      "location": "query",
      "pattern": null,
      "additionalProperties": null,
      "type": "string",
      "annotations": null
    },
  [...]
  "revision": null
}

The above should look like this:

$ discovery1 apis get-rest discovery v1
{
 "kind": "discovery#restDescription",
 "etag": "\"ye6orv2F-1npMW3u9suM3a7C5Bo/rJ-Wlqqs_yJDjtCFAIylPtmqXPY\"",
 "discoveryVersion": "v1",
 "id": "discovery:v1",
 "name": "discovery",
 "version": "v1",
 "title": "APIs Discovery Service",
 "description": "Lets you discover information about other Google APIs, such as what APIs are available, the resource and method details for each API.",
 "ownerDomain": "google.com",
 "ownerName": "Google",
 "icons": {
  "x16": "http://www.google.com/images/icons/feature/filing_cabinet_search-g16.png",
  "x32": "http://www.google.com/images/icons/feature/filing_cabinet_search-g32.png"
 },
 "documentationLink": "https://developers.google.com/discovery/",
 "protocol": "rest",
 "baseUrl": "https://www.googleapis.com/discovery/v1/",
 "basePath": "/discovery/v1/",
 "rootUrl": "https://www.googleapis.com/",
 "servicePath": "discovery/v1/",
 "batchPath": "batch",
 "parameters": {
  "alt": {
   "type": "string",
   "description": "Data format for the response.",
   "default": "json",
   "enum": [
    "json"
   ],
   "enumDescriptions": [
    "Responses with Content-Type of application/json"
   ],
   "location": "query"
  },
  "fields": {
   "type": "string",
   "description": "Selector specifying which fields to include in a partial response.",
   "location": "query"
  },
  "key": {
   "type": "string",
   "description": "API key. Your API key identifies your project and provides you with API access, quota, and reports. Required unless you provide an OAuth 2.0 token.",
   "location": "query"
  },
  "oauth_token": {
   "type": "string",
   "description": "OAuth 2.0 token for the current user.",
   "location": "query"
  },
  "prettyPrint": {
   "type": "boolean",
   "description": "Returns response with indentations and line breaks.",
   "default": "true",
   "location": "query"
  },
  "quotaUser": {
   "type": "string",
   "description": "Available to use for quota purposes for server-side applications. Can be any arbitrary string assigned to a user, but should not exceed 40 characters. Overrides userIp if both are provided.",
   "location": "query"
  },
  "userIp": {
   "type": "string",
   "description": "IP address of the site where the request originates. Use this if you want to enforce per-user limits.",
   "location": "query"
  }
 },
 "schemas": {
  "DirectoryList": {
   "id": "DirectoryList",
   "type": "object",
   "properties": {
    "discoveryVersion": {
     "type": "string",
     "description": "Indicate the version of the Discovery API used to generate this doc.",
     "default": "v1"
    },
    "items": {
     "type": "array",
     "description": "The individual directory entries. One entry per api/version pair.",
     "items": {
      "type": "object",
      "properties": {
       "description": {
        "type": "string",
        "description": "The description of this API."
       },
       "discoveryLink": {
        "type": "string",
        "description": "A link to the discovery document."
       },
       "discoveryRestUrl": {
        "type": "string",
        "description": "The URL for the discovery REST document."
       },
       "documentationLink": {
        "type": "string",
        "description": "A link to human readable documentation for the API."
       },
       "icons": {
        "type": "object",
        "description": "Links to 16x16 and 32x32 icons representing the API.",
        "properties": {
         "x16": {
          "type": "string",
          "description": "The URL of the 16x16 icon."
         },
         "x32": {
          "type": "string",
          "description": "The URL of the 32x32 icon."
         }
        }
       },
       "id": {
        "type": "string",
        "description": "The id of this API."
       },
       "kind": {
        "type": "string",
        "description": "The kind for this response.",
        "default": "discovery#directoryItem"
       },
       "labels": {
        "type": "array",
        "description": "Labels for the status of this API, such as labs or deprecated.",
        "items": {
         "type": "string"
        }
       },
       "name": {
        "type": "string",
        "description": "The name of the API."
       },
       "preferred": {
        "type": "boolean",
        "description": "True if this version is the preferred version to use."
       },
       "title": {
        "type": "string",
        "description": "The title of this API."
       },
       "version": {
        "type": "string",
        "description": "The version of the API."
       }
      }
     }
    },
    "kind": {
     "type": "string",
     "description": "The kind for this response.",
     "default": "discovery#directoryList"
    }
   }
  },
 [...]
}

I'm trying to implement custom type serializer for the BigInt type. So, the first thing that I've made is a look into the serde docs. But this case a little bit complex, that described inside the doc.

That's a code, which I'm using for a serializing:

use num::bigint::{BigInt, Sign};  // other crate
use serde::ser;

pub struct Serializer<W>{
    writer: W
}

impl<W> ser::Serializer for Serializer<W> where W: io::Write {
  // some code there...
}

impl ser::Serialize for BigInt {
    fn serialize<S>(&self, serializer: &mut S) -> Result<()> where S: Serializer {
        let (sign, bytes) = *self.to_bytes_le();
        let length = bytes.len();
        match length {
            0...255 => self.get_small_big_number(sign, length as u8, bytes),
            _ => self.get_large_big_number(sign, length as u32, bytes),
        }
    }
}

As a result, the complier generates an error:

src/serializers.rs:591:71: 591:81 error: `Serializer` is not a trait [E0404]
src/serializers.rs:591     fn serialize<S>(&self, serializer: &mut S) -> Result<()> where S: Serializer {
                                                                                             ^~~~~~~~~~
src/serializers.rs:591:71: 591:81 help: run `rustc --explain E0404` to see a detailed explanation
error: cannot continue compilation due to previous error
error: Could not compile `bert`.

How can I connect all this stuff together and provide method to serialize BigInt type? I've also trying to implement serializing for BigInt type from the num crate as described there, but getting the same error as I said earlier.

This is an alternative implementation of #214, #198, #208 and implements #90 and #216. This allows one to write:

#[derive(Serialize, Deserialize)]
struct Struct {
    #[serde(default="123")]
    a: i32,
    #[serde(skip_serializing_if="self.b == 123")]
    b: i32,
    #[serde(serialize_with="(self.b == 123).serialize(serializer)"]
    c: i32,
    #[serde(deserialize_with="Ok(if try!(bool::deserialize(deserializer) { 123) } else { 0 })"]
    d: i32,
}

cc @oli-obk, @target-san, @arcnmx

• Added codegen for disallow_unknown
• ... with new default to ignore unknown values during deserialization
• Added ContainerAttrs

Original pull request against v0.6.x: https://github.com/serde-rs/serde/pull/201

When using CBOR based parsers, a common occurrence is that a struct such as

struct ProtocolElement {
    title: String,
    value: u16,
    unit: Option<String>,
}

needs to be deserialized from CBOR that, in diagnostic notation, looks like this:

{
    0: "My title",
    5: 23,
    7: "knots",
}

Right now, implementing this manually is rather complex for a task that's very common with CBOR (and possibly other serialization formats that lean towards numeric keys).

If derivation were allowed as #[serde(rename_numeric = 0)], these structs would be trivial to use.

Often, CBOR uses numeric keys where JSON uses string keys (while otherwise sharing the structure). It'd be tempting to suggest the same mechanism for #[serde(alias_numeric = 0)], but that would (even though barely) not allow using the same type for both CBOR and JSON serialization, because while it'd process CBOR and JSON correctly (tolerating JSON-style input on the CBOR side), this would not quite cut it on the serialization side (because we can't tell CBOR to use the alias where JSON uses the name), so I'd keep that aspect out of scope in the first iteration.

My original problem is similar to #1402 but the proposed layering of enums tagged inside of untagged does not seem to work with yaml.

AFAICT this restriction is due to the now changed code.

It's very slooow to visit https://serde.rs/ in some countries. How about to publish it on https://docs.rs/ as a crate serde-docs or just a mod of the serde crate enabled by a feature docs?

I'm adding serde support to deflect, and I'd like to test that my implementations of Serialize over reflected values match the serializations produced by #[derive(Serialize)] on those values. The serde_test crate could support this use-case, if it provided a assert_ser_eq helper to check that two values serialize to the same sequence of Tokens.

I'm working on a serializer for a format I came across in the wild which represents all structs with explicit type names, and as a result I need access to the struct name during serialization. The format looks like this:

Parent {
    a = b

    Child {
        enabled = True
        type = SomeValue
        SomeValueField = 2
    }

    Child {
        enabled = False
        type = OtherValue
        OtherValueField = True
    }
}

The way I'd like to write my Rust structs is as follows:

#[derive(serde::Serialize)]
struct Parent {
    a: String,
    children: Vec<Child>,
}

#[derive(serde::Serialize)]
struct Child {
    enabled: bool,

    #[serde(flatten)]
    variant: ChildVariant,
}

#[derive(serde::Serialize)]
#[serde(tag = "type")]
enum ChildVariant {
    SomeValue {
        SomeValueField: u64,
    },
    OtherValue {
        OtherValueField: bool,
    },
}

However with #[serde(flatten)] present, serde_derive calls serialize_map directly when serializing Child, meaning that I don't have access to the struct name anymore and must use a fallback value — Child formats itself as:

AnonymousMap {
    enabled = True
    type = SomeValue
    SomeValueField = 2
}

The root of this behavior appears to be that structs with flattened members are serialized as if they were maps: https://github.com/serde-rs/serde/blob/2062a3c16d44a611558e79a6eaeb7ea1e718db9c/serde_derive/src/ser.rs#L301-L305

While this is understandable (as serialize_struct expects a list of fields, and we can't compute this statically if we're flattened with a HashMap), I don't think there's an a priori reason to lose access to the struct name in this case.

Perhaps we could add a Serializer::serialize_flattened_struct(self, name: &'static str, len: Option<usize>) -> SerializeFlatMap with a default impl pointing at Serializer::serialize_map? I'd be happy to PR this if you're amenable.