Hi @Byron! Allowing the indent character is a great idea. Filtering out the nulls is a little more tricky due to the streaming nature of serialization. In order to do this, we'd have to delay serializing the object key until we know which value we're serializing. Ideally without having to buffer the serialized key.

Hm. If we added a key_buf: String to Serializer, we could do this with:

• serializes keys to key_buf, set serializing_map_elt to true.
• In all visit_* methods but visit_unit, add a check if serializing_map_elt is true, serialize the key first.
• In visit_unit, if the skip_null is true, skip the value, otherwise serialize the key and null.

It's a little ugly, and would probably slow down the common case. I wonder if there's a better way to do this...

Hi @erickt ! Judging from my inability to follow your statements starting with key_buf: String, I fear I am currently not of much help in this conversation. But I agree that something like skipping values should be easy - I guess a look-ahead is needed to do that (i.e. don't serialize key unless value is not Null). Ideally, things like this can be added to the system without slowing down the common case, I totally agree. Maybe it's possible to use some sort of monomorphism to have a high-speed (i.e. the current) version of the serialization, and one that supports all sorts of flags.
After all, for end-user serialization it doesn't matter if we make 400MB/s, or just 100Mb/s :).

In any case, I am looking forward to the progress on this issue, as it will be very relevant to me when the advanced CLIs for Google Services are due for release.

@Byron: Hehe :) Yeah sometimes my thinking out loud isn't particularly clear. The real challenge here is how to enable the serializer to do some lookahead. I had thought that this was only necessary for deserializers.

@erickt I have updated the issue to be more general, and added a more prominent example to show the importance of the 'ignore empty' feature. Interesting enough, some remote servers like the ones for Google Drive, seem to ignore null values when uploading files. I will keep trying to find out exactly what happens. NOTE: Didn't work, after all, thus server error must be caused by null fields.

@erickt Thanks for your help ! This is a major issue for me and I can help if you give me some hints on how to solve this. It's about time I do something for serde :).

@erickt There has been no feedback on this matter for 8 days now, and with the 15th of May approaching, I feel like I have to step in and try to implement what I need myself and do so in an unguided fashion. This worries me as I am not at all familiar with the code-base. Additionally I am not comfortable submitting PRs anymore as these were usually ignored for quite a while until they have been superseded by code coming in from the maintainers.
The latter is something I usually very much appreciate, but it's the lack of somewhat timely feedback that makes contributing hard and very unrewarding, putting me at risk wasting my time if it's not for the experience points earned in the process ;).
Therefore, if you read this, please consider letting me know what you think about this just to be sure you either direct me down the path you want for serde, or let me know when you think this feature will be implemented anyway.
As I said before, it's blocking a primary ability of my CLIs (i.e. uploading files to youtube), which makes sitting it out not an option I have.
Thanks for your consideration

During lunch I cleared my mind and came to realize that attempting the implement this feature in serde has low chances of success, and that it will be more economic to just implement a client-side filter that operates on the generated string. That way, I can have my CLI remove the unwanted NULL values and just use corresponding serde features as they arrive.

Have a look at the relevant function for json: http://serde-rs.github.io/serde/src/serde/json/ser.rs.html#248-258

One could simply call value.serialize(empty_tester) where empty_tester is some serializer that doesn't to anything but set a boolean to true if it encounters a null, none, empty sequence or empty struct (maybe even configurable to a subset of these). Since the serialize function doesn't consume its self argument, everything should be fine.

Thanks for the information !
By now I have already solved that problem, in quite a different way though.

@Byron: hey there! I've been very distracted with trying to get serde_macros working with stable, so I haven't had much time for serde proper. Another option too which I thought of earlier is that you could serialize into a json::Value type, and then from there you can recursively filter out all the null fields before you serialize it to a string. The only limitation is that it might cause problems for the custom enum encoding, but as long as you aren't using them it shouldn't be a problem. So something like this:

use serde::json::value::{self, Value};

fn remove_nulls(value: &mut Value) {
    match value {
        Value::Object(ref mut map) => {
            let to_remove = Vec::new();

            for (key, value) in map {
                if value.is_null() {
                    to_remove.push(key.clone());
                } else {
                    remove_nulls(value);
                }
            }

            for key in to_remove {
                map.delete(key);
            }
        }
        _ => {}
    }
}

#[derive(Serialize, Deserialize)]
struct Point { x: u32, y: u32, name: Option<String> }

let point = Point { x: 0, y: 0, name: None };
let mut value = serde::json::value::to_value(&point);
remove_nulls(&mut value);
println!("{}", serde::json::to_string(&value));

I haven't tried running it, but something like that should probably work.

@erickt Thanks so much for this hint ! It solved my problem right away. Doing that could be a general workaround any kind of filtering one would wish, or could be used to implement custom Serializers which offer more options when pretty-printing.

This can be worked around today using serde's new attributes. Define a trait and blanket impl for Option<T>:

trait ShouldSkip {
    fn should_skip(&self) -> bool;
}

impl<T> ShouldSkip for Option<T> {
    fn should_skip(&self) -> bool {
        self.is_none()
    }
}

Then in the struct definition

struct Thing {
    /// Optional string where null is undesirable
    #[serde(skip_serializing_if="ShouldSkip::should_skip")]
    field: Option<String>
}

Even so, it would be nice to have some better default behavior for the format.

You don't need the ShouldSkip trait:

struct Thing {
    #[serde(skip_serializing_if="Option::is_none")]
    field: Option<String>
}

Closing this because both solutions are sufficiently simple and I think the current behavior makes sense as the default.

Why is the current behavior a sensible default, if I may ask? I haven't seen many consumers expecting null. Most I've used are happy with skipping the nulled fields (as it actually saves bandwidth).

I think the current default is at least not confusing. Pretty sure if nulls were skipped by default it would confuse quite a few people.

Pretty sure it wouldn't. Which framework does output null by default, and doesn't allow a global overriding? That is short to say that there should be some sort of API with this form:

let encoder = serde_json::Encoder::new_with_some_options(x,y,z);
let str = try!(encoder.to_string(&item));

where in this case options would look like ``IgnoreNone` or something like it.
It also works the other way round, where you want to change default behaviour for handling unknown fields during Decoding, for instance.

At the moment it's very unreadable. This is a small snippet out of a large, hierarchical structure I'm using.

    ...
    #[serde(skip_serializing_if="Option::is_none")]
    pub DayHoursStart: Option<Int>,
    #[serde(skip_serializing_if="Option::is_none")]
    pub DayHours: Option<Int>,
    #[serde(skip_serializing_if="Option::is_none")]
    pub NightHoursStart: Option<Int>,
    #[serde(skip_serializing_if="Option::is_none")]
    pub NightHours: Option<Int>,
    #[serde(skip_serializing_if="Option::is_none")]
    pub SpeedOfTime: Option<Vec<Float>>,
    #[serde(skip_serializing_if="Option::is_none")]
    pub TrackInfrastructure: Option<Vec<Int>>,
    #[serde(skip_serializing_if="Option::is_none")]
    pub DryCoverage: Option<Vec<String>>,
    #[serde(skip_serializing_if="Option::is_none")]
    pub WetCoverage: Option<Vec<String>>,
    #[serde(skip_serializing_if="Option::is_none")]
    pub MinimumPlayers: Option<Int>,
    #[serde(skip_serializing_if="Option::is_none")]
    pub MaximumPlayers: Option<Int>,
    #[serde(skip_serializing_if="Option::is_none")]
    pub Interval: Option<Int>,
    #[serde(skip_serializing_if="Option::is_none")]
    pub NextSlotOffsetSeconds: Option<Int>,
    #[serde(skip_serializing_if="Option::is_none")]
    pub LockoutTime: Option<Int>,
    #[serde(skip_serializing_if="Option::is_none")]
    pub MatchedIntoTimeout: Option<Int>,
    #[serde(skip_serializing_if="Option::is_none")]
    pub EndOfRace: Option<Int>,
    #[serde(skip_serializing_if="Option::is_none")]
    pub VehicleSelect: Option<Int>,
    #[serde(skip_serializing_if="Option::is_none")]
    pub TimeTrialDuration: Option<Int>,
    #[serde(skip_serializing_if="Option::is_none")]
    pub DryToWet: Option<Bool>,
    #[serde(skip_serializing_if="Option::is_none")]
    pub Events: Option<Vec<Event>>,
    ...

@norru i am not sure what serialization frameworks do but a naive serialization of a data structure to JSON in all environments I have seen emit a null. That includes dynamic languages as well as JSON libraries in C and C++. Especially because of web apis that accept arbitrary JSON without schema it's important that null is emitted as otherwise the entire key is missing which then would require the consumer of that data blob to be aware of the schema.

It's the other way round. I have never used an API which requires null, omitting the field seems to be the common case. On the contrary, some APIs I've used in the past won't accept "null". I generally remove nulls whenever possible for 1) readability 2) message size 3) Any Other Business I don't feel the need to justify here. I don't know if those APIs do it by design or by accident, and you may well dismiss those APIs as "broken" (which is not necessarily so), but, hey! It's a wild wild world out there and that's life.

If "send null" is a default for Option let's at least have a mechanism to override it once and for all instead of for Every. Single. Field. The code above is not fun to read.

+1 for more configurability

@norru:
While I agree with @mitsuhiko that APIs returning JSON data should not skip null fields by default, I have another use case in which it is sensible to skip them. JSON-LD schema.org fields are often optional and in a web context a smaller JSON-LD output is preferable over lots of null fields.

@norru:
I am reasonably satisfied with the following format:

#[allow(non_snake_case)]
#[derive(Clone, Serialize)]
pub struct LooseJson {
    #[serde(skip_serializing_if="Option::is_none")] pub DayHoursStart: Option<Int>,
    #[serde(skip_serializing_if="Option::is_none")] pub DayHours: Option<Int>,
    #[serde(skip_serializing_if="Option::is_none")] pub NightHoursStart: Option<Int>,
}

With the quality of syntax highlighting in IntelliJ IDEA this is even more readable.

Thanks for the discussion everybody. Our hands are sort of tied here by non-self-describing binary formats. If Serde were for JSON only, I would agree that skip none may make sense by default. However, this is overruled by the desire to have the default generated code work in all data formats supported by Serde.

How would JSON folks feel about a container attribute to enable this behavior for all Option fields of a particular struct?

#[derive(Serialize)]
#[serde(skip_serializing_none)]
struct Norru {
    pub DayHoursStart: Option<Int>,
    pub DayHours: Option<Int>,
    pub NightHoursStart: Option<Int>,
    pub NightHours: Option<Int>,
    pub SpeedOfTime: Option<Vec<Float>>,
    pub TrackInfrastructure: Option<Vec<Int>>,
    pub DryCoverage: Option<Vec<String>>,
    pub WetCoverage: Option<Vec<String>>,
    pub MinimumPlayers: Option<Int>,
    pub MaximumPlayers: Option<Int>,
    pub Interval: Option<Int>,
    pub NextSlotOffsetSeconds: Option<Int>,
    pub LockoutTime: Option<Int>,
    pub MatchedIntoTimeout: Option<Int>,
    pub EndOfRace: Option<Int>,
    pub VehicleSelect: Option<Int>,
    pub TimeTrialDuration: Option<Int>,
    pub DryToWet: Option<Bool>,
    pub Events: Option<Vec<Event>>,
}

What about making it possible to apply #[serde(skip_serializing_if="Option::is_none")] to an entire struct, and optionally override it for particular fields (with a skip_serializing or skip_serializing_if annotation, or some variant thereof)? That seems more general and a good idea regardless, since it makes it more ergonomic to do this sort of thing. Though perhaps something like skip_serializing_field_if would make more sense... not sure on the naming.

@Mark-Simulacrum: what would happen when not all fields have Option type?
@dtolnay: @Mark-Simulacrum's suggestion skip_serializing_field_if with some more generic filter function appeals to me more than skip_serializing_none.

@sanmai-NL formatting-only solution doesn't work for me, looks very cluttered. Yes, it's marginally better, but: how did you set up rustfmt so it only keeps attributes in line for the serde structs? Sure you don't format your code by hand?

On the other hand, I like the per-struct attribute, which can be overridden per-field (specifying skip_serializing_if="MyStruct::never" or something that produces the same outcome).

@Mark-Simulacrum: what would happen when not all fields have Option type?

If not all fields have the Option type, then individual fields can be overridden with other functions. If they're not, then it would be a compiler error, which though potentially hard to make it have a good error message...

So there is still NOT a macro to cover a whole struct with non-option and option attributes? Do I really have to add #[serde(skip_serializing_if = "Option::is_none")] before every option key?