Seconding what @alexcrichton said, publishing breaking changes as a "patch" version breaks our published crate (which has a ^ dependency) pretty regularly. Semantic versioning exists to address this issue, and publishing breaking changes as a patch violates semver and makes useless the entire syntax for dependency version specification.

Correctly incrementing versions according to semver would allow the 275 crates that depend on Serde to synchronize their dependencies in a meaningful way, instead of discovering breaking changes when published crates suddenly break. When semver is followed correctly, a ^ dependency allows bugfixes that are compatible, and will not cause published crates to break. It may be difficult but it is at least correct and meaningful.

Except that anything goes before a project is 1.0 in SemVer, and Serde is very much pre-1.0, so if you are using SemVer's rules for your argument, it doesn't apply here.

I think the way it works now, though painful, is acceptable and correct. The pain we're feeling is because we're depending on a bleeding edge library for a bleeding edge feature of a young programming language. Monkeying around with SemVer at the library level is just using low quality band-aids in an attempt to treat the fundamental issue, which is that this is all built on top of an unstable compiler feature. No one is forced to use Serde if they need more stability. Our pain should just be a signal back to the Rust team about how important a long term solution for this is. And they are already aware. That's what the macro reform RFCs are for.

Ultimately, my concern is a practical one - I'd like to be reasonably sure that when I publish a crate, it won't be broken retroactively by a patch to one of its dependencies. I think serde is mature enough and widely enough adopted that it should make an effort to avoid that if it can.

It is true that "anything goes" pre-1.0.0 with SemVer, but that doesn't mean the idea behind it should be thrown out just because it technically can. The goal is to avoid dependency hell, and I think there's a perfectly reasonable middle ground between following post-1.0.0 semver to the letter and publishing large, breaking changes in a "patch" release.

@adimarco

Ultimately, my concern is a practical one - I'd like to be reasonably sure that when I publish a crate, it won't be broken retroactively by a patch to one of its dependencies

Then what is the problem with you doing what @dtolnay advised in the issue itself?

Where this is unacceptable, you could use an = dependency.

I think part of the disagreement here stems from the fact that I don't care about nightly builds at all - the crate I'm publishing is targeted at stable Rust.

I guess that's the root of it. For users of nightly these aren't breaking changes because they don't even use Syntex. It is the opposite - the nightly compiler broke their code, we fix it for them, and because there is no major version change they get the fix immediately.

That makes this is a decision about whether Serde caters toward nightly or stable users.

If nightly, the current versioning behavior is correct. Stable users occasionally get broken in exchange for having a share of the codegen fun.

If stable, we should just never bump Syntex versions and stop supporting nightly - no breakage ever.

The best of both worlds is that we keep the current behavior of not changing major versions on Syntex bumps, and stable users use an = dependency.

If there were an extra digit in semver (0.x.y.z) we could use a scheme where "z" is incremented on non-breaking changes, "y" is incremented on syntex bumps, and "x" is incremented on breaking changes. Nightly users would depend on ^0.x and stable users would depend on ^0.x.y. Does anyone know whether we can make this work?

The only difference from an = dependency is that stable users receive non-breaking changes...

@jmesmon - The problem with an = dependency, as @indiv0 points out in another discussion is:

Wouldn't that fix result in any crate which uses both rusoto and another crate depending on serde failing to compile unless the serde versions match exactly?

I guess that's what you want but that means that downstream crates can't lock the serde dependency to a common version between rusoto and other packages, unless it's exactly the one you specified.

But as @dtolnay points out above, the root question here is whether the priority is nightly builds (which are actually fixed by these updates) or stable (which has published code broken by them). As my crate targets stable Rust, I obviously fall in the latter camp, but I suspect the answer is going to be a priority of nightly.

I'll have to think on it, but requiring stable users to use an = dependency for the desired behavior seems reasonable to me at first glance. It would just need to be pointed out loudly in the crate documentation.

If there were an extra digit in semver (0.x.y.z) we could use a scheme where "z" is incremented on non-breaking changes, "y" is incremented on syntex bumps, and "x" is incremented on breaking changes. Nightly users would depend on ^0.x and stable users would depend on ^0.x.y. Does anyone know whether we can make this work?

@dtolnay - it looks like that might work - http://semver.org/#spec-item-9

Wouldn't that fix result in any crate which uses both rusoto and another crate depending on serde failing to compile unless the serde versions match exactly?

I would think additional major version changes only make this worse.

If there were an extra digit in semver (0.x.y.z) we could use a scheme where "z" is incremented on non-breaking changes, "y" is incremented on syntex bumps, and "x" is incremented on breaking changes. Nightly users would depend on ^0.x and stable users would depend on ^0.x.y. Does anyone know whether we can make this work?

@dtolnay - it looks like that might work - http://semver.org/#spec-item-9

Good find. If Cargo supports this, let's start doing "0.x.y-z".

Ah the awkward thing will be those are "pre-release" versions, so if Serde is on 0.7.8-0 then ^0.7.8 will not see it. I suspect this will cause a lot of confusion.

How about we bite the bullet and release the next breaking change as 1.0.0, then follow an x.y.z scheme where z is incremented on non-breaking changes, y is incremented on syntex bumps, and x is incremented on breaking changes? We don't need to complicate things to stay away from 1.0 - it means only what we say it does.

Whatever we decide, I filed #357 for me to remember to document it in the readme.

How about we bite the bullet and release the next breaking change as 1.0.0, then follow an x.y.z scheme where z is incremented on non-breaking changes, y is incremented on syntex bumps, and x is incremented on breaking changes?

Once 1.0 is reached, minor versions are not supposed to cause breaking changes, but the entire discussion revolves around syntax updates being a breaking change. Using this model would cause serde to deviate from semver significantly.

Right, but we discussed our version numbers are catered toward nightly, for which the minor version changes would not be breaks (they would un-break things). The proposal conforms to semver for nightly users, and allows stable users to take advantage of semantic information in the version number (although different from semver).

The proposed schemes so far are:

• 0.x.yz (current scheme)
  • Nightly users depend on ^0.x
  • Stable users depend on =0.x.yz
  • Pros:
    • incumbent
    • simplest
  • Cons:
    • stable doesn't receive non-breaking changes
• 0.x.y-z
  • Nightly users depend on ^0.x
  • Stable users depend on ^0.x.(y-1) (I think...?)
  • Pros:
    • semver
    • nightly and stable both receive non-breaking changes
  • Cons:
    • unconventional for Rust crates
    • the (y-1) is confusing
• x.y.z
  • Nightly users depend on ^x
  • Stable users depend on ~x.y
  • Pros:
    • nightly and stable both receive non-breaking changes
    • second-simplest
  • Cons:
    • not conformant with semver for stable users
    • tilde requirement is unusual
    • "serde is now 1.0" (whatever that means)
• 0.xy.z
  • Nightly and stable both depend on ^0.xy
  • Pros:
    • same dependency for both nightly and stable
    • conforms to semver for stable
  • Cons:
    • nightly users don't receive non-breaking changes after the first syntex bump
    • nightly users don't get un-broken for free
    • requires many more version bumps for nightly users

I vote for the current scheme, with second choice being x.y.z. Note that the latter is SemVer-compatible, because all of this is unstable.

Keep in mind downstream crates can also do >= 0.x.y, < 0.u.v, where u > x. (e.g. >= 0.31.0, < 0.33.0) and then bump the maximum version as they test it for breakages (unless they've changed it recently, this is what diesel-rs does).

Also I don't like the categorical separation of "nightly" and "stable" users. I write all my code on nightly, but test it on both nightly and stable. I never let builds break on master for stable, though.

Also I don't like the categorical separation of "nightly" and "stable" users.

I am using "nightly" to mean "users of the compiler plugin" and "stable" to mean "users of syntex."

Unfortunately Serde doesn't have any control over this categorical separation. We need the same serde-codegen code to compile against both Syntex and nightly rustc libsyntax. When libsyntax breaks serde-codegen, we fix serde-codegen (or quasi or aster) which requires committing the new libsyntax code into Syntex which turns into a breaking change for some Syntex users.

Essentially we end up with versions A and B of Serde where A doesn't compile on nightly and B doesn't compile on stable. Given a single Serde dependency, there would be no way for you to declare it in such a way that it consistently works on both.

I write all my code on nightly, but test it on both nightly and stable. I never let builds break on master for stable, though.

I guess maintain separate Cargo.toml and Cargo.lock files for the two. For the nightly one use ^0.x and for the stable one use =0.x.y (or as appropriate, based on what scheme we settle on).

There is probably some lift here that could be provided by tooling, if we are able to distill this into a Cargo feature request.

Can we make this work through features? Only bumping the syntex dep for the nightly feature?

@dtolnay

Thanks for opening this issue! I'll see if I can weigh in and perhaps help out a bit. The motivation for my comment was that I have a few things here and there which are using serde, but I personally try to avoid nightly like the plague so they're all on stable Rust. From my perspective it should be the case that usage of stable serde is like Rust, it's stable and you have a nice experience beyond the ergonomics.

I think it makes sense for nightly users to continuously have to upgrade serde in order to literally fix compatibility with upstream libsyntax, but one major release of serde to me needs to pin to one version of syntex. This basically means that the entire 0.7 release series on stable Rust has only one version of syntex_syntax and can parse basically one revision of the language. This in turn means that all active usage of syntex_syntax in codegen scripts (e.g. when you register generators) will all work together.

I disagree that the solution here is = dependencies on stable and ^ dependencies on nightly. In theory we want the usage of serde on stable to be idiomatic, right? Additionally, as a stable serde user, I'd also like to pick up bug fixes without having to perform major upgrades. Unfortunately it's also a violation of Cargo's interpretation of semver for 0.a.b to break compatibility with 0.a.(b-1), and this ends up causing a lot of problems in the ecosystem as that's the idiomatic way to depend on things.

The codegen crates understand if they're using nightly Rust, right? In theory they could have two code paths, one for the nightly version of syntax and one for the stable version of syntex_syntax I would think. This would be a pain to maintain but would probably be less work overall than trying to drag both the nightly and stable ecosystems forward.

I guess basically what I'm getting at is that if we really want to promote serde we can't hamstring the stable usage. There either needs to be a way to solve this at the heart (e.g. don't make syntex_syntax a public dependency, not sure if this is possible), or I think it's worth going to great lengths to ensure that stable serde has the most ergonomic experience possible.

Just to clarify, when people are saying "major" version in this thread, they are actually meaning the minor version according to SemVer, right? i.e. MAJOR.MINOR.PATCH

To me this is less so about strictly following the semver spec and instead delivering stability as a feature of serde itself. This is then communicated in Cargo through semver versions, but the crux of it is what the stability story of serde is.

Cargo today considers 0.7.1 semver compatible with 0.7.2, despite what the spec says otherwise. This means that the practical expectation of the Rust community is that the two are compatible and only bug fixes or new additions were made in 0.7.2.

I think when I personally refer to a major version upgrade what I really mean is any semver-incompatible upgrade, not necessarily specifically the major version in the semver number.

It's worth considering (as has been hinted at in previous comments) that pure semver (or rust's "practical expectation" semver) could be insufficient for serde to provide a stability guarantee while still achieving it's development goals (ie: allow use with nightly rust to obtain actual line numbers, or stable rust for deployment).

Perhaps we should consider what extensions to semver would be needed for practical use.

I added the naive implementation of Alex's proposal to the chart in #356 (comment) as 0.xy.z (as distinct from 0.x.yz). That means increment the middle number both for ordinary breaking changes and for Syntex bumps which are breaking changes only for stable.

I filed #358 to investigate whether we can do away with the public dependency on Syntex. That way Syntex bumps would not be breaking changes for anybody (except in the rare case that Rust itself breaks backward compatibility).

This would solve all problems right?

@dtolnay yeah if syntex is not a public dependency that'd definitely solve the problem! (not sure if that's possible though)

Note that my suggestion is to have serde 0.x.* track essentially just be bug fixes, and the way it would work is that when the nightly feature is enabled on serde-codegen it links to libsyntax and exports those APIs (issuing bug fix releases to update this dependency when it breaks) and when the nightly feature is not enabled it just tracks one stable version of syntex_syntax. This should mean that the codegen crate itself may get complicated as it has two divergent code paths, but the benefit is stability which is quite nice!

PR #362 allows using serde_codegen on stable without exposing the Syntex dependency.

Serde 0.7.9 adds an alternative way of writing build.rs that avoids exposing Serde's dependency on Syntex.

// build.rs
pub fn main() {
    let out_dir = env::var_os("OUT_DIR").unwrap();
    let src = Path::new("src/main.rs.in");
    let dst = Path::new(&out_dir).join("main.rs");

    // no more public Syntex dependency
    serde_codegen::expand(&src, &dst).unwrap();
}

For stable users using serde_codegen::expand, Serde bumping its Syntex dependency will no longer be a breaking change so we will plan to stick with the current versioning scheme. We can remove serde_codegen::register in 0.8.0 unless someone has a reason to keep it - that discussion can continue on #358.

Thanks everyone for the ideas and feedback.

@dtolnay: this is great! nice job. For 0.8, we should just get rid of serde_codegen::register if it'll fix these problems. Having users have to do multiple passes doesn't seem to be that bad of an option.