Here I created a flat simple token based output, this can be used for syntax colouring and better change diffs. In some cases the output is not very nice yet, especially generics. And sometimes the ID comes peeking out, in structs in enum variants. The code organisation/API is of course open for discussion.

And build rustdoc JSON inside of cargo test instead of using pre-built rustdoc JSON that we version in git. This has several advantages:

• It becomes much easier to add regression tests since we have a test-crate we can add a public item to

• The git repo becomes smaller since it is not filled with large amounts of raw rustdoc JSON.

• We remove the need to be in possession of bat and syntect source code (to update the rustdoc JSON)

• We improve code coverage of the library code by quite a bit in relevant files. See below:

Code coverage sample before this commit:

Filename                         Regions Covered
--------------------------------------------------
intermediate_public_item.rs      78.53%
item_iterator.rs                 92.86%

after this commit:

Filename                         Regions Covered
--------------------------------------------------
intermediate_public_item.rs      84.29%
item_iterator.rs                 94.29%

The down-side is that we lose the benefit of testing against actual, non-trivial, real-world public APIs. But I think we can compensate for that.

I have become more and more convinced that it would be beneficial to turn public-api and cargo-public-api into a mono-repo. Only on the repo level though. It still make a lot of sense for public-api and cargo-public-api to remain two distinct crates.

So all code would live in https://github.com/Enselic/cargo-public-api. And long-term possibly in https://github.com/cargo-public-api/cargo-public-api.

Pros

Easier development flow

There would only be a single project and git repo to manage in VS Code (or your IDE of choice).

And the default setup can be that cargo-public-api uses the local version of public-api. Right now that is an extra step, which makes development slightly less convenient than it could be.

Ability to re-use code for testing

The public-api repo has a very useful comprehensive_api test crate, for example. cargo-public-api would get access to that crate too.

Less duplicate efforts to make commits

There are many examples in the past where I essentially make the same commit to both projects, but separately. It would have been a lot more convenient to only have to make a single commit. TODO: Add examples

Easier to sync releases

Whenever we make a public-api release, we practically always also want to make a cargo-public-api release. Because we want to set the public-api dependency of cargo-public-api to the latest version, to ensure that users gets the latest and greatest functionality.

Having everything in the same repo makes it easier to sync releases.

It is however very possible that we in the future want to make only cargo-public-api releases, without bumping public-api. But a mono-repo does not prevent us from doing that.

Cons

I can't really think of any big cons. Losing some github stars, maybe? The project is still in the very early stages, so I think we can tolerate that :) Of course; the public-api repo will point towards the cargo-public-api repo.

Current status

This is still an idea. Before a final decision, we at the very least need a prototype commit to see how a mono-repo would look like. That is the next step for me.

@douweschulte Naturally, I would also love your feedback on this. (But as usual, there is no stress or urgency. I think this will be the last time I explicitly mention that :)

So that we don't get an error when doing something like this:

% RUSTDOCFLAGS='-Z unstable-options --output-format json' cargo +nightly doc  --manifest-path tests/crates/comprehensive_api/Cargo.toml --lib --no-deps
% ./target/debug/public-api tests/crates/comprehensive_api/target/doc/comprehensive_api.json | head -n 1
pub async fn comprehensive_api::functions::async_fn() -> impl Future<Output = ()>
Error: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }

Comparing the rustdoc HTML output of https://github.com/rust-lang/rust/blob/master/src/test/rustdoc/higher-ranked-trait-bounds.rs with the output of public-api, there are differences. To see them, first generate rustdoc HTML output for that file:

% ~/.rustup/toolchains/nightly-2022-03-31-aarch64-apple-darwin/bin/rustdoc ~/src/rust/src/test/rustdoc/higher-ranked-trait-bounds.rs -Z unstable-options --output-format html -o /tmp/hrtb

and then look at e.g. open /tmp/hrtb/foo/fn.test3.html which looks like this:

pub fn test3<F>() 
where
    F: for<'a, 'b> Fn(&'a u8, &'b u8), 

and then generate output with public-api:

% ~/.rustup/toolchains/nightly-2022-03-31-aarch64-apple-darwin/bin/rustdoc ~/src/rust/src/test/rustdoc/higher-ranked-trait-bounds.rs -Z unstable-options --output-format json -o /tmp/hrtb
% public-api /tmp/hrtb/foo.json 
pub fn foo::Foo::bar<T>() where T: Trait<'a>
pub fn foo::test1<T>() where &'a T: Iterator
pub fn foo::test2<T>() where &'a T: Trait<'b>
pub fn foo::test3<F>() where F: Fn(&'a u8, &'b u8)<'a, 'b>
pub mod foo
pub struct field foo::Bar::bar: &'a Trait<'b> + Unpin
pub struct field foo::Bar::baz: &'a Unpin + Trait<'b><'b>
pub struct field foo::Foo::some_func: fn(&'c i32) -> i32
pub struct field foo::Foo::some_trait: &'a Trait<'b>
pub struct foo::Bar<'a>
pub struct foo::Foo<'a>
pub trait foo::B<'x>
pub trait foo::Trait<'x>

Normalizing whitespace and linewlines and aligning, we have this vs that:

pub fn      test3<F>() where F: for<'a, 'b> Fn(&'a u8, &'b u8), 
pub fn foo::test3<F>() where F:             Fn(&'a u8, &'b u8)<'a, 'b>

There is a FIXME related to HRTB in rust-lang/rust/src/librustdoc/json/conversions.rs, so not sure we can fix them all, but it looks like we can fix the particular example I brought up.

For reference HRTB, didn't work properly in rustdoc HTML either for quite a while (see https://github.com/rust-lang/rust/issues/78482), but was fixed quite recently, which also added the FIXME (see https://github.com/rust-lang/rust/pull/84814).

To fix HRTBs in rustdoc HTML, the DynTrait object was introduced in the above PR. So fixing all HRTB problems might be related to fixing https://github.com/Enselic/cargo-public-api/issues/40, in case it requires upstream changes.

See discussion in https://github.com/Enselic/cargo-public-items/issues/17

But let's wait with renaming public_items until at least https://github.com/Enselic/public_items/pull/53 has been merged, because otherwise there will be massive amounts of merge conflicts

Should be pretty easy to add because the JSON for a function has the necessary info:

                "header": {
                    "const": false,
                    "unsafe": false,
                    "async": true,
                    "abi": "Rust"
                }

I am a big fan of keeping things as simple as possible. This makes it easier for others to use and contribute to this library.

One thing that IMHO would simplify things is to remove TokenStream from both the public API and the implementation. This significantly simplifies the API, without making the implementation more unergonomic to work with. It arguably becomes easier to work with since everyone knows how to work with Vecs.

I also took this opportunity to hide PublicItem::tokens behind a getter, to make the API more future-proof.

Here is how this commit impacts the public API:

% cargo public-api --diff-git-checkouts origin/main remove-tokenstream
Removed items from the public API
=================================
-pub fn public_api::tokens::TokenStream::clone(&self) -> TokenStream
-pub fn public_api::tokens::TokenStream::cmp(&self, other: &TokenStream) -> $crate::cmp::Ordering
-pub fn public_api::tokens::TokenStream::default() -> TokenStream
-pub fn public_api::tokens::TokenStream::eq(&self, other: &TokenStream) -> bool
-pub fn public_api::tokens::TokenStream::extend(&mut self, tokens: impl Into<Self>)
-pub fn public_api::tokens::TokenStream::fmt(&self, f: &mut $crate::fmt::Formatter<'_>) -> $crate::fmt::Result
-pub fn public_api::tokens::TokenStream::fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result
-pub fn public_api::tokens::TokenStream::from(token: Token) -> Self
-pub fn public_api::tokens::TokenStream::from(tokens: &[Token]) -> Self
-pub fn public_api::tokens::TokenStream::from(tokens: Vec<Token>) -> Self
-pub fn public_api::tokens::TokenStream::is_empty(&self) -> bool
-pub fn public_api::tokens::TokenStream::len(&self) -> usize
-pub fn public_api::tokens::TokenStream::ne(&self, other: &TokenStream) -> bool
-pub fn public_api::tokens::TokenStream::partial_cmp(&self, other: &TokenStream) -> $crate::option::Option<$crate::cmp::Ordering>
-pub fn public_api::tokens::TokenStream::tokens(&self) -> impl Iterator<Item = &Token> + '_
-pub fn public_api::tokens::TokenStream::tokens_len(&self) -> usize
-pub struct field public_api::PublicItem::tokens: TokenStream
-pub struct field public_api::tokens::TokenStream::tokens: Vec<Token>
-pub struct public_api::tokens::TokenStream

Changed items in the public API
===============================
(none)

Added items to the public API
=============================
+pub fn public_api::PublicItem::tokens(&self) -> impl Iterator<Item = &Token>

Here is how this impacts cargo-public-api: https://github.com/Enselic/cargo-public-api/pull/27

Step-by-step

1. cargo doc --manifest-path ./tests/crates/comprehensive_api_proc_macro/Cargo.toml && open ./target/doc/comprehensive_api_proc_macro/attr.simple_proc_macro_attribute.html
2. Notice that the item is represented as #[simple_proc_macro_attribute]
3. Look in ./tests/expected_output/comprehensive_api_proc_macro.txt

Expected

There is a line

pub proc macro comprehensive_api_proc_macro::#[simple_proc_macro_attribute]

Actual

There is a line

pub proc macro comprehensive_api_proc_macro::simple_proc_macro_attribute!

because that is how the library represents that item. We should represent it the same as cargo doc.

See https://github.com/aDotInTheVoid/rustdoc-types/issues/6

It is beneficial to wait with bumping rustdoc-types to v0.8.0 until we implement support for GATs (or until we need to bump for other reasons), because a reader based on rustdoc-types v0.7.0 can still read output produced by rustdoc-types v0.8.0 (nightly-2022-03-04 and onwards), since the only difference is the addition of JSON fields.

It might also make sense to wait with supporting this until GAT lands in stable.

It should be possible to opt-in to omit blanket implementations. It is usually not interesting be shown e.g. the below items for every struct in a crate:

pub fn public_items::PublicItem::borrow(&self) -> &T
pub fn public_items::PublicItem::borrow_mut(&mut self) -> &mut T
pub fn public_items::PublicItem::from(t: T) -> T
pub fn public_items::PublicItem::into(self) -> U
pub fn public_items::PublicItem::try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>
pub fn public_items::PublicItem::try_into(self) -> Result<U, <U as TryFrom<T>>::Error>
pub fn public_items::PublicItem::type_id(&self) -> TypeId

Seems like we can use Impl::blanket_impl to know if an impl is a blanket impl.