This is necessary so that these nodes exist on the stack when we serialize the children nodes and may need the parent for context. Addresses servo/servo#14975

Opening this PR for feedback/review

~~This does work and solves the above bug. I'm not sure if this is the best implementation, as there's a bit of duplicated code. I thought about adding a boolean flag to start_elem that would determine whether the writes should be performed. But then I thought maybe when not writing, it's best not to call a function returning io::Result<()>?~~

Also not 100% sure if the xml implementation is the right thing to do

Anyway, please let me know of any improvements/changes to make. Thanks :)

This change is 

This large commit concerns #43. It adds everything that is currently defined in spec.

Included inside:

• Basic machinery - Tokenizer/States/Tree builder/etc. that implements following spec: Ygg01/xml5_draft
• Tests, which reference: Ygg01/xml5lib-test
• Some basic examples

Missing:

• Namespace resolution
• Doctype handling as per Ygg01/xml5_draft#2
• Entity replacement (this is mostly C'n'P from html5ever character tokenizer), so all html5 entities are valid (e.g. ‐ Á etc.)
• Unification of xml5 and html5 syntax
• C API

(I'm aware 0.25.1 is the latest version, if this is fixed on master let me know and I can close this. 0.25 removed RcDom so it would take some set up on my end to switch)

Docs.rs uses html5ever to parse HTML files generated by rustdoc. We recently updated from 0.22 to 0.24 and docs.rs started using so much CPU that all requests turned into 500 errors. The 0.24 version was deployed from 20:04 to 20:20 in the following diagram:

See also https://github.com/rust-lang/docs.rs/pull/669 (and many other PRs linked from there, we had to bisect in production).

Do you know what could have caused the slowdown? I saw that no new dependencies were introduced so it must have been something internal to html5ever. I can try to reproduce this on a single file if you're not sure, but right now it's preventing us from upgrading.

Ok, this is a large one.

Fixes #266, fixes #261, fixes #210.

It moves html5ever into separate folder, renames html5ever macros markup5ever and stores common code there.

Here is short summary of what I know is and isn't done.

• [x] Make every crate in the repo use a single workspace
• [x] Make sure Travis-CI is running every test
• [x] Rename the the html5ever_atoms crate to markup5ever and update html5ever and xml5ever to use it.
• [x] Increment version numbers
• [x] Make it so that users of either html5ever or xml5ever don’t need to have an explicit dependency to markup5ever
• [x] Export QualName #210
• [x] let markup5ever generate entities.json #261
• [x] Move TokenSink to markup5ever
• [x] Move TreeSink to markup5ever
• [x] Move BufferQueue to markup5ever
• [x] Move SmallCharSet to markup5ever
• [x] Deal with driver.rs

This change is 

We're willing to use html5ever's tokenizer for streaming transformation of HTML (that is, replacing specific attributes on specific tags only - e.g. src in img or src in script or href in link). For that, it would be useful to have lazy mode that would not force parser to parse attributes and content that we don't care about and then force us to serialize them back properly, thus 1) preserve formatting as closely to original as possible and 2) improve performance by avoiding lots of unnecessary work.

Would be that possible to implement? I'm pretty new to Rust, but I would be happy to take an attempt to do this myself if given some basic pointers and directions on where to look / what might need to be changed.

Thanks.

P.S. Theoretically, Servo could benefit from lazy parsing, too, if decoding attributes would happen only on first access as it might safely ignore many attributes for rendering and call such lazy decoder only when actually accessing them e.g. via DOM API (whether they are data-* attributes or any other that renderer doesn't care about and only JS code might do).

I propose the change because I had this problem when I did cargo test:

[....]$ RUST_BACKTRACE=1 cargo test
     Running target/debug/html5ever-ebbb862bf0664388

running 20 tests
....................
test result: ok. 20 passed; 0 failed; 0 ignored; 0 measured

     Running target/debug/serializer-bd6e4b6794d0e313

running 40 tests
........................................
test result: ok. 40 passed; 0 failed; 0 ignored; 0 measured

     Running target/debug/tokenizer-9fee19ffe67616b5
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { repr: Os { code: 2, message: "No such file or directory" } }', ../src/libcore/result.rs:788
stack backtrace:
   1:        0x10f9e54bb - std::sys::backtrace::tracing::imp::write::h46e546df6e4e4fe6
   2:        0x10f9e775a - std::panicking::default_hook::_$u7b$$u7b$closure$u7d$$u7d$::h077deeda8b799591
   3:        0x10f9e738a - std::panicking::default_hook::heb8b6fd640571a4f
   4:        0x10f9dc178 - std::panicking::rust_panic_with_hook::hd7b83626099d3416
   5:        0x10f9e7d36 - std::panicking::begin_panic::h941ea76fc945d925
   6:        0x10f9dcdb8 - std::panicking::begin_panic_fmt::h30280d4dd3f149f5
   7:        0x10f9e798f - rust_begin_unwind
   8:        0x10fa0d570 - core::panicking::panic_fmt::h2d3cc8234dde51b4
   9:        0x10f903469 - core::result::unwrap_failed::h77e8f7274d900d66
  10:        0x10f902d9a - _<core..result..Result<T, E>>::unwrap::h832a73c540c5d590
  11:        0x10f902782 - tokenizer::foreach_html5lib_test::foreach_html5lib_test::h2ce9421430543c73
  12:        0x10f901f8d - tokenizer::tests::h3cbd3447622b6aa9
  13:        0x10f905c2f - tokenizer::main::h4149a347b29284d8
  14:        0x10f9e6f4d - std::panicking::try::call::hca715a47aa047c49
  15:        0x10f9e7e0b - __rust_try
  16:        0x10f9e7da5 - __rust_maybe_catch_panic
  17:        0x10f9e6d71 - std::rt::lang_start::h162055cb2e4b9fe7
  18:        0x10f907399 - main
error: test failed

This change is

When the "unstable" Cargo feature is given, html5ever’s build script will use html5ever_macros as a library to generate or update src/tree_builder/rules.expanded.rs, which is part of the source tree.

(This feature also enables the corresponding features of tendril and string-cache.)

html5ever_macros is not a plugin anymore, but it still uses libsyntax and still behaves a lot like a syntax extension.

When "unstable" is not given, the build script checks that rules.expanded.rs is up to date (based on a hash of rules.rs) and fails if it’s not.

Tests also run on stable Rust on Travis-CI, but unfortunately most of them are disabled as they dynamically generated using internals of the test crate.

r? @Manishearth

Fixes #53 Closes #151

I was reading through the code and noticed that the formatting was different from what I understand to be the currently agreed upon convention. So I ran rustfmt on the entire repo. I hope that's welcome, it made reading the code more pleasant. Tests run as previously.

phf 0.8 depends on rand 0.7 This causes rand to be included twice as most other projects have updated to rand 0.8.

phf 0.9 uses rand 0.8 so update to avoid the duplicate dependency.

While running valgrind on my program, I found this

==28112== Conditional jump or move depends on uninitialised value(s)
==28112==    at 0xB28252: html5ever::tree_builder::data::doctype_error_and_quirks (data.rs:157)
==28112==    by 0x682617: <html5ever::tree_builder::TreeBuilder<Handle,Sink> as html5ever::tokenizer::interface::TokenSink>::process_token (mod.rs:472)
==28112==    by 0x6D0AA8: _ZN9html5ever9tokenizer21Tokenizer$LT$Sink$GT$13process_token17ha27fb7b70a53cc47E.llvm.10445680363282595522 (mod.rs:237)
==28112==    by 0x6D926A: html5ever::tokenizer::Tokenizer<Sink>::step (mod.rs:0)
==28112==    by 0x6D390A: _ZN9html5ever9tokenizer21Tokenizer$LT$Sink$GT$3run17h29926122ee9878e2E.llvm.10445680363282595522 (mod.rs:369)
==28112==    by 0x62196F: feed<html5ever::tree_builder::TreeBuilder<kuchiki::tree::NodeRef, kuchiki::parser::Sink>> (mod.rs:224)
==28112==    by 0x62196F: <html5ever::driver::Parser<Sink> as tendril::stream::TendrilSink<tendril::fmt::UTF8>>::process (driver.rs:109)
==28112==    by 0x5E9504: process<html5ever::driver::Parser<kuchiki::parser::Sink>,tendril::tendril::NonAtomic> (stream.rs:180)
==28112==    by 0x5E9504: one<tendril::stream::Utf8LossyDecoder<html5ever::driver::Parser<kuchiki::parser::Sink>, tendril::tendril::NonAtomic>,tendril::fmt::Bytes,tendril::tendril::NonAtomic,&[u8]> (stream.rs:48)
==28112==    by 0x5E9504: cratesfyi::utils::html::extract_head_and_body (html.rs:28)
==28112==    by 0x76F641: cratesfyi::web::rustdoc::rustdoc_html_server_handler (rustdoc.rs:312)
==28112==    by 0x751A1B: call<fn(&mut iron::request::Request) -> core::result::Result<iron::response::Response, iron::error::IronError>,(&mut iron::request::Request)> (function.rs:72)
==28112==    by 0x751A1B: <F as iron::middleware::Handler>::handle (mod.rs:312)
==28112==    by 0xA3ED50: <alloc::boxed::Box<dyn iron::middleware::Handler> as iron::middleware::Handler>::handle (mod.rs:318)
==28112==    by 0x7009C6: <cratesfyi::web::metrics::RequestRecorder as iron::middleware::Handler>::handle (metrics.rs:185)
==28112==    by 0xA3ED50: <alloc::boxed::Box<dyn iron::middleware::Handler> as iron::middleware::Handler>::handle (mod.rs:318)
==28112==  Uninitialised value was created by a stack allocation
==28112==    at 0xB27BC0: html5ever::tree_builder::data::doctype_error_and_quirks (data.rs:91)

gdb then pointed out the source of the problem, which is this snippet

html5ever::tree_builder::data::doctype_error_and_quirks (doctype=<optimized out>, iframe_srcdoc=false) at ~/.cargo/registry/src/github.com-1ecc6299db9ec823/html5ever-0.25.1/src/tree_builder/data.rs:157
157             (_, Some(ref s)) if QUIRKY_SYSTEM_MATCHES.contains(s) => Quirks,

I'm working on an application where the push-tokenizer that is built into html5ever is not very ergonomic.

Instead of making a sink and having process_token get called on it, I would rather enqueue a Tendril, and get an Iterator of Tokens that are parsed and returned on-demand.

My question is: would this be an acceptable option for html5ever? I wouldn't mind doing the implementation.

Updates the requirements on criterion to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Updates the requirements on typed-arena to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Using html5ever in async code is painful, because of Cell in StrTendril. As far as I can tell, that choice is for performance reasons (the documentation claims that the use of non-thread-safe primitives is explained in README.md, but no such explanation exists anywhere I can find) but in my use case I'm quite happy to trade a bit of performance for massively enhanced ergonomics with async.

To that end, I'd like to see something which swaps out StrTendril (aka Tendril<UTF8, NonAtomic>) for Tendril<UTF8, Atomic>. A feature to enable atomicity seems reasonable, but there might be a better option. As far as I can tell that's a completely source-compatible change, so this can probably be implemented with something like:

#[cfg(not(feature="atomic"))]
type H5ETendril = tendril::StrTendril;
#[cfg(feature="atomic")]
type H5ETendril = tendril::Tendril<tendril::UTF8, tendril::Atomic>;

and then replacing references to tendril::StrTendril with crate::H5ETendril.

Obviously the names can be bikeshedded. Those are the first ones I came up with.

Whether this belongs in this crate or tendril is also up for debate, but in my opinion: people using tendril directly already have plenty of ways to give themselves atomicity, and changing defaults through a feature there would just lead to dependency hell.

I can put in a PR, if it does turn out to be that simple.

See #476

If you use "*" for the dependency version, cargo cannot resolve the versions correctly:

[dependencies]
html5ever = "*"
markup5ever_rcdom = "*"

Fix: use exact dependencies

[dependencies]
html5ever = "0.25.2"
markup5ever_rcdom = "0.1.0"

Not sure if this is a bug in how the versions have been published, or a bug in cargo itself.

Re: https://github.com/servo/html5ever/pull/475

1. Make sure your rustc version is 1.61.0 or greater

   rustup update
   rustup override unset
   rustup default stable
   rustc --version
   

2. Create a new project

   mkdir -p ./my-print-rcdom/
   pushd ./my-print-rcdom/
   cargo init
   

3. Update Cargo.toml to include the dependencies (also, 2018 appears to be the required edition)

   [package]
   name = "example01"
   version = "0.1.0"
   authors = ["Joe Doe <[email protected]>"]
   edition = "2018"
   
   # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
   
   [dependencies]
   html5ever = "0.25.2"
   markup5ever_rcdom = "0.1.0"
   

4. Replace src/main.rs with the contents of, for example, ./rcdom/examples/print-rcdom.rs

   rm ./src/main.rs
   curl -L -o ./src/main.rs https://raw.githubusercontent.com/servo/html5ever/1b34cf17c/rcdom/examples/print-rcdom.rs
   

5. Install, build, run (with input):

   cargo build
   curl https://example.com | cargo run
   

Troubleshooting

Weird bug:

warning: unused import: `html5ever::tendril::TendrilSink`
  --> src/main.rs:20:5
   |
20 | use html5ever::tendril::TendrilSink;
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` on by default

Some errors have detailed explanations: E0277, E0599.
For more information about an error, try `rustc --explain E0277`.

Changing the name field in Cargo.toml (to anything at all), saving, and running cargo install again.

A room full of rust developers, including one of the most popular teachers on udemy in the room here with us, cannot figure this out.