The list of new lints are:

• clippy::all
• clippy::pedantic
• absolute_paths_not_starting_with_crate
• rustdoc::invalid_html_tags
• missing_copy_implementations
• missing_debug_implementations
• semicolon_in_expressions_from_macros
• unreachable_pub
• unused_extern_crates
• variant_size_differences
• ~clippy::missing_const_for_fn~

Though the intention is to avoid using unsafe Rust, the current implementation of html5gum::arrayvec::ArrayVec needs much improvement and the there's no real unsafety if we just use the arrayvec crate.

Consider html5gum::arrayvec::ArrayVec::push():

    pub fn push(&mut self, item: T) {
        self.content[self.len] = item;
        self.len += 1;
    }

Keep pushing and you are guaranteed to get an runtime index out of bound error.

While arrayvec::ArrayVec::push() checks the capacity and there's no unsafety or run-time error.

    fn push(&mut self, element: Self::Item) {
        self.try_push(element).unwrap()
    }

    fn try_push(&mut self, element: Self::Item) -> Result<(), CapacityError<Self::Item>> {
        if self.len() < Self::CAPACITY {
            unsafe {
                self.push_unchecked(element);
            }
            Ok(())
        } else {
            Err(CapacityError::new(element))
        }
    }

Consider html5gum::arrayvec::ArrayVec::drain()

    pub fn drain(&mut self) -> &[T] {
        let rv = &self.content[..self.len];
        self.len = 0;
        rv
    }

It doesn't drain anything actually. It just gives you a view of the content.

I propose to use arrayvec::ArrayVec instead. If you wish, we can use new type pattern and only expose new(), push(), and drain() function.

Hello there,

First, thanks for this crate, i was using Html5Ever before this, but this crate seems to be faster, and less dependencies.

However, i just noticed there was a new version available (v0.5.0), but i get errors now with my current implementation.

It seems to be using a HtmlString struct now, instead of a vec[u8]. Even though that struct is pub, the emitter module is private.

My current implementation can be found here: https://github.com/BlackDex/vaultwarden/blob/main/src/api/icons.rs#L419-L476 https://github.com/BlackDex/vaultwarden/blob/main/src/api/icons.rs#L827-L959

And it gets called from here: https://github.com/BlackDex/vaultwarden/blob/main/src/api/icons.rs#L557-L558

Not sure if this is a bug, or that i need to change some other items on my side for this new version to work? If you have any suggestions, please let me know.

Thanks in advance!

When I and SimonSapin were looking into html5ever speed up, one low-hanging fruit left was iterating over bytes rather than on Characters. It seems iterating on chars does a lot of UTF8 bounds check.

With bytes iteration and memchr, you might get a few speed-ups.

as per #12, change everything to run on bytes. Statemachine transitions are now per-byte, character boundaries are validated in emitter.

Open questions:

• [x] do we want to allow reading from raw bytes? currently BufReadReader does not validate any input, leading to panics in emitter when converting to string
• [x] where do we validate the characters of the input stream? maybe we want to do that in the emitter now, when assembling tokens? -- terrible idea, breaks tests in irreparable ways
• [x] serious performance regression with hyperlink
• [x] surrogate detection possible? seems too hard after initial attempt -- let's not
• [x] fuzzing in comparison mode
• [x] separate fuzzing testcases from rest
• [x] change fuzzing code such that it fuzzes arbitrary bytes
• [x] fuzz against crashes

I think most people who use an HTML5 tokenizer will want <script><b>test</b></script> to be tokenized as

StartTag(StartTag { self_closing: false, name: "script", attributes: {} })
String("<b>test</b>")
EndTag(EndTag { name: "script" })

instead of

StartTag(StartTag { self_closing: false, name: "script", attributes: {} })
StartTag(StartTag { self_closing: false, name: "b", attributes: {} })
String("test")
EndTag(EndTag { name: "b" })
EndTag(EndTag { name: "script" })

Unless I am missing something that doesn't seem to be possible with the current API?

Sidenote: It would also be nice to have some convenience utility that automatically dealt with the state implications of script, style, title, textarea, iframe etc. For example the html tokenizer of the Python standard library automatically takes care of script and style.

this is likely because I am bad at rust but I struggled to get this working (though in theory it shouldn't be too difficult?)

I was hoping to be able to do something like:

let resp = reqwest::get(u).await?;

for token in html5gum::Tokenizer::new(&resp) { ... }

or even with bytes_stream

let resp = reqwest::get(u).await?.bytes_stream();

for token in html5gum::Tokenizer::new(&resp) { ... }

fell back on

let resp = reqwest::get(u).await?.text().await?;

for token in html5gum::Tokenizer::new(&resp) { ... }

but I'm pretty sure that's not going to stream the response and it's going to convert it back and forth from Bytes -> String -> Vec -> String unnecessarily

Bumps tests/html5lib-tests from e3e6e15 to dd0d815.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps tests/html5lib-tests from e3e6e15 to 038c066.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

@lebensterben pointed out in #32 that it's currently hard to understand why certain potential panics (supposedly) don't occur in practice.

We should

• start documenting the relevant invariants in code comments
• write more explicit assertion messages when those fail (either by adding more debug_asserts on top or doing something else)
• statically enforce the above (this is probably impossible)

labelling as documentation as I'm not aware of actual panics being hit in usage.

Added a wrapper type html5gum::arrayvec::ArrayVec which in turn uses arrayvec::ArrayVec but only exposes the following interface:

• new() -> html5gum::arrayvec::ArrayVec<T, CAP>
• push(&mut html5gum::arrayvec::ArrayVec<T, CAP>)
• drain(&mut html5gum::arrayvec::ArrayVec<T, CAP>) -> arrayvec::Drain<T, CAP>

closes #32

While implementing https://github.com/lycheeverse/lychee/pull/480 I realized how slow the default emitter really is. It makes link extraction 10-40% slower than html5ever. It is currently not really possible to beat html5ever at all unless a custom emitter is implemented.

We could:

• build another emitter that reuses strings, and calls a callback with borrowed strings instead. Therefore much closer to lol-html's API.
• allow for custom allocators for all the strings we create -- similar to strtendril magic html5ever does (but definetly not using that crate)

lone surrogates are invalid utf8, so html5gum 0.3.0, which takes &str/String, is not able to handle those.

after merging #25, html5gum will be able to read arbitrary bytes. at this point the expectation might be that lone surrogates produce error tokens, but they do not.

note: lone surrogates have no impact on parsing behavior. only some error tokens are missing from token stream.

We should add a real treebuilder and lol-html like api on top of that to this crate. As part of that we need to move the tokenizer into a submodule and rework the readme

Hey, thanks for this library ... it looks really promising :) I am working on an HTML linter for which I require the spans of parser errors, tag names, attribute names and attribute values. These spans would ideally be reported as core::ops::Range<usize>, so that I can pass them directly to the codespan_reporting library (codespan_reporting::diagnostic::Label::range in particular). Since span tracking is of course overhead it would be behind an off-by-default feature flag.

I recently implemented this in my fork of the html5ever tokenizer ... which I frankly would love to abandon for a more sound library :) If you are interested in this I can probably implement it.

The tokenizer type itself allocates some strings on self that one should be able to replace with their own buffers and allocation behavior. Create another trait much like Emitter and expose push/pop/clear-type methods.