html5tokenizer - Fork of html5gum with code span support

Age	Commit message (Collapse)	Author
2023-09-03	docs: the span logic no longer assumes UTF-8	Martin Fischer

2023-09-03	fix!: make comment data spans encoding-independent	Martin Fischer

2023-09-03	fix!: make set_self_closing encoding-independent	Martin Fischer

2023-09-03	fix!: make attribute spans encoding-independent	Martin Fischer

2023-09-03	fix!: make start/end tag name spans encoding-independent	Martin Fischer

2023-09-03	fix!: make PosTrackingReader encoding-independent	Martin Fischer
	While much of the span logic currently assumes UTF-8, we also want to support other character encodings, such as e.g. UTF-16 where characters can take up more or less bytes than in UTF-8.
2023-09-03	fix!: calculate tag offsets in Tokenizer instead of Emitter impl	Martin Fischer

2023-09-03	fix: too small char ref error spans	Martin Fischer

2023-09-03	fix: off-by-one missing-semicolon-after-character-reference span	Martin Fischer

2023-09-03	fix!: off-by-one end-tag-with-trailing-solidus span	Martin Fischer

2023-09-03	fix: most error spans mistakenly being empty	Martin Fischer
	With codespan_reporting an empty span shows up exactly like a one-byte span, which is why I didn't notice this mistake earlier.
2023-09-03	fix: off-by-one eof error spans	Martin Fischer

2023-09-03	break!: make Emitter::emit_error take span	Martin Fischer

2023-09-03	fix!: wrong attribute value spans for char refs	Martin Fischer

2023-09-03	feat: make DefaultEmitter public again	Martin Fischer

2023-09-03	fix!: remove adjusted_current_node_present_and_not_in_html_namespace	Martin Fischer
	Conceptually the tokenizer emits tokens, which are then handled in the tree construction stage (which this crate doesn't yet implement). While the tokenizer can operate almost entirely based on its state (which may be changed via Tokenizer::set_state) and its internal state, there is the exception of the 'Markup declaration open state'[1], the third condition of which depends on the "adjusted current node", which in turn depends on the "stack of open elements" only known to the tree constructor. In 82898967320f90116bbc686ab7ffc2f61ff456c4 I tried to address this by adding the adjusted_current_node_present_and_not_in_html_namespace method to the Emitter trait. What I missed was that adding this method to the Emitter trait effectively crippled the composability of the API. You should be able to do the following: struct TreeConstructor<R, O> { tokenizer: Tokenizer<R, O, SomeEmitter<O>>, stack_of_open_elements: Vec<NodeId>, // ... } However this doesn't work if the implementation of SomeEmitter depends on the stack_of_open_elements field. This commits remedies this oversight by removing this method and instead making the Tokenizer yield values of a new Event enum: enum Event<T> { Token(T), CdataOpen } Event::CdataOpen signals that the new Tokenizer::handle_cdata_open method has to be called, which accepts a CdataAction: enum CdataAction { Cdata, BogusComment } the variants of which correspond exactly to the possible outcomes of the third condition of the 'Markup declaration open state'. Removing this method also has the added benefit that the DefaultEmitter is now again spec-compliant, which lets us expose it again in the next commit in good conscience (previously it just hard-coded the method implementation to return false, which is why I had removed the DefaultEmitter from the public API in the last release). [1]: https://html.spec.whatwg.org/multipage/parsing.html#markup-declaration-open-state
2023-09-03	fix: BufReadReader skips line on invalid UTF-8	Martin Fischer

2023-09-03	fix!: add missing `R: Position<O>` bounds	Martin Fischer
	It doesn't make sense that you're able to construct a Tokenizer/NaiveParser that you're unable to iterate over.
2023-09-03	docs: add changelog	Martin Fischer