aboutsummaryrefslogtreecommitdiff
path: root/CHANGELOG.md
AgeCommit message (Collapse)Author
2023-09-28break!: move offsets out of TokenMartin Fischer
Previously the Token enum contained the offsets using the O generic type parameter, which could be a usize if you're tracking offsets or a zero-sized type if you didn't care about offsets. This commit moves all the byte offset and syntax information to a new Trace enum, which has several advantages: * Traces can now easily be stored separately, while the tokens are fed to the tree builder. (The tree builder only has to keep track of which tree nodes originate from which tokens.) * No needless generics for functions that take a token but don't care about offsets (a tree construction implementation is bound to have many of such functions). * The FromIterator<(String, String)> impl for AttributeMap no longer has to specify arbitrary values for the spans and the value_syntax). * The PartialEq implementation of Token is now much more useful (since it no longer includes all the offsets). * The Debug formatting of Token is now more readable (since it no longer includes all the offsets). * Function pointers to functions accepting tokens are possible. (Since function pointer types may not have generic parameters.)
2023-09-28chore: add BasicEmitter stubMartin Fischer
2023-09-28break!: rename DefaultEmitter to TracingEmitterMartin Fischer
2023-09-28break!: remove NaiveParser::new_with_spansMartin Fischer
The method never really made much sense since you could just as well use NaiveParser::new.
2023-09-28break!: make DefaultEmitter::emit_eof emit EndOfFileMartin Fischer
2023-09-28break!: add Token::EndOfFileMartin Fischer
While the end-of-file token can also be represented by None, this is less clear than having an explicit variant. Especially when it comes to tree construction, the spec explicitly has conditions named "An end-of-file token", and it's nice if the code for tree construction can match the spec text closely.
2023-09-28break!: emit chars instead of stringsMartin Fischer
The HTML spec specifies that the tokenizer emits character tokens. That html5gum always emitted strings instead was probably just done to make the token consumption more convenient. When it comes to tree construction character tokens are however actually more convenient than string tokens since the spec defines that specific character tokens should be ignored in specific states (and character tokens let us avoid string manipulation for these conditions). This should also make the DefaultEmitter more performant for cases where you don't actually need the strings at all (or only a few) since it avoids string allocations. Though I haven't benchmarked it.
2023-09-28break!: remove Token::ErrorMartin Fischer
An error isn't a token (in general and also according to the spec). You shouldn't have to filter out errors when you're just interested in tokens but most importantly having errors in the Token enum is annoying when implementing tree construction (since the spec conditions exhaustively cover all Token variants except Token::Error).
2023-09-28break!: rename Emitter::emit_error to report_errorMartin Fischer
2023-09-28feat: add blanket impl of Reader for boxed readersMartin Fischer
2023-09-27break!: remove Emitter::pop_token, use Iterator insteadMartin Fischer
2023-09-05docs: fix heading style in changelogMartin Fischer
2023-09-05fix(docs): Comment data_offset field was changedMartin Fischer
I forgot to document this breaking change in e993f19c2b8ef00b32f17f9ed32306f3ceb21bc3.
2023-09-03chore: bump version to 0.5.1v0.5.1Martin Fischer
2023-09-03feat: add Doctype::name_spanMartin Fischer
2023-09-03break!: make Doctype name field optionalMartin Fischer
2023-09-03docs: the span logic no longer assumes UTF-8Martin Fischer
2023-09-03fix!: make comment data spans encoding-independentMartin Fischer
2023-09-03fix!: make set_self_closing encoding-independentMartin Fischer
2023-09-03fix!: make attribute spans encoding-independentMartin Fischer
2023-09-03fix!: make start/end tag name spans encoding-independentMartin Fischer
2023-09-03fix!: make PosTrackingReader encoding-independentMartin Fischer
While much of the span logic currently assumes UTF-8, we also want to support other character encodings, such as e.g. UTF-16 where characters can take up more or less bytes than in UTF-8.
2023-09-03fix!: calculate tag offsets in Tokenizer instead of Emitter implMartin Fischer
2023-09-03fix: too small char ref error spansMartin Fischer
2023-09-03fix: off-by-one missing-semicolon-after-character-reference spanMartin Fischer
2023-09-03fix!: off-by-one end-tag-with-trailing-solidus spanMartin Fischer
2023-09-03fix: most error spans mistakenly being emptyMartin Fischer
With codespan_reporting an empty span shows up exactly like a one-byte span, which is why I didn't notice this mistake earlier.
2023-09-03fix: off-by-one eof error spansMartin Fischer
2023-09-03break!: make Emitter::emit_error take spanMartin Fischer
2023-09-03fix!: wrong attribute value spans for char refsMartin Fischer
2023-09-03feat: make DefaultEmitter public againMartin Fischer
2023-09-03fix!: remove adjusted_current_node_present_and_not_in_html_namespaceMartin Fischer
Conceptually the tokenizer emits tokens, which are then handled in the tree construction stage (which this crate doesn't yet implement). While the tokenizer can operate almost entirely based on its state (which may be changed via Tokenizer::set_state) and its internal state, there is the exception of the 'Markup declaration open state'[1], the third condition of which depends on the "adjusted current node", which in turn depends on the "stack of open elements" only known to the tree constructor. In 82898967320f90116bbc686ab7ffc2f61ff456c4 I tried to address this by adding the adjusted_current_node_present_and_not_in_html_namespace method to the Emitter trait. What I missed was that adding this method to the Emitter trait effectively crippled the composability of the API. You should be able to do the following: struct TreeConstructor<R, O> { tokenizer: Tokenizer<R, O, SomeEmitter<O>>, stack_of_open_elements: Vec<NodeId>, // ... } However this doesn't work if the implementation of SomeEmitter depends on the stack_of_open_elements field. This commits remedies this oversight by removing this method and instead making the Tokenizer yield values of a new Event enum: enum Event<T> { Token(T), CdataOpen } Event::CdataOpen signals that the new Tokenizer::handle_cdata_open method has to be called, which accepts a CdataAction: enum CdataAction { Cdata, BogusComment } the variants of which correspond exactly to the possible outcomes of the third condition of the 'Markup declaration open state'. Removing this method also has the added benefit that the DefaultEmitter is now again spec-compliant, which lets us expose it again in the next commit in good conscience (previously it just hard-coded the method implementation to return false, which is why I had removed the DefaultEmitter from the public API in the last release). [1]: https://html.spec.whatwg.org/multipage/parsing.html#markup-declaration-open-state
2023-09-03fix: BufReadReader skips line on invalid UTF-8Martin Fischer
2023-09-03fix!: add missing `R: Position<O>` boundsMartin Fischer
It doesn't make sense that you're able to construct a Tokenizer/NaiveParser that you're unable to iterate over.
2023-09-03docs: add changelogMartin Fischer