Fork of html5gum with code span support http://git.push-f.com/html5tokenizer/atom/integration_tests/tests?h=v0.5.1 2023-09-03T21:00:05Z 2023-09-03T21:00:05Z Martin Fischer martin@push-f.com 2023-09-03T08:47:44Z urn:sha1:d56686deab81c8b50207b75a485cf26ec8502383 2023-09-03T21:00:05Z Martin Fischer martin@push-f.com 2023-08-28T17:04:58Z urn:sha1:f31bffb8426f04aaadea911e7c42b130a9ee80a5 Conceptually the tokenizer emits tokens, which are then handled in the tree construction stage (which this crate doesn't yet implement). While the tokenizer can operate almost entirely based on its state (which may be changed via Tokenizer::set_state) and its internal state, there is the exception of the 'Markup declaration open state'[1], the third condition of which depends on the "adjusted current node", which in turn depends on the "stack of open elements" only known to the tree constructor. In 82898967320f90116bbc686ab7ffc2f61ff456c4 I tried to address this by adding the adjusted_current_node_present_and_not_in_html_namespace method to the Emitter trait. What I missed was that adding this method to the Emitter trait effectively crippled the composability of the API. You should be able to do the following: struct TreeConstructor<R, O> { tokenizer: Tokenizer<R, O, SomeEmitter<O>>, stack_of_open_elements: Vec<NodeId>, // ... } However this doesn't work if the implementation of SomeEmitter depends on the stack_of_open_elements field. This commits remedies this oversight by removing this method and instead making the Tokenizer yield values of a new Event enum: enum Event<T> { Token(T), CdataOpen } Event::CdataOpen signals that the new Tokenizer::handle_cdata_open method has to be called, which accepts a CdataAction: enum CdataAction { Cdata, BogusComment } the variants of which correspond exactly to the possible outcomes of the third condition of the 'Markup declaration open state'. Removing this method also has the added benefit that the DefaultEmitter is now again spec-compliant, which lets us expose it again in the next commit in good conscience (previously it just hard-coded the method implementation to return false, which is why I had removed the DefaultEmitter from the public API in the last release). [1]: https://html.spec.whatwg.org/multipage/parsing.html#markup-declaration-open-state 2023-08-19T11:41:55Z Martin Fischer martin@push-f.com 2023-08-19T09:46:27Z urn:sha1:8eb2a95a483bddc8c6a6a8a30dff68c290277075 2023-08-19T11:41:55Z Martin Fischer martin@push-f.com 2023-08-18T14:43:03Z urn:sha1:0c495ba984436cccc6caeed66639a2b61095dbad In the next commit I'm adding a test that compares the content of files and pretty_assertions doesn't omit large portions of unchanged lines in its diff[1] (contrary to similar-asserts). (Sidenote: We already depend on similar via insta.) [1]: https://github.com/rust-pretty-assertions/rust-pretty-assertions/issues/114 2023-08-19T11:41:55Z Martin Fischer martin@push-f.com 2023-08-17T16:07:32Z urn:sha1:11316f041985345dd3a712d14bea749790f937a4 Display impls should return human-readable strings. After this commit we're able to introduce a proper Display impl in the future without that being a breaking change. 2023-08-19T11:41:55Z Martin Fischer martin@push-f.com 2023-08-16T16:54:35Z urn:sha1:c07334e7bb613774d3535663433c90b6dde86943 Just a bit more succinct. And now rustdoc also no longer cuts off the names of these Emitter methods in its sidebar. 2023-08-19T11:41:55Z Martin Fischer martin@push-f.com 2023-08-16T13:58:48Z urn:sha1:0c87a9ec25a45efc9b6b5ab7883cd19ded483909 Making this change made me realize that adding an `impl IntoIterator for T` can be a breaking change if `impl IntoIterator for &T` already exists. See also the cargo-semver-checks issue[1] I filed about that. [1]: https://github.com/obi1kenobi/cargo-semver-checks/issues/518 2023-08-19T11:41:55Z Martin Fischer martin@push-f.com 2023-08-16T07:45:18Z urn:sha1:65aca9cbf0318bd3a2f936641b4f5bc3729c98c2 This has a number of benefits: * it hides the implementation of the map * it hides the type used for the map values (which lets us e.g. change name_span to name_offset while still being able to provide a convenient `Attribute::name_span` method.) * it lets us provide convenience impls for the map such as `FromIterator<(String, String)>` 2023-08-19T09:41:55Z Martin Fischer martin@push-f.com 2023-08-17T13:21:32Z urn:sha1:4d9cf7171836625b61dcfe675bdf9452766166c0 2023-08-19T04:41:55Z Martin Fischer martin@push-f.com 2023-08-12T09:50:46Z urn:sha1:69fdbee250ebfce135bff2671226097bc536d953 This is primarily done to make the rustdoc more readable (by grouping Reader, IntoReader, StringReader and BufReadReader in the reader module). Ideally IntoReader is already implemented for your input type and you don't have to concern yourself with these traits / types at all.