aboutsummaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2023-09-28chore: bump version to 0.5.2HEADv0.5.2mainMartin Fischer
2023-09-28docs: add errors exampleMartin Fischer
2023-09-28docs: add more context to early history in changelogMartin Fischer
2023-09-28chore: remove unnecessary generic param boundsMartin Fischer
2023-09-28chore: rename integration testsMartin Fischer
2023-09-28chore: update html5lib-tests submodule to latestMartin Fischer
2023-09-28fix: include CHANGELOG.md in .crate fileMartin Fischer
The file is linked from the README.md, so it should also be included.
2023-09-28fix(tokenizer): don't lowercase temp chars in ScriptDataEndTagNameMartin Fischer
This bug resulted in e.g. "<script></SCRI" being wrongly tokenized as: StartTag(StartTag { name: "script", self_closing: false, attributes: {} }) Char('<') Char('/') Char('s') Char('c') Char('r') Char('i') EndOfFile Note that the Char tokens should be uppercase. (This bug could only be observed when properly doing state switching via tree construction.)
2023-09-28docs: improve tokenize example output formatMartin Fischer
2023-09-28chore: move StartTag::name field upMartin Fischer
The Debug formatting is more readable when the name comes first.
2023-09-28feat: prettier Debug formatting for AttributeMap & AttributeMartin Fischer
2023-09-28break!: remove State::{ScriptDataEscaped, ScriptDataDoubleEscaped}Martin Fischer
These variants actually don't need to be exposed (to the tree builder).
2023-09-28break!: make AttributeMap::get return Option<&str>Martin Fischer
Most of the time you'll only be interested in the attribute value, so the `get` method should directly return it instead of a wrapper type. This also makes the API more predictable since e.g. the DOM getAttribute method also returns a string. Lastly previously it was quite confusing that map[key] wasn't equivalent to map.get(key).unwrap().
2023-09-28feat: impl Clone for Token & EventMartin Fischer
2023-09-28chore: rename internal states as wellMartin Fischer
See the previous commit.
2023-09-28break!: rename State variantsMartin Fischer
The spec refers to them only as RCDATA, RAWTEXT and PLAINTEXT. See https://rust-lang.github.io/api-guidelines/naming.html.
2023-09-28break!: remove CdataActionMartin Fischer
Which action the tokenizer takes depending on whether or not an adjusted current node is present but not in the HTML namespace, is an implementation detail and shouldn't be exposed in the API.
2023-09-28feat: add span to Trace::CharMartin Fischer
2023-09-28chore: use let...elseMartin Fischer
2023-09-28refactor: remove needless match & mapMartin Fischer
2023-09-28chore: rename variableMartin Fischer
2023-09-28feat: add offset to Trace::EndOfFileMartin Fischer
2023-09-28feat: implement BasicEmitterMartin Fischer
2023-09-28break!: move offsets out of TokenMartin Fischer
Previously the Token enum contained the offsets using the O generic type parameter, which could be a usize if you're tracking offsets or a zero-sized type if you didn't care about offsets. This commit moves all the byte offset and syntax information to a new Trace enum, which has several advantages: * Traces can now easily be stored separately, while the tokens are fed to the tree builder. (The tree builder only has to keep track of which tree nodes originate from which tokens.) * No needless generics for functions that take a token but don't care about offsets (a tree construction implementation is bound to have many of such functions). * The FromIterator<(String, String)> impl for AttributeMap no longer has to specify arbitrary values for the spans and the value_syntax). * The PartialEq implementation of Token is now much more useful (since it no longer includes all the offsets). * The Debug formatting of Token is now more readable (since it no longer includes all the offsets). * Function pointers to functions accepting tokens are possible. (Since function pointer types may not have generic parameters.)
2023-09-28refactor: make TracingEmitter only work with usizesMartin Fischer
2023-09-28chore: add BasicEmitter stubMartin Fischer
2023-09-28break!: rename DefaultEmitter to TracingEmitterMartin Fischer
2023-09-28refactor: decouple run_test_inner from DefaultEmitterMartin Fischer
2023-09-28refactor: simplify attribute logic in DefaultEmitterMartin Fischer
2023-09-28refactor: clean up DefaultEmitter codeMartin Fischer
2023-09-28[0/8]: move offsets out of TokenMartin Fischer
2023-09-28chore: reorder DefaultEmitter methods as wellMartin Fischer
See the previous commit.
2023-09-28docs: put Emitter methods in a more logical orderMartin Fischer
2023-09-28refactor: make NaiveParser new* signatures more readableMartin Fischer
2023-09-28break!: remove NaiveParser::new_with_spansMartin Fischer
The method never really made much sense since you could just as well use NaiveParser::new.
2023-09-28break!: make DefaultEmitter::emit_eof emit EndOfFileMartin Fischer
2023-09-28break!: add Token::EndOfFileMartin Fischer
While the end-of-file token can also be represented by None, this is less clear than having an explicit variant. Especially when it comes to tree construction, the spec explicitly has conditions named "An end-of-file token", and it's nice if the code for tree construction can match the spec text closely.
2023-09-28break!: emit chars instead of stringsMartin Fischer
The HTML spec specifies that the tokenizer emits character tokens. That html5gum always emitted strings instead was probably just done to make the token consumption more convenient. When it comes to tree construction character tokens are however actually more convenient than string tokens since the spec defines that specific character tokens should be ignored in specific states (and character tokens let us avoid string manipulation for these conditions). This should also make the DefaultEmitter more performant for cases where you don't actually need the strings at all (or only a few) since it avoids string allocations. Though I haven't benchmarked it.
2023-09-28refactor: proxy emit_string calls through utilsMartin Fischer
This is done separately so that the next commit has a cleaner diff.
2023-09-28refactor: move machine impl details to machine moduleMartin Fischer
This commit separates the public API (the "Tokenizer") from the internal implementation (the "Machine") to make the code more readable.
2023-09-28refactor: move utils module under tokenizer::machineMartin Fischer
2023-09-28refactor: only use InternalState re-export for feature-gated internal APIMartin Fischer
2023-09-28refactor: move machine module under tokenizerMartin Fischer
2023-09-28//: remove wrong commentMartin Fischer
Methods defined in another module don't have access to private fields, so the function could very well have been implemented as a method.
2023-09-28break!: remove Token::ErrorMartin Fischer
An error isn't a token (in general and also according to the spec). You shouldn't have to filter out errors when you're just interested in tokens but most importantly having errors in the Token enum is annoying when implementing tree construction (since the spec conditions exhaustively cover all Token variants except Token::Error).
2023-09-28refactor: remove DefaultEmitter::push_error helper fnMartin Fischer
2023-09-28break!: rename Emitter::emit_error to report_errorMartin Fischer
2023-09-28chore: move emit_error method upMartin Fischer
2023-09-28chore: build html5lib_tests::Output laterMartin Fischer
This is done separately so that the next commit has a cleaner diff.
2023-09-28refactor: make labeler closures take NaiveParserMartin Fischer
The second next commit will move errors out of the Token enum but we still want to be able to test that the spans of errors are character encoding independent.
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-02 10:09:36 +0000