aboutsummaryrefslogtreecommitdiff
path: root/src/tokenizer/machine/utils.rs
AgeCommit message (Collapse)Author
2023-09-28feat: add span to Trace::CharMartin Fischer
2023-09-28feat: add offset to Trace::EndOfFileMartin Fischer
2023-09-28break!: emit chars instead of stringsMartin Fischer
The HTML spec specifies that the tokenizer emits character tokens. That html5gum always emitted strings instead was probably just done to make the token consumption more convenient. When it comes to tree construction character tokens are however actually more convenient than string tokens since the spec defines that specific character tokens should be ignored in specific states (and character tokens let us avoid string manipulation for these conditions). This should also make the DefaultEmitter more performant for cases where you don't actually need the strings at all (or only a few) since it avoids string allocations. Though I haven't benchmarked it.
2023-09-28refactor: proxy emit_string calls through utilsMartin Fischer
This is done separately so that the next commit has a cleaner diff.
2023-09-28refactor: move machine impl details to machine moduleMartin Fischer
This commit separates the public API (the "Tokenizer") from the internal implementation (the "Machine") to make the code more readable.
2023-09-28refactor: move utils module under tokenizer::machineMartin Fischer