aboutsummaryrefslogtreecommitdiff
path: root/examples/tokenize.rs
AgeCommit message (Collapse)Author
2023-08-19refactor!: make Emitter generic over offset instead of readerMartin Fischer
Emitters should not have access to the reader at all. Also the current position of the reader, at the time an Emitted method is called, very much depends on machine implementation details such as if `Tokenizer::unread_char` is used. Having the Emitter methods take offsets lets the machine take care of providing the right offsets, as evidenced by the next commit.
2023-08-19refactor: add default for S type param of DefaultEmitterMartin Fischer
2023-08-19break!: merge Tokenizer::new_with_emitter into Tokenizer::newMartin Fischer
The Tokenizer does not perform any state switching, since proper state switching requires a feedback loop between tokenization and DOM tree building. Using the Tokenizer directly therefore is a bit of a pitfall, since you might not expect it to e.g. tokenize `<script><b>` as: StartTag(StartTag { name: "script", .. }) StartTag(StartTag { name: "b", .. }) Since we don't want to make walking into pitfalls particularly easy, this commit changes the Tokenizer::new method so that you have to specify the Emitter. Since this makes new_with_emitter redundant it is removed.
2021-12-05rename to html5tokenizer, bump versionv0.4.0Martin Fischer
2021-12-03Fix typo and add example (#9)Martin Fischer