aboutsummaryrefslogtreecommitdiff
path: root/examples/tokenize.rs
AgeCommit message (Collapse)Author
2023-09-28chore: add BasicEmitter stubMartin Fischer
2023-09-28break!: rename DefaultEmitter to TracingEmitterMartin Fischer
2023-09-28break!: remove Token::ErrorMartin Fischer
An error isn't a token (in general and also according to the spec). You shouldn't have to filter out errors when you're just interested in tokens but most importantly having errors in the Token enum is annoying when implementing tree construction (since the spec conditions exhaustively cover all Token variants except Token::Error).
2023-08-19refactor!: make Emitter generic over offset instead of readerMartin Fischer
Emitters should not have access to the reader at all. Also the current position of the reader, at the time an Emitted method is called, very much depends on machine implementation details such as if `Tokenizer::unread_char` is used. Having the Emitter methods take offsets lets the machine take care of providing the right offsets, as evidenced by the next commit.
2023-08-19refactor: add default for S type param of DefaultEmitterMartin Fischer
2023-08-19break!: merge Tokenizer::new_with_emitter into Tokenizer::newMartin Fischer
The Tokenizer does not perform any state switching, since proper state switching requires a feedback loop between tokenization and DOM tree building. Using the Tokenizer directly therefore is a bit of a pitfall, since you might not expect it to e.g. tokenize `<script><b>` as: StartTag(StartTag { name: "script", .. }) StartTag(StartTag { name: "b", .. }) Since we don't want to make walking into pitfalls particularly easy, this commit changes the Tokenizer::new method so that you have to specify the Emitter. Since this makes new_with_emitter redundant it is removed.
2021-12-05rename to html5tokenizer, bump versionv0.4.0Martin Fischer
2021-12-03Fix typo and add example (#9)Martin Fischer