html5tokenizer/src/tokenizer/machine, branch mainFork of html5gum with code span supporthttp://git.push-f.com/html5tokenizer/atom/src/tokenizer/machine?h=main2023-09-28T08:36:08Zchore: rename internal states as well2023-09-28T08:36:08ZMartin Fischermartin@push-f.com2023-09-04T18:47:52Zurn:sha1:bd446b6123e0b077a48e9f1e836affac78822f44
See the previous commit.
feat: add span to Trace::Char2023-09-28T08:36:08ZMartin Fischermartin@push-f.com2023-09-15T07:40:55Zurn:sha1:d46de6ab592e57a31fef13cfc015c4ce818e8f47feat: add offset to Trace::EndOfFile2023-09-28T08:36:08ZMartin Fischermartin@push-f.com2023-09-15T06:15:29Zurn:sha1:e0434737b20b32a8d7238a134c3662fa9075eb6dbreak!: emit chars instead of strings2023-09-28T08:36:08ZMartin Fischermartin@push-f.com2023-08-29T05:24:55Zurn:sha1:df5a2ae14c4976d404195e83e52b8db62be63988
The HTML spec specifies that the tokenizer emits character tokens.
That html5gum always emitted strings instead was probably just done
to make the token consumption more convenient. When it comes to tree
construction character tokens are however actually more convenient
than string tokens since the spec defines that specific character
tokens should be ignored in specific states (and character tokens
let us avoid string manipulation for these conditions).
This should also make the DefaultEmitter more performant for cases
where you don't actually need the strings at all (or only a few)
since it avoids string allocations. Though I haven't benchmarked it.
refactor: proxy emit_string calls through utils2023-09-28T08:36:08ZMartin Fischermartin@push-f.com2023-09-09T20:15:54Zurn:sha1:394c52260e861e911e2d8706d4904136a920da87
This is done separately so that the next commit has a cleaner diff.
refactor: move machine impl details to machine module2023-09-28T08:36:08ZMartin Fischermartin@push-f.com2023-09-09T19:53:22Zurn:sha1:5aa3b82fbe62882da8007b0a4548b979c845aa97
This commit separates the public API (the "Tokenizer")
from the internal implementation (the "Machine")
to make the code more readable.
refactor: move utils module under tokenizer::machine2023-09-28T08:36:08ZMartin Fischermartin@push-f.com2023-09-09T19:42:17Zurn:sha1:2c73901944e2d22747a2a4ebcc11881b3f8c2ad3