html5tokenizer/examples, branch v0.5.0Fork of html5gum with code span supporthttp://git.push-f.com/html5tokenizer/atom/examples?h=v0.5.02023-08-19T11:53:58Zfeat: introduce NaiveParser2023-08-19T11:53:58ZMartin Fischermartin@push-f.com2023-08-12T10:58:08Zurn:sha1:0d9cd9ed44b676ccd4991cea27dc620b94ebe7e7refactor!: make Emitter generic over offset instead of reader2023-08-19T11:41:55ZMartin Fischermartin@push-f.com2023-08-17T07:40:47Zurn:sha1:c15895d44d17984386d3684e2aa85aca386ba3bf
Emitters should not have access to the reader at all. Also the
current position of the reader, at the time an Emitted method is
called, very much depends on machine implementation details such
as if `Tokenizer::unread_char` is used. Having the Emitter
methods take offsets lets the machine take care of providing
the right offsets, as evidenced by the next commit.
refactor: add default for S type param of DefaultEmitter2023-08-19T09:40:58ZMartin Fischermartin@push-f.com2023-08-17T06:57:15Zurn:sha1:c899f2f36df0371af776f21231ccdf0d30a2aaabbreak!: merge Tokenizer::new_with_emitter into Tokenizer::new2023-08-19T04:41:55ZMartin Fischermartin@push-f.com2023-08-12T09:06:02Zurn:sha1:9f1019afa7a8e9102d67356d85bd632044eb2d0c
The Tokenizer does not perform any state switching, since
proper state switching requires a feedback loop between
tokenization and DOM tree building. Using the Tokenizer
directly therefore is a bit of a pitfall, since you might
not expect it to e.g. tokenize `<script><b>` as:
StartTag(StartTag { name: "script", .. })
StartTag(StartTag { name: "b", .. })
Since we don't want to make walking into pitfalls
particularly easy, this commit changes the Tokenizer::new
method so that you have to specify the Emitter.
Since this makes new_with_emitter redundant it is removed.
break!: remove StartTag::next_state2023-08-19T04:41:03ZMartin Fischermartin@push-f.com2023-08-18T12:14:52Zurn:sha1:bfff0560a0e448eef41ac2b4c7e8deb0a77e4167
You shouldn't manually have to match tokens yielded by the
tokenizer iterator just to correctly handle state transitions.
A better NaiveParser API will be introduced.
rename to html5tokenizer, bump version2021-12-05T02:52:22ZMartin Fischermartin@push-f.com2021-12-05T02:19:03Zurn:sha1:b00714411306ee6500e4ee34a81bd7f4a111169eintroduce StartTag::next_state2021-12-05T01:45:31ZMartin Fischermartin@push-f.com2021-11-30T15:16:47Zurn:sha1:7e400c29bf14bd75154c74e2dd9ba0814f83bec7
Closes #11.
Fix typo and add example (#9)2021-12-03T14:59:37ZMartin Fischermartin@push-f.com2021-12-03T14:59:37Zurn:sha1:c9c8fcf4daa594c7b3ffe08a64f563f152c22f29