html5tokenizer/examples, branch v0.5.2Fork of html5gum with code span supporthttp://git.push-f.com/html5tokenizer/atom/examples?h=v0.5.22023-09-28T09:00:01Zdocs: add errors example2023-09-28T09:00:01ZMartin Fischermartin@push-f.com2023-09-28T08:14:21Zurn:sha1:70b92eda9a38c2f0e093c0f74cafb6398f97b58edocs: improve tokenize example output format2023-09-28T08:36:08ZMartin Fischermartin@push-f.com2023-09-28T08:32:45Zurn:sha1:ade1034c46f3c5e50bdf07fd1556b7957011fe98break!: move offsets out of Token2023-09-28T08:36:08ZMartin Fischermartin@push-f.com2023-09-10T17:37:34Zurn:sha1:852d5c6f2e65a5ab466662ae1c649a0ed25c70a9
Previously the Token enum contained the offsets using the O generic
type parameter, which could be a usize if you're tracking offsets or
a zero-sized type if you didn't care about offsets. This commit moves
all the byte offset and syntax information to a new Trace enum,
which has several advantages:
* Traces can now easily be stored separately, while the tokens are
fed to the tree builder. (The tree builder only has to keep track
of which tree nodes originate from which tokens.)
* No needless generics for functions that take a token but don't
care about offsets (a tree construction implementation is bound
to have many of such functions).
* The FromIterator<(String, String)> impl for AttributeMap no longer
has to specify arbitrary values for the spans and the value_syntax).
* The PartialEq implementation of Token is now much more useful
(since it no longer includes all the offsets).
* The Debug formatting of Token is now more readable
(since it no longer includes all the offsets).
* Function pointers to functions accepting tokens are possible.
(Since function pointer types may not have generic parameters.)
chore: add BasicEmitter stub2023-09-28T08:36:08ZMartin Fischermartin@push-f.com2023-09-12T07:03:56Zurn:sha1:14bc6f2cceed0fa578d6a1195266885bf57a5d4cbreak!: rename DefaultEmitter to TracingEmitter2023-09-28T08:36:08ZMartin Fischermartin@push-f.com2023-09-12T06:19:00Zurn:sha1:ad6ac5f0a825775c231e76cdc9016e61e54f4141break!: remove Token::Error2023-09-28T08:36:01ZMartin Fischermartin@push-f.com2023-08-29T11:09:44Zurn:sha1:826907487e2b593f1c54e98b59fe2f6eb8cb6937
An error isn't a token (in general and also according to the spec).
You shouldn't have to filter out errors when you're just interested
in tokens but most importantly having errors in the Token enum is
annoying when implementing tree construction (since the spec conditions
exhaustively cover all Token variants except Token::Error).
docs: add spans example2023-09-03T21:02:01ZMartin Fischermartin@push-f.com2023-09-03T21:02:01Zurn:sha1:0c590e90564870fa8600460c866c7395b7d865cdfeat: introduce NaiveParser2023-08-19T11:53:58ZMartin Fischermartin@push-f.com2023-08-12T10:58:08Zurn:sha1:0d9cd9ed44b676ccd4991cea27dc620b94ebe7e7refactor!: make Emitter generic over offset instead of reader2023-08-19T11:41:55ZMartin Fischermartin@push-f.com2023-08-17T07:40:47Zurn:sha1:c15895d44d17984386d3684e2aa85aca386ba3bf
Emitters should not have access to the reader at all. Also the
current position of the reader, at the time an Emitted method is
called, very much depends on machine implementation details such
as if `Tokenizer::unread_char` is used. Having the Emitter
methods take offsets lets the machine take care of providing
the right offsets, as evidenced by the next commit.
refactor: add default for S type param of DefaultEmitter2023-08-19T09:40:58ZMartin Fischermartin@push-f.com2023-08-17T06:57:15Zurn:sha1:c899f2f36df0371af776f21231ccdf0d30a2aaab