html5tokenizer/tests, branch mainFork of html5gum with code span supporthttp://git.push-f.com/html5tokenizer/atom/tests?h=main2023-09-28T09:00:01Zchore: rename integration tests2023-09-28T09:00:01ZMartin Fischermartin@push-f.com2023-09-27T07:25:12Zurn:sha1:b027ecdb397c2e378491f847660f8eeb740e8cf6feat: add span to Trace::Char2023-09-28T08:36:08ZMartin Fischermartin@push-f.com2023-09-15T07:40:55Zurn:sha1:d46de6ab592e57a31fef13cfc015c4ce818e8f47feat: add offset to Trace::EndOfFile2023-09-28T08:36:08ZMartin Fischermartin@push-f.com2023-09-15T06:15:29Zurn:sha1:e0434737b20b32a8d7238a134c3662fa9075eb6dbreak!: move offsets out of Token2023-09-28T08:36:08ZMartin Fischermartin@push-f.com2023-09-10T17:37:34Zurn:sha1:852d5c6f2e65a5ab466662ae1c649a0ed25c70a9
Previously the Token enum contained the offsets using the O generic
type parameter, which could be a usize if you're tracking offsets or
a zero-sized type if you didn't care about offsets. This commit moves
all the byte offset and syntax information to a new Trace enum,
which has several advantages:
* Traces can now easily be stored separately, while the tokens are
fed to the tree builder. (The tree builder only has to keep track
of which tree nodes originate from which tokens.)
* No needless generics for functions that take a token but don't
care about offsets (a tree construction implementation is bound
to have many of such functions).
* The FromIterator<(String, String)> impl for AttributeMap no longer
has to specify arbitrary values for the spans and the value_syntax).
* The PartialEq implementation of Token is now much more useful
(since it no longer includes all the offsets).
* The Debug formatting of Token is now more readable
(since it no longer includes all the offsets).
* Function pointers to functions accepting tokens are possible.
(Since function pointer types may not have generic parameters.)
refactor: make TracingEmitter only work with usizes2023-09-28T08:36:08ZMartin Fischermartin@push-f.com2023-09-12T07:26:06Zurn:sha1:a03cea75d9d120a7519be91ec872b143b5d74276chore: add BasicEmitter stub2023-09-28T08:36:08ZMartin Fischermartin@push-f.com2023-09-12T07:03:56Zurn:sha1:14bc6f2cceed0fa578d6a1195266885bf57a5d4cbreak!: rename DefaultEmitter to TracingEmitter2023-09-28T08:36:08ZMartin Fischermartin@push-f.com2023-09-12T06:19:00Zurn:sha1:ad6ac5f0a825775c231e76cdc9016e61e54f4141break!: remove Token::Error2023-09-28T08:36:01ZMartin Fischermartin@push-f.com2023-08-29T11:09:44Zurn:sha1:826907487e2b593f1c54e98b59fe2f6eb8cb6937
An error isn't a token (in general and also according to the spec).
You shouldn't have to filter out errors when you're just interested
in tokens but most importantly having errors in the Token enum is
annoying when implementing tree construction (since the spec conditions
exhaustively cover all Token variants except Token::Error).
refactor: make labeler closures take NaiveParser2023-09-28T05:26:44ZMartin Fischermartin@push-f.com2023-09-04T07:27:30Zurn:sha1:82961e1f8c85d6961410fd007ca2127cee594848
The second next commit will move errors out of the Token enum
but we still want to be able to test that the spans of errors
are character encoding independent.
refactor: make parser helper fn generic over reader2023-09-28T05:26:44ZMartin Fischermartin@push-f.com2023-09-04T07:15:30Zurn:sha1:9dd2ac1c95c949066e341837650c3211769342c3