diff options
| author | Martin Fischer <martin@push-f.com> | 2023-08-29 13:17:59 +0200 |
|---|---|---|
| committer | Martin Fischer <martin@push-f.com> | 2023-09-28 10:36:08 +0200 |
| commit | bb7c64a8245769f20742502cfa942f33ccb08af7 (patch) | |
| tree | 4d92489782c9cb277ac3a8f5b655808bac603bdd /integration_tests/tests/tokenizer.rs | |
| parent | df5a2ae14c4976d404195e83e52b8db62be63988 (diff) | |
break!: add Token::EndOfFile
While the end-of-file token can also be represented by None,
this is less clear than having an explicit variant. Especially when
it comes to tree construction, the spec explicitly has conditions
named "An end-of-file token", and it's nice if the code for tree
construction can match the spec text closely.
Diffstat (limited to 'integration_tests/tests/tokenizer.rs')
0 files changed, 0 insertions, 0 deletions