break!: add Token::EndOfFile

While the end-of-file token can also be represented by None,
this is less clear than having an explicit variant. Especially when
it comes to tree construction, the spec explicitly has conditions
named "An end-of-file token", and it's nice if the code for tree
construction can match the spec text closely.

1 files changed, 1 insertions, 0 deletions

diff --git a/integration_tests/tests/test_html5lib.rs b/integration_tests/tests/test_html5lib.rs
index bede29a..8fedc1a 100644
--- a/integration_tests/tests/test_html5lib.rs
+++ b/integration_tests/tests/test_html5lib.rs
@@ -152,6 +152,7 @@ fn run_test_inner<R: Reader>(
                 system_id: doctype.system_id,
                 force_quirks: doctype.force_quirks,
             }),
+            Token::EndOfFile => {}
         };
     }