Age | Commit message (Collapse) | Author | |
---|---|---|---|
2023-09-28 | [0/8]: move offsets out of Token | Martin Fischer | |
2023-09-28 | chore: reorder DefaultEmitter methods as well | Martin Fischer | |
See the previous commit. | |||
2023-09-28 | docs: put Emitter methods in a more logical order | Martin Fischer | |
2023-09-28 | refactor: make NaiveParser new* signatures more readable | Martin Fischer | |
2023-09-28 | break!: remove NaiveParser::new_with_spans | Martin Fischer | |
The method never really made much sense since you could just as well use NaiveParser::new. | |||
2023-09-28 | break!: make DefaultEmitter::emit_eof emit EndOfFile | Martin Fischer | |
2023-09-28 | break!: add Token::EndOfFile | Martin Fischer | |
While the end-of-file token can also be represented by None, this is less clear than having an explicit variant. Especially when it comes to tree construction, the spec explicitly has conditions named "An end-of-file token", and it's nice if the code for tree construction can match the spec text closely. | |||
2023-09-28 | break!: emit chars instead of strings | Martin Fischer | |
The HTML spec specifies that the tokenizer emits character tokens. That html5gum always emitted strings instead was probably just done to make the token consumption more convenient. When it comes to tree construction character tokens are however actually more convenient than string tokens since the spec defines that specific character tokens should be ignored in specific states (and character tokens let us avoid string manipulation for these conditions). This should also make the DefaultEmitter more performant for cases where you don't actually need the strings at all (or only a few) since it avoids string allocations. Though I haven't benchmarked it. | |||
2023-09-28 | refactor: proxy emit_string calls through utils | Martin Fischer | |
This is done separately so that the next commit has a cleaner diff. | |||
2023-09-28 | refactor: move machine impl details to machine module | Martin Fischer | |
This commit separates the public API (the "Tokenizer") from the internal implementation (the "Machine") to make the code more readable. | |||
2023-09-28 | refactor: move utils module under tokenizer::machine | Martin Fischer | |
2023-09-28 | refactor: only use InternalState re-export for feature-gated internal API | Martin Fischer | |
2023-09-28 | refactor: move machine module under tokenizer | Martin Fischer | |
2023-09-28 | //: remove wrong comment | Martin Fischer | |
Methods defined in another module don't have access to private fields, so the function could very well have been implemented as a method. | |||
2023-09-28 | break!: remove Token::Error | Martin Fischer | |
An error isn't a token (in general and also according to the spec). You shouldn't have to filter out errors when you're just interested in tokens but most importantly having errors in the Token enum is annoying when implementing tree construction (since the spec conditions exhaustively cover all Token variants except Token::Error). | |||
2023-09-28 | refactor: remove DefaultEmitter::push_error helper fn | Martin Fischer | |
2023-09-28 | break!: rename Emitter::emit_error to report_error | Martin Fischer | |
2023-09-28 | chore: move emit_error method up | Martin Fischer | |
2023-09-28 | chore: build html5lib_tests::Output later | Martin Fischer | |
This is done separately so that the next commit has a cleaner diff. | |||
2023-09-28 | refactor: make labeler closures take NaiveParser | Martin Fischer | |
The second next commit will move errors out of the Token enum but we still want to be able to test that the spans of errors are character encoding independent. | |||
2023-09-28 | feat: add blanket impl of Reader for boxed readers | Martin Fischer | |
2023-09-28 | refactor: make parser helper fn generic over reader | Martin Fischer | |
2023-09-28 | chore: make assert_char_encoding_independence more readable | Martin Fischer | |
2023-09-28 | chore: rename internal tokenizer helper fn to parser | Martin Fischer | |
2023-09-28 | [0/10]: remove Token::Error | Martin Fischer | |
2023-09-27 | break!: remove Emitter::pop_token, use Iterator instead | Martin Fischer | |
2023-09-27 | chore: move bounds to where clause | Martin Fischer | |
2023-09-12 | docs: move warning from DefaultEmitter to Tokenizer | Martin Fischer | |
2023-09-11 | chore: move DefaultEmitter to own module | Martin Fischer | |
2023-09-09 | refactor: merge token types with attr to new token module | Martin Fischer | |
2023-09-09 | chore: group public modules together | Martin Fischer | |
2023-09-09 | docs: stop referencing Emitter from token types | Martin Fischer | |
2023-09-05 | docs: fix heading style in changelog | Martin Fischer | |
2023-09-05 | fix(docs): Comment data_offset field was changed | Martin Fischer | |
I forgot to document this breaking change in e993f19c2b8ef00b32f17f9ed32306f3ceb21bc3. | |||
2023-09-03 | chore: bump version to 0.5.1v0.5.1 | Martin Fischer | |
2023-09-03 | chore: bump edition to 2021 | Martin Fischer | |
2023-09-03 | chore: add categories to Cargo.toml | Martin Fischer | |
2023-09-03 | docs: add spans example | Martin Fischer | |
2023-09-03 | feat: add Doctype::name_span | Martin Fischer | |
2023-09-03 | break!: make Doctype name field optional | Martin Fischer | |
2023-09-03 | refactor: make test_and_annotate test encoding-independence | Martin Fischer | |
2023-09-03 | docs: the span logic no longer assumes UTF-8 | Martin Fischer | |
2023-09-03 | fix!: make comment data spans encoding-independent | Martin Fischer | |
2023-09-03 | test: test comment data spans more thoroughly | Martin Fischer | |
2023-09-03 | fix: make doctype id spans encoding-independent | Martin Fischer | |
2023-09-03 | fix!: make set_self_closing encoding-independent | Martin Fischer | |
2023-09-03 | fix!: make attribute spans encoding-independent | Martin Fischer | |
2023-09-03 | fix!: make start/end tag name spans encoding-independent | Martin Fischer | |
2023-09-03 | fix: don't assume UTF-8 in machine/tokenizer | Martin Fischer | |
2023-09-03 | refactor: inline internal method only used once | Martin Fischer | |