Age | Commit message (Collapse) | Author | |
---|---|---|---|
2023-09-28 | chore: rename variable | Martin Fischer | |
2023-09-28 | feat: add offset to Trace::EndOfFile | Martin Fischer | |
2023-09-28 | feat: implement BasicEmitter | Martin Fischer | |
2023-09-28 | break!: move offsets out of Token | Martin Fischer | |
Previously the Token enum contained the offsets using the O generic type parameter, which could be a usize if you're tracking offsets or a zero-sized type if you didn't care about offsets. This commit moves all the byte offset and syntax information to a new Trace enum, which has several advantages: * Traces can now easily be stored separately, while the tokens are fed to the tree builder. (The tree builder only has to keep track of which tree nodes originate from which tokens.) * No needless generics for functions that take a token but don't care about offsets (a tree construction implementation is bound to have many of such functions). * The FromIterator<(String, String)> impl for AttributeMap no longer has to specify arbitrary values for the spans and the value_syntax). * The PartialEq implementation of Token is now much more useful (since it no longer includes all the offsets). * The Debug formatting of Token is now more readable (since it no longer includes all the offsets). * Function pointers to functions accepting tokens are possible. (Since function pointer types may not have generic parameters.) | |||
2023-09-28 | refactor: make TracingEmitter only work with usizes | Martin Fischer | |
2023-09-28 | chore: add BasicEmitter stub | Martin Fischer | |
2023-09-28 | break!: rename DefaultEmitter to TracingEmitter | Martin Fischer | |
2023-09-28 | refactor: simplify attribute logic in DefaultEmitter | Martin Fischer | |
2023-09-28 | refactor: clean up DefaultEmitter code | Martin Fischer | |
2023-09-28 | chore: reorder DefaultEmitter methods as well | Martin Fischer | |
See the previous commit. | |||
2023-09-28 | docs: put Emitter methods in a more logical order | Martin Fischer | |
2023-09-28 | refactor: make NaiveParser new* signatures more readable | Martin Fischer | |
2023-09-28 | break!: remove NaiveParser::new_with_spans | Martin Fischer | |
The method never really made much sense since you could just as well use NaiveParser::new. | |||
2023-09-28 | break!: make DefaultEmitter::emit_eof emit EndOfFile | Martin Fischer | |
2023-09-28 | break!: add Token::EndOfFile | Martin Fischer | |
While the end-of-file token can also be represented by None, this is less clear than having an explicit variant. Especially when it comes to tree construction, the spec explicitly has conditions named "An end-of-file token", and it's nice if the code for tree construction can match the spec text closely. | |||
2023-09-28 | break!: emit chars instead of strings | Martin Fischer | |
The HTML spec specifies that the tokenizer emits character tokens. That html5gum always emitted strings instead was probably just done to make the token consumption more convenient. When it comes to tree construction character tokens are however actually more convenient than string tokens since the spec defines that specific character tokens should be ignored in specific states (and character tokens let us avoid string manipulation for these conditions). This should also make the DefaultEmitter more performant for cases where you don't actually need the strings at all (or only a few) since it avoids string allocations. Though I haven't benchmarked it. | |||
2023-09-28 | refactor: proxy emit_string calls through utils | Martin Fischer | |
This is done separately so that the next commit has a cleaner diff. | |||
2023-09-28 | refactor: move machine impl details to machine module | Martin Fischer | |
This commit separates the public API (the "Tokenizer") from the internal implementation (the "Machine") to make the code more readable. | |||
2023-09-28 | refactor: move utils module under tokenizer::machine | Martin Fischer | |
2023-09-28 | refactor: only use InternalState re-export for feature-gated internal API | Martin Fischer | |
2023-09-28 | refactor: move machine module under tokenizer | Martin Fischer | |
2023-09-28 | //: remove wrong comment | Martin Fischer | |
Methods defined in another module don't have access to private fields, so the function could very well have been implemented as a method. | |||
2023-09-28 | break!: remove Token::Error | Martin Fischer | |
An error isn't a token (in general and also according to the spec). You shouldn't have to filter out errors when you're just interested in tokens but most importantly having errors in the Token enum is annoying when implementing tree construction (since the spec conditions exhaustively cover all Token variants except Token::Error). | |||
2023-09-28 | refactor: remove DefaultEmitter::push_error helper fn | Martin Fischer | |
2023-09-28 | break!: rename Emitter::emit_error to report_error | Martin Fischer | |
2023-09-28 | chore: move emit_error method up | Martin Fischer | |
2023-09-28 | feat: add blanket impl of Reader for boxed readers | Martin Fischer | |
2023-09-27 | break!: remove Emitter::pop_token, use Iterator instead | Martin Fischer | |
2023-09-27 | chore: move bounds to where clause | Martin Fischer | |
2023-09-12 | docs: move warning from DefaultEmitter to Tokenizer | Martin Fischer | |
2023-09-11 | chore: move DefaultEmitter to own module | Martin Fischer | |
2023-09-09 | refactor: merge token types with attr to new token module | Martin Fischer | |
2023-09-09 | chore: group public modules together | Martin Fischer | |
2023-09-09 | docs: stop referencing Emitter from token types | Martin Fischer | |
2023-09-03 | docs: add spans example | Martin Fischer | |
2023-09-03 | feat: add Doctype::name_span | Martin Fischer | |
2023-09-03 | break!: make Doctype name field optional | Martin Fischer | |
2023-09-03 | fix!: make comment data spans encoding-independent | Martin Fischer | |
2023-09-03 | fix: make doctype id spans encoding-independent | Martin Fischer | |
2023-09-03 | fix!: make set_self_closing encoding-independent | Martin Fischer | |
2023-09-03 | fix!: make attribute spans encoding-independent | Martin Fischer | |
2023-09-03 | fix!: make start/end tag name spans encoding-independent | Martin Fischer | |
2023-09-03 | fix: don't assume UTF-8 in machine/tokenizer | Martin Fischer | |
2023-09-03 | refactor: inline internal method only used once | Martin Fischer | |
2023-09-03 | fix!: make PosTrackingReader encoding-independent | Martin Fischer | |
While much of the span logic currently assumes UTF-8, we also want to support other character encodings, such as e.g. UTF-16 where characters can take up more or less bytes than in UTF-8. | |||
2023-09-03 | refactor: also use some_offset for start/end tags | Martin Fischer | |
2023-09-03 | fix!: calculate tag offsets in Tokenizer instead of Emitter impl | Martin Fischer | |
2023-09-03 | fix: too small char ref error spans | Martin Fischer | |
2023-09-03 | chore: rename doctype_offset field to some_offset | Martin Fischer | |
We'll reuse the field for another offset in the next commit. | |||
2023-09-03 | refactor: proxy init_doctype through Tokenizer | Martin Fischer | |