aboutsummaryrefslogtreecommitdiff
path: root/src/tokenizer.rs
AgeCommit message (Collapse)Author
2023-09-03chore: move ControlToken enum definition to machineMartin Fischer
2023-09-03fix!: add missing `R: Position<O>` boundsMartin Fischer
It doesn't make sense that you're able to construct a Tokenizer/NaiveParser that you're unable to iterate over.
2023-08-19feat: introduce NaiveParserMartin Fischer
2023-08-19break!: remove type param defaults from TokenizerMartin Fischer
2023-08-19docs: correct `a HTML` to `an HTML` and fix 2 typosMartin Fischer
2023-08-19docs: link multipage version of HTML specMartin Fischer
2023-08-19chore: use link reference definitions in MarkdownMartin Fischer
2023-08-19feat!: add span and offsets to DoctypeMartin Fischer
2023-08-19feat!: add all-inclusive spans to tagsMartin Fischer
Also more performant since we no longer have to update the name span on every Emitter::push_tag_name call.
2023-08-19fix: fix lots of position off-by-onesMartin Fischer
Previously the PosTrackingReader always mysteriously subtracted 1 from the current position ... this wasn't sound at all ... the machine just happens to often call `Tokenizer::unread_char` ... but not always. E.g. for proper comments it didn't which resulted in their offset and spans being off-by-one, which is fixed by this commit (see test_spans.rs).
2023-08-19refactor!: make Emitter generic over offset instead of readerMartin Fischer
Emitters should not have access to the reader at all. Also the current position of the reader, at the time an Emitted method is called, very much depends on machine implementation details such as if `Tokenizer::unread_char` is used. Having the Emitter methods take offsets lets the machine take care of providing the right offsets, as evidenced by the next commit.
2023-08-19chore: move type param bounds to where clauseMartin Fischer
2023-08-19refactor!: remove Span trait, just use RangeMartin Fischer
`std::mem::size_of::<Range<NoopOffset>>()` is 0 so there's no need to abstract over Range.
2023-08-19refactor!: make Position generic over offset typeMartin Fischer
Previously Span was generic over R just so that it could provide the method: fn from_reader(reader: &R) -> Self; and properly implementing that method again relied on R implementing the Position trait: impl<P: Position> Span<P> for Range<usize> { .. } which was a very roundabout and awkward way of doing things. It makes much more sense to make the Position trait generic over the return type of its method (which previously always had to be usize). Which lets us provide a blanket implementation: impl<R: Reader> Position<NoopOffset> for R { .. }
2023-08-19refactor!: remove current_is_appropriate_end_tag_token from EmitterMartin Fischer
2023-08-19refactor: proxy essential Emitter methods through TokenizerMartin Fischer
2023-08-19break!: stop re-exporting reader traits & typesMartin Fischer
This is primarily done to make the rustdoc more readable (by grouping Reader, IntoReader, StringReader and BufReadReader in the reader module). Ideally IntoReader is already implemented for your input type and you don't have to concern yourself with these traits / types at all.
2023-08-19break!: merge Tokenizer::new_with_emitter into Tokenizer::newMartin Fischer
The Tokenizer does not perform any state switching, since proper state switching requires a feedback loop between tokenization and DOM tree building. Using the Tokenizer directly therefore is a bit of a pitfall, since you might not expect it to e.g. tokenize `<script><b>` as: StartTag(StartTag { name: "script", .. }) StartTag(StartTag { name: "b", .. }) Since we don't want to make walking into pitfalls particularly easy, this commit changes the Tokenizer::new method so that you have to specify the Emitter. Since this makes new_with_emitter redundant it is removed.
2023-08-19docs: move note about Reader impls to Reader traitMartin Fischer
2023-08-19break!: remove InfallibleTokenizer in favor of Iterator::flattenMartin Fischer
2023-08-19break!: rename Readable to IntoReaderMartin Fischer
The trait of the standard library is also called IntoIterator and not Iterable.
2023-08-19fix(docs): remove outdated list of Readable implsMartin Fischer
dced8066f77f570dd3e396ec3570c71aa86c454e introduced a Readable impl for std::io::BufReader. Manually listing impls in a doc comment is a bad idea since such lists will just get out of date and there's no need for that since rustdoc automatically lists all implementations on the trait page.
2023-08-19break!: remove set_last_start_tag from EmitterMartin Fischer
2021-12-05spans: add spans to Token::ErrorMartin Fischer
2021-12-05spans: make Emitter generic over ReaderMartin Fischer
2021-12-05spans: make Emitter generic over SpanMartin Fischer
2021-12-05allow setting the Tokenizer to Data, PlainText, RcData, RawText and ↵Martin Fischer
ScriptData states
2021-12-05prepare for introduction of public State enumMartin Fischer
2021-11-28clarify what html5gum isn't, fix #5Markus Unterwaditzer
2021-11-27fix crash in try_read_stringMarkus Unterwaditzer
2021-11-27split up match-arms and tokenizer to isolate some tokenizer-internal stateMarkus Unterwaditzer
purpose: don't want to expose self.to_reconsume to the consume() method