| Age | Commit message (Collapse) | Author |
|---|
|
This bug resulted in e.g. "<script></SCRI" being wrongly tokenized as:
StartTag(StartTag { name: "script", self_closing: false, attributes: {} })
Char('<')
Char('/')
Char('s')
Char('c')
Char('r')
Char('i')
EndOfFile
Note that the Char tokens should be uppercase. (This bug could only be
observed when properly doing state switching via tree construction.)
|
|
See the previous commit.
|
|
Which action the tokenizer takes depending on whether or not an
adjusted current node is present but not in the HTML namespace,
is an implementation detail and shouldn't be exposed in the API.
|
|
|
|
|
|
|
|
|
|
Previously the Token enum contained the offsets using the O generic
type parameter, which could be a usize if you're tracking offsets or
a zero-sized type if you didn't care about offsets. This commit moves
all the byte offset and syntax information to a new Trace enum,
which has several advantages:
* Traces can now easily be stored separately, while the tokens are
fed to the tree builder. (The tree builder only has to keep track
of which tree nodes originate from which tokens.)
* No needless generics for functions that take a token but don't
care about offsets (a tree construction implementation is bound
to have many of such functions).
* The FromIterator<(String, String)> impl for AttributeMap no longer
has to specify arbitrary values for the spans and the value_syntax).
* The PartialEq implementation of Token is now much more useful
(since it no longer includes all the offsets).
* The Debug formatting of Token is now more readable
(since it no longer includes all the offsets).
* Function pointers to functions accepting tokens are possible.
(Since function pointer types may not have generic parameters.)
|
|
This is done separately so that the next commit has a cleaner diff.
|
|
This commit separates the public API (the "Tokenizer")
from the internal implementation (the "Machine")
to make the code more readable.
|
|
|
|
|
|
|
