| Age | Commit message (Collapse) | Author |
|---|
|
|
|
|
|
Previously the Token enum contained the offsets using the O generic
type parameter, which could be a usize if you're tracking offsets or
a zero-sized type if you didn't care about offsets. This commit moves
all the byte offset and syntax information to a new Trace enum,
which has several advantages:
* Traces can now easily be stored separately, while the tokens are
fed to the tree builder. (The tree builder only has to keep track
of which tree nodes originate from which tokens.)
* No needless generics for functions that take a token but don't
care about offsets (a tree construction implementation is bound
to have many of such functions).
* The FromIterator<(String, String)> impl for AttributeMap no longer
has to specify arbitrary values for the spans and the value_syntax).
* The PartialEq implementation of Token is now much more useful
(since it no longer includes all the offsets).
* The Debug formatting of Token is now more readable
(since it no longer includes all the offsets).
* Function pointers to functions accepting tokens are possible.
(Since function pointer types may not have generic parameters.)
|
|
|
|
The HTML spec specifies that the tokenizer emits character tokens.
That html5gum always emitted strings instead was probably just done
to make the token consumption more convenient. When it comes to tree
construction character tokens are however actually more convenient
than string tokens since the spec defines that specific character
tokens should be ignored in specific states (and character tokens
let us avoid string manipulation for these conditions).
This should also make the DefaultEmitter more performant for cases
where you don't actually need the strings at all (or only a few)
since it avoids string allocations. Though I haven't benchmarked it.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
With codespan_reporting an empty span shows up exactly like a
one-byte span, which is why I didn't notice this mistake earlier.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Conceptually the tokenizer emits tokens, which are then handled in the
tree construction stage (which this crate doesn't yet implement).
While the tokenizer can operate almost entirely based on its state
(which may be changed via Tokenizer::set_state) and its internal state,
there is the exception of the 'Markup declaration open state'[1], the third
condition of which depends on the "adjusted current node", which in turn
depends on the "stack of open elements" only known to the tree constructor.
In 82898967320f90116bbc686ab7ffc2f61ff456c4 I tried to address this
by adding the adjusted_current_node_present_and_not_in_html_namespace
method to the Emitter trait. What I missed was that adding this method
to the Emitter trait effectively crippled the composability of the API.
You should be able to do the following:
struct TreeConstructor<R, O> {
tokenizer: Tokenizer<R, O, SomeEmitter<O>>,
stack_of_open_elements: Vec<NodeId>,
// ...
}
However this doesn't work if the implementation of SomeEmitter
depends on the stack_of_open_elements field.
This commits remedies this oversight by removing this method and
instead making the Tokenizer yield values of a new Event enum:
enum Event<T> { Token(T), CdataOpen }
Event::CdataOpen signals that the new Tokenizer::handle_cdata_open
method has to be called, which accepts a CdataAction:
enum CdataAction { Cdata, BogusComment }
the variants of which correspond exactly to the possible outcomes
of the third condition of the 'Markup declaration open state'.
Removing this method also has the added benefit that the DefaultEmitter
is now again spec-compliant, which lets us expose it again in the next
commit in good conscience (previously it just hard-coded the method
implementation to return false, which is why I had removed the
DefaultEmitter from the public API in the last release).
[1]: https://html.spec.whatwg.org/multipage/parsing.html#markup-declaration-open-state
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Just a bit more succinct. And now rustdoc also no longer
cuts off the names of these Emitter methods in its sidebar.
|
|
|
|
Note that while making this breaking change, we're also swapping
the parameter order for more consistency so that the reader
parameter always comes last in Emitter methods.
|
|
This spares us two usizes per AttrInternal<Range<usize>>.
So on a 64 bit target where a usize is 8 bytes this spares
us 16 bytes of memory per attribute (if spans are enabled,
... for Token<()> this obviously doesn't change anything).
And the DefaultEmitter now also no longer has to update the
spans on each Emitter::push_attribute_(name|value) call.
The spans are now calculated on demand by the Attribute methods,
which is fine since the assumption is that API users are only
interested in a few specific spans (rather than all spans).
|
|
|
|
This has a number of benefits:
* it hides the implementation of the map
* it hides the type used for the map values
(which lets us e.g. change name_span to name_offset while still
being able to provide a convenient `Attribute::name_span` method.)
* it lets us provide convenience impls for the map
such as `FromIterator<(String, String)>`
|
|
This is done separately so that the following commit has a cleaner diff.
|
|
Also more performant since we no longer have to update
the name span on every Emitter::push_tag_name call.
|
|
|
|
Previously the PosTrackingReader always mysteriously subtracted 1
from the current position ... this wasn't sound at all ... the machine
just happens to often call `Tokenizer::unread_char` ... but not always.
E.g. for proper comments it didn't which resulted in their offset and
spans being off-by-one, which is fixed by this commit (see test_spans.rs).
|
|
Emitters should not have access to the reader at all. Also the
current position of the reader, at the time an Emitted method is
called, very much depends on machine implementation details such
as if `Tokenizer::unread_char` is used. Having the Emitter
methods take offsets lets the machine take care of providing
the right offsets, as evidenced by the next commit.
|
|
|
|
`std::mem::size_of::<Range<NoopOffset>>()` is 0
so there's no need to abstract over Range.
|
