diff options
| author | Martin Fischer <martin@push-f.com> | 2023-09-01 14:59:08 +0200 |
|---|---|---|
| committer | Martin Fischer <martin@push-f.com> | 2023-09-03 23:00:05 +0200 |
| commit | 153acbee5650b9b006c7de215fb1421a64516fb4 (patch) | |
| tree | ed363c6e00035cb227b183170bbd42495c84c0ca /integration_tests/tests/tokenizer.rs | |
| parent | eea8c60fba5962d5575c8515b837fb6304f73c9d (diff) | |
fix!: make PosTrackingReader encoding-independent
While much of the span logic currently assumes UTF-8, we also
want to support other character encodings, such as e.g. UTF-16
where characters can take up more or less bytes than in UTF-8.
Diffstat (limited to 'integration_tests/tests/tokenizer.rs')
0 files changed, 0 insertions, 0 deletions