aboutsummaryrefslogtreecommitdiffstats
path: root/src/tokenizer.rs (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Refactor to reuse some tokenizer codeLibravatar Titus Wormer2022-08-091-69/+58
|
* Add peeking to unindented flow linesLibravatar Titus Wormer2022-08-091-4/+2
|
* Add support for passing `ok`, `nok` as separate states to attemptsLibravatar Titus Wormer2022-08-091-123/+77
|
* Rewrite algorithm to not pass around boxed functionsLibravatar Titus Wormer2022-08-091-227/+785
| | | | | | * Pass state names from an enum around instead of boxed functions * Refactor to simplify attempts a lot * Use a subtokenizer for the the `document` content type
* Refactor to remove most closuresLibravatar Titus Wormer2022-08-021-3/+145
|
* Add missing docs, refactor some codeLibravatar Titus Wormer2022-08-011-57/+43
|
* Refactor to work on bytes (`u8`)Libravatar Titus Wormer2022-07-291-65/+61
|
* Refactor to use `debug_assert`Libravatar Titus Wormer2022-07-281-10/+10
|
* Refactor to work on `char`sLibravatar Titus Wormer2022-07-281-88/+192
| | | | | | | | | | | | | | | Previously, a custom char implementation was used. This was easier to work with, as sometimes “virtual” characters are injected, or characters are ignored. This replaces that with working on actual `char`s. In the hope of in the future working on `u8`s, even. This simplifies the state machine somewhat, as only `\n` is fed, regardless of whether it was a CRLF, CR, or LF. It also feeds `' '` instead of virtual spaces. The BOM, if present, is now available as a `ByteOrderMark` event.
* Refactor to simplify tokenizerLibravatar Titus Wormer2022-07-261-123/+77
|
* Remove unneeded parameterLibravatar Titus Wormer2022-07-261-20/+16
|
* Refactor to not pass codes aroundLibravatar Titus Wormer2022-07-251-32/+26
|
* Refactor to store less dataLibravatar Titus Wormer2022-07-251-47/+43
|
* Remove no longer needed field in `State::Ok`Libravatar Titus Wormer2022-07-251-9/+9
|
* Refactor to remove need for cloning codesLibravatar Titus Wormer2022-07-251-33/+21
|
* Improve performance w/ a single feed loopLibravatar Titus Wormer2022-07-251-97/+90
|
* Refactor to remove unneeded `Option`Libravatar Titus Wormer2022-07-221-14/+18
|
* Refactor to use a single shared edit mapLibravatar Titus Wormer2022-07-221-16/+7
|
* Refactor to reserve some more vecsLibravatar Titus Wormer2022-07-221-5/+9
|
* Refactor to pass more references aroundLibravatar Titus Wormer2022-07-221-26/+39
|
* Refactor to remove unneeded tuples in every statesLibravatar Titus Wormer2022-07-221-60/+50
|
* Refactor to pass ints instead of vecs aroundLibravatar Titus Wormer2022-07-221-74/+47
|
* Refactor to move `index` field to `point`Libravatar Titus Wormer2022-07-211-19/+10
|
* Refactor to move some event fields to `link`Libravatar Titus Wormer2022-07-211-11/+24
|
* Refactor to share edit mapLibravatar Titus Wormer2022-07-201-2/+17
|
* Refactor to use less vecs for eventsLibravatar Titus Wormer2022-07-201-7/+5
|
* Remove a couple of clonesLibravatar Titus Wormer2022-07-191-3/+2
|
* Refactor to remove cloning in `edit_map`Libravatar Titus Wormer2022-07-191-0/+1
|
* Use `edit_map` in `subtokenize`Libravatar Titus Wormer2022-07-191-2/+0
|
* Remove an unneeded `HashMap`Libravatar Titus Wormer2022-07-191-15/+43
|
* Fix token that should be voidLibravatar Titus Wormer2022-07-181-1/+10
|
* Add docs and refactor listLibravatar Titus Wormer2022-07-151-1/+1
|
* Add docs and refactor document content typeLibravatar Titus Wormer2022-07-151-1/+10
|
* Add missing docs to tokenizerLibravatar Titus Wormer2022-07-151-14/+16
|
* Fix annoying bug around virtual spaces in containersLibravatar Titus Wormer2022-07-151-18/+22
|
* Add support for container stateLibravatar Titus Wormer2022-07-141-0/+10
| | | | | | * Fix to parse list item continuation based on how big the initial list item prefix was * Fix list items that start with blank lines
* Add support for lazy linesLibravatar Titus Wormer2022-07-111-1/+0
|
* Fix block quote bugsLibravatar Titus Wormer2022-07-111-5/+6
|
* Fix positional info on line ending after shiftLibravatar Titus Wormer2022-07-081-4/+17
|
* Fix closing of flow when exiting containersLibravatar Titus Wormer2022-07-081-14/+30
|
* Add support for concrete constructsLibravatar Titus Wormer2022-07-081-0/+4
|
* Refactor to move token types to `token`Libravatar Titus Wormer2022-07-071-1774/+6
|
* Fix some small to dosLibravatar Titus Wormer2022-07-071-1/+2
|
* Add docs for block quote tokensLibravatar Titus Wormer2022-07-071-1/+76
|
* Fix order of tokensLibravatar Titus Wormer2022-07-071-77/+77
|
* Add basic support for block quotesLibravatar Titus Wormer2022-07-071-12/+26
|
* Add support for `Flow` content typeLibravatar Titus Wormer2022-07-071-0/+2
|
* Add `go_until` to tokenizerLibravatar Titus Wormer2022-07-071-7/+37
|
* Refactor to do some to dosLibravatar Titus Wormer2022-07-051-18/+130
|
* Refactor code styleLibravatar Titus Wormer2022-07-051-78/+0
|