aboutsummaryrefslogtreecommitdiffstats
path: root/src/tokenizer.rs (unfollow)
Commit message (Collapse)AuthorFilesLines
2022-08-10Refactor some code for document parsingLibravatar Titus Wormer1-12/+4
2022-08-10Remove boxes when calling statesLibravatar Titus Wormer1-354/+328
Interestingly, this cuts performance by 20%.
2022-08-10Add `State::Retry`Libravatar Titus Wormer1-11/+37
2022-08-10Rename `State::Fn` to `State::Next`Libravatar Titus Wormer1-23/+23
2022-08-09Refactor to reuse some tokenizer codeLibravatar Titus Wormer1-69/+58
2022-08-09Add peeking to unindented flow linesLibravatar Titus Wormer1-4/+2
2022-08-09Add support for passing `ok`, `nok` as separate states to attemptsLibravatar Titus Wormer1-123/+77
2022-08-09Rewrite algorithm to not pass around boxed functionsLibravatar Titus Wormer1-227/+785
* Pass state names from an enum around instead of boxed functions * Refactor to simplify attempts a lot * Use a subtokenizer for the the `document` content type
2022-08-02Refactor to remove most closuresLibravatar Titus Wormer1-3/+145
2022-08-01Add missing docs, refactor some codeLibravatar Titus Wormer1-57/+43
2022-07-29Refactor to work on bytes (`u8`)Libravatar Titus Wormer1-65/+61
2022-07-28Refactor to use `debug_assert`Libravatar Titus Wormer1-10/+10
2022-07-28Refactor to work on `char`sLibravatar Titus Wormer1-88/+192
Previously, a custom char implementation was used. This was easier to work with, as sometimes “virtual” characters are injected, or characters are ignored. This replaces that with working on actual `char`s. In the hope of in the future working on `u8`s, even. This simplifies the state machine somewhat, as only `\n` is fed, regardless of whether it was a CRLF, CR, or LF. It also feeds `' '` instead of virtual spaces. The BOM, if present, is now available as a `ByteOrderMark` event.
2022-07-26Refactor to simplify tokenizerLibravatar Titus Wormer1-123/+77
2022-07-26Remove unneeded parameterLibravatar Titus Wormer1-20/+16
2022-07-25Refactor to not pass codes aroundLibravatar Titus Wormer1-32/+26
2022-07-25Refactor to store less dataLibravatar Titus Wormer1-47/+43
2022-07-25Remove no longer needed field in `State::Ok`Libravatar Titus Wormer1-9/+9
2022-07-25Refactor to remove need for cloning codesLibravatar Titus Wormer1-33/+21
2022-07-25Improve performance w/ a single feed loopLibravatar Titus Wormer1-97/+90
2022-07-22Refactor to remove unneeded `Option`Libravatar Titus Wormer1-14/+18
2022-07-22Refactor to use a single shared edit mapLibravatar Titus Wormer1-16/+7
2022-07-22Refactor to reserve some more vecsLibravatar Titus Wormer1-5/+9
2022-07-22Refactor to pass more references aroundLibravatar Titus Wormer1-26/+39
2022-07-22Refactor to remove unneeded tuples in every statesLibravatar Titus Wormer1-60/+50
2022-07-22Refactor to pass ints instead of vecs aroundLibravatar Titus Wormer1-74/+47
2022-07-21Refactor to move `index` field to `point`Libravatar Titus Wormer1-19/+10
2022-07-21Refactor to move some event fields to `link`Libravatar Titus Wormer1-11/+24
2022-07-20Refactor to share edit mapLibravatar Titus Wormer1-2/+17
2022-07-20Refactor to use less vecs for eventsLibravatar Titus Wormer1-7/+5
2022-07-19Remove a couple of clonesLibravatar Titus Wormer1-3/+2
2022-07-19Refactor to remove cloning in `edit_map`Libravatar Titus Wormer1-0/+1
2022-07-19Use `edit_map` in `subtokenize`Libravatar Titus Wormer1-2/+0
2022-07-19Remove an unneeded `HashMap`Libravatar Titus Wormer1-15/+43
2022-07-18Fix token that should be voidLibravatar Titus Wormer1-1/+10
2022-07-15Add docs and refactor listLibravatar Titus Wormer1-1/+1
2022-07-15Add docs and refactor document content typeLibravatar Titus Wormer1-1/+10
2022-07-15Add missing docs to tokenizerLibravatar Titus Wormer1-14/+16
2022-07-15Fix annoying bug around virtual spaces in containersLibravatar Titus Wormer1-18/+22
2022-07-14Add support for container stateLibravatar Titus Wormer1-0/+10
* Fix to parse list item continuation based on how big the initial list item prefix was * Fix list items that start with blank lines
2022-07-11Add support for lazy linesLibravatar Titus Wormer1-1/+0
2022-07-11Fix block quote bugsLibravatar Titus Wormer1-5/+6
2022-07-08Fix positional info on line ending after shiftLibravatar Titus Wormer1-4/+17
2022-07-08Fix closing of flow when exiting containersLibravatar Titus Wormer1-14/+30
2022-07-08Add support for concrete constructsLibravatar Titus Wormer1-0/+4
2022-07-07Refactor to move token types to `token`Libravatar Titus Wormer1-1774/+6
2022-07-07Fix some small to dosLibravatar Titus Wormer1-1/+2
2022-07-07Add docs for block quote tokensLibravatar Titus Wormer1-1/+76
2022-07-07Fix order of tokensLibravatar Titus Wormer1-77/+77
2022-07-07Add basic support for block quotesLibravatar Titus Wormer1-12/+26
generated by cgit (git 2.34.1) at 2025-04-30 14:17:20 +0000