aboutsummaryrefslogtreecommitdiffstats
path: root/src/tokenizer.rs (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Refactor to work on `char`sLibravatar Titus Wormer2022-07-281-88/+192
| | | | | | | | | | | | | | | Previously, a custom char implementation was used. This was easier to work with, as sometimes “virtual” characters are injected, or characters are ignored. This replaces that with working on actual `char`s. In the hope of in the future working on `u8`s, even. This simplifies the state machine somewhat, as only `\n` is fed, regardless of whether it was a CRLF, CR, or LF. It also feeds `' '` instead of virtual spaces. The BOM, if present, is now available as a `ByteOrderMark` event.
* Refactor to simplify tokenizerLibravatar Titus Wormer2022-07-261-123/+77
|
* Remove unneeded parameterLibravatar Titus Wormer2022-07-261-20/+16
|
* Refactor to not pass codes aroundLibravatar Titus Wormer2022-07-251-32/+26
|
* Refactor to store less dataLibravatar Titus Wormer2022-07-251-47/+43
|
* Remove no longer needed field in `State::Ok`Libravatar Titus Wormer2022-07-251-9/+9
|
* Refactor to remove need for cloning codesLibravatar Titus Wormer2022-07-251-33/+21
|
* Improve performance w/ a single feed loopLibravatar Titus Wormer2022-07-251-97/+90
|
* Refactor to remove unneeded `Option`Libravatar Titus Wormer2022-07-221-14/+18
|
* Refactor to use a single shared edit mapLibravatar Titus Wormer2022-07-221-16/+7
|
* Refactor to reserve some more vecsLibravatar Titus Wormer2022-07-221-5/+9
|
* Refactor to pass more references aroundLibravatar Titus Wormer2022-07-221-26/+39
|
* Refactor to remove unneeded tuples in every statesLibravatar Titus Wormer2022-07-221-60/+50
|
* Refactor to pass ints instead of vecs aroundLibravatar Titus Wormer2022-07-221-74/+47
|
* Refactor to move `index` field to `point`Libravatar Titus Wormer2022-07-211-19/+10
|
* Refactor to move some event fields to `link`Libravatar Titus Wormer2022-07-211-11/+24
|
* Refactor to share edit mapLibravatar Titus Wormer2022-07-201-2/+17
|
* Refactor to use less vecs for eventsLibravatar Titus Wormer2022-07-201-7/+5
|
* Remove a couple of clonesLibravatar Titus Wormer2022-07-191-3/+2
|
* Refactor to remove cloning in `edit_map`Libravatar Titus Wormer2022-07-191-0/+1
|
* Use `edit_map` in `subtokenize`Libravatar Titus Wormer2022-07-191-2/+0
|
* Remove an unneeded `HashMap`Libravatar Titus Wormer2022-07-191-15/+43
|
* Fix token that should be voidLibravatar Titus Wormer2022-07-181-1/+10
|
* Add docs and refactor listLibravatar Titus Wormer2022-07-151-1/+1
|
* Add docs and refactor document content typeLibravatar Titus Wormer2022-07-151-1/+10
|
* Add missing docs to tokenizerLibravatar Titus Wormer2022-07-151-14/+16
|
* Fix annoying bug around virtual spaces in containersLibravatar Titus Wormer2022-07-151-18/+22
|
* Add support for container stateLibravatar Titus Wormer2022-07-141-0/+10
| | | | | | * Fix to parse list item continuation based on how big the initial list item prefix was * Fix list items that start with blank lines
* Add support for lazy linesLibravatar Titus Wormer2022-07-111-1/+0
|
* Fix block quote bugsLibravatar Titus Wormer2022-07-111-5/+6
|
* Fix positional info on line ending after shiftLibravatar Titus Wormer2022-07-081-4/+17
|
* Fix closing of flow when exiting containersLibravatar Titus Wormer2022-07-081-14/+30
|
* Add support for concrete constructsLibravatar Titus Wormer2022-07-081-0/+4
|
* Refactor to move token types to `token`Libravatar Titus Wormer2022-07-071-1774/+6
|
* Fix some small to dosLibravatar Titus Wormer2022-07-071-1/+2
|
* Add docs for block quote tokensLibravatar Titus Wormer2022-07-071-1/+76
|
* Fix order of tokensLibravatar Titus Wormer2022-07-071-77/+77
|
* Add basic support for block quotesLibravatar Titus Wormer2022-07-071-12/+26
|
* Add support for `Flow` content typeLibravatar Titus Wormer2022-07-071-0/+2
|
* Add `go_until` to tokenizerLibravatar Titus Wormer2022-07-071-7/+37
|
* Refactor to do some to dosLibravatar Titus Wormer2022-07-051-18/+130
|
* Refactor code styleLibravatar Titus Wormer2022-07-051-78/+0
|
* Add support for unicode punctuationLibravatar Titus Wormer2022-07-041-1/+1
|
* Update list of todosLibravatar Titus Wormer2022-07-041-2/+0
|
* Add support for attention (emphasis, strong)Libravatar Titus Wormer2022-07-041-0/+9
|
* Make paragraphs really fastLibravatar Titus Wormer2022-07-011-0/+3
| | | | | | | | | | | | | | | | | | The approach that `micromark-js` takes is as follows: to parse a paragraph, check whether each line starts with something else. If it does, exit, otherwise continue. That is slow, because our actual flow parser does similar things: the work was being done twice. To fix this, this commit introduces parsing each line of a paragraph separately. And finally, when done with flow, combining adjacent paragraphs. This same mechanism is reused for setext headings. Additionally, this commit adds support for interrupting things (or not). E.g., HTML (flow, complete) cannot interrupt paragraphs. Definitions cannot interrupt paragraphs, and connect be interrupted either, but they can follow each other.
* Add support for trimming whitespace around string, textLibravatar Titus Wormer2022-06-301-1/+8
| | | | | | | | | | | | | This commit introduces trimming initial and final whitespace around the whole string or text, or around line endings inside that string or text. * Add `register_resolver_before`, to run resolvers earlier than others, used for labels * Add resolver to merge `data` events, which are the most frequent token that occurs, and can happen adjacently. In `micromark-js` this sped up parsing a lot * Fix a bug where a virtual space was not seen as an okay event * Refactor to enable all turned off whitespace tests
* Add docs on resolver, clean feedLibravatar Titus Wormer2022-06-301-77/+97
|
* Refactor to reorder token typesLibravatar Titus Wormer2022-06-301-303/+303
|
* Add docs to image, link, and other media tokensLibravatar Titus Wormer2022-06-301-16/+434
|
generated by cgit (git 2.34.1) at 2025-02-16 20:42:08 +0000