aboutsummaryrefslogtreecommitdiffstats
path: root/src/tokenizer.rs (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Move `log` to optional dependenciesLibravatar pinkforest(she/her)2023-02-101-1/+20
| | | Closes GH-48.
* Refactor some code to improve coverageLibravatar Titus Wormer2022-10-131-14/+14
|
* Add support for parsing MDX ESM, expressionsLibravatar Titus Wormer2022-09-191-0/+4
| | | | | | | | | | | | | | | | This commit adds support for hooks that lets a user integrate another parser with `micromark-rs`, to parse ESM and expressions according to a particular grammar (such as a programming language, typically JavaScript). For an example integrating with SWC, see `tests/test_utils/mod.rs`. The integration occurs with two functions passed in `options`: `mdx_expression_parse` and `mdx_esm_parse`. The can signal back to micromark when they are successful, whether there is an error at the end (in which case micromark will try to parse more), or whether there is a syntax error (in which case micromark will crash).
* Fix to prefer flow over definitions, setext headingsLibravatar Titus Wormer2022-09-141-5/+18
| | | | | | | | | | | | | | An undocumented part of CommonMark is how to deal with things in definition labels or definition titles (which both can span multiple lines). Can flow (or containers?) interrupt them? They can according to the `cmark` reference parser, so this was implemented here. This adds a new `Content` content type, which houses zero or more definitions, and then zero-or-one paragraphs. Content can be followed by a setext heading underline, which either turns into a setext heading when the content ends in a paragraph, or turns into the start of the following paragraph when it is followed by content that starts with a paragraph, or turns into a stray paragraph.
* Fix containers piercing into indented codeLibravatar Titus Wormer2022-09-121-1/+1
|
* Refactor to move byte, char info to own fileLibravatar Titus Wormer2022-09-091-17/+3
|
* Fix debug messageLibravatar Titus Wormer2022-09-071-1/+1
|
* Add support for recoverable syntax errorsLibravatar Titus Wormer2022-09-071-5/+12
|
* Fix to improve debug loggingLibravatar Titus Wormer2022-09-051-2/+16
|
* Add support for GFM tablesLibravatar Titus Wormer2022-08-311-3/+6
|
* Add support for math (flow)Libravatar Titus Wormer2022-08-261-0/+3
|
* Add support for GFM footnotesLibravatar Titus Wormer2022-08-251-1/+51
|
* Fix lazy paragraph after definitionLibravatar Titus Wormer2022-08-221-3/+3
|
* Add support for GFM task list itemLibravatar Titus Wormer2022-08-221-5/+5
|
* Refactor to move more things to `util/`Libravatar Titus Wormer2022-08-191-2/+1
|
* Add `no_std + alloc`Libravatar Titus Wormer2022-08-161-0/+1
|
* Add support for frontmatterLibravatar Titus Wormer2022-08-161-0/+5
|
* Refactor to proof docs, grammarsLibravatar Titus Wormer2022-08-151-44/+46
|
* Add derive of `Eq` where possibleLibravatar Titus Wormer2022-08-151-1/+1
|
* Refactor to move `content` to `construct`Libravatar Titus Wormer2022-08-151-5/+5
|
* Refactor to improve enteringLibravatar Titus Wormer2022-08-121-29/+22
|
* Refactor to improve some namesLibravatar Titus Wormer2022-08-121-12/+12
|
* Refactor to improve docs of each functionLibravatar Titus Wormer2022-08-121-5/+6
|
* Refactor attempts to remove unneeded state nameLibravatar Titus Wormer2022-08-111-12/+7
|
* Refactor to handle definitions when parsingLibravatar Titus Wormer2022-08-111-5/+9
|
* Remove unneeded `balance` info on label startsLibravatar Titus Wormer2022-08-111-21/+20
|
* Remove boxes around resolversLibravatar Titus Wormer2022-08-111-56/+42
|
* Refactor to move some code to `event.rs`Libravatar Titus Wormer2022-08-111-96/+38
|
* Refactor to move some code to `state.rs`Libravatar Titus Wormer2022-08-111-640/+7
|
* Refactor internal docs, code style of tokenizerLibravatar Titus Wormer2022-08-111-262/+256
|
* Add improved container exit injectionLibravatar Titus Wormer2022-08-111-13/+10
|
* Refactor some code for document parsingLibravatar Titus Wormer2022-08-101-12/+4
|
* Remove boxes when calling statesLibravatar Titus Wormer2022-08-101-354/+328
| | | | Interestingly, this cuts performance by 20%.
* Add `State::Retry`Libravatar Titus Wormer2022-08-101-11/+37
|
* Rename `State::Fn` to `State::Next`Libravatar Titus Wormer2022-08-101-23/+23
|
* Refactor to reuse some tokenizer codeLibravatar Titus Wormer2022-08-091-69/+58
|
* Add peeking to unindented flow linesLibravatar Titus Wormer2022-08-091-4/+2
|
* Add support for passing `ok`, `nok` as separate states to attemptsLibravatar Titus Wormer2022-08-091-123/+77
|
* Rewrite algorithm to not pass around boxed functionsLibravatar Titus Wormer2022-08-091-227/+785
| | | | | | * Pass state names from an enum around instead of boxed functions * Refactor to simplify attempts a lot * Use a subtokenizer for the the `document` content type
* Refactor to remove most closuresLibravatar Titus Wormer2022-08-021-3/+145
|
* Add missing docs, refactor some codeLibravatar Titus Wormer2022-08-011-57/+43
|
* Refactor to work on bytes (`u8`)Libravatar Titus Wormer2022-07-291-65/+61
|
* Refactor to use `debug_assert`Libravatar Titus Wormer2022-07-281-10/+10
|
* Refactor to work on `char`sLibravatar Titus Wormer2022-07-281-88/+192
| | | | | | | | | | | | | | | Previously, a custom char implementation was used. This was easier to work with, as sometimes “virtual” characters are injected, or characters are ignored. This replaces that with working on actual `char`s. In the hope of in the future working on `u8`s, even. This simplifies the state machine somewhat, as only `\n` is fed, regardless of whether it was a CRLF, CR, or LF. It also feeds `' '` instead of virtual spaces. The BOM, if present, is now available as a `ByteOrderMark` event.
* Refactor to simplify tokenizerLibravatar Titus Wormer2022-07-261-123/+77
|
* Remove unneeded parameterLibravatar Titus Wormer2022-07-261-20/+16
|
* Refactor to not pass codes aroundLibravatar Titus Wormer2022-07-251-32/+26
|
* Refactor to store less dataLibravatar Titus Wormer2022-07-251-47/+43
|
* Remove no longer needed field in `State::Ok`Libravatar Titus Wormer2022-07-251-9/+9
|
* Refactor to remove need for cloning codesLibravatar Titus Wormer2022-07-251-33/+21
|
generated by cgit (git 2.34.1) at 2024-11-21 23:44:57 +0000