aboutsummaryrefslogtreecommitdiffstats
path: root/src/tokenizer.rs (unfollow)
Commit message (Collapse)AuthorFilesLines
2022-09-19Add support for parsing MDX ESM, expressionsLibravatar Titus Wormer1-0/+4
This commit adds support for hooks that lets a user integrate another parser with `micromark-rs`, to parse ESM and expressions according to a particular grammar (such as a programming language, typically JavaScript). For an example integrating with SWC, see `tests/test_utils/mod.rs`. The integration occurs with two functions passed in `options`: `mdx_expression_parse` and `mdx_esm_parse`. The can signal back to micromark when they are successful, whether there is an error at the end (in which case micromark will try to parse more), or whether there is a syntax error (in which case micromark will crash).
2022-09-14Fix to prefer flow over definitions, setext headingsLibravatar Titus Wormer1-5/+18
An undocumented part of CommonMark is how to deal with things in definition labels or definition titles (which both can span multiple lines). Can flow (or containers?) interrupt them? They can according to the `cmark` reference parser, so this was implemented here. This adds a new `Content` content type, which houses zero or more definitions, and then zero-or-one paragraphs. Content can be followed by a setext heading underline, which either turns into a setext heading when the content ends in a paragraph, or turns into the start of the following paragraph when it is followed by content that starts with a paragraph, or turns into a stray paragraph.
2022-09-12Fix containers piercing into indented codeLibravatar Titus Wormer1-1/+1
2022-09-09Refactor to move byte, char info to own fileLibravatar Titus Wormer1-17/+3
2022-09-07Fix debug messageLibravatar Titus Wormer1-1/+1
2022-09-07Add support for recoverable syntax errorsLibravatar Titus Wormer1-5/+12
2022-09-05Fix to improve debug loggingLibravatar Titus Wormer1-2/+16
2022-08-31Add support for GFM tablesLibravatar Titus Wormer1-3/+6
2022-08-26Add support for math (flow)Libravatar Titus Wormer1-0/+3
2022-08-25Add support for GFM footnotesLibravatar Titus Wormer1-1/+51
2022-08-22Fix lazy paragraph after definitionLibravatar Titus Wormer1-3/+3
2022-08-22Add support for GFM task list itemLibravatar Titus Wormer1-5/+5
2022-08-19Refactor to move more things to `util/`Libravatar Titus Wormer1-2/+1
2022-08-16Add `no_std + alloc`Libravatar Titus Wormer1-0/+1
2022-08-16Add support for frontmatterLibravatar Titus Wormer1-0/+5
2022-08-15Refactor to proof docs, grammarsLibravatar Titus Wormer1-44/+46
2022-08-15Add derive of `Eq` where possibleLibravatar Titus Wormer1-1/+1
2022-08-15Refactor to move `content` to `construct`Libravatar Titus Wormer1-5/+5
2022-08-12Refactor to improve enteringLibravatar Titus Wormer1-29/+22
2022-08-12Refactor to improve some namesLibravatar Titus Wormer1-12/+12
2022-08-12Refactor to improve docs of each functionLibravatar Titus Wormer1-5/+6
2022-08-11Refactor attempts to remove unneeded state nameLibravatar Titus Wormer1-12/+7
2022-08-11Refactor to handle definitions when parsingLibravatar Titus Wormer1-5/+9
2022-08-11Remove unneeded `balance` info on label startsLibravatar Titus Wormer1-21/+20
2022-08-11Remove boxes around resolversLibravatar Titus Wormer1-56/+42
2022-08-11Refactor to move some code to `event.rs`Libravatar Titus Wormer1-96/+38
2022-08-11Refactor to move some code to `state.rs`Libravatar Titus Wormer1-640/+7
2022-08-11Refactor internal docs, code style of tokenizerLibravatar Titus Wormer1-262/+256
2022-08-11Add improved container exit injectionLibravatar Titus Wormer1-13/+10
2022-08-10Refactor some code for document parsingLibravatar Titus Wormer1-12/+4
2022-08-10Remove boxes when calling statesLibravatar Titus Wormer1-354/+328
Interestingly, this cuts performance by 20%.
2022-08-10Add `State::Retry`Libravatar Titus Wormer1-11/+37
2022-08-10Rename `State::Fn` to `State::Next`Libravatar Titus Wormer1-23/+23
2022-08-09Refactor to reuse some tokenizer codeLibravatar Titus Wormer1-69/+58
2022-08-09Add peeking to unindented flow linesLibravatar Titus Wormer1-4/+2
2022-08-09Add support for passing `ok`, `nok` as separate states to attemptsLibravatar Titus Wormer1-123/+77
2022-08-09Rewrite algorithm to not pass around boxed functionsLibravatar Titus Wormer1-227/+785
* Pass state names from an enum around instead of boxed functions * Refactor to simplify attempts a lot * Use a subtokenizer for the the `document` content type
2022-08-02Refactor to remove most closuresLibravatar Titus Wormer1-3/+145
2022-08-01Add missing docs, refactor some codeLibravatar Titus Wormer1-57/+43
2022-07-29Refactor to work on bytes (`u8`)Libravatar Titus Wormer1-65/+61
2022-07-28Refactor to use `debug_assert`Libravatar Titus Wormer1-10/+10
2022-07-28Refactor to work on `char`sLibravatar Titus Wormer1-88/+192
Previously, a custom char implementation was used. This was easier to work with, as sometimes “virtual” characters are injected, or characters are ignored. This replaces that with working on actual `char`s. In the hope of in the future working on `u8`s, even. This simplifies the state machine somewhat, as only `\n` is fed, regardless of whether it was a CRLF, CR, or LF. It also feeds `' '` instead of virtual spaces. The BOM, if present, is now available as a `ByteOrderMark` event.
2022-07-26Refactor to simplify tokenizerLibravatar Titus Wormer1-123/+77
2022-07-26Remove unneeded parameterLibravatar Titus Wormer1-20/+16
2022-07-25Refactor to not pass codes aroundLibravatar Titus Wormer1-32/+26
2022-07-25Refactor to store less dataLibravatar Titus Wormer1-47/+43
2022-07-25Remove no longer needed field in `State::Ok`Libravatar Titus Wormer1-9/+9
2022-07-25Refactor to remove need for cloning codesLibravatar Titus Wormer1-33/+21
2022-07-25Improve performance w/ a single feed loopLibravatar Titus Wormer1-97/+90
2022-07-22Refactor to remove unneeded `Option`Libravatar Titus Wormer1-14/+18
generated by cgit (git 2.34.1) at 2024-11-22 11:53:03 +0000