Make paragraphs really fast

The approach that `micromark-js` takes is as follows: to parse a
paragraph, check whether each line starts with something else.
If it does, exit, otherwise continue.
That is slow, because our actual flow parser does similar things: the work was
being done twice.

To fix this, this commit introduces parsing each line of a paragraph separately.
And finally, when done with flow, combining adjacent paragraphs.

This same mechanism is reused for setext headings.

Additionally, this commit adds support for interrupting things (or not).
E.g., HTML (flow, complete) cannot interrupt paragraphs.
Definitions cannot interrupt paragraphs, and connect be interrupted either,
but they can follow each other.

1 files changed, 9 insertions, 2 deletions

diff --git a/src/construct/code_indented.rs b/src/construct/code_indented.rs
index 99445b9..9bdfd71 100644
--- a/src/construct/code_indented.rs
+++ b/src/construct/code_indented.rs
@@ -59,8 +59,13 @@ use crate::tokenizer::{Code, State, StateFnResult, TokenType, Tokenizer};
 /// > filled line (that it has a non-whitespace character), because blank lines
 /// > are parsed already, so we never run into that.
 pub fn start(tokenizer: &mut Tokenizer, code: Code) -> StateFnResult {
-    tokenizer.enter(TokenType::CodeIndented);
-    tokenizer.go(space_or_tab_min_max(TAB_SIZE, TAB_SIZE), at_break)(tokenizer, code)
+    // Do not interrupt paragraphs.
+    if tokenizer.interrupt {
+        (State::Nok, None)
+    } else {
+        tokenizer.enter(TokenType::CodeIndented);
+        tokenizer.go(space_or_tab_min_max(TAB_SIZE, TAB_SIZE), at_break)(tokenizer, code)
+    }
 }
 
 /// At a break.
@@ -110,6 +115,8 @@ fn content(tokenizer: &mut Tokenizer, code: Code) -> StateFnResult {
 /// ```
 fn after(tokenizer: &mut Tokenizer, code: Code) -> StateFnResult {
     tokenizer.exit(TokenType::CodeIndented);
+    // Feel free to interrupt.
+    tokenizer.interrupt = false;
     (State::Ok, Some(vec![code]))
 }