Add code of conduct, improve docs

4 files changed, 352 insertions, 171 deletions

diff --git a/.github/code-of-conduct.md b/.github/code-of-conduct.md
new file mode 100644
index 0000000..8e24a76
--- /dev/null
+++ b/.github/code-of-conduct.md
@@ -0,0 +1,136 @@
+# Code of conduct
+
+## Our pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, caste, color, religion, or sexual
+identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming,
+diverse, inclusive, and healthy community.
+
+## Our standards
+
+Examples of behavior that contributes to a positive environment for our
+community include:
+
+*   Demonstrating empathy and kindness toward other people
+*   Being respectful of differing opinions, viewpoints, and experiences
+*   Giving and gracefully accepting constructive feedback
+*   Accepting responsibility and apologizing to those affected by our mistakes,
+    and learning from the experience
+*   Focusing on what is best not just for us as individuals, but for the overall
+    community
+
+Examples of unacceptable behavior include:
+
+*   The use of sexualized language or imagery, and sexual attention or advances
+    of any kind
+*   Trolling, insulting or derogatory comments, and personal or political
+    attacks
+*   Public or private harassment
+*   Publishing others’ private information, such as a physical or email address,
+    without their explicit permission
+*   Other conduct which could reasonably be considered inappropriate in a
+    professional setting
+
+## Enforcement responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of
+acceptable behavior and will take appropriate and fair corrective action in
+response to any behavior that they deem inappropriate, threatening, offensive,
+or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this code of conduct, and will communicate reasons for moderation
+decisions when appropriate.
+
+## Scope
+
+This code of conduct applies within all community spaces, and also applies when
+an individual is officially representing the community in public spaces.
+Examples of representing our community include using an official email address,
+posting via an official social media account, or acting as an appointed
+representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the community leaders responsible for enforcement at
+`tituswormer@gmail.com`.
+All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the
+reporter of any incident.
+
+## Enforcement guidelines
+
+Community leaders will follow these guidelines in determining
+the consequences for any action they deem in violation of this code of conduct:
+
+### 1. Correction
+
+**Community impact**: Use of inappropriate language or other behavior deemed
+unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing
+clarity around the nature of the violation and an explanation of why the
+behavior was inappropriate.
+A public apology may be requested.
+
+### 2. Warning
+
+**Community impact**: A violation through a single incident or series of
+actions.
+
+**Consequence**: A warning with consequences for continued behavior.
+No interaction with the people involved, including unsolicited interaction with
+those enforcing the code of conduct, for a specified period of time.
+This includes avoiding interactions in community spaces as well as external
+channels like social media.
+Violating these terms may lead to a temporary or permanent ban.
+
+### 3. Temporary ban
+
+**Community impact**: A serious violation of community standards, including
+sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public
+communication with the community for a specified period of time.
+No public or private interaction with the people involved, including
+unsolicited interaction with those enforcing the code of conduct, is allowed
+during this period.
+Violating these terms may lead to a permanent ban.
+
+### 4. Permanent ban
+
+**Community impact**: Demonstrating a pattern of violation of community
+standards, including sustained inappropriate behavior, harassment of an
+individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the
+community.
+
+## Attribution
+
+This code of conduct is adapted from the [contributor covenant][homepage],
+version 2.1, available at
+[`contributor-covenant.org/version/2/1/code_of_conduct.html`][v2.1].
+
+Community impact guidelines were inspired by
+[Mozilla’s code of conduct enforcement ladder][mozilla-coc].
+
+For answers to common questions about this code of conduct, see the FAQ at
+[`contributor-covenant.org/faq`][faq].
+
+[homepage]: https://www.contributor-covenant.org
+
+[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
+
+[mozilla-coc]: https://github.com/mozilla/inclusion
+
+[faq]: https://www.contributor-covenant.org/faq
diff --git a/.github/contribute.md b/.github/contribute.md
index 18ef1ee..a59fe01 100644
--- a/.github/contribute.md
+++ b/.github/contribute.md
@@ -1,5 +1,9 @@
 # Contribute
 
+> 👉 **Important**: this project has a [code of conduct][coc].
+> By interacting with this repository and community you agree to abide by its
+> terms.
+
 This article explains how to contribute.
 Please read through the following guidelines.
 
@@ -33,34 +37,34 @@ See [Project][] for more info.
 
 ## Submitting an issue
 
-- the issue tracker is for issues, discussions are for for questions
-- search the issue tracker (including closed issues) before opening a new
-  issue
-- ensure you’re using the latest versions of packages and other tools
-- use a clear and descriptive title
-- include as much information as possible: steps to reproduce the issue,
-  error message, version, operating system, etcetera
-- the more time you put into an issue, the better help you can get
-- the best issue report is a [failing test][unit-test] proving it
+*   the issue tracker is for issues, discussions are for questions
+*   search the issue tracker (including closed issues) before opening a new
+    issue
+*   ensure you’re using the latest versions of packages and other tools
+*   use a clear and descriptive title
+*   include as much information as possible: steps to reproduce the issue,
+    error message, version, operating system, etcetera
+*   the more time you put into an issue, the better help you can get
+*   the best issue report is a [failing test][unit-test] proving it
 
 ## Submitting a pull request
 
-- run `cargo fmt` and `cargo test` locally to format and test your changes
-- non-trivial changes are often best discussed in an issue first, to prevent
-  you from doing unnecessary work
-- for ambitious tasks, you should try to get your work in front of the
-  community for feedback as soon as possible
-- new features should be accompanied by tests and documentation
-- don’t include unrelated changes
-- write a convincing description of why your pull request should land:
-  it’s your job to be convincing
+*   run `cargo fmt` and `cargo test` locally to format and test your changes
+*   non-trivial changes are often best discussed in an issue first, to prevent
+    you from doing unnecessary work
+*   for ambitious tasks, you should try to get your work in front of the
+    community for feedback as soon as possible
+*   new features should be accompanied by tests and documentation
+*   don’t include unrelated changes
+*   write a convincing description of why your pull request should land:
+    it’s your job to be convincing
 
 ## Resources
 
-- [how to contribute to open source](https://opensource.guide/how-to-contribute/)
-- [making your first contribution](https://medium.com/@vadimdemedes/making-your-first-contribution-de6576ddb190)
-- [using pull requests](https://help.github.com/articles/about-pull-requests/)
-- [GitHub help](https://help.github.com)
+*   [how to contribute to open source](https://opensource.guide/how-to-contribute/)
+*   [making your first contribution](https://medium.com/@vadimdemedes/making-your-first-contribution-de6576ddb190)
+*   [using pull requests](https://help.github.com/articles/about-pull-requests/)
+*   [GitHub help](https://help.github.com)
 
 ## License
 
@@ -69,10 +73,15 @@ See [Project][] for more info.
 <!-- Definitions -->
 
 [license]: https://creativecommons.org/licenses/by/4.0/
+
 [author]: https://wooorm.com
-[coc]: https://github.com/remarkjs/.github/blob/main/code-of-conduct.md
+
 [unit-test]: https://twitter.com/sindresorhus/status/579306280495357953
+
 [support]: support.md
-[collective]: https://opencollective.com/unified
+
+[coc]: code-of-conduct.md
+
 [sponsor]: https://github.com/wooorm/micromark-rs/#sponsor
+
 [project]: https://github.com/wooorm/micromark-rs/#project
diff --git a/.github/support.md b/.github/support.md
index 7fd182d..afc8aaf 100644
--- a/.github/support.md
+++ b/.github/support.md
@@ -1,5 +1,9 @@
 # Support
 
+> 👉 **Important**: this project has a [code of conduct][coc].
+> By interacting with this repository and community you agree to abide by its
+> terms.
+
 This article explains how and where to get help.
 Please read through the following guidelines.
 
@@ -12,23 +16,24 @@ Spend time framing questions and add links and resources.
 Spending the extra time up front helps save everyone time in the long run.
 Here are some tips:
 
-- see [this article on _How do I ask a good question_][how-to-ask] for help
-- [talk to a duck][rubberduck]!
-- don’t fall for the [XY problem][xy]
-- search to find out if a similar question has been asked
-- try to define what you need help with:
-  - is there something in particular you want to do?
-  - what problem are you encountering and what steps have you taken to try
-    and fix it?
-  - is there a concept you don’t understand?
-- provide sample code, if possible
-- screenshots can help, but if there’s important text such as code or error
-  messages in them, please also provide those as text
-- the more time you put into asking your question, the better we can help you
+*   see [*How do I ask a good question* by `StackOverflow`][how-to-ask] for a
+    good guide
+*   [talk to a duck][rubberduck]!
+*   don’t fall for the [XY problem][xy]
+*   search to find out if a similar question has been asked
+*   try to define what you need help with:
+    *   is there something in particular you want to do?
+    *   what problem are you encountering and what steps have you taken to try
+        and fix it?
+    *   is there a concept you don’t understand?
+*   provide sample code, if possible
+*   screenshots can help, but if there’s important text such as code or error
+    messages in them, please also provide those as text
+*   the more time you put into asking your question, the better we can help you
 
 ## Contributions
 
-See [`contributing.md`][contributing] on how to contribute.
+See [`contribute.md`][contribute] on how to contribute.
 
 ## License
 
@@ -37,10 +42,17 @@ See [`contributing.md`][contributing] on how to contribute.
 <!-- Definitions -->
 
 [license]: https://creativecommons.org/licenses/by/4.0/
+
 [author]: https://wooorm.com
-[coc]: https://github.com/remarkjs/.github/blob/main/code-of-conduct.md
+
 [rubberduck]: https://rubberduckdebugging.com
+
 [xy]: https://meta.stackexchange.com/questions/66377/what-is-the-xy-problem/66378#66378
-[chat]: https://github.com/remarkjs/remark/discussions
-[contributing]: contributing.md
+
+[chat]: https://github.com/wooorm/micromark-rs/discussions
+
+[contribute]: contribute.md
+
+[coc]: code-of-conduct.md
+
 [how-to-ask]: https://stackoverflow.com/help/how-to-ask
diff --git a/readme.md b/readme.md
index 8bbbbe2..83d861b 100644
--- a/readme.md
+++ b/readme.md
@@ -5,63 +5,74 @@
 <!-- <img align="right" width="106" height="106" alt="" src="https://raw.githubusercontent.com/wooorm/micromark-rs/14f1ad0/logo.svg?sanitize=true"> -->
 
 <!-- To do: enable badges when repo is public/published -->
+
 <!-- To do: link `Downloads`/`crate-badge` to `crate` instead of temporary site. -->
 
 <!-- [![Build][build-badge]][build] -->
+
 <!-- [![Downloads][crate-badge]][docs] -->
+
 <!-- [![Coverage][coverage-badge]][coverage] -->
 
-[![Sponsors][sponsors-badge]][opencollective]
-[![Backers][backers-badge]][opencollective]
 [![Chat][chat-badge]][chat]
 
-A [`CommonMark`][commonmark-spec] compliant markdown parser in [Rust][] with
-positional info, concrete tokens, and extensions.
+CommonMark compliant markdown parser in Rust with ASTs and extensions.
 
 ## Feature highlights
 
-- [x] **[compliant][commonmark]** (100% to CommonMark)
-- [x] **[extensions][]** (100% GFM, 100% MDX, frontmatter, math)
-- [x] **[ast][mdast]** (mdast)
-- [x] **[safe][security]** (100% safe rust, also 100% safe HTML by default)
-- [x] **[robust][test]** (2300+ tests, 100% coverage, fuzz testing)
+*   [x] **[compliant][commonmark]** (100% to CommonMark)
+*   [x] **[extensions][]** (100% GFM, 100% MDX, frontmatter, math)
+*   [x] **[safe][security]** (100% safe Rust, also 100% safe HTML by default)
+*   [x] **[robust][test]** (2300+ tests, 100% coverage, fuzz testing)
+*   [x] **[ast][mdast]** (mdast)
 
-It’s also `#![no_std]` + `alloc` and has tons of docs.
+## When should I use this?
 
-> 🐣 **Note**: coverage is currently within progress.
+*   If you *just* want to turn markdown into HTML (with maybe a few extensions)
+*   If you want to do *really complex things* with markdown
 
-## When to use this
+## What is this?
 
-- If you _just_ want to turn markdown into HTML (with maybe a few extensions)
-- If you want to do _really complex things_ with markdown
+micromark is an open source markdown parser written in Rust.
+It’s implemented as a state machine (`#![no_std]` + `alloc`) that emits
+concrete tokens, so that every byte is accounted for, with positional info.
+The API then exposes this information as an AST, which is easier to work with,
+or it compiles directly to HTML.
 
-See [§ Comparison][comparison] for more info
+While most markdown parsers work towards compliancy with CommonMark (or GFM),
+this project goes further by following how the reference parsers (`cmark`,
+`cmark-gfm`) work, which is confirmed with thousands of extra tests.
 
-## Intro
+Other than CommonMark and GFM, this project also supports common extensions
+to markdown such as MDX, math, and frontmatter.
 
-micromark is markdown parser in Rust.
-It uses a state machine to parse the entirety of markdown into concrete
-tokens.
-Its API compiles to HTML, but its parts are made to be used separately, so as to
-generate syntax trees or compile to other output formats.
-`micromark-rs` has a sibling in JavaScript, [`micromark-js`][micromark-js].
+## Questions
 
-<!-- To do: link to unified etc if this repo gets moved there? -->
-
-- to learn markdown, see this [cheatsheet and tutorial][cheat]
-- for questions, see [Discussions][chat]
-- to help, see [contribute][] or [sponsor][] below
+*   to learn markdown, see this [cheatsheet and tutorial][cheat]
+*   for the API, see the [crate docs][docs]
+*   for questions, see [Discussions][chat]
+*   to help, see [contribute][] or [sponsor][] below
 
 ## Contents
 
-- [Install](#install)
-- [Use](#use)
-- [API](#api)
-- [Extensions](#extensions)
-- [Examples](#examples)
-- [Markdown](#markdown)
-- [Project](#project)
-- [License](#license)
+*   [Install](#install)
+*   [Use](#use)
+*   [API](#api)
+*   [Extensions](#extensions)
+*   [Examples](#examples)
+    *   [Example: syntax highlighting code](#example-syntax-highlighting-code)
+*   [Markdown](#markdown)
+    *   [CommonMark](#commonmark)
+    *   [Grammar](#grammar)
+*   [Project](#project)
+    *   [Overview](#overview)
+    *   [File structure](#file-structure)
+    *   [Test](#test)
+    *   [Version](#version)
+    *   [Security](#security)
+    *   [Contribute](#contribute)
+    *   [Sponsor](#sponsor)
+*   [License](#license)
 
 ## Install
 
@@ -149,6 +160,7 @@ Root { children: [Heading { children: [Text { value: "Hey, ", position: Some(1:3
 [`micromark_to_mdast`](https://wooorm.com/micromark-rs/micromark/fn.micromark_to_mdast.html),
 [`Options`](https://wooorm.com/micromark-rs/micromark/struct.Options.html),
 and a few other structs and enums.
+
 See the [crate docs][docs] for more info.
 
 ## Extensions
@@ -157,29 +169,26 @@ micromark supports extensions to `CommonMark`.
 These extensions are maintained in this project.
 They are not enabled by default but can be turned on with options.
 
-- frontmatter
-- GFM
-  - autolink literal
-  - footnote
-  - strikethrough
-  - table
-  - tagfilter
-  - task list item
-- math
-- MDX
-  - ESM
-  - expressions
-  - JSX
+*   frontmatter
+*   GFM
+    *   autolink literal
+    *   footnote
+    *   strikethrough
+    *   table
+    *   tagfilter
+    *   task list item
+*   math
+*   MDX
+    *   ESM
+    *   expressions
+    *   JSX
 
 It is not a goal of this project to support lots of different extensions.
-It’s instead a goal to support incredibly common, somewhat standardized,
-extensions.
+It’s instead a goal to support very common and mostly standardized extensions.
 
 ## Examples
 
-<!-- To do: example section with more full-fledged examples, on GFM, math, frontmatter, etc. -->
-
-> 🚧 **To do**.
+<!-- To do: math example; syntax highlighting in Rust -->
 
 ### Example: syntax highlighting code
 
@@ -266,9 +275,10 @@ The code example in the markdown as HTML will first look like this:
 </code></pre>
 ```
 
-Opening that page in a browser, we’d see that being swapped with:
+Opening the document in a browser, we’d see it being swapped with:
 
 <!-- prettier-ignore -->
+
 ```html
 <pre><code class="language-js"><span class="pl-en">console</span>.<span class="pl-c1">log</span>(<span class="pl-s"><span class="pl-pds">'</span>it works!<span class="pl-pds">'</span></span>)
 </code></pre>
@@ -306,9 +316,10 @@ markdown = .*
 ```
 
 No, that’s [not a typo](http://trevorjim.com/a-specification-for-markdown/):
-markdown has no syntax errors; anything thrown at it renders _something_.
+markdown has no syntax errors; anything thrown at it renders *something*.
 
-For more practical examples of how things roughly work in BNF, see the module docs of each `src/construct`.
+For more practical examples of how things roughly work in BNF, see the module
+docs of each `src/construct`.
 
 ## Project
 
@@ -331,38 +342,32 @@ The process to parse markdown looks like this:
 
 The files in `src/` are as follows:
 
-- `construct/*.rs`
-  — CommonMark, GFM, and other extension constructs used in micromark
-- `util/*.rs`
-  — helpers often needed when parsing markdown
-- `event.rs`
-  — things with meaning happening somewhere
-- `lib.rs`
-  — core module
-- `mdast.rs`
-  — syntax tree
-- `parser.rs`
-  — turn a string of markdown into events
-- `resolve.rs`
-  — steps to process events
-- `state.rs`
-  — steps of the state machine
-- `subtokenize.rs`
-  — handle content in other content
-- `to_html.rs`
-  — turns events into a string of HTML
-- `to_mdast.rs`
-  — turns events into a syntax tree
-- `tokenizer.rs`
-  — glue the states of the state machine together
-- `unist.rs`
-  — point and position, used in mdast
-
-### Comparison
-
-> 🚧 **To do**.
-
-<!-- To do. -->
+*   `construct/*.rs`
+    — CommonMark, GFM, and other extension constructs used in micromark
+*   `util/*.rs`
+    — helpers often needed when parsing markdown
+*   `event.rs`
+    — things with meaning happening somewhere
+*   `lib.rs`
+    — public API
+*   `mdast.rs`
+    — syntax tree
+*   `parser.rs`
+    — turn a string of markdown into events
+*   `resolve.rs`
+    — steps to process events
+*   `state.rs`
+    — steps of the state machine
+*   `subtokenize.rs`
+    — handle content in other content
+*   `to_html.rs`
+    — turns events into a string of HTML
+*   `to_mdast.rs`
+    — turns events into a syntax tree
+*   `tokenizer.rs`
+    — glue the states of the state machine together
+*   `unist.rs`
+    — point and position, used in mdast
 
 ### Test
 
@@ -373,37 +378,37 @@ These tests reach all branches in the code, which means that this project has
 100% code coverage.
 Fuzz testing is used to check for things that might fall through coverage.
 
-The following scripts are useful when working on this project:
-
-- run examples:
-  ```sh
-  RUST_BACKTRACE=1 RUST_LOG=debug cargo run --example lib
-  ```
-- format:
-  ```sh
-  cargo fmt
-  ```
-- lint:
-  ```sh
-  cargo fmt --check && cargo clippy --examples --tests --benches
-  ```
-- test:
-  ```sh
-  RUST_BACKTRACE=1 cargo test
-  ```
-- docs:
-  ```sh
-  cargo doc --document-private-items
-  ```
-- fuzz:
-  ```sh
-  cargo install cargo-fuzz
-  cargo +nightly fuzz run micromark
-  ```
+The following bash scripts are useful when working on this project:
+
+*   run examples:
+    ```sh
+    RUST_BACKTRACE=1 RUST_LOG=debug cargo run --example lib
+    ```
+*   format:
+    ```sh
+    cargo fmt
+    ```
+*   lint:
+    ```sh
+    cargo fmt --check && cargo clippy --examples --tests --benches
+    ```
+*   test:
+    ```sh
+    RUST_BACKTRACE=1 cargo test
+    ```
+*   docs:
+    ```sh
+    cargo doc --document-private-items
+    ```
+*   fuzz:
+    ```sh
+    cargo install cargo-fuzz
+    cargo +nightly fuzz run micromark
+    ```
 
 ### Version
 
-micromark adheres to [SemVer](https://semver.org).
+micromark follows [SemVer](https://semver.org).
 
 ### Security
 
@@ -439,8 +444,8 @@ For more information on markdown sanitation, see
 
 See [`contributing.md`][contributing] for ways to help.
 See [`support.md`][support] for ways to get help.
-
-<!-- To do: CoC. -->
+See [`code-of-conduct.md`][coc] for how to communicate in and around this
+project.
 
 ### Sponsor
 
@@ -450,11 +455,11 @@ See [`support.md`][support] for ways to get help.
 
 Support this effort and give back by sponsoring:
 
-- [GitHub Sponsors](https://github.com/sponsors/wooorm)
-  (personal; monthly or one-time)
-- [OpenCollective](https://opencollective.com/unified) or
-  [GitHub Sponsors](https://github.com/sponsors/unifiedjs)
-  (unified; monthly or one-time)
+*   [GitHub Sponsors](https://github.com/sponsors/wooorm)
+    (personal; monthly or one-time)
+*   [OpenCollective](https://opencollective.com/unified) or
+    [GitHub Sponsors](https://github.com/sponsors/unifiedjs)
+    (unified; monthly or one-time)
 
 <!-- To do: origin story -->
 
@@ -463,36 +468,55 @@ Support this effort and give back by sponsoring:
 [MIT][license] © [Titus Wormer][author]
 
 <!-- To do: public/publish -->
+
 <!-- [build-badge]: https://github.com/wooorm/micromark-rs/workflows/main/badge.svg -->
+
 <!-- [build]: https://github.com/wooorm/micromark-rs/actions -->
+
 <!-- [crate-badge]: https://img.shields.io/crates/d/micromark.svg -->
+
 <!-- [crate]: https://crates.io/crates/micromark -->
 
-[sponsors-badge]: https://opencollective.com/unified/sponsors/badge.svg
-[backers-badge]: https://opencollective.com/unified/backers/badge.svg
-[opencollective]: https://opencollective.com/unified
 [docs]: https://wooorm.com/micromark-rs/micromark/
+
 [chat-badge]: https://img.shields.io/badge/chat-discussions-success.svg
+
 [chat]: https://github.com/wooorm/micromark-rs/discussions
+
 [commonmark-spec]: https://spec.commonmark.org
+
 [cheat]: https://commonmark.org/help/
-[gfm-spec]: https://github.github.com/gfm/
+
 [rust]: https://www.rust-lang.org
-[cmsm]: https://github.com/micromark/common-markup-state-machine
-[micromark-js]: https://github.com/micromark/micromark
+
 [xss]: https://en.wikipedia.org/wiki/Cross-site_scripting
+
 [improper]: https://github.com/ChALkeR/notes/blob/master/Improper-markup-sanitization.md
+
 [chalker]: https://github.com/ChALkeR
+
 [license]: https://github.com/micromark/micromark/blob/main/license
+
 [author]: https://wooorm.com
+
 [mdast]: https://github.com/syntax-tree/mdast
+
 [starry-night]: https://github.com/wooorm/starry-night
+
 [contribute]: #contribute
+
 [sponsor]: #sponsor
+
 [commonmark]: #commonmark
+
 [extensions]: #extensions
+
 [security]: #security
+
 [test]: #test
-[comparison]: #comparison
+
 [contributing]: .github/contribute.md
+
 [support]: .github/support.md
+
+[coc]: .github/code-of-conduct.md