Refactor to work on `char`s

Previously, a custom char implementation was used.
This was easier to work with, as sometimes “virtual” characters are injected,
or characters are ignored.

This replaces that with working on actual `char`s.
In the hope of in the future working on `u8`s, even.

This simplifies the state machine somewhat, as only `\n` is fed, regardless of
whether it was a CRLF, CR, or LF.
It also feeds `' '` instead of virtual spaces.

The BOM, if present, is now available as a `ByteOrderMark` event.

1 files changed, 13 insertions, 1 deletions

diff --git a/src/token.rs b/src/token.rs
index a0479e1..db3bffc 100644
--- a/src/token.rs
+++ b/src/token.rs
@@ -157,6 +157,17 @@ pub enum Token {
     ///   | b
     /// ```
     BlockQuotePrefix,
+    /// Byte order mark.
+    ///
+    /// ## Info
+    ///
+    /// *   **Context**:
+    ///     optional first event
+    /// *   **Content model**:
+    ///     void
+    /// *   **Construct**:
+    ///     [`document`][crate::content::document]
+    ByteOrderMark,
     /// Whole character escape.
     ///
     /// ## Info
@@ -1822,13 +1833,14 @@ pub enum Token {
 }
 
 /// List of void tokens, used to make sure everything is working good.
-pub const VOID_TOKENS: [Token; 39] = [
+pub const VOID_TOKENS: [Token; 40] = [
     Token::AttentionSequence,
     Token::AutolinkEmail,
     Token::AutolinkMarker,
     Token::AutolinkProtocol,
     Token::BlankLineEnding,
     Token::BlockQuoteMarker,
+    Token::ByteOrderMark,
     Token::CharacterEscapeMarker,
     Token::CharacterEscapeValue,
     Token::CharacterReferenceMarker,