Refactor to work on `char`s

Previously, a custom char implementation was used.
This was easier to work with, as sometimes “virtual” characters are injected,
or characters are ignored.

This replaces that with working on actual `char`s.
In the hope of in the future working on `u8`s, even.

This simplifies the state machine somewhat, as only `\n` is fed, regardless of
whether it was a CRLF, CR, or LF.
It also feeds `' '` instead of virtual spaces.

The BOM, if present, is now available as a `ByteOrderMark` event.

1 files changed, 7 insertions, 7 deletions

diff --git a/src/parser.rs b/src/parser.rs
index 0f71daf..cc9c256 100644
--- a/src/parser.rs
+++ b/src/parser.rs
@@ -1,19 +1,18 @@
 //! Turn a string of markdown into events.
 
 use crate::content::document::document;
-use crate::tokenizer::{Code, Event, Point};
-use crate::util::codes::parse as parse_codes;
+use crate::tokenizer::{Event, Point};
 use crate::{Constructs, Options};
 
 /// Information needed, in all content types, when parsing markdown.
 ///
 /// Importantly, this contains a set of known definitions.
-/// It also references the input value as [`Code`][]s.
+/// It also references the input value as a `Vec<char>`.
 #[derive(Debug)]
 pub struct ParseState<'a> {
     pub constructs: &'a Constructs,
-    /// List of codes.
-    pub codes: Vec<Code>,
+    /// List of chars.
+    pub chars: Vec<char>,
     /// Set of defined identifiers.
     pub definitions: Vec<String>,
 }
@@ -24,7 +23,8 @@ pub struct ParseState<'a> {
 pub fn parse<'a>(value: &str, options: &'a Options) -> (Vec<Event>, ParseState<'a>) {
     let mut parse_state = ParseState {
         constructs: &options.constructs,
-        codes: parse_codes(value),
+        // To do: change to `u8`s?
+        chars: value.chars().collect::<_>(),
         definitions: vec![],
     };
 
@@ -33,8 +33,8 @@ pub fn parse<'a>(value: &str, options: &'a Options) -> (Vec<Event>, ParseState<'
         Point {
             line: 1,
             column: 1,
-            offset: 0,
             index: 0,
+            vs: 0,
         },
     );