Refactor to work on `char`s

Previously, a custom char implementation was used.
This was easier to work with, as sometimes “virtual” characters are injected,
or characters are ignored.

This replaces that with working on actual `char`s.
In the hope of in the future working on `u8`s, even.

This simplifies the state machine somewhat, as only `\n` is fed, regardless of
whether it was a CRLF, CR, or LF.
It also feeds `' '` instead of virtual spaces.

The BOM, if present, is now available as a `ByteOrderMark` event.

1 files changed, 3 insertions, 3 deletions

diff --git a/src/construct/paragraph.rs b/src/construct/paragraph.rs
index 4bce6a4..5d230d3 100644
--- a/src/construct/paragraph.rs
+++ b/src/construct/paragraph.rs
@@ -33,7 +33,7 @@
 //! [html]: https://html.spec.whatwg.org/multipage/grouping-content.html#the-p-element
 
 use crate::token::Token;
-use crate::tokenizer::{Code, ContentType, EventType, State, Tokenizer};
+use crate::tokenizer::{ContentType, EventType, State, Tokenizer};
 use crate::util::skip::opt as skip_opt;
 
 /// Before a paragraph.
@@ -44,7 +44,7 @@ use crate::util::skip::opt as skip_opt;
 /// ```
 pub fn start(tokenizer: &mut Tokenizer) -> State {
     match tokenizer.current {
-        Code::None | Code::CarriageReturnLineFeed | Code::Char('\n' | '\r') => {
+        None | Some('\n') => {
             unreachable!("unexpected eol/eof")
         }
         _ => {
@@ -63,7 +63,7 @@ pub fn start(tokenizer: &mut Tokenizer) -> State {
 /// ```
 fn inside(tokenizer: &mut Tokenizer) -> State {
     match tokenizer.current {
-        Code::None | Code::CarriageReturnLineFeed | Code::Char('\n' | '\r') => {
+        None | Some('\n') => {
             tokenizer.exit(Token::Data);
             tokenizer.exit(Token::Paragraph);
             tokenizer.register_resolver_before("paragraph".to_string(), Box::new(resolve));