Commit message (Expand) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | Refactor to move token types to `token` | 2022-07-07 | 1 | -11/+12 | |
* | Refactor code style | 2022-07-05 | 1 | -43/+37 | |
* | Add `attempt_opt` to tokenizer | 2022-06-22 | 1 | -2/+2 | |
* | Add docs for token types | 2022-06-22 | 1 | -0/+5 | |
* | Refactor to improve a bunch of states | 2022-06-21 | 1 | -12/+8 | |
* | Add improved whitespace handling | 2022-06-20 | 1 | -5/+4 | |
* | Remove unneeded `content` content type | 2022-06-20 | 1 | -3/+3 | |
* | Remove unneeded `pub` keywords | 2022-06-20 | 1 | -32/+29 | |
* | Fix support for deep subtokenization | 2022-06-14 | 1 | -10/+14 | |
* | Add docs for html (text) | 2022-06-14 | 1 | -55/+272 | |
* | Add basic html (text) | 2022-06-13 | 1 | -0/+480 |