Commit message (Collapse) | Author | Files | Lines | ||
---|---|---|---|---|---|
2022-06-22 | Refactor some unneeded assignments | Titus Wormer | 1 | -2/+1 | |
2022-06-22 | Add docs for token types | Titus Wormer | 1 | -1/+3 | |
2022-06-21 | Add docs for `subtokenize` | Titus Wormer | 1 | -2/+51 | |
2022-06-21 | Update todo list | Titus Wormer | 1 | -8/+1 | |
2022-06-20 | Add support for BOM | Titus Wormer | 1 | -0/+4 | |
2022-06-20 | Remove unneeded `content` content type | Titus Wormer | 1 | -6/+3 | |
2022-06-14 | Fix support for deep subtokenization | Titus Wormer | 1 | -9/+19 | |
* Fix a couple of forgotten line ending handling in html (text) * Fix missing initial case for html (text) not having a `<` 😬 * Add line ending handling to `text` construct | |||||
2022-06-14 | Reorganize to split util | Titus Wormer | 1 | -6/+4 | |
2022-06-14 | Add docs for html (text) | Titus Wormer | 1 | -0/+1 | |
2022-06-13 | Add basic html (text) | Titus Wormer | 1 | -3/+9 | |
* Add all states for html (text) * Fix to link paragraph tokens together * Add note about uncovered bug where linking paragraph tokens together doesn’t work 😅 | |||||
2022-06-10 | Add text content type | Titus Wormer | 1 | -4/+10 | |
* Add character reference and character escapes in text * Add recursive subtokenization | |||||
2022-06-10 | Add proper support for subtokenization | Titus Wormer | 1 | -50/+116 | |
- Add “content” content type - Add paragraph - Add skips - Add linked tokens | |||||
2022-06-09 | Add basic subtokenization, string content in fenced code | Titus Wormer | 1 | -0/+67 | |