Commit message (Expand) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | Refactor some unneeded assignments | Titus Wormer | 2022-06-22 | 1 | -2/+1 |
* | Add docs for token types | Titus Wormer | 2022-06-22 | 1 | -1/+3 |
* | Add docs for `subtokenize` | Titus Wormer | 2022-06-21 | 1 | -2/+51 |
* | Update todo list | Titus Wormer | 2022-06-21 | 1 | -8/+1 |
* | Add support for BOM | Titus Wormer | 2022-06-20 | 1 | -0/+4 |
* | Remove unneeded `content` content type | Titus Wormer | 2022-06-20 | 1 | -6/+3 |
* | Fix support for deep subtokenization | Titus Wormer | 2022-06-14 | 1 | -9/+19 |
* | Reorganize to split util | Titus Wormer | 2022-06-14 | 1 | -6/+4 |
* | Add docs for html (text) | Titus Wormer | 2022-06-14 | 1 | -0/+1 |
* | Add basic html (text) | Titus Wormer | 2022-06-13 | 1 | -3/+9 |
* | Add text content type | Titus Wormer | 2022-06-10 | 1 | -4/+10 |
* | Add proper support for subtokenization | Titus Wormer | 2022-06-10 | 1 | -50/+116 |
* | Add basic subtokenization, string content in fenced code | Titus Wormer | 2022-06-09 | 1 | -0/+67 |