aboutsummaryrefslogtreecommitdiffstats
path: root/src/subtokenize.rs (unfollow)
Commit message (Expand)AuthorFilesLines
2022-07-20Refactor to use less vecs for eventsLibravatar Titus Wormer1-2/+4
2022-07-19Refactor to remove cloning in `edit_map`Libravatar Titus Wormer1-2/+2
2022-07-19Use `edit_map` in `subtokenize`Libravatar Titus Wormer1-67/+40
2022-07-19Remove an unneeded `HashMap`Libravatar Titus Wormer1-1/+1
2022-07-15Fix annoying bug around virtual spaces in containersLibravatar Titus Wormer1-1/+1
2022-07-07Add support for `Flow` content typeLibravatar Titus Wormer1-2/+4
2022-07-05Refactor to do some to dosLibravatar Titus Wormer1-3/+2
2022-07-04Add support for unicode punctuationLibravatar Titus Wormer1-1/+1
2022-07-04Update list of todosLibravatar Titus Wormer1-2/+0
2022-06-28Fix jumps in `edit_map`Libravatar Titus Wormer1-101/+99
2022-06-24Add link, images (resource)Libravatar Titus Wormer1-12/+26
2022-06-22Refactor some unneeded assignmentsLibravatar Titus Wormer1-2/+1
2022-06-22Add docs for token typesLibravatar Titus Wormer1-1/+3
2022-06-21Add docs for `subtokenize`Libravatar Titus Wormer1-2/+51
2022-06-21Update todo listLibravatar Titus Wormer1-8/+1
2022-06-20Add support for BOMLibravatar Titus Wormer1-0/+4
2022-06-20Remove unneeded `content` content typeLibravatar Titus Wormer1-6/+3
2022-06-14Fix support for deep subtokenizationLibravatar Titus Wormer1-9/+19
2022-06-14Reorganize to split utilLibravatar Titus Wormer1-6/+4
2022-06-14Add docs for html (text)Libravatar Titus Wormer1-0/+1
2022-06-13Add basic html (text)Libravatar Titus Wormer1-3/+9
2022-06-10Add text content typeLibravatar Titus Wormer1-4/+10
2022-06-10Add proper support for subtokenizationLibravatar Titus Wormer1-50/+116
2022-06-09Add basic subtokenization, string content in fenced codeLibravatar Titus Wormer1-0/+67