Word Tokenizer
Splits printed lines into the words a reader actually says.
Which marks may live inside a word is supplied by the language. A writing system without spaces needs another tokenizer rather than a different set of marks.
A character, as Unicode clusters it, is a letter or a space by its base: the code point its combining marks sit on, past any sign prepended to it. A private-use character is never a letter, whatever a font draws for it.
Where a character ends and whether it is a letter follow the Unicode data of the runtime, the JDK's or Android's, so an older runtime can cut a newer character differently.
Properties
Functions
Splits every line into drawable word-and-punctuation segments.
Splits one line while preserving every original character and space.
Returns the UTF-16 offsets of words in one line.
Returns all spoken words with their positions in the passage.