WordTokenizer

class WordTokenizer(val interiorMarks: Set<Int>)(source)

Splits printed lines into the words a reader actually says.

Which marks may live inside a word is supplied by the language. A writing system without spaces needs another tokenizer rather than a different set of marks.

A character, as Unicode clusters it, is a letter or a space by its base: the code point its combining marks sit on, past any sign prepended to it. A private-use character is never a letter, whatever a font draws for it.

Where a character ends and whether it is a letter follow the Unicode data of the runtime, the JDK's or Android's, so an older runtime can cut a newer character differently.

Constructors

Link copied to clipboard
constructor(interiorMarks: Set<Int>)
constructor(interiorMarks: String)

Properties

Link copied to clipboard

Code points that remain part of a word after the word has begun, the marks the work's data names for its script, such as an apostrophe or a hyphen.

Functions

Link copied to clipboard

Splits every line into drawable word-and-punctuation segments.

Splits one line while preserving every original character and space.

Link copied to clipboard

Returns the UTF-16 offsets of words in one line.

Link copied to clipboard
fun words(passage: Passage): List<SpokenWord>

Returns all spoken words with their positions in the passage.