text_quality.feature.tokenizer

Module Contents

Classes

Tokenizer	Helper class that provides a standard way to create an ABC using
NautilusOcrTokenizer	Helper class that provides a standard way to create an ABC using

class text_quality.feature.tokenizer.Tokenizer

Bases: abc.ABC

Helper class that provides a standard way to create an ABC using inheritance.

abstract tokenize(text: str) → List[str]

class text_quality.feature.tokenizer.NautilusOcrTokenizer

Bases: Tokenizer

Helper class that provides a standard way to create an ABC using inheritance.

_HYPHENS

tokenize(text: str) → List[str]

Nautilus-OCR tokenizer