tokenizer

Classes

LightningIRTokenizer(*args[, query_length, ...])

Base class for LightningIR tokenizers.