ai.tokenize

ai.tokenize

Model-aware token counting and context budgeting.

Functions

Name	Description
encode	Tokenize text with an explicitly identified tokenizer.

encode

ai.tokenize.encode(
    text,
    *,
    tokenizer,
    revision=None,
    add_special_tokens=False,
    backend='huggingface',
)

Tokenize text with an explicitly identified tokenizer.