Technology
What does Token mean?
Language models do not process text as whole words but as tokens – small units roughly corresponding to a word fragment. A typical sentence consists of more tokens than words.
Tokens are also the unit of account for cost and limits: both the input and the output of a model are measured in tokens. Understanding this helps to estimate the effort and cost of AI applications.
Ready to hand off the routine work?
In a short conversation we find the use case with the biggest leverage in your business – with a clear goal and a measurable result.