Tokens
Tokens are nothing but a group of characters.
Last updated
Tokens are nothing but a group of characters.
Last updated
1 token is equal to 4 characters.
For example, the following sentence “The quick brown fox jumps over the lazy dog” has 11 tokens.
Each plan has a different token quota. .
Tokens are used by both the input (what is being sent to the AI model) and output (what is being received to the AI model).
For example, when you train a AI model with the website www.wotSABot.ai here’s how many tokens would be consumed:
Training – Total number of characters extracted from the website (Counted once, as you only train once)
Input tokens – All the tokens used to send the question to OpenAI are counted here. They include:
Chunks – When a question is asked to the AI model, the system would fetch a chunk that closely matches the question.
Input question – Question asked by the user
Prompt – System prompt and the instructions defined
Output tokens – Size of the response received from OpenAI
Token calculation would vary depending on the size of your AI model, how brief are your prompts/instructions, and how big are the responses from the AI model.