Count the number of Tokens in Your Prompt Azure Open AI .NET

In this Microsoft Azure Open AI blog post, we will show you how to count the number of tokens your prompt uses before or after it is sent to the API.

When working with an Azure OpenAI resource or with the OpenAI API, the service has a token limit that measures the size of your prompt in tokens.

In some cases, knowing the size of your prompt in tokens before sending it to the API is crucial, and in this post, we will show how to use the Microsoft ML tokenizers library to count the number of tokens before or after you send it.

Install Library

Installing the following library to count the number of tokens in your prompt.

dotnet add package Microsoft.ML.Tokenizers --version 0.21.1

Once installed, add the following code to your application.

//  Microsoft.ML.Tokenizers library 
using Microsoft.ML.Tokenizers;

Download the following vocabulary files and copy them to the root directory of your application.

https://huggingface.co/openai-community/gpt2/tree/main

• vocab.json
• merges.txt

Declare Variables

Add the following variables.

// vocabulary files used to encode the text into tokens
var vocabFilePath = @"vocab.json";
var mergeFilePath = @"merges.txt";
var tokenizer = new Tokenizer(new Bpe(vocabFilePath, mergeFilePath));

Count and print token count

Use the following code to count and print the number of tokens used in the prompt.

// The user's input is taken and encoded using the tokenizer.
var input = userPrompt; // can also be a string for testing
// The encoded result is stored in 'tokenizerEncodedResult'.
var tokenizerEncodedResult = tokenizer.Encode(input);
// The number of tokens used in the encoding is counted.
tokenizerEncodedResult.Tokens.Count();
// The console's foreground color is set to Magenta.
Console.ForegroundColor = ConsoleColor.Magenta;
// The number of tokens used is printed to the console in Magenta color.
Console.WriteLine($"Number of tokens used: {tokenizerEncodedResult.Tokens.Count()}");
// The console's color settings are reset to default.
Console.ResetColor();

In my case, I’m printing the tokens count after and the output looks like this.

To get started with Azure Open AI, visit the following articles:

• Get Started With Azure Open AI .NET SDK
• Use Azure OpenAI with C# Application
• Deploy Azure OpenAI Resource With Terraform
• Connect to Azure OpenAI Using Postman
• Azure AI vs Azure Open AI
• Azure AI Studio vs Microsoft Copilot Studio
• Get Started With Azure AI Studio
• Create Prompt Flow for an LLM App in Azure AI Studio
• Detect Language using Azure AI Services
• Use GitHub Copilot to Write and Manage Code

More Articles

• Create Admin User for Kubernetes Dashboard (Web UI) Access
• Create Multiple Azure Resources With Terraform
• How To Output Multiple Count Items With Terraform
• Use Postman With Microsoft Graph API
• Prompt for Values With Bicep