articleschannelstagsspacestoolkit
Microsoft Dev Blogs

Android tokenizer for OpenAI

View Original
thumbnail
  • Tokenizer selection: The author discusses the need for an accurate tokenizer for OpenAI models on Android devices and introduces JTokkit, an Android-compatible open-source tokenizer.
  • Accuracy comparison: The author compares the token counts of different strings using the original estimate method and the JTokkit tokenizer. The JTokkit package shows more accurate results provided by OpenAI's algorithm.
  • Updating the class: The author explains the process of adding the JTokkit package in the project's build.gradle.kts file and updating the class to use the more accurate token count and string truncation.
  • Improved efficiency: By using the more accurate token count, the sliding window algorithm in the JetchatAI demo can include more context in each API request without risking exceeding the token count. This can result in cost savings when using GPT-3.5 Turbo API.
  • Limitations: Though the token count for user-generated queries and responses can be accurately calculated, determining the exact size of the payload processed by the model is not possible due to the construction of chat completion requests. The author provides a JTokkit recipe for getting a more accurate total token count for a request.
  • Resources and feedback: The author suggests referring to the Azure OpenAI documentation for more information on available services and welcomes feedback from readers on the post and any tips or tricks they have discovered while working with ChatGPT prompts.