A fast, accurate tokenizer built for OpenAI models. Instantly estimate token usage, validate prompt size, and avoid unexpected API costs — all with model-aware encoding.
Tokenizer is a native macOS utility that uses OpenAI’s tokenization logic (via the Tiktoken library) to reliably count tokens as they would be processed by OpenAI language models.
- Instant token-count for any text input.
- Select encoding based on model: e.g.,
o200k_basefor GPT-4o,cl100k_basefor GPT-4/GPT-3.5,p50k_basefor older Codex models. - Real-time updates as you type or paste text.
- Clean, native macOS UI built with Swift & SwiftUI.
- Ideal for API developers, prompt engineers and anyone working with language-model usage.
Contributions are welcome! If you spot a bug, have feature requests or want to improve token-encoding support:
- Fork the repository.
- Create a feature branch.
- Write tests (if applicable).
- Submit a pull request with a clear description of your changes.
The original Tiktoken Swift implementation was built by asepinilla, and later modified by me.
This project is licensed under the MIT License. See LICENSE for details.