Skip to content

Conversation

jalateras
Copy link

Summary

This PR adds comprehensive examples for counting tokens when using structured responses with the response_format parameter, addressing the gap identified in issue #2102.

Changes

Added a new section (Section 8) to the tiktoken notebook that covers:

  • Token counting for JSON mode (response_format={"type": "json_object"})
  • Token counting for structured outputs with JSON schemas
  • Comparison of token usage across different response formats
  • Helper function num_tokens_for_structured_response() to calculate schema overhead

Implementation Details

The solution introduces:

  1. A helper function that calculates base message tokens plus schema overhead
  2. Practical examples showing real-world use cases (sentiment analysis, book information extraction)
  3. Verification against actual OpenAI API responses to validate accuracy
  4. Clear documentation of overhead estimates for different response formats

Testing

All code examples have been tested to ensure:

  • Functions correctly calculate token estimates
  • Examples execute without errors
  • Token estimates align reasonably with actual API usage

Impact

This enhancement helps developers:

  • Better estimate API costs when using structured outputs
  • Understand the token overhead of different response formats
  • Plan for context window management with JSON schemas
  • Make informed decisions about schema complexity vs token usage

Fixes #2102

…ebook

Added comprehensive examples for counting tokens when using structured outputs:
- JSON mode token counting with minimal overhead estimation
- Structured output with JSON schema token counting
- Comparison of token usage across different response formats
- Helper function to calculate schema overhead for accurate cost estimation

This addresses the need for understanding token usage with response_format parameter,
helping developers better estimate costs and manage context windows when using
structured outputs.

Fixes openai#2102
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add example to count tokens for structured responses
1 participant