Skip to content

o1 release breaks token usage stats #1556

@lucapericlp

Description

@lucapericlp

Describe the bug

Traceback (most recent call last):
  File "/home/luca/.cache/pypoetry/virtualenvs/<NAME>-p2Ot1sDh-py3.12/bin/oaieval", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/luca/.cache/pypoetry/virtualenvs/<NAME>-p2Ot1sDh-py3.12/lib/python3.12/site-packages/evals/cli/oaieval.py", line 304, in main
    run(args)
  File "/home/luca/.cache/pypoetry/virtualenvs/<NAME>-p2Ot1sDh-py3.12/lib/python3.12/site-packages/evals/cli/oaieval.py", line 227, in run
    add_token_usage_to_result(result, recorder)
  File "/home/luca/.cache/pypoetry/virtualenvs/<NAME>-p2Ot1sDh-py3.12/lib/python3.12/site-packages/evals/cli/oaieval.py", line 279, in add_token_usage_to_result
    key: sum(u[key] if u[key] is not None else 0 for u in usage_events)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: unsupported operand type(s) for +: 'int' and 'dict'

To Reproduce

Run any eval

Code snippets

No response

OS

macOS

Python version

v3.12

Library version

3.0.1.post1

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions