docs: update README with latest benchmarks and new sections #489
Conversation
| | gemini-2.5-flash-preview-04-17 | 10 | 0 | 100% | | ||
| | gemini-2.5-pro-preview-03-25 | 10 | 0 | 100% | | ||
| | gemma-3-27b-it | 8 | 2 | 80% | | ||
| | AWS Bedrock Claude 3.7 Sonnet | 10 | 0 | 100% | |
There was a problem hiding this comment.
To improve the clarity of the benchmark, could we clarify if AWS Bedrock is just the access layer? It might be better to list the core model, 'Claude 3.7 Sonnet', to ensure we're comparing the models directly.
There was a problem hiding this comment.
We will add a column for the llm provider in the final benchmark report for completeness and reproducibility. I have seen differences in behavior of the model across different providers (in bedrock case, there are some prompt specific changes as well that I am not super sure of) and when we include secondary metrics e.g. cost, latency etc, it will become even more important. This will also be critical for open models where one inference stack (llama.cpp, vllm) affects even the accuracy of the same model.
/cc @noahlwest
No description provided.