Add OpenAiClient stream handling, update types

[declark1]

This adds handling to return a stream response when stream: true and applies type updates based on initial testing with vLLM & llama-3-8b-instruct.

Notes:

• The stream error handling probably needs more work, we can do this as part of the orchestrator chat implementation

[evaline-ju]

some small initial comments, thanks for the addition!

The stream error handling probably needs more work, we can do this as part of the orchestrator chat implementation

Let’s make sure to note this as part of #192 or whatever appropriate issue?

[evaline-ju]

LGTM with nit

[evaline-ju]

nit: is role actually optional? I didn't happen to see a or null on https://platform.openai.com/docs/api-reference/chat/streaming

[declark1]

I made it Option for the ChatCompletionDelta object (used in streaming responses) since their documentation has examples without it included, e.g. the second one below.

{"id":"chatcmpl-123","object":"chat.completion.chunk","created":1694268190,"model":"gpt-4o-mini", "system_fingerprint": "fp_44709d6fcb", "choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]}

{"id":"chatcmpl-123","object":"chat.completion.chunk","created":1694268190,"model":"gpt-4o-mini", "system_fingerprint": "fp_44709d6fcb", "choices":[{"index":0,"delta":{"content":"Hello"},"logprobs":null,"finish_reason":null}]}

[declark1]

I just remembered. I believe this is optional (although not properly documented) for the streaming responses because the chunk is part of the assistant's response. In the example above, {"content":"Hello"} is part of the assistant's response, so it keeps the messages smaller by not repeating {"role":"assistant"} in each chunk.