在higress中配置azure模型提供者，不能很好兼容流式的请求 || Configuring the Azure model provider in higress is not very compatible with streaming requests

[gitYuZui]

在higress中配置azure模型提供者，流式请求有bug。

当我配置stream 为false时：

在grafana的日志中如下：
{"ai_log":"{"answer":"Hello! 😊 How can I assist you today?","api":"-","chat_id":"chatcmpl-CDlyzAngTLVPHHNtghqPmAhi4mt3e","chat_round":1,"input_token":9,"llm_service_duration":1316,"model":"gpt-4o-2024-11-20","output_token":11,"question":"Hello!","response_type":"normal"}","authority":"172.16.4.175:8080","bytes_received":"109","bytes_sent":"1244","downstream_local_address":"172.19.0.2:8080","downstream_remote_address":"172.16.9.113:54044","duration":"1323","istio_policy_status":"-","method":"POST","path":"/test/v1/chat/completions/openai/deployments/MultiModelTestingGPT4omini/chat/completions?api-version=2025-03-01-preview","protocol":"HTTP/1.1","request_id":"d1fae16e-7e4e-4231-8b93-a5b8cf0db908","requested_server_name":"-","response_code":"200","response_flags":"-","route_name":"ai-route-MultiModelTestingGPT4omini.internal","start_time":"2025-09-09T06:19:04.217Z","trace_id":"24285a6e4ae43609c91f90d57abf42b6","upstream_cluster":"outbound|443||llm-MultiModelTesting_test_fee.internal.dns","upstream_host":"51.12.73.214:443","upstream_local_address":"172.19.0.2:35400","upstream_service_time":"1308","upstream_transport_failure_reason":"-","user_agent":"Apifox/1.0.0 (https://apifox.com)","x_forwarded_for":"172.16.9.113","response_code_details":"via_upstream"}

当我配置stream 为 true时：

在grafana的日志中如下：
{"ai_log":"{"api":"-","chat_id":"chatcmpl-CDmhmZtp8w5W2RnkbmXVOS6PazIZa","chat_round":1,"llm_first_token_duration":1586,"llm_service_duration":3419,"response_type":"stream"}","authority":"172.16.4.175:8080","bytes_received":"139","bytes_sent":"48174","downstream_local_address":"172.19.0.2:8080","downstream_remote_address":"172.16.9.113:58421","duration":"3424","istio_policy_status":"-","method":"POST","path":"/test/v1/chat/completions/openai/deployments/MultiModelTestingGPT4omini/chat/completions?api-version=2025-03-01-preview","protocol":"HTTP/1.1","request_id":"89d3eacd-80c7-462b-8307-4d90658d8ffe","requested_server_name":"-","response_code":"200","response_flags":"-","route_name":"ai-route-MultiModelTestingGPT4omini.internal","start_time":"2025-09-09T07:05:21.505Z","trace_id":"8f351f1a061c7e9691dbaca0e6e1f147","upstream_cluster":"outbound|443||llm-MultiModelTesting_test_fee.internal.dns","upstream_host":"51.12.73.214:443","upstream_local_address":"172.19.0.2:36174","upstream_service_time":"1574","upstream_transport_failure_reason":"-","user_agent":"Apifox/1.0.0 (https://apifox.com)","x_forwarded_for":"172.16.9.113","response_code_details":"via_upstream"}

---

问题：

这里会导致在grafana面板中也无法统计，包含（模型、answer、question、input token、output token 等重要字段）都无法获取到。

请问我该如何解决这个问题呢？

---

Configure the Azure model provider in higress, there are bugs in streaming requests.

When I configure stream to false:

In the logs of grafana, as follows:
{"ai_log":"{"answer":"Hello! 😊 How can I assist you today?","api":"-","chat_id":"chatcmpl-CDlyzAngTLVPHHNtghqPmAhi4mt3e","chat_round":1,"input_token":9,"llm_service_duration":1316,"model":"gpt-4o-2024-11-20","output_token":11,"question":"Hello!","response_type":"normal"}","authority":"172.16.4.175:8080","bytes _received":"109","bytes_sent":"1244","downstream_local_address":"172.19.0.2:8080","downstream_remote_address":"172.16.9.113:54044","duration":"1323","istio_policy_status":"-","method":"POST","path":"/test/v1/chat/completions/openai/deployments/MultiModelTestingGPT4omini/chat/completions?a pi-version=2025-03-01-preview","protocol":"HTTP/1.1","request_id":"d1fae16e-7e4e-4231-8b93-a5b8cf0db908","requested_server_name":"-","response_code":"200","response_flags":"-","route_name":"ai-route-MultiModelTestingGPT4omini.internal","start_time":"2025-09-09T06:19:04.217Z","trace_id":" 24285a6e4ae43609c91f90d57abf42b6","upstream_cluster":"outbound|443||llm-MultiModelTesting_test_fee.internal.dns","upstream_host":"51.12.73.214:443","upstream_local_address":"172.19.0.2:35400","upstream_service_time":"1308","upstream_transport_failure_reason":"-","user_agent":"Apifox/1.0.0 (https://apifox.com)","x_forwarded_for":"172.16.9.113","response_code_details":"via_upstream"}

When I configure stream to true:

In the logs of grafana, as follows:
{"ai_log":"{"api":"-","chat_id":"chatcmpl-CDmhmZtp8w5W2RnkbmXVOS6PazIZa","chat_round":1,"llm_first_token_duration":1586,"llm_service_duration":3419,"response_type":"stream"}","authority":"172.16.4.175:8080","bytes_received":"139","bytes_sent":"48174","down stream_local_address":"172.19.0.2:8080","downstream_remote_address":"172.16.9.113:58421","duration":"3424","istio_policy_status":"-","method":"POST","path":"/test/v1/chat/completions/openai/deployments/MultiModelTestingGPT4omini/chat/completions?api-version=2025-03-01-previe w","protocol":"HTTP/1.1","request_id":"89d3eacd-80c7-462b-8307-4d90658d8ffe","requested_server_name":"-","response_code":"200","response_flags":"-","route_name":"ai-route-MultiModelTestingGPT4omini.internal","start_time":"2025-09-09T07:05:21.505Z","trace_id":"8f351f1a061c7e 9691dbaca0e6e1f147","upstream_cluster":"outbound|443||llm-MultiModelTesting_test_fee.internal.dns","upstream_host":"51.12.73.214:443","upstream_local_address":"172.19.0.2:36174","upstream_service_time":"1574","upstream_transport_failure_reason":"-","user_agent":"Apifox/1.0.0 (https://apifox.com)","x_forwarded_for":"172.16.9.113","response_code_details":"via_upstream"}

---

question:

This will make it impossible to count in the grafana panel, and all important fields including (model, answer, question, input token, output token and other important fields) cannot be obtained.

How can I solve this problem?