higress-gateway访问一段时间后出现 503 no_healthy_upstream

[liuyang0508]

{"ai_log":"-","authority":"10.245.192.64:32280","bytes_received":"0","bytes_sent":"19","downstream_local_address":"10.245.219.173:8001","downstream_remote_address":"10.245.192.64:45975","duration":"0","istio_policy_status":"-","method":"POST","path":"/api/digital_man/api/v1/mht/status","protocol":"HTTP/1.1","request_id":"c2b94fbc-2c7d-4d1d-a5fd-69432eb396af","requested_server_name":"-","response_code":"503","response_flags":"UH","route_name":"618354638671841312","start_time":"2025-09-04T02:25:23.074Z","trace_id":"-","upstream_cluster":"outbound|8027||digital_man.static","upstream_host":"-","upstream_local_address":"-","upstream_service_time":"-","upstream_transport_failure_reason":"-","user_agent":"Java/11.0.14.1","x_forwarded_for":"10.245.192.64","response_code_details":"no_healthy_upstream"}

[liuyang0508]

istio-proxy@higress-gateway-5467bd58c7-hrtct:/$ curl -iv 10.246.4.77:8027/health

• Trying 10.246.4.77:8027...
• Connected to 10.246.4.77 (10.246.4.77) port 8027 (#0)

GET /health HTTP/1.1
Host: 10.246.4.77:8027
User-Agent: curl/7.81.0
Accept: /

• Mark bundle as not supporting multiuse
  < HTTP/1.1 200
  HTTP/1.1 200
  < Content-Type: application/json;charset=UTF-8
  Content-Type: application/json;charset=UTF-8
  < Transfer-Encoding: chunked
  Transfer-Encoding: chunked
  < Date: Thu, 04 Sep 2025 02:28:19 GMT
  Date: Thu, 04 Sep 2025 02:28:19 GMT

<

• Connection #0 to host 10.246.4.77 left intact
  {"msg":"success","code":200}istio-proxy@higress-gateway-5467bd58c7-hrtct:/$ curl http://localhost:15000/config_dump | grep -C 200 618354638671841312
  % Total % Received % Xferd Average Speed Time Time Time Current
  Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 },
  "max_grpc_timeout": "0s"
  },
  "metadata": {
  "filter_metadata": {
  "istio": {
  "config": "/apis/networking.istio.io/v1alpha3/namespaces/higress-system/virtual-service/istio-autogenerated-k8s-ingress-higress-system-default-61af79b04c9c8e70"
  }
  }
  },
  "decorator": {
  "operation": "testmanager.dns:80/api/testmanager"
  },
  "name": "618342463177754656"
  },
  {
  "match": {
  "prefix": "/api/testmanager/",
  "case_sensitive": true,
  "headers": [
  {
  "name": ":method",
  "string_match": {
  "safe_regex": {
  "google_re2": {},
  "regex": "POST|GET"
  }
  }
  }
  ]
  },
  "route": {
  "cluster": "outbound|80||testmanager.dns",
  "prefix_rewrite": "/",
  "host_rewrite_literal": "cicd-sit-8080.msxf.msxfyun.test",
  "timeout": "0s",
  "retry_policy": {
  "retry_on": "5xx",
  "num_retries": 1,
  "per_try_timeout": "0s",
  "retry_host_predicate": [
  {
  "name": "envoy.retry_host_predicates.previous_hosts",
  "typed_config": {
  "@type": "type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate"
  }
  }
  ],
  "host_selection_retry_max_attempts": "5",
  "retriable_request_headers": [
  {
  "name": ":method",
  "invert_match": true,
  "string_match": {
  "safe_regex": {
  "google_re2": {},
  "regex": "POST|PATCH|LOCK"
  }
  }
  }
  ]
  },
  "max_grpc_timeout": "0s"
  },
  "metadata": {
  "filter_metadata": {
  "istio": {
  "config": "/apis/networking.istio.io/v1alpha3/namespaces/higress-system/virtual-service/istio-autogenerated-k8s-ingress-higress-system-default-61af79b04c9c8e70"
  }

[liuyang0508]

istio-proxy@higress-gateway-5467bd58c7-hrtct:/$ curl http://localhost:15000/clusters | grep digital_man
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0outbound|8027||digital_man_test.static::observability_name::outbound|8027||digital_man_test.static
outbound|8027||digital_man_test.static::default_priority::max_connections::10000
outbound|8027||digital_man_test.static::default_priority::max_pending_requests::10000
outbound|8027||digital_man_test.static::default_priority::max_requests::10000
outbound|8027||digital_man_test.static::default_priority::max_retries::4294967295
outbound|8027||digital_man_test.static::high_priority::max_connections::1024
outbound|8027||digital_man_test.static::high_priority::max_pending_requests::1024
outbound|8027||digital_man_test.static::high_priority::max_requests::1024
outbound|8027||digital_man_test.static::high_priority::max_retries::3
outbound|8027||digital_man_test.static::added_via_api::true
outbound|8027||digital_man_test.static::eds_service_name::outbound|8027||digital_man_test.static
outbound|8027||digital_man_test.static::10.246.4.77:8027::cx_active::0
outbound|8027||digital_man_test.static::10.246.4.77:8027::cx_connect_fail::0
outbound|8027||digital_man_test.static::10.246.4.77:8027::cx_total::0
outbound|8027||digital_man_test.static::10.246.4.77:8027::rq_active::0
outbound|8027||digital_man_test.static::10.246.4.77:8027::rq_error::0
outbound|8027||digital_man_test.static::10.246.4.77:8027::rq_success::0
outbound|8027||digital_man_test.static::10.246.4.77:8027::rq_timeout::0
outbound|8027||digital_man_test.static::10.246.4.77:8027::rq_total::0
outbound|8027||digital_man_test.static::10.246.4.77:8027::hostname::
outbound|8027||digital_man_test.static::10.246.4.77:8027::health_flags::healthy
outbound|8027||digital_man_test.static::10.246.4.77:8027::weight::1
outbound|8027||digital_man_test.static::10.246.4.77:8027::region::
outbound|8027||digital_man_test.static::10.246.4.77:8027::zone::
outbound|8027||digital_man_test.static::10.246.4.77:8027::sub_zone::
outbound|8027||digital_man_test.static::10.246.4.77:8027::canary::false
outbound|8027||digital_man_test.static::10.246.4.77:8027::priority::0
outbound|8027||digital_man_test.static::10.246.4.77:8027::success_rate::-1
outbound|8027||digital_man_test.static::10.246.4.77:8027::local_origin_success_rate::-1
outbound|8027||digital_man.static::observability_name::outbound|8027||digital_man.static
outbound|8027||digital_man.static::default_priority::max_connections::10000
outbound|8027||digital_man.static::default_priority::max_pending_requests::10000
outbound|8027||digital_man.static::default_priority::max_requests::10000
outbound|8027||digital_man.static::default_priority::max_retries::4294967295
outbound|8027||digital_man.static::high_priority::max_connections::1024
outbound|8027||digital_man.static::high_priority::max_pending_requests::1024
outbound|8027||digital_man.static::high_priority::max_requests::1024
outbound|8027||digital_man.static::high_priority::max_retries::3
outbound|8027||digital_man.static::added_via_api::true
outbound|8027||digital_man.static::eds_service_name::outbound|8027||digital_man.static
100 186k 0 186k 0 0 102M 0 --:--:-- --:--:-- --:--:-- 181M
istio-proxy@higress-gateway-5467bd58c7-hrtct:/$

[johnlanni]

response_flags":"UH"

这个是没有可用的后端节点了

---

response_flags":"UH"

There is no backend node available