ingress 切换到higress 使用https 请求报错 Connection reset by peer

[vvlisn]

ingress 配置

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: cnca-xxx-t1
  annotations:
    #  key: string
  creationTimestamp: '2025-05-13T14:04:07Z'
  generation: 1
  labels:
    {}
    #  key: string
  namespace: certification-platform
spec:
  ingressClassName: ingress-nginx-t1-nginx-t1
  rules:
    - host: cnca.stg.xxx.net
      http:
        paths:
          - backend:
              service:
                name: cnca-console
                port:
                  number: 8080
            path: /
            pathType: Prefix
  tls:
    - hosts:
        - cnca.stg.xxx.net
      secretName: cnca.stg.cert
  backend:
    {}

cacheObject:
  useNestedBackendField: true
  showPathType: true
__clone: true

测试

higress-gateway 日志

{"ai_log":"-","authority":"-","bytes_received":"0","bytes_sent":"0","downstream_local_address":"192.168.239.10:443","downstream_remote_address":"192.168.239.10:56992","duration":"40","istio_policy_status":"-","method":"-","path":"-","protocol":"-","request_id":"-","requested_server_name":"-","response_code":"0","response_flags":"NR","route_name":"-","start_time":"2025-09-09T02:33:42.159Z","trace_id":"-","upstream_cluster":"-","upstream_host":"-","upstream_local_address":"-","upstream_service_time":"-","upstream_transport_failure_reason":"-","user_agent":"-","x_forwarded_for":"-","response_code_details":"filter_chain_not_found"}

ingress 配置了tls ，但是看报错higress 好像无法找到证书

[vvlisn]

higress 部署指定已存在的ingressclass

[CH3CHO]

拉一下Gateway的运行时配置吧，官网有操作方法。

---

Pull the runtime configuration of Gateway, the official website has an operation method.

[vvlisn]

配置如下

{
         "filter_chain_match": {
          "server_names": [
           "cnca.stg.xxx.net"
          ]
         },
         "filters": [
          {
           "name": "envoy.filters.network.http_connection_manager",
           "typed_config": {
            "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
            "stat_prefix": "outbound_0.0.0.0_443",
            "http_filters": [
             {
              "name": "envoy.filters.http.custom_response",
              "typed_config": {
               "@type": "type.googleapis.com/envoy.extensions.filters.http.custom_response.v3.CustomResponse"
              }
             },
             {
              "name": "envoy.filters.http.compressor",
              "typed_config": {
               "@type": "type.googleapis.com/envoy.extensions.filters.http.compressor.v3.Compressor",
               "compressor_library": {
                "name": "text_optimized",
                "typed_config": {
                 "@type": "type.googleapis.com/envoy.extensions.compression.gzip.compressor.v3.Gzip",
                 "memory_level": 5,
                 "compression_level": "COMPRESSION_LEVEL_9",
                 "window_bits": 12
                }
               },
               "request_direction_config": {
                "common_config": {
                 "enabled": {
                  "default_value": false,
                  "runtime_key": "request_compressor_enabled"
                 }
                }
               },
               "response_direction_config": {
                "common_config": {
                 "min_content_length": 1024,
                 "content_type": [
                  "text/html",
                  "application/json",
                  "text/css",
                  "application/javascript",
                  "application/xhtml+xml",
                  "image/svg+xml"
                 ]
                },
                "disable_on_etag_header": true
               }
              }
             },
             {
              "name": "envoy.filters.http.cors",
              "typed_config": {
               "@type": "type.googleapis.com/envoy.extensions.filters.http.cors.v3.Cors"
              }
             },
             {
              "name": "envoy.filters.http.rbac",
              "typed_config": {
               "@type": "type.googleapis.com/envoy.extensions.filters.http.rbac.v3.RBAC"
              }
             },
             {
              "name": "envoy.filters.http.local_ratelimit",
              "typed_config": {
               "@type": "type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit",
               "stat_prefix": "http_local_rate_limiter"
              }
             },
             {
              "name": "envoy.filters.http.fault",
              "typed_config": {
               "@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
              }
             },
             {
              "name": "envoy.filters.http.router",
              "typed_config": {
               "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
              }
             }
            ],
            "tracing": {
             "client_sampling": {
              "value": 100
             },
             "random_sampling": {
              "value": 1
             },
             "overall_sampling": {
              "value": 100
             },
             "custom_tags": [
              {
               "tag": "istio.authorization.dry_run.allow_policy.name",
               "metadata": {
--
            ],
            "use_remote_address": true,
            "forward_client_cert_details": "SANITIZE_SET",
            "set_current_client_cert_details": {
             "subject": true,
             "cert": true,
             "dns": true,
             "uri": true
            },
            "upgrade_configs": [
             {
              "upgrade_type": "websocket"
             }
            ],
            "stream_idle_timeout": "180s",
            "max_request_headers_kb": 60,
            "normalize_path": true,
            "scoped_routes": {
             "name": "scoped-route",
             "scope_key_builder": {
              "fragments": [
               {
                "local_port_value_extractor": {}
               },
               {
                "host_value_extractor": {}
               }
              ]
             },
             "rds_config_source": {
              "ads": {},
              "initial_fetch_timeout": "0s",
              "resource_api_version": "V3"
             },
             "scoped_rds": {
              "scoped_rds_config_source": {
               "ads": {},
               "initial_fetch_timeout": "0s",
               "resource_api_version": "V3"
              }
             }
            },
            "common_http_protocol_options": {
             "idle_timeout": "180s"
            },
            "request_id_extension": {
             "typed_config": {
              "@type": "type.googleapis.com/envoy.extensions.request_id.uuid.v3.UuidRequestIdConfig",
              "use_request_id_for_trace_sampling": true
             }
            },
            "path_with_escaped_slashes_action": "KEEP_UNCHANGED"
           }
          }
         ],
         "transport_socket": {
          "name": "envoy.transport_sockets.tls",
          "typed_config": {
           "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext",
           "common_tls_context": {
            "tls_params": {
             "tls_minimum_protocol_version": "TLSv1_0",
             "tls_maximum_protocol_version": "TLSv1_3",
             "cipher_suites": [
              "ECDHE-ECDSA-AES128-GCM-SHA256",
              "ECDHE-ECDSA-CHACHA20-POLY1305",
              "ECDHE-RSA-AES128-GCM-SHA256",
              "ECDHE-RSA-CHACHA20-POLY1305",
              "ECDHE-ECDSA-AES128-SHA",
              "ECDHE-RSA-AES128-SHA",
              "AES128-GCM-SHA256",
              "AES128-SHA",
              "ECDHE-ECDSA-AES256-GCM-SHA384",
              "ECDHE-RSA-AES256-GCM-SHA384",
              "ECDHE-ECDSA-AES256-SHA",
              "ECDHE-RSA-AES256-SHA",
              "AES256-GCM-SHA384",
              "AES256-SHA"
             ]
            },
            "alpn_protocols": [
             "h2",
             "http/1.1"
            ],
            "tls_certificate_sds_secret_configs": [
             {
              "name": "kubernetes-ingress://Kubernetes/certification-platform/cnca.stg.cert",
              "sds_config": {
               "ads": {},
               "resource_api_version": "V3"
              }
             }
            ]
           },
           "require_client_certificate": false
          }
         },
         "transport_socket_connect_timeout": "15s"
        },

[CH3CHO]

我怀疑是你的 curl 没有加 SNI 的原因。看一下这个帖子：https://stackoverflow.com/questions/12941703/use-curl-with-sni-server-name-indication

---

I suspect it is the reason why your curl is not added with SNI. Take a look at this post: https://stackoverflow.com/questions/12941703/use-curl-with-sni-server-name-indication

[vvlisn]

添加SNI 测试结果

curl -vk --resolve cnca.stg.xxx.net:443:192.168.239.10 https://cnca.stg.xxx.net/
* Added cnca.stg.xxx.net:443:192.168.239.10 to DNS cache
* Hostname cnca.stg.xxx.net was found in DNS cache
*   Trying 192.168.239.10:443...
* Connected to cnca.stg.xxx.net (192.168.239.10) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=Washington; L=Seattle; O=xxx Coffee Company; OU=IT; CN=cnca.stg.xxx.net
*  start date: Apr 22 05:50:45 2025 GMT
*  expire date: Apr 22 05:50:45 2026 GMT
*  issuer: C=US; ST=Washington; L=Seattle; O=xxx Coffee Company; OU=IT; CN=xxx Corporate Issuing CA 07
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x55c8ea7f19f0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: cnca.stg.xxx.net
> user-agent: curl/7.81.0
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200 
< server: istio-envoy
< date: Tue, 09 Sep 2025 06:41:22 GMT
< content-type: text/html
< content-length: 505
< last-modified: Thu, 28 Aug 2025 06:13:48 GMT
< vary: Accept-Encoding
< etag: "68aff39c-1f9"
< accept-ranges: bytes
< req-cost-time: 1
< req-arrive-time: 1757400082984
< resp-start-time: 1757400082985
< x-envoy-upstream-service-time: 1
< 
<!doctype html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <link rel="icon" type="image/svg+xml" href="/vite.svg" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>SBUX CNCA</title>
    <script type="module" crossorigin src="/assets/index-BTkmmMpY.js"></script>
    <link rel="stylesheet" crossorigin href="/assets/index-BC8UUqpm.css">
  </head>
  <body style="margin: 0; padding: 0; overflow: hidden;">
    <div id="root"></div>
  </body>
</html>
* Connection #0 to host cnca.stg.xxx.net left intact
istio-proxy@higress-gateway-cbd577d46-f9rfm:/$ 
istio-proxy@higress-gateway-cbd577d46-f9rfm:/$ 
istio-proxy@higress-gateway-cbd577d46-f9rfm:/$ curl -v --resolve cnca.stg.xxx.net:443:192.168.239.10 https://cnca.stg.xxx.net/
* Added cnca.stg.xxx.net:443:192.168.239.10 to DNS cache
* Hostname cnca.stg.xxx.net was found in DNS cache
*   Trying 192.168.239.10:443...
* Connected to cnca.stg.xxx.net (192.168.239.10) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

[CH3CHO]

证书是自签的吗，本机信任吗？

---

Is the certificate self-signed? Does the local machine trust it?

[vvlisn]

现在这个发送SNI 应该是正常请求的，现在只是客户端没有证书的错误

---

Now this sending SNI should be a normal request, but now it is just an error that the client does not have a certificate

[vvlisn]

大佬，请求是必须包含SNI 才可以吗？能否关闭

---

Boss, does the request must include SNI? Can you close

[CH3CHO]

大佬，请求是必须包含SNI 才可以吗？能否关闭

Boss, does the request must include SNI? Can you close

你的默认域名配证书了吗？

[vvlisn]

这里是指higress TLS 证书全局配置么。默认部署没有做额外的配置。现在的问题是curl 发送SNI 是可以正常访问的。那么问题应该是higress 前面的代理没有传递SNI 吧

---

Is this the global configuration of the higress TLS certificate? The default deployment does not have additional configuration. The problem now is that curl sends SNI can be accessed normally. Then the problem should be that the agent in front of higress does not pass SNI

[CH3CHO]

这里是指higress TLS 证书全局配置么。默认部署没有做额外的配置。现在的问题是curl 发送SNI 是可以正常访问的。那么问题应该是higress 前面的代理没有传递SNI 吧

建议你先了解一下SNI是什么

---

Is this the global configuration of the higress TLS certificate? The default deployment does not have additional configuration. The problem now is that curl sends SNI can be accessed normally. Then the problem should be that the agent in front of higress does not pass SNI

It is recommended that you understand what SNI is

[vvlisn]

好吧！我了解了，可是ingress tls 指定了证书。为何还要配置默认的证书

---

All right! I understand, but ingress tls specifies the certificate. Why do you need to configure the default certificate

[CH3CHO]

好吧！我了解了，可是ingress tls 指定了证书。为何还要配置默认的证书

我的理解是，你这个 Ingress 指定的证书是用在请求 Ingress 所对应的域名的时候的，也就是只有客户端通过 SNI 指定了这个域名，网关返回返回其对应的 TLS 证书。但如果用户没有发送 SNI 信息，那么网关只能返回 443 端口所对应的默认证书。如果你没配默认证书，那这个 TLS 握手流程就走不下去了。

---

OK! I understand, but ingress tls specifies the certificate. Why do you need to configure the default certificate?

My understanding is that the certificate specified by your Ingress is used when requesting the domain name corresponding to Ingress, that is, only the client specifies the domain name through SNI, and the gateway returns the corresponding TLS certificate. However, if the user does not send SNI information, the gateway can only return the default certificate corresponding to port 443. If you do not have a default certificate, then this TLS handshake process will not be able to continue.