Error: IncompleteMessage: connection closed before message completed #2136
Description
Hey, I'm experiencing a weird behavior with the hyper client when using https.
Sometimes my app in production fails to perform the request, but the same request works most of the time. I performed a load test locally to try to reproduce the problem, and I could reproduce: it is occurring ~0.02% of the times.
I guess that it could be something related to the hyper-tls, so I switched to hyper-rustls, but the same problem continue to occur.
So I tried to hit the url using http instead of https and the error went away!
The error I receive from hyper::Client::get is: hyper::Error(IncompleteMessage): connection closed before message completed.
Follow a minimal working example to reproduce the error:
Cargo.toml:
[dependencies]
hyper = "0.13"
tokio = { version = "0.2", features = ["full"] }
hyper-tls = "0.4.1"
src/main.rs:
use std::convert::Infallible;
use std::net::SocketAddr;
use hyper::service::{make_service_fn, service_fn};
use hyper::{Body, Client, Response, Server, Uri};
use hyper_tls::HttpsConnector;
pub type HttpClient = Client<HttpsConnector<hyper::client::connect::HttpConnector>>;
#[tokio::main]
async fn main() {
let addr = SocketAddr::from(([0, 0, 0, 0], 8100));
let client = Client::builder().build::<_, hyper::Body>(HttpsConnector::new());
let make_service = make_service_fn(move |_| {
let client = client.clone();
async move { Ok::<_, Infallible>(service_fn(move |_req| handle(client.clone()) )) }
});
let server = Server::bind(&addr).serve(make_service);
println!("Listening on http://{}", addr);
if let Err(e) = server.await {
eprintln!("server error: {}", e);
}
}
async fn handle(client: HttpClient) -> Result<Response<Body>, hyper::Error> {
let url = "https://url-here"; // CHANGE THE URL HERE!
match client.get(url.parse::<Uri>().unwrap()).await {
Ok(resp) => Ok(resp),
Err(err) => { eprintln!("{:?} {}", err, err); Err(err) }
}
}PS: replace the url value with a valid https url. In my tests I used a small file on aws s3.
I performed a local load test using hey:
$ hey -z 120s -c 150 http://localhost:8100 Running the test for 2 minutes (-z 120s) was enough to see some errors appearing.
Could anyone help me out? If I need to provide more information, or anything, just let me know.
Thank you!