"SSH Error: data could not be sent to remote host" during "Gathering Facts" step when provisioning cluster

[sjenning]

Happened 4 times in the last ~10hrs

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/18323/test_pull_request_origin_extended_conformance_gce/15311/

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/18102/test_pull_request_origin_extended_conformance_gce/15298/

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/18102/test_pull_request_origin_extended_conformance_gce/15298/

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/18121/test_pull_request_origin_extended_conformance_gce/15293/

[stevekuznetsov]

We've seen this manifest in a couple of different ways for a while now but the root cause seems to be EC2 instance connection instability. I'm fairly certain the Ansible connection logic has a retry, I'm not sure what else we can do here.

[sdodson]

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/openshift_openshift-ansible/7007/test_pull_request_openshift_ansible_extended_conformance_gce/515/

[sdodson]

openshift/openshift-ansible#7056 make sure that it waits until ansible is actually able to establish an ssh connection hopefully fixing this.