re-enable router integration tests

[deads2k]

They required a golang 1.9 node and were failing without it, so they were disabled in the rebase here: https://github.com/openshift/origin/blob/master/Makefile#L188 .

Specifically, we no longer run COVERAGE_SPEC=' ' DETECT_RACES='false' TIMEOUT='10m' hack/test-go.sh ./test/end-to-end

@openshift/sig-networking

[stevekuznetsov]

Can we determine today:

• is it possible to run these tests without direct access to the Docker daemon
• is it possible to run these tests using DIND?
• can we allocate the bandwidth to get these tests into a container right now?

This is going to become more and more of a pain point the longer we wait. We've been asking for containerized tests in this place and for the registry (/cc @bparees @mfojtik @dmage @miminar ) for a while.

[dmage]

@stevekuznetsov you are talking about test/end-to-end/core.sh, right? Can we split it into smaller pieces and run each piece with its own master api and DinD? Though, to run DinD we still need direct access to the Docker daemon.

[stevekuznetsov]

No, I mean the integration suite that was written for the registry and uses the Docker socket: https://github.com/openshift/image-registry/blob/master/pkg/testframework/master.go#L60-L67

These router tests and those registry tests need to be written to work like e2e -- assume an OpenShift cluster is running, deploy your application under test into the cluster using the Go client, test it, etc.

The mess of test/end-to-end should most likely be re-thought and potentially not live in Bash but for the most part it already just uses the normal oc client to do it's job.

[dmage]

@stevekuznetsov for a while? Those tests were written a month ago and they are written the way that they can be executed in parallel. We have global OpenShift resources called image, and we care a lot about the openshift.io/image.managed annotation, to prevent interference from other tests we need a separate instance of OpenShift for some tests. Also we have the stop-the-world routine pruning, the test for it must inspect all images and image streams in all namespaces, so it requires a separate instance of OpenShift as well.

So, unless we have to deal with global objects in our tests, we cannot use a single OpenShift instance.

[stevekuznetsov]

Those tests were written a month ago

I may be misremembering but when we tried to put integration tests into a container in January or so there were router and registry tests that needed Docker. If I'm not remembering that correctly, my apologies.

So, unless we have to deal with global objects in our tests, we cannot use a single OpenShift instance.

How difficult would it be to make your tests non-disruptive and clean up after themselves? I believe the contract with e.g. test-cmd suites or the normal Ginkgo e2es is similar -- keep track of your global state mutation and clean it up. I wouldn't necessarily ask for you to run your tests in parallel, but your test suite not being in the business of starting OpenShift would be nice.

[bparees]

How difficult would it be to make your tests non-disruptive and clean up after themselves?

basically impossible.

They could run serially to possibly deal w/ the pruning challenges (except that @smarterclayton yells at us every time we create a serial test), but they still need unique configuration for the registry so at a minimum i think they'd need to delete the deployed registry and recreate it. There may be master configuration required as well, at which point there's no solution but to start a new cluster for the test.

[stevekuznetsov]

Are they mostly testing through the Origin API or the registry API? Can we deploy multiple registries, or are they all under the assumption they're a singleton in the cluster?

[dmage]

They are testing through both API, Origin is a configuration storage for the registry. Deploying multiple registries is the way how to scale, they should serve the same content (if they are started with the proper storage backed). You cannot have two integrated registries that will serve different images.

[stevekuznetsov]

Right now are the tests serial? I understand your point @bparees about serial but if it's not changing the status quo and stopping us from having to deploy a separate cluster per test to parallelize it ...