ContainerCannotRun: devmapper: Error activating devmapper device for ''...'': devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed

[tnozicka]

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17751/test_pull_request_origin_extended_conformance_install/4109/

  containerStatuses:
  - containerID: docker://b7f14c8009aee731cd5199c9cce74b5b46912ffd0175c26b3ff73f2139a8212b
    image: openshift/origin-deployer:551f02f
    imageID: docker://sha256:ed03e11e54a7b36812b1a76d0f821357f727b59f3fa56fd290efa5a94fd745a6
    lastState: {}
    name: deployment
    ready: false
    restartCount: 0
    state:
      terminated:
        containerID: docker://b7f14c8009aee731cd5199c9cce74b5b46912ffd0175c26b3ff73f2139a8212b
        exitCode: 128
        finishedAt: 2017-12-14T13:59:13Z
        message: 'devmapper: Error activating devmapper device for ''46d82310de547f7f9e3e87d95f6a5af6420fe4b69ea3e1142bfc86486999e0f0'':
          devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed'
        reason: ContainerCannotRun
        startedAt: 2017-12-14T13:59:13Z

[jwhonce]

@rhvgoyal excerpts from logs. Looks like your patch for additional debugging is not in this build.

Client:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-68.gitec8512b.el7.x86_64
 Go version:      go1.8.3
 Git commit:      ec8512b/1.12.6
 Built:           Thu Nov 16 15:19:17 2017
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-68.gitec8512b.el7.x86_64
 Go version:      go1.8.3
 Git commit:      ec8512b/1.12.6
 Built:           Thu Nov 16 15:19:17 2017
 OS/Arch:         linux/amd64

Dec 14 13:59:12 ip-172-18-2-50.ec2.internal oci-umount[112015]: umounthook <debug>: prestart container_id:f92f4b74698a rootfs:/var/lib/docker/devicemapper/mnt/451a040fd9afc89f24aae64bd046d1755e841c4c25e5ad515f7b5d6abe05d487/rootfs
Dec 14 13:59:12 ip-172-18-2-50.ec2.internal oci-umount[112015]: umounthook <error>: f92f4b74698a: Failed to read directory /usr/share/oci-umount/oci-umount.d: No such file or directory
Dec 14 13:59:14 ip-172-18-2-50.ec2.internal dockerd-current[16589]: time="2017-12-14T13:59:14.909237523Z" level=error msg="devmapper: Error unmounting device 46d82310de547f7f9e3e87d95f6a5af6420fe4b69ea3e1142bfc86486999e0f0: invalid argument"
Dec 14 13:59:14 ip-172-18-2-50.ec2.internal dockerd-current[16589]: time="2017-12-14T13:59:14.909278876Z" level=error msg="Error unmounting container b7f14c8009aee731cd5199c9cce74b5b46912ffd0175c26b3ff73f2139a8212b: invalid argument"
Dec 14 13:59:14 ip-172-18-2-50.ec2.internal dockerd-current[16589]: time="2017-12-14T13:59:14.911250303Z" level=error msg="Handler for POST /v1.24/containers/b7f14c8009aee731cd5199c9cce74b5b46912ffd0175c26b3ff73f2139a8212b/start returned error: devmapper: Error activating devmapper device for '46d82310de547f7f9e3e87d95f6a5af6420fe4b69ea3e1142bfc86486999e0f0': devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed"
Dec 14 13:59:14 ip-172-18-2-50.ec2.internal dockerd-current[16589]: time="2017-12-14T13:59:14.911285913Z" level=error msg="Handler for POST /v1.24/containers/b7f14c8009aee731cd5199c9cce74b5b46912ffd0175c26b3ff73f2139a8212b/start returned error: devmapper: Error activating devmapper device for '46d82310de547f7f9e3e87d95f6a5af6420fe4b69ea3e1142bfc86486999e0f0': devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed"
Dec 14 13:59:16 ip-172-18-2-50.ec2.internal oci-register-machine[112541]: 2017/12/14 13:59:16 Register machine: prestart 158ff1a13e78d92571cf9cd2de8df046b399b82a124cb134ce9b6e07bfc82912 112533 /var/lib/docker/devicemapper/mnt/e56517f04a762ed475c067b149aa6020d1cc8d09a64041a0a967511951aab1d3/rootfs
Dec 14 13:59:16 ip-172-18-2-50.ec2.internal oci-umount[112548]: umounthook <debug>: prestart container_id:158ff1a13e78 rootfs:/var/lib/docker/devicemapper/mnt/e56517f04a762ed475c067b149aa6020d1cc8d09a64041a0a967511951aab1d3/rootfs

[tnozicka]

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/18060/test_pull_request_origin_extended_conformance_install/5312/

[stevekuznetsov]

The AMIs now have docker-1.12.6-71.git3e8e77d.el7

[bparees]

@jwhonce do we think this would have also been addressed by the devmapper fix in the new docker? it's a different error.