[FEAT] Note switch to nvidia-container-toolkit

[Fyko]

Is this a new feature request?

• I have searched the existing issues

Wanted change

As of NVIDIA/nvidia-docker@69b790e, https://github.com/NVIDIA/nvidia-docker is deprecated. Everyone should switch to https://github.com/NVIDIA/nvidia-container-toolkit.

Reason for change

nvidia-docker deprecated

Proposed code change

documentation (i can take on this feat)

[github-actions]

Thanks for opening your first issue here! Be sure to follow the relevant issue templates, or risk having this issue marked as invalid.

[TopCheddar27]

I just wanted to add my experience here testing this change. In short, I do not believe that the necessary changes have been made to properly send ffmpeg requests to the new Nvidia-Container-Toolkit integration. You see the job make it to the GPU, and you even see some compute happen. However it crashes out after a second or two.

The kicker is, if you select "Convert Automatically", it does seem to do some consistent compute for playback. It did seem to be a drop in replacement for the previous nvidia-docker setup. It should use the same "nvidia" runtime as defined within the deamon.json.

Not sure what needs to be changed / tested, but am willing to test in my environment! Below is some specs.

Relevant Specs: Ubuntu 20.04 LTS - 1650 Super - NVIDIA Driver 470.223.02-0ubuntu0.20.04.1

Edit: Adding Some relevant screenshots and info for how Plex interacts with the Hosts GPU. When playing the original file, all is well. When switching to a encode option (This example was 8mbps), you can see TWO jobs enter into the GPU (pictured below in NVTOP). It will stay there for 2-3 seconds, then they will both immediately be purged from the process list. Playback for the client during this time will hang, then crash back to Home.

Are the flags "NVIDIA_DRIVER_CAPABILITIES" and "NVIDIA_VISIBLE_DEVICES" still needed? Documentation for the new toolkit is very slim.

Has anyone else been successful getting the new toolkit working with current drivers?

My configuration seems solid according to: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/sample-workload.html. As the container correctly runs nvidia-smi and correctly lists the GPU.

[cohenchris]

I haven't dived very deep into this, but it seems that this issue has broken my instance. When creating the plex container, I see this error message:

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: nvml error: driver/library version mismatch: unknown

The relevant parts of my docker-compose file look like this:

deploy:                                                                                                                        
  resources:                                                                                                                     
    reservations:                                                                                                                  
      devices:                                                                                                                       
        - capabilities: [gpu]
environment:
 - PGID
 - PUID
 - TZ
 - DOCKER_MODS=gilbn/theme.park:plex # custom CSS 
 - PLEX_CLAIM=${PLEX_CLAIM}
 - NVIDIA_VISIBLE_DEVICES=all
 - NVIDIA_DRIVER_CAPABILITIES=compute,video,utility

I have changed nothing recently.

Edit: This ended up being an issue with me updating nvidia drivers on the system. Rebooting fixed it.

[Fyko]

@cohenchris with this new toolkit, you don't need to provision the GPU to the container. since it's a custom docker runtime, it handles the resources (at least the GPU, in this case) when you add runtime: nvidia

[LinuxServer-CI]

This issue has been automatically marked as stale because it has not had recent activity. This might be due to missing feedback from OP. It will be closed if no further activity occurs. Thank you for your contributions.

[Fyko]

stale me not

[LinuxServer-CI]

This issue has been automatically marked as stale because it has not had recent activity. This might be due to missing feedback from OP. It will be closed if no further activity occurs. Thank you for your contributions.