Linux OS with .NET- The configured user limit (128) on the number of inotify instances has been reached

[travisgosselin]

This issue is directly related to and a followup of dotnet/aspnetcore#7531 since that issue has been locked. The same issue occurs now in .NET Core 3.1. Following the previous thread, I was able to specify a different user in my Dockerfile to limit this problem which occurs when putting too many docker containers on the same host that are using the same user UID. Our teams now use a randomized UID to ensure each immutable build has its own ID:

RUN useradd --uid $(shuf -i 2000-65000 -n 1) app
USER app

This works to limit the issue, but the problem still continues if you intend to run the same instance of that immutable container with 9 or more instances on the same container orchestrator host. For example if I use the following image: "FROM mcr.microsoft.com/dotnet/core/aspnet:3." to run a base default / empty ASP.NET Core application with the following Program.cs (and usage of zero-configuration files):

Host.CreateDefaultBuilder(args)
                .ConfigureWebHost(builder =>
                {
                    builder
                        .UseKestrel()
                        .UseStartup<Startup>();
                });

Run a default app in the container above, and attach your shell and execute the following to count the number of watchers:

apt-get install lsof
lsof | grep inotify

.NET Core 3.1 shows me generally 15 or 16, while .NET Core 2.1 default app shows more like 40.
If I am running 8 containers each taking at least 15 instances of the 128 limit - I run out pretty quickly.

General concerns:

1. Scaling is the first concern I have for obvious reasons (depending on my placement strategy). Is the expectation that I should be increasing this limit on my orchestrator hosts to allow for this?
2. With a default execution of ASP.NET Core 3.1 that uses no configuration files, why am I seeing 15 instances? Typically speaking in a container there is very little reason I even want to watch files (especially not configuration files) since its entirely immutable. Is there a way to entirely turn off the inotify watcher instances all together (which would solve this problem entirely)? Or is there an underlying bug in ASP.NET Core here?

EDIT by @Rick-Anderson #28387 mitigates the problem.

---

Associated WorkItem - 64021

[odin88]

+1 I face the same issue.

[johnnyreilly]

I'm also bumping hard on this issue; mostly in the context of being able to run integration tests inside a devcontainer. After a certain number tests in my project have been run I start to experience this problem. I'd love for this issue to be solved generally in dot net core.

I do wonder if it might be possible for Host.CreateDefaultBuilder to have an overload which
did not use reloadOnChange: true and instead using reloadOnChange: false. If that was in place then people would be able to pivot easily to at least using less watchers.

The ideal solution is the underlying issue being fixed though 😄

https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/generic-host?view=aspnetcore-3.1

[travisgosselin]

Thanks for jumping in @johnnyreilly - yes we are experiencing in our dev containers as well during builds. It can be recreated just based on the shared host. So it could be a build agent (single host) running many tests from different projects that hit this problem randomly, or it can be a single project WITH A LOT OF TESTS. frustrating to have to worry about this issue horizontally and vertically as well. Still trying to figure out what to do next... and additionally wondering why there are any default file watchers that I cannot disable, especially running in an immutable container in most instances....

Started to blog this journey a bit for our organization here: http://blog.travisgosselin.com/configured-user-limit-inotify-instances/

[BrennanConroy]

We should look into adding a doc about ulimit on mac and linux.

Also, mention using the polling file watcher to reduce inotify instances.

[Rick-Anderson]

@travisgosselin what doc should this go in?

[travisgosselin]

Good question @Rick-Anderson - Ideally it wouldn't be a concern at all and be fixed, but for the Docs specifically, it seems prudent to call out these best practices out in any location that discusses Dockerfile usage patterns for ASP.NET Core. Existing, "The Dockerfile" section here might be a starting place; it already indicates COPY of csproj and sln files as a good practice for docker layer caching (https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/docker/building-net-docker-images?view=aspnetcore-3.1). Calling out additionally some of the security practices described here for not running as root and the side effects from a port security perspective (i.e. has to be above port 1000) might make sense?

Perhaps here as well? https://docs.microsoft.com/en-us/dotnet/core/docker/build-container?tabs=windows
Or perhaps this belongs closer to security? A new page as a sibling or close to something like this related to hosting: https://docs.microsoft.com/en-us/aspnet/core/security/docker-https?view=aspnetcore-3.1

[vasnab]

+1

[Rick-Anderson]

@BrennanConroy @travisgosselin several places report fixing this via reloadOnChange: false);

.AddJsonFile($"appsettings.json", optional: true, reloadOnChange: false);

See https://stackoverflow.com/a/56547116/502537