io.grpc.netty.shaded.io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 2097152 byte(s) of direct memory (used: 501219607, max: 503316480)

[z514987467]

Issue Description

Type: bug report or feature request

Describe what happened (or what feature you want)

使用三台机器安装Docker，部署Nacos集群，配置如下

docker-compose.yaml

version: "3.7"
services:
  nacos:
    image: nacos/nacos-server:v2.2.2
    container_name: nacos
    networks:
      - nacos_net
    volumes:
      - ./cluster-logs/nacos2:/home/nacos/logs
    ports:
      - "8848:8848"
      - "9848:9848"
      - "9849:9849"
      - "7848:7848"
    env_file:
      - ./env/nacos-ip.env
    restart: on-failure
    
networks:
  nacos_net:
    driver: bridge

nacos-ip.env

NACOS_SERVERS=172.16.30.51:8848 172.16.12.235:8848 172.16.12.29:8848
NACOS_SERVER_IP=172.16.12.235
JVM_XMS=512m
JVM_XMX=512m
JVM_XMN=256m
SPRING_DATASOURCE_PLATFORM=mysql
MYSQL_SERVICE_HOST=172.16.12.235
MYSQL_SERVICE_DB_NAME=nacos
MYSQL_SERVICE_PORT=3306
MYSQL_SERVICE_USER=root
MYSQL_SERVICE_PASSWORD=123456
MYSQL_SERVICE_DB_PARAM=characterEncoding=utf8&connectTimeout=1000&socketTimeout=3000&autoReconnect=true&useSSL=false&serverTimezone=UTC
Dio.grpc.netty.shaded.io.netty.allocation.cacheTrimIntervalMillis=90000

运行一段时间后后端就会报错

Exception in thread "nacos-cluster-grpc-executor-101975" io.grpc.netty.shaded.io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 2097152 byte(s) of direct memory (used: 501219607, max: 503316480)
	at io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:845)
	at io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:774)
	at io.grpc.netty.shaded.io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:649)
	at io.grpc.netty.shaded.io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:624)
	at io.grpc.netty.shaded.io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:203)
	at io.grpc.netty.shaded.io.netty.buffer.PoolArena.tcacheAllocateSmall(PoolArena.java:173)
	at io.grpc.netty.shaded.io.netty.buffer.PoolArena.allocate(PoolArena.java:134)
	at io.grpc.netty.shaded.io.netty.buffer.PoolArena.allocate(PoolArena.java:126)
	at io.grpc.netty.shaded.io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:396)
	at io.grpc.netty.shaded.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
	at io.grpc.netty.shaded.io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:124)
	at io.grpc.netty.shaded.io.grpc.netty.NettyWritableBufferAllocator.allocate(NettyWritableBufferAllocator.java:51)
	at io.grpc.internal.MessageFramer.writeKnownLengthUncompressed(MessageFramer.java:230)
	at io.grpc.internal.MessageFramer.writeUncompressed(MessageFramer.java:169)
	at io.grpc.internal.MessageFramer.writePayload(MessageFramer.java:142)
	at io.grpc.internal.AbstractStream.writeMessage(AbstractStream.java:65)
	at io.grpc.internal.ServerCallImpl.sendMessageInternal(ServerCallImpl.java:172)
	at io.grpc.internal.ServerCallImpl.sendMessage(ServerCallImpl.java:154)
	at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onNext(ServerCalls.java:380)
	at com.alibaba.nacos.core.remote.grpc.GrpcRequestAcceptor.request(GrpcRequestAcceptor.java:91)
	at com.alibaba.nacos.core.remote.grpc.BaseGrpcServer.lambda$addServices$0(BaseGrpcServer.java:141)
	at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
	at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
	at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
	at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
	at io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:355)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:867)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

使用的SDK为nacos-sdk-csharp1.3.5

Describe what you expected to happen

How to reproduce it (as minimally and precisely as possible)

Tell us your environment

Anything else we need to know?

[stone-98]

你这个堆内存可以设置大一点。

[Bo-Qiu]

通过 -XX:MaxDirectMemorySize 参数来指定 DirectMemory 的最大值，把堆外内存设置大些试试。
未设置最大堆外内存大小的话，默认是用的-XMX，你的-XMS -XMX看起来都配置的有些小，建议多分配些

[KomachiSion]

堆外内存超出了， 一般情况下出现在短时间内大量请求的情况。

可以通过-DMaxDirectMemorySize设置最大值，

同时通过-Dio.grpc.netty.shaded.io.netty.allocation.cacheTrimIntervalMillis 调整回收堆外内存频率

[z514987467]

堆外内存超出了， 一般情况下出现在短时间内大量请求的情况。

可以通过-DMaxDirectMemorySize设置最大值，

同时通过-Dio.grpc.netty.shaded.io.netty.allocation.cacheTrimIntervalMillis 调整回收堆外内存频率

好的，谢谢，我试试

[KomachiSion]

No more response from author, I think this is a usage problem.