help request: Program deadlock, cpu usage reaches 100%

[math-chao]

Description

version: apisix:2.15.0-centos in k8s
run for 13 days, and Program deadlock,

error log
prometheus.lua:920: log_error(): Error while setting 'node_info{hostname=""}' to '1': 'no memory', client: xxx.xxxx.xxxx, server: , request: "GET /apisix/prometheus/metrics HTTP/1.1", host: "xxxx"
prometheus.lua:920: log_error(): Error while setting 'etcd_modify_indexes{key="x_etcd_index"}' to '8553921': 'no memory', client: xxx.xxxx.xxx, server: , request: "GET /apisix/prometheus/metrics HTTP/1.1", host: "xxxx"

debug by strace, found a lot of futex calls
futex(0x7fc9d4ded080, FUTEX_WAIT, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7fc9d4ded080, FUTEX_WAKE, 1) = 0
futex(0x7fc9d4ded080, FUTEX_WAIT, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7fc9d4ded080, FUTEX_WAKE, 1) = 0
futex(0x7fc9d4ded080, FUTEX_WAKE, 1) = 0
futex(0x7fc9d4ded080, FUTEX_WAKE, 1) = 0

debug by perf
Samples: 31K of event 'cycles', Event count (approx.): 18419237386
Children Self Command Shared Object Symbol

• 87.95% 0.00% openresty [unknown] [k] 0x0000000000000080
• 84.44% 83.62% openresty nginx [.] ngx_shmtx_lock
• 4.85% 0.10% openresty [kernel.kallsyms] [k] entry_SYSCALL_64_after_hwframe
• 4.57% 0.03% openresty [kernel.kallsyms] [k] do_syscall_64
• 4.36% 0.07% openresty [kernel.kallsyms] [k] __x64_sys_futex
• 4.30% 0.08% openresty [kernel.kallsyms] [k] do_futex

pstack results are as follows
#0 0x0000000000449bf3 in ngx_shmtx_lock (mtx=0x7fc9d4ded068) at src/core/ngx_shmtx.c:78
#1 0x0000000000572caf in ngx_meta_lua_ffi_shdict_get (zone=0x1281138, key=0x7fc9d1d69080 "__ngx_prom__delete_count", key_len=24, value_type=0x7fc9de85f828, str_value_buf=0x7fc9de844380, str_value_len=0x7fc9de875ef8, num_value=0x7fc9de85f868, user_flags=0x7fc9de85f848, get_stale=0, is_stale=0x7fc9de8636f8, err=0x7fc9de8723a8) at /tmp/tmp.YN7822nAJt/openresty-1.21.4.1/../apisix-nginx-module-1.9.0/src/meta/ngx_meta_lua_shdict.c:1631
#2 0x00007fca19caa8d7 in ?? ()
#3 0x00007fc9de85f868 in ?? ()
#4 0x00007fc9de85f848 in ?? ()
#5 0x0000000000000000 in ?? ()

What's the problem？help please

Environment

• APISIX version (run apisix version): 2.15.0-centos
• Operating system (run uname -a): Linux 5.10.131-1.el8.x86_64 change: added doc of how to load plugin. #1 SMP Sun Jul 17 11:20:32 CST 2022 x86_64 x86_64 x86_64 GNU/Linux
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[flearc]

I think you need to check the prometheus shared dict size first. Check metricsapisix_shared_dict_capacity_bytes and apisix_shared_dict_free_space_bytes. If the memory ran out, increase the size in your config file.

[Aaron199]

I have the same problem, and I have found some clues that I hope can help solve the issue.

[shreemaan-abhishek]

@math-chao do you have any updates here?

[github-actions]

Due to lack of the reporter's response this issue has been labeled with "no response". It will be close in 3 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.

[github-actions]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.

[zhaoqiang1980]

@math-chao @Aaron199 my cpu is full,i use perf and flamegraph see that..

I'd like to know how you finally handled this. Did you scale up the shared_dict? thank you..

@leslie-tsang Is this issue fixed? Which version was it fixed in?