「Docker」- runtime/cgo: pthread_create failed: No space left on device @20210317

问题描述在执行 docker ps 命令时（运行容器、其他命令），产生如下错误：# docker psruntime/cgo: pthread_create failed: No space left on deviceSIGABRT: abortPC=0x7f1a69cace97 m=0 sigcode=18446744073709551610goroutine 0 [idle]:runtime

研究林纳斯的那个系统

2484人浏览 · 2021-03-17 10:20:20

研究林纳斯的那个系统 · 2021-03-17 10:20:20 发布

问题描述

在执行 docker ps 命令时（运行容器、其他命令），产生如下错误：

# docker ps
runtime/cgo: pthread_create failed: No space left on device
SIGABRT: abort
PC=0x7f1a69cace97 m=0 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7f1a69cace97
stack: frame={sp:0x7ffd418bd5d0, fp:0x0} stack=[0x7ffd410bec10,0x7ffd418bdc50)
00007ffd418bd4d0:  0000000000000000  0000000000000000
...
00007ffd418bd540:  000000006e43a318  00005601a65818ac
00007ffd418bd550:  0000000000000000  00005601a508d93e <runtime.callCgoMmap+62>
00007ffd418bd560:  00007ffd418bd568  0000000000000000
00007ffd418bd570:  0000000000000000  00007ffd418bd5b8
00007ffd418bd580:  00005601a50855ba <runtime.mmap.func1+90>  0000000000000000
00007ffd418bd590:  0000000000210800  0000002200000003
00007ffd418bd5a0:  00000000ffffffff  00007f1a67a5d000
00007ffd418bd5b0:  00007ffd418bd5f8  00007ffd418bd630
00007ffd418bd5c0:  00005601a5032113 <runtime.mmap+179>  00007ffd418bd600
00007ffd418bd5d0: <0000000000000000  00007ffd418bd628
...
00007ffd418bd600:  000000006e43a318  00005601a65818ac
00007ffd418bd610:  0000000000210800  00005601a508d93e <runtime.callCgoMmap+62>
00007ffd418bd620:  00007ffd418bd628  00007ffd418bd5f8
00007ffd418bd630:  00007ffd418bd680  00007ffd418bd678
00007ffd418bd640:  00005601a50855ba <runtime.mmap.func1+90>  0000000000000000
00007ffd418bd650:  fffffffe7fffffff  ffffffffffffffff
...
00007ffd418bd6c0:  ffffffffffffffff  ffffffffffffffff
runtime: unknown pc 0x7f1a69cace97
stack: frame={sp:0x7ffd418bd5d0, fp:0x0} stack=[0x7ffd410bec10,0x7ffd418bdc50)
00007ffd418bd4d0:  0000000000000000  0000000000000000
...
00007ffd418bd540:  000000006e43a318  00005601a65818ac
00007ffd418bd550:  0000000000000000  00005601a508d93e <runtime.callCgoMmap+62>
00007ffd418bd560:  00007ffd418bd568  0000000000000000
00007ffd418bd570:  0000000000000000  00007ffd418bd5b8
00007ffd418bd580:  00005601a50855ba <runtime.mmap.func1+90>  0000000000000000
00007ffd418bd590:  0000000000210800  0000002200000003
00007ffd418bd5a0:  00000000ffffffff  00007f1a67a5d000
00007ffd418bd5b0:  00007ffd418bd5f8  00007ffd418bd630
00007ffd418bd5c0:  00005601a5032113 <runtime.mmap+179>  00007ffd418bd600
00007ffd418bd5d0: <0000000000000000  00007ffd418bd628
...
00007ffd418bd600:  000000006e43a318  00005601a65818ac
00007ffd418bd610:  0000000000210800  00005601a508d93e <runtime.callCgoMmap+62>
00007ffd418bd620:  00007ffd418bd628  00007ffd418bd5f8
00007ffd418bd630:  00007ffd418bd680  00007ffd418bd678
00007ffd418bd640:  00005601a50855ba <runtime.mmap.func1+90>  0000000000000000
00007ffd418bd650:  fffffffe7fffffff  ffffffffffffffff
...
00007ffd418bd690:  ffffffffffffffff  ffffffffffffffff

goroutine 1 [running, locked to thread]:
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc0000986c8 sp=0xc0000986c0 pc=0x5601a50899b0
runtime.newproc(0x560100000000, 0x5601a7220820)
        /usr/local/go/src/runtime/proc.go:3255 +0x70 fp=0xc000098710 sp=0xc0000986c8 pc=0x5601a5067500
runtime.init.5()
        /usr/local/go/src/runtime/proc.go:242 +0x37 fp=0xc000098730 sp=0xc000098710 pc=0x5601a505faa7
runtime.doInit(0x5601a8134ac0)
        /usr/local/go/src/runtime/proc.go:5222 +0x8c fp=0xc000098760 sp=0xc000098730 pc=0x5601a506beac
runtime.main()
        /usr/local/go/src/runtime/proc.go:150 +0xcb fp=0xc0000987e0 sp=0xc000098760 pc=0x5601a505f78b
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000987e8 sp=0xc0000987e0 pc=0x5601a508bab1

rax    0x0
rbx    0x7f1a6a05a840
rcx    0x7f1a69cace97
rdx    0x0
rdi    0x2
rsi    0x7ffd418bd5d0
rbp    0x5601a686fb54
rsp    0x7ffd418bd5d0
r8     0x0
r9     0x7ffd418bd5d0
r10    0x8
r11    0x246
r12    0x5601a8e5c3f0
r13    0x0
r14    0x5601a68173ac
r15    0x0
rip    0x7f1a69cace97
rflags 0x246
cs     0x33
fs     0x0
gs     0x0

问题原因

Kernel not freeing memory cgroup causing no space left on device · Issue #29638 · moby/moby

解决方案

我们未尝试过以下解决方案，这里仅根据 GitHub Issue 记录：
1）升级到 4.18 内核（This never happened again after I upgrade Linux to 4.18）
2）定期清理缓存：echo 3 > /proc/sys/vm/drop_caches（参考 Kernel not freeing memory cgroup causing no space left on device 解释）