Bug#949171: linux: Please, enable COMPACTION for armel/marvell to mitigate OOMs
Source: linux
Version: 4.19.67-2+deb10u2
Severity: important
X-Debbugs-CC: debian-arm@lists.debian.org
Hi, Ben and others.
I've been getting some OOM errors on my armel/kirkwood device running Debian
stable. One of the reasons for this is, according to the dmesg logs, that
COMPACT (memory compaction) is tried, but not available:
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
$ grep -i compact /boot/config-4.19.0-6-marvell
# CONFIG_COMPACTION is not set
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Here is a snippet of the relevant part of the dmesg logs from an earlier
kernel, but the problem still manifests:
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(...)
[3558478.729941] kswapd0: page allocation failure: order:0, mode:0x480020(GFP_ATOMIC), nodemask=(null)
[3558478.738990] CPU: 0 PID: 20 Comm: kswapd0 Not tainted 4.19.0-5-marvell #1 Debian 4.19.37-5
[3558478.747317] Hardware name: Marvell Orion5x (Flattened Device Tree)
[3558478.753697] [<c0016c6c>] (unwind_backtrace) from [<c001370c>] (show_stack+0x18/0x1c)
[3558478.761620] [<c001370c>] (show_stack) from [<c00f5af0>] (warn_alloc+0x90/0x13c)
[3558478.769110] [<c00f5af0>] (warn_alloc) from [<c00f61f8>] (__alloc_pages_nodemask+0x5c4/0xa30)
[3558478.777725] [<c00f61f8>] (__alloc_pages_nodemask) from [<c00f6744>] (page_frag_alloc+0x64/0x150)
[3558478.786693] [<c00f6744>] (page_frag_alloc) from [<c02f9e84>] (__netdev_alloc_skb+0x84/0xfc)
[3558478.795268] [<c02f9e84>] (__netdev_alloc_skb) from [<bf3f12a0>] (rxq_refill+0x74/0x174 [mv643xx_eth])
[3558478.804739] [<bf3f12a0>] (rxq_refill [mv643xx_eth]) from [<bf3f222c>] (mv643xx_eth_poll+0x524/0x558 [mv643xx_eth])
[3558478.815289] [<bf3f222c>] (mv643xx_eth_poll [mv643xx_eth]) from [<c030f01c>] (net_rx_action+0x124/0x380)
[3558478.824857] [<c030f01c>] (net_rx_action) from [<c000a1c8>] (__do_softirq+0x1c0/0x2b0)
[3558478.832862] [<c000a1c8>] (__do_softirq) from [<c0025ae0>] (irq_exit+0x80/0xd4)
[3558478.840273] [<c0025ae0>] (irq_exit) from [<c00567c8>] (__handle_domain_irq+0x7c/0xa8)
[3558478.848298] [<c00567c8>] (__handle_domain_irq) from [<c023b22c>] (orion_handle_irq+0x7c/0x88)
[3558478.857002] [<c023b22c>] (orion_handle_irq) from [<c00099e8>] (__irq_svc+0x68/0x84)
[3558478.864818] Exception stack(0xc79a9d10 to 0xc79a9d58)
[3558478.870045] 9d00: c0731338 b4ace000 c795f5a0 c23fe858
[3558478.878392] 9d20: c79a9d74 c0aad000 00000001 c059e4fa 00000319 c04829bd 00000001 b4ace000
[3558478.886733] 9d40: 00cd914f c79a9d60 c011dcc8 c011dbb0 a0000013 ffffffff
[3558478.893548] [<c00099e8>] (__irq_svc) from [<c011dbb0>] (page_vma_mapped_walk+0x40/0x168)
[3558478.901830] [<c011dbb0>] (page_vma_mapped_walk) from [<c011e400>] (page_referenced_one+0x54/0x164)
[3558478.910969] [<c011e400>] (page_referenced_one) from [<c011e778>] (rmap_walk_anon+0x94/0x12c)
[3558478.919588] [<c011e778>] (rmap_walk_anon) from [<c011fccc>] (page_referenced+0x140/0x174)
[3558478.927937] [<c011fccc>] (page_referenced) from [<c0103268>] (shrink_active_list+0x2c0/0x408)
[3558478.936636] [<c0103268>] (shrink_active_list) from [<c0103700>] (shrink_node+0x350/0x764)
[3558478.944981] [<c0103700>] (shrink_node) from [<c01044d0>] (kswapd+0x41c/0x5d4)
[3558478.952293] [<c01044d0>] (kswapd) from [<c003b7bc>] (kthread+0x120/0x138)
[3558478.959264] [<c003b7bc>] (kthread) from [<c00090e0>] (ret_from_fork+0x14/0x34)
[3558478.966640] Exception stack(0xc79a9fb0 to 0xc79a9ff8)
[3558478.971864] 9fa0: 00000000 00000000 00000000 00000000
[3558478.980211] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[3558478.988552] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[3558478.995319] Mem-Info:
[3558478.997805] active_anon:3221 inactive_anon:2017 isolated_anon:32
active_file:9147 inactive_file:8585 isolated_file:0
unevictable:0 dirty:199 writeback:0 unstable:0
slab_reclaimable:3447 slab_unreclaimable:1480
mapped:3434 shmem:1 pagetables:422 bounce:0
free:985 free_pcp:6 free_cma:0
[3558479.031259] Node 0 active_anon:12884kB inactive_anon:8068kB active_file:36588kB inactive_file:34340kB unevictable:0kB isolated(anon):128kB isolated(file):0kB mapped:13736kB dirty:796kB writeback:0kB shmem:4kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[3558479.054496] Normal free:3940kB min:2048kB low:2560kB high:3072kB active_anon:12884kB inactive_anon:8068kB active_file:36588kB inactive_file:34340kB unevictable:0kB writepending:796kB present:131072kB managed:124152kB mlocked:0kB kernel_stack:600kB pagetables:1688kB bounce:0kB free_pcp:24kB local_pcp:24kB free_cma:0kB
[3558479.082604] lowmem_reserve[]: 0 0 0
[3558479.086277] Normal: 13*4kB (H) 8*8kB (H) 9*16kB (H) 5*32kB (H) 7*64kB (H) 4*128kB (H) 0*256kB 1*512kB (H) 0*1024kB 1*2048kB (H) 0*4096kB = 3940kB
[3558479.099586] 18154 total pagecache pages
[3558479.103589] 421 pages in swap cache
[3558479.107247] Swap cache stats: add 184262, delete 183841, find 531277/573632
[3558479.114360] Free swap = 462844kB
[3558479.117841] Total swap = 524284kB
[3558479.121323] 32768 pages RAM
[3558479.124286] 0 pages HighMem/MovableOnly
[3558479.128287] 1730 pages reserved
[3558479.234279] apt: page allocation failure: order:0, mode:0x480020(GFP_ATOMIC), nodemask=(null)
[3558479.242980] CPU: 0 PID: 25580 Comm: apt Not tainted 4.19.0-5-marvell #1 Debian 4.19.37-5
[3558479.251220] Hardware name: Marvell Orion5x (Flattened Device Tree)
[3558479.257599] [<c0016c6c>] (unwind_backtrace) from [<c001370c>] (show_stack+0x18/0x1c)
[3558479.265521] [<c001370c>] (show_stack) from [<c00f5af0>] (warn_alloc+0x90/0x13c)
[3558479.273013] [<c00f5af0>] (warn_alloc) from [<c00f61f8>] (__alloc_pages_nodemask+0x5c4/0xa30)
[3558479.281627] [<c00f61f8>] (__alloc_pages_nodemask) from [<c00f6744>] (page_frag_alloc+0x64/0x150)
[3558479.290595] [<c00f6744>] (page_frag_alloc) from [<c02f9e84>] (__netdev_alloc_skb+0x84/0xfc)
[3558479.299171] [<c02f9e84>] (__netdev_alloc_skb) from [<bf3f12a0>] (rxq_refill+0x74/0x174 [mv643xx_eth])
[3558479.308625] [<bf3f12a0>] (rxq_refill [mv643xx_eth]) from [<bf3f222c>] (mv643xx_eth_poll+0x524/0x558 [mv643xx_eth])
[3558479.319175] [<bf3f222c>] (mv643xx_eth_poll [mv643xx_eth]) from [<c030f01c>] (net_rx_action+0x124/0x380)
[3558479.328741] [<c030f01c>] (net_rx_action) from [<c000a1c8>] (__do_softirq+0x1c0/0x2b0)
[3558479.336746] [<c000a1c8>] (__do_softirq) from [<c0025ae0>] (irq_exit+0x80/0xd4)
[3558479.344147] [<c0025ae0>] (irq_exit) from [<c00567c8>] (__handle_domain_irq+0x7c/0xa8)
[3558479.352169] [<c00567c8>] (__handle_domain_irq) from [<c023b22c>] (orion_handle_irq+0x7c/0x88)
[3558479.360876] [<c023b22c>] (orion_handle_irq) from [<c00099e8>] (__irq_svc+0x68/0x84)
[3558479.368684] Exception stack(0xc0b1fb28 to 0xc0b1fb70)
[3558479.373917] fb20: c7fab600 00000001 00000000 00010408 c7fab600 c7f5eda0
[3558479.382261] fb40: 00005930 c7fab604 c0b1fbf0 c05dadb4 c055d8e0 00000100 00000002 c0b1fb78
[3558479.390596] fb60: c00f49f0 c00f2a70 60000013 ffffffff
[3558479.395838] [<c00099e8>] (__irq_svc) from [<c00f2a70>] (free_pcp_prepare+0x40/0x98)
[3558479.403672] [<c00f2a70>] (free_pcp_prepare) from [<c00f49f0>] (free_unref_page_list+0x84/0x160)
[3558479.412547] [<c00f49f0>] (free_unref_page_list) from [<c01017f4>] (shrink_page_list+0x88/0xee8)
[3558479.421418] [<c01017f4>] (shrink_page_list) from [<c0102e08>] (shrink_inactive_list+0x224/0x3c4)
[3558479.430376] [<c0102e08>] (shrink_inactive_list) from [<c0103a84>] (shrink_node+0x6d4/0x764)
[3558479.438896] [<c0103a84>] (shrink_node) from [<c0103fd8>] (try_to_free_pages+0x4c4/0x5a0)
[3558479.447159] [<c0103fd8>] (try_to_free_pages) from [<c00f6160>] (__alloc_pages_nodemask+0x52c/0xa30)
[3558479.456384] [<c00f6160>] (__alloc_pages_nodemask) from [<c00faa7c>] (__do_page_cache_readahead+0x11c/0x1a0)
[3558479.466305] [<c00faa7c>] (__do_page_cache_readahead) from [<c00ef2e4>] (generic_file_read_iter+0x2c8/0x8c4)
[3558479.476230] [<c00ef2e4>] (generic_file_read_iter) from [<c01326d4>] (new_sync_read+0xec/0x10c)
[3558479.485025] [<c01326d4>] (new_sync_read) from [<c0134744>] (vfs_read+0x98/0x14c)
[3558479.492602] [<c0134744>] (vfs_read) from [<c0134c34>] (ksys_read+0x44/0x7c)
[3558479.499743] [<c0134c34>] (ksys_read) from [<c0009000>] (ret_fast_syscall+0x0/0x50)
[3558479.507468] Exception stack(0xc0b1ffa8 to 0xc0b1fff0)
[3558479.512702] ffa0: b6e2f780 be90bf4c 00000004 02510059 00007f7a 00000000
[3558479.521053] ffc0: b6e2f780 be90bf4c 00007f7a 00000003 be90be00 02510059 00007f7a 00000000
[3558479.529387] ffe0: 0000006c be90bd70 b6e29768 b6b847e0
[3558627.671129] git invoked oom-killer: gfp_mask=0x7080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), nodemask=(null), order=1, oom_score_adj=0
[3558627.682817] COMPACTION is disabled!!!
[3558627.686762] CPU: 0 PID: 25994 Comm: git Not tainted 4.19.0-5-marvell #1 Debian 4.19.37-5
[3558627.695079] Hardware name: Marvell Orion5x (Flattened Device Tree)
[3558627.701529] [<c0016c6c>] (unwind_backtrace) from [<c001370c>] (show_stack+0x18/0x1c)
[3558627.709530] [<c001370c>] (show_stack) from [<c00f2104>] (dump_header.constprop.4+0x7c/0x1c4)
[3558627.718251] [<c00f2104>] (dump_header.constprop.4) from [<c00f1378>] (oom_kill_process+0x90/0x404)
[3558627.727471] [<c00f1378>] (oom_kill_process) from [<c00f1ff0>] (out_of_memory+0x394/0x3e0)
[3558627.735904] [<c00f1ff0>] (out_of_memory) from [<c00f6474>] (__alloc_pages_nodemask+0x840/0xa30)
[3558627.744862] [<c00f6474>] (__alloc_pages_nodemask) from [<c001f688>] (copy_process.part.4+0x118/0x16d0)
[3558627.754424] [<c001f688>] (copy_process.part.4) from [<c0020de4>] (_do_fork+0xe0/0x3a4)
[3558627.762603] [<c0020de4>] (_do_fork) from [<c0021190>] (sys_clone+0x24/0x2c)
[3558627.769822] [<c0021190>] (sys_clone) from [<c0009000>] (ret_fast_syscall+0x0/0x50)
[3558627.777636] Exception stack(0xc4a11fa8 to 0xc4a11ff0)
[3558627.782956] 1fa0: b68654c8 b6865460 003d0f00 b6864f98 b68654c8 b6865920
[3558627.791377] 1fc0: b68654c8 b6865460 be817efc 00000078 be817efa b6865460 be817efc be817f44
[3558627.799792] 1fe0: 003d0f00 be817e80 b6ea2194 b6e0c858
[3558627.805077] Mem-Info:
[3558627.807633] active_anon:2575 inactive_anon:1136 isolated_anon:0
active_file:11462 inactive_file:7830 isolated_file:6
unevictable:0 dirty:25 writeback:1 unstable:0
slab_reclaimable:3121 slab_unreclaimable:1482
mapped:10690 shmem:2 pagetables:475 bounce:0
free:1201 free_pcp:36 free_cma:0
[3558627.841339] Node 0 active_anon:10300kB inactive_anon:4544kB active_file:45848kB inactive_file:31320kB unevictable:0kB isolated(anon):0kB isolated(file):24kB mapped:42760kB dirty:100kB writeback:4kB shmem:8kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[3558627.864566] Normal free:4804kB min:2048kB low:2560kB high:3072kB active_anon:10300kB inactive_anon:4544kB active_file:45848kB inactive_file:31320kB unevictable:0kB writepending:104kB present:131072kB managed:124152kB mlocked:0kB kernel_stack:656kB pagetables:1900kB bounce:0kB free_pcp:144kB local_pcp:144kB free_cma:0kB
[3558627.892933] lowmem_reserve[]: 0 0 0
[3558627.896686] Normal: 229*4kB (UMEH) 8*8kB (H) 9*16kB (H) 5*32kB (H) 7*64kB (H) 4*128kB (H) 0*256kB 1*512kB (H) 0*1024kB 1*2048kB (H) 0*4096kB = 4804kB
[3558627.910421] 19527 total pagecache pages
[3558627.914496] 230 pages in swap cache
[3558627.918222] Swap cache stats: add 185216, delete 184986, find 531476/573984
[3558627.925407] Free swap = 461564kB
[3558627.928955] Total swap = 524284kB
[3558627.932505] 32768 pages RAM
[3558627.935538] 0 pages HighMem/MovableOnly
[3558627.939606] 1730 pages reserved
[3558627.942971] Tasks state (memory values in pages):
[3558627.947902] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[3558627.956758] [ 134] 0 134 15558 283 75776 163 0 systemd-journal
[3558627.966037] [ 152] 0 152 1180 0 12288 47 0 blkmapd
[3558627.974624] [ 160] 0 160 4604 32 18432 140 -1000 systemd-udevd
[3558627.983731] [ 165] 108 165 5779 536 20480 138 0 systemd-network
[3558627.993020] [ 226] 0 226 850 0 10240 52 0 rpc.idmapd
[3558628.001877] [ 228] 107 228 5631 200 24576 149 0 systemd-timesyn
[3558628.011156] [ 229] 113 229 1621 51 14336 78 0 rpcbind
[3558628.019743] [ 230] 0 230 8335 80 40960 6507 0 rpc.mountd
[3558628.028590] [ 240] 0 240 1867 136 16384 39 0 cron
[3558628.036920] [ 241] 0 241 3397 501 20480 138 0 systemd-logind
[3558628.046111] [ 243] 102 243 1789 422 14336 57 -900 dbus-daemon
[3558628.055045] [ 247] 103 247 1615 223 14336 68 0 avahi-daemon
[3558628.064090] [ 256] 0 256 1510 376 14336 154 0 smartd
[3558628.072598] [ 262] 0 262 2749 27 18432 157 -1000 sshd
[3558628.080931] [ 267] 103 267 1583 0 14336 72 0 avahi-daemon
[3558628.089952] [ 270] 0 270 1530 0 14336 32 0 agetty
[3558628.098454] [ 271] 0 271 590 82 10240 21 0 syslogd
[3558628.107052] [ 283] 33 283 2572 163 20480 303 0 lighttpd
[3558628.115727] [ 15189] 0 15189 11888 266 55296 445 0 smbd
[3558628.124053] [ 15191] 0 15191 11211 69 49152 457 0 smbd-notifyd
[3558628.133072] [ 15192] 0 15192 11214 73 49152 457 0 cleanupd
[3558628.141748] [ 15205] 0 15205 7792 266 36864 313 0 nmbd
[3558628.150079] [ 24970] 0 24970 3154 52 18432 203 0 sshd
[3558628.158405] [ 24973] 1000 24973 3677 48 20480 223 0 systemd
[3558628.166994] [ 24974] 1000 24974 8957 51 30720 433 0 (sd-pam)
[3558628.175667] [ 24987] 1000 24987 3239 225 18432 183 0 sshd
[3558628.183998] [ 24988] 1000 24988 2084 45 14336 94 0 bash
[3558628.192326] [ 24993] 1000 24993 2528 50 18432 96 0 su
[3558628.200482] [ 24994] 0 24994 2175 567 14336 47 0 bash
[3558628.208813] [ 25056] 104 25056 33975 134 94208 4667 0 minidlnad
[3558628.217584] [ 25954] 0 25954 13164 11044 57344 0 0 apt
[3558628.225825] [ 25959] 0 25959 13164 3085 30720 0 0 apt
[3558628.234066] [ 25960] 0 25960 501 276 8192 0 0 sh
[3558628.242223] [ 25961] 0 25961 501 296 10240 0 0 etckeeper
[3558628.250983] [ 25986] 0 25986 501 274 8192 0 0 50uncommitted-c
[3558628.260263] [ 25987] 0 25987 501 330 8192 0 0 etckeeper
[3558628.269022] [ 25993] 0 25993 501 303 10240 0 0 50test
[3558628.277524] [ 25994] 0 25994 4599 918 16384 0 0 git
[3558628.285758] Out of memory: Kill process 25954 (apt) score 68 or sacrifice child
[3558628.293325] Killed process 25959 (apt) total-vm:52656kB, anon-rss:10724kB, file-rss:1616kB, shmem-rss:0kB
[3558628.330213] oom_reaper: reaped process 25959 (apt), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[3559321.964279] systemd[1]: Stopping Journal Service...
[3559321.969754] systemd-journald[134]: Received SIGTERM from PID 1 (systemd).
[3559322.475409] systemd-journal: 56 output lines suppressed due to ratelimiting
[3559322.486605] systemd[1]: systemd-journald.service: Succeeded.
[3559322.498942] systemd[1]: Stopped Journal Service.
[3559322.568880] systemd[1]: Starting Journal Service...
[3559323.616592] systemd[1]: Started Journal Service.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Can CONFIG_COMPACT please be enabled for armel (or only marvel, if that's
the case)?
Thanks in advance,
Rogério Brito.
P.S.: Please, disregard the information collected by reportbug after this,
as I am writing this message on an amd64 system. I will possibly file a bug
report regarding the TAINT_WARN for this system that I am using after I
debug it a little bit.
-- System Information:
Debian Release: bullseye/sid
APT prefers testing
APT policy: (500, 'testing'), (200, 'unstable'), (150, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 5.4.0-2-amd64 (SMP w/4 CPU cores)
Kernel taint flags: TAINT_WARN
Locale: LANG=en_US.utf-8, LC_CTYPE=pt_BR.utf-8 (charmap=UTF-8), LANGUAGE=en_US.utf-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
--
Rogério Brito : rbrito@{ime.usp.br,gmail.com} : GPG key 4096R/BCFCAAAA
http://cynic.cc/blog/ : github.com/rbrito : profiles.google.com/rbrito
DebianQA: http://qa.debian.org/developer.php?login=rbrito%40ime.usp.br
Reply to: