Bug#939466: linux-image-4.19.0-5-amd64: Xen xl block-attach occasionally causes page allocation failure in DomU
Package: src:linux
Version: 4.19.37-5+deb10u2
Severity: normal
Tags: upstream
Dear Maintainer,
I have dozens of DomUs on several Debian Dom0s. I do backups of them by taking LVM snapshots on Dom0 and attaching them to DomU. Occasionally the block-attach fails with page allocation failure in DomU:
Sep 4 20:13:38 openvpn kernel: [696863.453080] xenwatch: page allocation failure: order:5,
I asked for help on xen-users mailing list and this is what I got:
You want upstream Linux kernel commit 1d5c76e66433382a1e170d in your guest kernels. This is part of Linux 5.2, there are stable backports in 5.1.9 and 4.19.50.
-- Package-specific info:
** Version:
Linux version 4.19.0-5-amd64 (debian-kernel@lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.37-5+deb10u2 (2019-08-08)
** Command line:
root=UUID=dc5ac2a4-96eb-4d9a-9682-ddf5d144bae4 ro root=/dev/xvda2 ro
** Not tainted
** Kernel log:
[ 13.697337] br72: port 2(tap72) entered blocking state
[ 13.697339] br72: port 2(tap72) entered forwarding state
[ 13.960185] NET: Registered protocol family 15
[ 14.207986] Initializing XFRM netlink socket
[ 5432.432860] blkfront: xvdz1: barrier or flush: disabled; persistent grants: enabled; indirect descriptors: enabled;
[ 5453.406762] EXT4-fs (xvdz1): 4 orphan inodes deleted
[ 5453.406770] EXT4-fs (xvdz1): recovery complete
[ 5486.102597] EXT4-fs (xvdz1): mounted filesystem with ordered data mode. Opts: (null)
[33634.037988] FS-Cache: Loaded
[33634.084083] FS-Cache: Netfs 'nfs' registered for caching
[33634.111960] Key type dns_resolver registered
[33634.143794] NFS: Registering the id_resolver key type
[33634.143807] Key type id_resolver registered
[33634.143809] Key type id_legacy registered
[92355.053919] blkfront: xvdz1: barrier or flush: disabled; persistent grants: enabled; indirect descriptors: enabled;
[92355.863062] EXT4-fs (xvdz1): 4 orphan inodes deleted
[92355.863071] EXT4-fs (xvdz1): recovery complete
[92355.941764] EXT4-fs (xvdz1): mounted filesystem with ordered data mode. Opts: (null)
[178602.427415] blkfront: xvdz1: barrier or flush: disabled; persistent grants: enabled; indirect descriptors: enabled;
[178603.228916] EXT4-fs (xvdz1): 5 orphan inodes deleted
[178603.228924] EXT4-fs (xvdz1): recovery complete
[178603.339181] EXT4-fs (xvdz1): mounted filesystem with ordered data mode. Opts: (null)
[264577.972563] blkfront: xvdz1: barrier or flush: disabled; persistent grants: enabled; indirect descriptors: enabled;
[264578.739261] EXT4-fs (xvdz1): 5 orphan inodes deleted
[264578.739270] EXT4-fs (xvdz1): recovery complete
[264578.858172] EXT4-fs (xvdz1): mounted filesystem with ordered data mode. Opts: (null)
[351512.679429] blkfront: xvdz1: barrier or flush: disabled; persistent grants: enabled; indirect descriptors: enabled;
[351513.719269] EXT4-fs (xvdz1): 5 orphan inodes deleted
[351513.719277] EXT4-fs (xvdz1): recovery complete
[351513.833490] EXT4-fs (xvdz1): mounted filesystem with ordered data mode. Opts: (null)
[437227.779596] blkfront: xvdz1: barrier or flush: disabled; persistent grants: enabled; indirect descriptors: enabled;
[437229.897525] EXT4-fs (xvdz1): 5 orphan inodes deleted
[437229.897533] EXT4-fs (xvdz1): recovery complete
[437230.078057] EXT4-fs (xvdz1): mounted filesystem with ordered data mode. Opts: (null)
[524139.971324] blkfront: xvdz1: barrier or flush: disabled; persistent grants: enabled; indirect descriptors: enabled;
[524143.381537] EXT4-fs (xvdz1): 5 orphan inodes deleted
[524143.381545] EXT4-fs (xvdz1): recovery complete
[524150.368760] EXT4-fs (xvdz1): mounted filesystem with ordered data mode. Opts: (null)
[610296.225246] blkfront: xvdz1: barrier or flush: disabled; persistent grants: enabled; indirect descriptors: enabled;
[610296.774381] EXT4-fs (xvdz1): 5 orphan inodes deleted
[610296.774390] EXT4-fs (xvdz1): recovery complete
[610296.893241] EXT4-fs (xvdz1): mounted filesystem with ordered data mode. Opts: (null)
[649806.990839] TCP: request_sock_TCP: Possible SYN flooding on port 1194. Sending cookies. Check SNMP counters.
[696863.453080] xenwatch: page allocation failure: order:5, mode:0x60c0c0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null)
[696863.453093] xenwatch cpuset=/ mems_allowed=0
[696863.453099] CPU: 0 PID: 19 Comm: xenwatch Not tainted 4.19.0-5-amd64 #1 Debian 4.19.37-5+deb10u2
[696863.453103] Call Trace:
[696863.453111] dump_stack+0x5c/0x80
[696863.453116] warn_alloc.cold.117+0x6c/0xec
[696863.453120] __alloc_pages_slowpath+0xc73/0xcb0
[696863.453127] ? test_reply.isra.1+0x40/0x40
[696863.453131] ? test_reply.isra.1+0x40/0x40
[696863.453134] __alloc_pages_nodemask+0x28b/0x2b0
[696863.453139] kmalloc_order+0x14/0x30
[696863.453142] kmalloc_order_trace+0x1d/0xa0
[696863.453144] ? test_reply.isra.1+0x40/0x40
[696863.453149] talk_to_blkback+0xc0/0xe00 [xen_blkfront]
[696863.453152] ? xenbus_gather+0x13b/0x160
[696863.453155] ? test_reply.isra.1+0x40/0x40
[696863.453157] blkback_changed+0x133/0xbf1 [xen_blkfront]
[696863.453160] ? test_reply.isra.1+0x40/0x40
[696863.453162] ? xenbus_read_driver_state+0x39/0x60
[696863.453165] ? test_reply.isra.1+0x40/0x40
[696863.453168] xenwatch_thread+0xcf/0x170
[696863.453173] ? finish_wait+0x80/0x80
[696863.453177] kthread+0x112/0x130
[696863.453180] ? kthread_bind+0x30/0x30
[696863.453184] ret_from_fork+0x35/0x40
[696863.453193] Mem-Info:
[696863.453197] active_anon:48394 inactive_anon:1305 isolated_anon:0
active_file:19592 inactive_file:27796 isolated_file:0
unevictable:0 dirty:24 writeback:0 unstable:0
slab_reclaimable:8236 slab_unreclaimable:6425
mapped:6505 shmem:1369 pagetables:569 bounce:0
free:4144 free_pcp:0 free_cma:0
[696863.453211] Node 0 active_anon:193576kB inactive_anon:5220kB active_file:78368kB inactive_file:111184kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:26020kB dirty:96kB writeback:0kB shmem:5476kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[696863.453219] Node 0 DMA free:8456kB min:100kB low:124kB high:148kB active_anon:2592kB inactive_anon:0kB active_file:1988kB inactive_file:1480kB unevictable:0kB writepending:8kB present:15996kB managed:15912kB mlocked:0kB kernel_stack:16kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[696863.453230] lowmem_reserve[]: 0 377 377 377 377
[696863.453233] Node 0 DMA32 free:8120kB min:2432kB low:3040kB high:3648kB active_anon:190984kB inactive_anon:5220kB active_file:76380kB inactive_file:109704kB unevictable:0kB writepending:88kB present:507904kB managed:477256kB mlocked:0kB kernel_stack:2064kB pagetables:2276kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[696863.453244] lowmem_reserve[]: 0 0 0 0 0
[696863.453248] Node 0 DMA: 506*4kB (UM) 346*8kB (UM) 147*16kB (UM) 35*32kB (UM) 3*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 8456kB
[696863.453258] Node 0 DMA32: 379*4kB (UMEH) 446*8kB (UMEH) 96*16kB (UMEH) 29*32kB (UMEH) 9*64kB (UH) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 8124kB
[696863.453268] 48760 total pagecache pages
[696863.453272] 0 pages in swap cache
[696863.453274] Swap cache stats: add 0, delete 0, find 0/0
[696863.453276] Free swap = 0kB
[696863.453277] Total swap = 0kB
[696863.453279] 130975 pages RAM
[696863.453282] 0 pages HighMem/MovableOnly
[696863.453283] 7683 pages reserved
[696863.453285] 0 pages hwpoisoned
[696863.453289] vbd vbd-268441857: 12 allocating ring_info structure
[696863.453347] vbd vbd-268441857: failed to write error node for device/vbd/268441857 (12 allocating ring_info structure)
[745807.497737] blkfront: xvdz1: barrier or flush: disabled; persistent grants: enabled; indirect descriptors: enabled;
[745822.942047] blkfront: xvdz1: barrier or flush: disabled; persistent grants: enabled; indirect descriptors: enabled;
[745856.942053] EXT4-fs (xvdz1): 5 orphan inodes deleted
[745856.942061] EXT4-fs (xvdz1): recovery complete
[745857.028389] EXT4-fs (xvdz1): mounted filesystem with ordered data mode. Opts: (null)
[752024.756193] device-mapper: uevent: version 1.0.3
[752024.756273] device-mapper: ioctl: 4.39.0-ioctl (2018-04-03) initialised: dm-devel@redhat.com
** Model information
** Loaded modules:
dm_mod
rpcsec_gss_krb5
auth_rpcgss
nfsv4
dns_resolver
nfs
lockd
grace
fscache
xfrm_user
xfrm4_tunnel
tunnel4
ipcomp
xfrm_ipcomp
esp4
ah4
af_key
xfrm_algo
tun
bridge
8021q
garp
stp
mrp
llc
ipt_REJECT
nf_reject_ipv4
xt_comment
xt_tcpudp
evdev
xt_state
xt_conntrack
intel_rapl
x86_pkg_temp_thermal
nft_counter
coretemp
crct10dif_pclmul
nft_chain_nat_ipv4
nf_nat_ipv4
crc32_pclmul
xt_nat
nf_nat
nf_conntrack
pcspkr
nf_defrag_ipv6
ghash_clmulni_intel
nf_defrag_ipv4
libcrc32c
intel_rapl_perf
nft_compat
nf_tables
nfnetlink
sunrpc
ip_tables
x_tables
autofs4
ext4
crc16
mbcache
jbd2
crc32c_generic
fscrypto
ecb
crc32c_intel
aesni_intel
xen_netfront
xen_blkfront
aes_x86_64
crypto_simd
cryptd
glue_helper
** PCI devices:
not available
** USB devices:
not available
-- System Information:
Debian Release: 10.0
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 4.19.0-5-amd64 (SMP w/1 CPU core)
Locale: LANG=C, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE=C (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages linux-image-4.19.0-5-amd64 depends on:
ii initramfs-tools [linux-initramfs-tool] 0.133
ii kmod 26-1
ii linux-base 4.6
Versions of packages linux-image-4.19.0-5-amd64 recommends:
pn apparmor <none>
ii firmware-linux-free 3.4
Versions of packages linux-image-4.19.0-5-amd64 suggests:
pn debian-kernel-handbook <none>
pn grub-pc | grub-efi-amd64 | extlinux <none>
pn linux-doc-4.19 <none>
Versions of packages linux-image-4.19.0-5-amd64 is related to:
pn firmware-amd-graphics <none>
pn firmware-atheros <none>
pn firmware-bnx2 <none>
pn firmware-bnx2x <none>
pn firmware-brcm80211 <none>
pn firmware-cavium <none>
pn firmware-intel-sound <none>
pn firmware-intelwimax <none>
pn firmware-ipw2x00 <none>
pn firmware-ivtv <none>
pn firmware-iwlwifi <none>
pn firmware-libertas <none>
pn firmware-linux-nonfree <none>
pn firmware-misc-nonfree <none>
pn firmware-myricom <none>
pn firmware-netxen <none>
pn firmware-qlogic <none>
pn firmware-realtek <none>
pn firmware-samsung <none>
pn firmware-siano <none>
pn firmware-ti-connectivity <none>
pn xen-hypervisor <none>
-- no debconf information
Reply to: