Issue with Xen PCI passthrough ("swiotlb buffer is full") after Debian Dom0 kernel update to 5.10.178-3 and later
Hi list,
Since an update of package linux-image-amd64 to version 5.10.0-22
(kernel 5.10.178-3) and later, I am experiencing an issue with PCI
passthrough. A WiFi card is still visible in DomU, but it is not working
anymore with the previous settings. I would expect the card to continue
working without any changes.
Does anybody experience similar behavior? Any hints on how to
troubleshoot? Should I file a bug report? For which package?
I have already sent a previous version of this description to xen-users,
but I wasn't able to find a solution yet. Please be aware that my
knowledge about IOMMU/swiotlb is very limited.
I am happy to provide more information if required.
Thanks,
Paul
Description:
------------
Initially, I was running Xen with a Debian Bullseye Dom0 (standard
packages, Xen 4.14) on an APU 4D4 system
(https://pcengines.ch/apu4d4.htm). CPU is AMD Embedded G series
GX-412TC, IOMMU is enabled in BIOS. A mPCIe WiFi card Compex WLE900VX is
passed through to an OpenWRT DomU in PV mode. So far, the system has
been running quite well (aside from some hickups in the early setup
process, with incidentally a very similar behavior which I solved by
increasing Dom0 memory).
In the meantime, I have upgraded Dom0 to Debian Bookworm with
linux-image-amd64 6.1.27-1 and Xen 4.17. The issue persists, seemingly
unchanged.
In DomU dmesg, there are several "swiotlb buffer is full" entries while
failing to load the driver during the boot process:
[ 7.979357] ath10k_pci 0000:00:00.0: Xen PCI mapped GSI40 to IRQ17
[ 7.982856] ath10k_pci 0000:00:00.0: pci irq msi oper_irq_mode 2
irq_mode 0 reset_mode 0
[ 8.258462] ath10k_pci 0000:00:00.0: qca988x hw2.0 target 0x4100016c
chip_id 0x043202ff sub 0000:0000
[ 8.259117] ath10k_pci 0000:00:00.0: kconfig debug 0 debugfs 1
tracing 0 dfs 1 testmode 0
[ 8.261937] ath10k_pci 0000:00:00.0: firmware ver 10.2.4.70.70 api 5
features no-p2p,raw-mode,mfp,allows-mesh-bcast crc32 3a226daa
[ 8.349462] ath10k_pci 0000:00:00.0: board_file api 1 bmi_id N/A
crc32 bebc7c08
[ 9.533804] ath10k_pci 0000:00:00.0: swiotlb buffer is full (sz: 2112
bytes), total 2048 (slots), used 2036 (slots)
[ 9.534677] ath10k_pci 0000:00:00.0: swiotlb buffer is full (sz: 2112
bytes), total 2048 (slots), used 2036 (slots)
[ 9.535954] ath10k_pci 0000:00:00.0: swiotlb buffer is full (sz: 2112
bytes), total 2048 (slots), used 2036 (slots)
[ 9.537023] ath10k_pci 0000:00:00.0: swiotlb buffer is full (sz: 2112
bytes), total 2048 (slots), used 2036 (slots)
[ 9.546441] ath10k_pci 0000:00:00.0: swiotlb buffer is full (sz: 2112
bytes), total 2048 (slots), used 2036 (slots)
[ 9.556979] ath10k_pci 0000:00:00.0: swiotlb buffer is full (sz: 2112
bytes), total 2048 (slots), used 2036 (slots)
[ 9.557754] ath10k_pci 0000:00:00.0: swiotlb buffer is full (sz: 2112
bytes), total 2048 (slots), used 2036 (slots)
[ 9.558593] ath10k_pci 0000:00:00.0: swiotlb buffer is full (sz: 2112
bytes), total 2048 (slots), used 2038 (slots)
[ 9.559396] ath10k_pci 0000:00:00.0: swiotlb buffer is full (sz: 2112
bytes), total 2048 (slots), used 2038 (slots)
[ 9.560553] ath10k_pci 0000:00:00.0: swiotlb buffer is full (sz: 2112
bytes), total 2048 (slots), used 2040 (slots)
[ 9.563413] ath10k_pci 0000:00:00.0: failed to refill htt rx ring: -12
[ 9.677728] ath10k_pci 0000:00:00.0: could not init core (-12)
[ 9.678924] ath10k_pci 0000:00:00.0: could not probe fw (-12)
[ 9.686262] kmodloader: done loading kernel modules from /etc/modul
The Debian Dom0 has 1 GB memory. I understand that 1 GB of memory for
Dom0 is not very much, but it was working well until the kernel update.
top in Dom0 shows that there is still free memory in Dom0 (although not
much).
What I tried:
-------------
1. Increasing the DomU memory to 384 MB didn't solve the issue.
2. Booting DomU with kernel option swiotlb=8192 and switolb=65536 leads
to a crash/reboot of the complete (!) system when the Openwrt boot
process reaches loading the kernel module for the WiFi card (dmesg from
DomU):
[ 7.780624] ath10k_pci 0000:00:00.0: Xen PCI mapped GSI40 to IRQ17
[ 7.785727] ath10k_pci 0000:00:00.0: pci irq msi oper_irq_mode 2
irq_mode 0 reset_mode 0
[ 7.903071] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[ 8.023071] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[ 8.142934] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[ 11.798704] ath10k_pci 0000:00:00.0: failed to read device register,
device is gone
[ 11.799297] ath10k_pci 0000:00:00.0: failed to wait for target after
cold reset: -5
[ 11.800178] ath10k_pci 0000:00:00.0: failed to reset chip: -5
3. Booting DomU with kernel option swiotlb=16394 leads to error messages
and a not working WiFi card (dmesg from DomU):
[ 8.908062] ath10k_pci 0000:00:00.0: Xen PCI mapped GSI40 to IRQ17
[ 8.912608] ath10k_pci 0000:00:00.0: pci irq msi oper_irq_mode 2
irq_mode 0 reset_mode 0
[ 9.025185] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[ 9.145555] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[ 9.265060] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[ 9.465046] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[ 9.585009] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[ 9.705125] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[ 9.794764] ath10k_pci 0000:00:00.0: Failed to get pcie state addr: -16
[ 9.795310] ath10k_pci 0000:00:00.0: failed to setup init config: -16
[ 9.795982] ath10k_pci 0000:00:00.0: could not power on hif bus (-16)
[ 9.796472] ath10k_pci 0000:00:00.0: could not probe fw (-16)
Workarounds:
------------
1. Booting Dom0 with kernel package linux-image-amd64 5.10.0-21 solves
this issue. In such a setup, dmesg from DomU looks like this:
[ 8.447495] ath10k_pci 0000:00:00.0: Xen PCI mapped GSI40 to IRQ13
[ 8.451706] ath10k_pci 0000:00:00.0: pci irq msi oper_irq_mode 2
irq_mode 0 reset_mode 0
[ 8.740643] ath10k_pci 0000:00:00.0: qca988x hw2.0 target 0x4100016c
chip_id 0x043202ff sub 0000:0000
[ 8.741348] ath10k_pci 0000:00:00.0: kconfig debug 0 debugfs 1
tracing 0 dfs 1 testmode 0
[ 8.744401] ath10k_pci 0000:00:00.0: firmware ver 10.2.4.70.70 api 5
features no-p2p,raw-mode,mfp,allows-mesh-bcast crc32 3a226daa
[ 8.836807] ath10k_pci 0000:00:00.0: board_file api 1 bmi_id N/A
crc32 bebc7c08
[ 10.044156] ath10k_pci 0000:00:00.0: htt-ver 2.1 wmi-op 5 htt-op 2
cal otp max-sta 128 raw 0 hwcrypto 1
2. Booting with kernel option iommu=soft solves this issue. In such a
workaround setup, dmesg from DomU looks like this:
[ 9.251808] ath10k_pci 0000:00:00.0: Xen PCI mapped GSI40 to IRQ17
[ 9.255480] ath10k_pci 0000:00:00.0: pci irq msi oper_irq_mode 2
irq_mode 0 reset_mode 0
[ 9.377406] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[ 9.496997] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[ 9.617334] ath10k_pci 0000:00:00.0: failed to poke copy engine: -16
[ 9.877658] ath10k_pci 0000:00:00.0: qca988x hw2.0 target 0x4100016c
chip_id 0x043202ff sub 0000:0000
[ 9.878453] ath10k_pci 0000:00:00.0: kconfig debug 0 debugfs 1
tracing 0 dfs 1 testmode 0
[ 9.881475] ath10k_pci 0000:00:00.0: firmware ver 10.2.4.70.70 api 5
features no-p2p,raw-mode,mfp,allows-mesh-bcast crc32 3a226daa
[ 9.963705] ath10k_pci 0000:00:00.0: board_file api 1 bmi_id N/A
crc32 bebc7c08
[ 11.112605] ath10k_pci 0000:00:00.0: htt-ver 2.1 wmi-op 5 htt-op 2
cal otp max-sta 128 raw 0 hwcrypto 1
In working setup, dmesg in the Dom0 shows some corresponding entries:
[ 303.813849] pciback 0000:05:00.0: xen_pciback: vpci: assign to
virtual slot 0
[ 303.819487] pciback 0000:05:00.0: registering for 5
[ 304.887437] xen-blkback: backend/vbd/5/51712: using 1 queues,
protocol 1 (x86_64-abi)
[ 324.941391] xen-blkback: backend/vbd/5/51712: prepare for reconnect
[ 326.400065] xen: registering gsi 40 triggering 0 polarity 1
[ 326.400129] Already setup the GSI :40
[ 326.403607] xen: registering gsi 40 triggering 0 polarity 1
[ 326.403630] Already setup the GSI :40
[ 326.406778] xen: registering gsi 40 triggering 0 polarity 1
[ 326.406826] Already setup the GSI :40
[ 326.409009] xen: registering gsi 40 triggering 0 polarity 1
[ 326.409025] Already setup the GSI :40
[ 326.411967] xen: registering gsi 40 triggering 0 polarity 1
[ 326.411988] Already setup the GSI :40
[ 326.414061] xen: registering gsi 40 triggering 0 polarity 1
[ 326.414075] Already setup the GSI :40
[ 326.415389] xen: registering gsi 40 triggering 0 polarity 1
[ 326.415399] Already setup the GSI :40
[ 326.497043] xen-blkback: backend/vbd/5/51712: using 1 queues,
protocol 1 (x86_64-abi) persistent grants
System information:
-------------------
xl dmesg :
(XEN) Xen version 4.17.2-pre (Debian 4.17.1+2-gb773c48e36-1)
(pkg-xen-devel@lists.alioth.debian.org) (x86_64-linux-gnu-gcc (Debian
12.2.0-14) 12.2.0) debug=n Thu May 18 19:26:30 UTC 2023
(XEN) build-id: 339ecfb2da005012eebb98bb98604c286f6e3160
(XEN) Bootloader: GRUB 2.06-13
(XEN) Command line: placeholder dom0_mem=1024M,max:1024M
dom0_max_vcpus=2 loglvl=all guest_loglvl=all iommu=verbose
com1=115200,8n1,pci console=com1
(XEN) Xen image load base address: 0xcf400000
(XEN) Video information:
(XEN) No VGA detected
(XEN) Disc information:
(XEN) Found 1 MBR signatures
(XEN) Found 1 EDD information structures
(XEN) CPU Vendor: AMD, Family 22 (0x16), Model 48 (0x30), Stepping 1
(raw 00730f01)
(XEN) Xen-e820 RAM map:
(XEN) [0000000000000000, 000000000009fbff] (usable)
(XEN) [000000000009fc00, 000000000009ffff] (reserved)
(XEN) [00000000000f0000, 00000000000fffff] (reserved)
(XEN) [0000000000100000, 00000000cfe95fff] (usable)
(XEN) [00000000cfe96000, 00000000cfffffff] (reserved)
(XEN) [00000000f8000000, 00000000fbffffff] (reserved)
(XEN) [00000000fec10000, 00000000fec10fff] (reserved)
(XEN) [00000000fed40000, 00000000fed44fff] (reserved)
(XEN) [0000000100000000, 000000012effffff] (usable)
(XEN) [000000012f000000, 000000012fffffff] (reserved)
(XEN) ACPI: RSDP 000F2990, 0024 (r2 COREv4)
(XEN) ACPI: XSDT CFEB00E0, 007C (r1 COREv4 COREBOOT 0 CORE 20200925)
(XEN) ACPI: FACP CFEB1E10, 0114 (r6 COREv4 COREBOOT 0 CORE 20200925)
(XEN) ACPI: DSDT CFEB0280, 1B87 (r2 COREv4 COREBOOT 10001 INTL 20200925)
(XEN) ACPI: FACS CFEB0240, 0040
(XEN) ACPI: SSDT CFEB1F30, 01F3 (r2 COREv4 COREBOOT 2A CORE 20200925)
(XEN) ACPI: MCFG CFEB2130, 003C (r1 COREv4 COREBOOT 0 CORE 20200925)
(XEN) ACPI: TPM2 CFEB2170, 004C (r4 COREv4 COREBOOT 0 CORE 20200925)
(XEN) ACPI: APIC CFEB21C0, 007E (r3 COREv4 COREBOOT 0 CORE 20200925)
(XEN) ACPI: HEST CFEB2240, 01D0 (r1 COREv4 COREBOOT 0 CORE 20200925)
(XEN) ACPI: IVRS CFEB2410, 00E0 (r2 COREv4 COREBOOT 0 CORE 20200925)
(XEN) ACPI: SSDT CFEB24F0, 48A6 (r2 AMD AGESA 2 MSFT 4000000)
(XEN) ACPI: SSDT CFEB6DA0, 07C8 (r1 AMD AGESA 1 AMD 1)
(XEN) ACPI: DRTM CFEB7570, 007C (r1 COREv4 COREBOOT 0 CORE 20200925)
(XEN) ACPI: HPET CFEB75F0, 0038 (r1 COREv4 COREBOOT 0 CORE 20200925)
(XEN) System RAM: 4078MB (4176084kB)
(XEN) No NUMA configuration found
(XEN) Faking a node at 0000000000000000-000000012f000000
(XEN) Domain heap initialised
(XEN) SMBIOS 3.0 present.
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x818 (32 bits)
(XEN) ACPI: v5 SLEEP INFO: control[0:0], status[0:0]
(XEN) ACPI: SLEEP INFO: pm1x_cnt[1:804,1:0], pm1x_evt[1:800,1:0]
(XEN) ACPI: 32/64X FACS address mismatch in FADT -
cfeb0240/0000000000000000, using 32
(XEN) ACPI: wakeup_vec[cfeb024c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 4, version 33, address 0xfec00000, GSI 0-23
(XEN) ACPI: IOAPIC (id[0x05] address[0xfec20000] gsi_base[24])
(XEN) IOAPIC[1]: apic_id 5, version 33, address 0xfec20000, GSI 24-55
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) ACPI: HPET id: 0x10228201 base: 0xfed00000
(XEN) PCI: MCFG configuration 0: base f8000000 segment 0000 buses 00 - 3f
(XEN) PCI: MCFG area at f8000000 reserved in E820
(XEN) PCI: Using MCFG for segment 0000 bus 00-3f
(XEN) HEST: Table parsing has been initialized
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 4 CPUs (0 hotplug CPUs)
(XEN) IRQ limits: 56 GSI, 776 MSI/MSI-X
(XEN) CPU0: 1000 (600 ... 1400) MHz
(XEN) xstate: size: 0x340 and states: 0x7
(XEN) CPU0: AMD Fam16h machine check reporting enabled
(XEN) Speculative mitigation facilities:
(XEN) Hardware hints:
(XEN) Hardware features:
(XEN) Compiled-in support: INDIRECT_THUNK SHADOW_PAGING
(XEN) Xen settings: BTI-Thunk RETPOLINE, SPEC_CTRL: No, Other:
BRANCH_HARDEN
(XEN) Support for HVM VMs: MSR_VIRT_SPEC_CTRL RSB
(XEN) Support for PV VMs: RSB
(XEN) XPTI (64-bit PV only): Dom0 disabled, DomU disabled (without PCID)
(XEN) PV L1TF shadowing: Dom0 disabled, DomU disabled
(XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
(XEN) Initializing Credit2 scheduler
(XEN) load_precision_shift: 18
(XEN) load_window_shift: 30
(XEN) underload_balance_tolerance: 0
(XEN) overload_balance_tolerance: -3
(XEN) runqueues arrangement: socket
(XEN) cap enforcement granularity: 10ms
(XEN) load tracking window length 1073741824 ns
(XEN) Platform timer is 14.318MHz HPET
(XEN) Detected 998.125 MHz processor.
(XEN) Freed 1024kB unused BSS memory
(XEN) alt table ffff82d040452190 -> ffff82d04045e1ae
(XEN) AMD-Vi: IOMMU Extended Features:
(XEN) - Peripheral Page Service Request
(XEN) - Invalidate All Command
(XEN) - Guest APIC
(XEN) - Performance Counters
(XEN) - Host Address Translation Size: 0x2
(XEN) - Guest Address Translation Size: 0
(XEN) - Guest CR3 Root Table Level: 0
(XEN) - Maximum PASID: 0x8
(XEN) - SMI Filter Register: 0x1
(XEN) - SMI Filter Register Count: 0x2
(XEN) - Guest Virtual APIC Modes: 0x1
(XEN) - Dual PPR Log: 0
(XEN) - Dual Event Log: 0
(XEN) - Device Table Segmentation: 0
(XEN) - Memory Access Routing and Control: 0
(XEN) - VM Table Size: 0
(XEN) AMD-Vi: Disabled HAP memory map sharing with IOMMU
(XEN) AMD-Vi: IOMMU 0 Enabled.
(XEN) I/O virtualisation enabled
(XEN) - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) nr_sockets: 1
(XEN) Enabling APIC mode: Flat. Using 2 I/O APICs
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using new ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=0 pin2=0
(XEN) Allocated console ring of 32 KiB.
(XEN) mwait-idle: does not run on family 22 model 48
(XEN) HVM: ASIDs enabled.
(XEN) SVM: Supported advanced features:
(XEN) - Nested Page Tables (NPT)
(XEN) - Last Branch Record (LBR) Virtualisation
(XEN) - Next-RIP Saved on #VMEXIT
(XEN) - DecodeAssists
(XEN) - Pause-Intercept Filter
(XEN) - Pause-Intercept Filter Threshold
(XEN) - TSC Rate MSR
(XEN) HVM: SVM enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) alt table ffff82d040452190 -> ffff82d04045e1ae
(XEN) Brought up 4 CPUs
(XEN) Scheduling granularity: cpu, 1 CPU per sched-resource
(XEN) Initializing Credit2 scheduler
(XEN) load_precision_shift: 18
(XEN) load_window_shift: 30
(XEN) underload_balance_tolerance: 0
(XEN) overload_balance_tolerance: -3
(XEN) runqueues arrangement: socket
(XEN) cap enforcement granularity: 10ms
(XEN) load tracking window length 1073741824 ns
(XEN) Adding cpu 0 to runqueue 0
(XEN) First cpu on runqueue, activating
(XEN) Adding cpu 1 to runqueue 0
(XEN) Adding cpu 2 to runqueue 0
(XEN) Adding cpu 3 to runqueue 0
(XEN) MCA: Use hw thresholding to adjust polling frequency
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) NX (Execute Disable) protection active
(XEN) Dom0 has maximum 472 PIRQs
(XEN) *** Building a PV Dom0 ***
(XEN) Xen kernel: 64-bit, lsb
(XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x3a00000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.: 0000000120000000->0000000124000000 (237615 pages
to be allocated)
(XEN) Init. ramdisk: 000000012d02f000->000000012efffaa3
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN) Loaded kernel: ffffffff81000000->ffffffff83a00000
(XEN) Phys-Mach map: 0000008000000000->0000008000200000
(XEN) Start info: ffffffff83a00000->ffffffff83a004b8
(XEN) Page tables: ffffffff83a01000->ffffffff83a22000
(XEN) Boot stack: ffffffff83a22000->ffffffff83a23000
(XEN) TOTAL: ffffffff80000000->ffffffff83c00000
(XEN) ENTRY ADDRESS: ffffffff82c3d1c0
(XEN) Dom0 has maximum 2 VCPUs
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM in background
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 624kB init memory
(XEN) PCI add device 0000:00:00.0
(XEN) PCI add device 0000:00:00.2
(XEN) PCI add device 0000:00:02.0
(XEN) PCI add device 0000:00:02.1
(XEN) PCI add device 0000:00:02.2
(XEN) PCI add device 0000:00:02.3
(XEN) PCI add device 0000:00:02.4
(XEN) PCI add device 0000:00:02.5
(XEN) PCI add device 0000:00:08.0
(XEN) PCI add device 0000:00:10.0
(XEN) SR-IOV device 0000:00:11.0 has its virtual functions already
enabled (01ab)
(XEN) PCI add device 0000:00:11.0
(XEN) PCI add device 0000:00:12.0
(XEN) PCI add device 0000:00:13.0
(XEN) PCI add device 0000:00:14.0
(XEN) PCI add device 0000:00:14.3
(XEN) PCI add device 0000:00:14.7
(XEN) PCI add device 0000:00:18.0
(XEN) PCI add device 0000:00:18.1
(XEN) PCI add device 0000:00:18.2
(XEN) PCI add device 0000:00:18.3
(XEN) PCI add device 0000:00:18.4
(XEN) PCI add device 0000:00:18.5
(XEN) PCI add device 0000:01:00.0
(XEN) PCI add device 0000:02:00.0
(XEN) PCI add device 0000:03:00.0
(XEN) PCI add device 0000:04:00.0
(XEN) PCI add device 0000:05:00.0
(XEN) HVM d3v0 save: CPU
(XEN) HVM d3v1 save: CPU
(XEN) HVM d3 save: PIC
(XEN) HVM d3 save: IOAPIC
(XEN) HVM d3v0 save: LAPIC
(XEN) HVM d3v1 save: LAPIC
(XEN) HVM d3v0 save: LAPIC_REGS
(XEN) HVM d3v1 save: LAPIC_REGS
(XEN) HVM d3 save: PCI_IRQ
(XEN) HVM d3 save: ISA_IRQ
(XEN) HVM d3 save: PCI_LINK
(XEN) HVM d3 save: PIT
(XEN) HVM d3 save: RTC
(XEN) HVM d3 save: HPET
(XEN) HVM d3 save: PMTIMER
(XEN) HVM d3v0 save: MTRR
(XEN) HVM d3v1 save: MTRR
(XEN) HVM d3 save: VIRIDIAN_DOMAIN
(XEN) HVM d3v0 save: CPU_XSAVE
(XEN) HVM d3v1 save: CPU_XSAVE
(XEN) HVM d3v0 save: VIRIDIAN_VCPU
(XEN) HVM d3v1 save: VIRIDIAN_VCPU
(XEN) HVM d3v0 save: VMCE_VCPU
(XEN) HVM d3v1 save: VMCE_VCPU
(XEN) HVM d3v0 save: TSC_ADJUST
(XEN) HVM d3v1 save: TSC_ADJUST
(XEN) HVM d3v0 save: CPU_MSR
(XEN) HVM d3v1 save: CPU_MSR
(XEN) HVM3 restore: CPU 0
(d3) HVM Loader
(d3) Detected Xen v4.17.2-pre
(d3) Xenbus rings @0xfeffc000, event channel 1
(d3) System requested SeaBIOS
(d3) CPU speed is 998 MHz
(d3) Relocating guest memory for lowmem MMIO space disabled
(d3) PCI-ISA link 0 routed to IRQ5
(d3) PCI-ISA link 1 routed to IRQ10
(d3) PCI-ISA link 2 routed to IRQ11
(d3) PCI-ISA link 3 routed to IRQ5
(d3) pci dev 01:3 INTA->IRQ10
(d3) pci dev 02:0 INTA->IRQ11
(d3) pci dev 04:0 INTA->IRQ5
(d3) No RAM in high memory; setting high_mem resource base to 100000000
(d3) pci dev 03:0 bar 10 size 002000000: 0f0000008
(d3) pci dev 02:0 bar 14 size 001000000: 0f2000008
(d3) pci dev 03:0 bar 30 size 000010000: 0f3000000
(d3) pci dev 03:0 bar 14 size 000001000: 0f3010000
(d3) pci dev 04:0 bar 24 size 000001000: 0f3011000
(d3) pci dev 02:0 bar 10 size 000000100: 00000c001
(d3) pci dev 04:0 bar 20 size 000000020: 00000c101
(d3) pci dev 01:1 bar 20 size 000000010: 00000c121
(d3) Multiprocessor initialisation:
(d3) - CPU0 ... 40-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done.
(d3) - CPU1 ... 40-bit phys ... fixed MTRRs ... var MTRRs [2/8] ... done.
(d3) Writing SMBIOS tables ...
(d3) Loading SeaBIOS ...
(d3) Creating MP tables ...
(d3) Loading ACPI ...
(d3) vm86 TSS at fc100280
(d3) BIOS map:
(d3) 10000-100e3: Scratch space
(d3) c0000-fffff: Main BIOS
(d3) E820 table:
(d3) [00]: 00000000:00000000 - 00000000:000a0000: RAM
(d3) HOLE: 00000000:000a0000 - 00000000:000c0000
(d3) [01]: 00000000:000c0000 - 00000000:00100000: RESERVED
(d3) [02]: 00000000:00100000 - 00000000:7f800000: RAM
(d3) HOLE: 00000000:7f800000 - 00000000:fc000000
(d3) [03]: 00000000:fc000000 - 00000000:fc00b000: NVS
(d3) [04]: 00000000:fc00b000 - 00000001:00000000: RESERVED
(d3) Invoking SeaBIOS ...
(d3) SeaBIOS (version 1.16.2-debian-1.16.2-1)
(d3) BUILD: gcc: (Debian 12.2.0-14) 12.2.0 binutils: (GNU Binutils for
Debian) 2.40
(d3)
(d3) Found Xen hypervisor signature at 40000000
(d3) Running on QEMU (i440fx)
(d3) xen: copy e820...
(d3) Relocating init from 0x000d48c0 to 0x7e7eb320 (size 85056)
(d3) Found 7 PCI devices (max PCI bus is 00)
(d3) Allocated Xen hypercall page at 7f7ff000
(d3) Detected Xen v4.17.2-pre
(d3) xen: copy BIOS tables...
(d3) Copying SMBIOS from 0x00010020 to 0x000f5ce0
(d3) Copying MPTABLE from 0xfc100180/fc100190 to 0x000f5be0
(d3) Copying PIR from 0x00010040 to 0x000f5b60
(d3) Copying ACPI RSDP from 0x000100c0 to 0x000f5b30
(d3) table(50434146)=0xfc00a370 (via xsdt)
(d3) Using pmtimer, ioport 0xb008
(d3) table(50434146)=0xfc00a370 (via xsdt)
(d3) ACPI: parse DSDT at 0xfc001040 (len 37539)
(d3) parse_termlist: parse error, skip from 16/27641
(d3) parse_termlist: parse error, skip from 87/6041
(d3) Scan for VGA option rom
(d3) Running option rom at c000:0003
(d3) pmm call arg1=0
(d3) Turning on vga text mode console
(d3) SeaBIOS (version 1.16.2-debian-1.16.2-1)
(d3) Machine UUID 02bb0854-b01d-42d1-96db-f25ddd6dd401
(d3) ATA controller 1 at 1f0/3f4/0 (irq 14 dev 9)
(d3) ATA controller 2 at 170/374/0 (irq 15 dev 9)
(d3) AHCI controller at 00:04.0, iobase 0xf3011000, irq 5
(d3) Searching bootorder for: /pci@i0cf8/*@4/drive@0/disk@0
(d3) AHCI/0: Set transfer mode to UDMA-5
(d3) Searching bios-geometry for: /pci@i0cf8/*@4/drive@0/disk@0
(d3) AHCI/0: registering: "AHCI/0: QEMU HARDDISK ATA-7 Hard-Disk (16384
MiBytes)"
(d3) PS2 keyboard initialized
(d3) Searching bootorder for: HALT
(d3) Found 0 lpt ports
(d3) Found 1 serial ports
(d3) All threads complete.
(d3) Scan for option roms
(d3)
(d3) Press ESC for boot menu.
(d3)
(d3) Searching bootorder for: HALT
(d3) drive 0x000f5a80: PCHS=16383/16/63 translation=lba LCHS=1024/255/63
s=33554432
(d3) Space available for UMB: ca000-e8000, f5500-f5a80
(d3) Returned 16769024 bytes of ZoneHigh
(d3) e820 map has 7 items:
(d3) 0: 0000000000000000 - 000000000009fc00 = 1 RAM
(d3) 1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
(d3) 2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
(d3) 3: 0000000000100000 - 000000007f7fe000 = 1 RAM
(d3) 4: 000000007f7fe000 - 000000007f800000 = 2 RESERVED
(d3) 5: 00000000fc000000 - 00000000fc00b000 = 4 NVS
(d3) 6: 00000000fc00b000 - 0000000100000000 = 2 RESERVED
(d3) enter handle_19:
(d3) NULL
(d3) Booting from Hard Disk...
(d3) Booting from 0000:7c00
(XEN) d3v0: upcall vector 93
(XEN) d3v1: upcall vector 93
xl info:
host : xxx
release : 5.10.0-21-amd64
version : #1 SMP Debian 5.10.162-1 (2023-01-21)
machine : x86_64
nr_cpus : 4
max_cpu_id : 3
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 1
cpu_mhz : 998.128
hw_caps :
178bf3ff:36d8220b:2e500800:044037ff:00000001:00000008:00000000:00000380
virt_caps : pv hvm hvm_directio pv_directio hap shadow
gnttab-v1 gnttab-v2
total_memory : 4078
free_memory : 181
sharing_freed_memory : 0
sharing_used_memory : 0
outstanding_claims : 0
free_cpus : 0
xen_major : 4
xen_minor : 17
xen_extra : .2-pre
xen_version : 4.17.2-pre
xen_caps : xen-3.0-x86_64 hvm-3.0-x86_32 hvm-3.0-x86_32p
hvm-3.0-x86_64
xen_scheduler : credit2
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset :
xen_commandline : placeholder dom0_mem=1024M,max:1024M
dom0_max_vcpus=2 loglvl=all guest_loglvl=all iommu=verbose
com1=115200,8n1,pci console=com1
cc_compiler : x86_64-linux-gnu-gcc (Debian 12.2.0-14) 12.2.0
cc_compile_by : pkg-xen-devel
cc_compile_domain : lists.alioth.debian.org
cc_compile_date : Thu May 18 19:26:30 UTC 2023
build_id : 339ecfb2da005012eebb98bb98604c286f6e3160
xend_config_format : 4
OpenWRT DomU configuration:
name = "xxx"
vcpus = 1
memory = 256
kernel = '/usr/lib/grub-xen/grub-x86_64-xen.bin'
root = ''
extra = '(xen/xvda1)/boot/grub/grub.cfg'
disk = ['phy:/dev/vg0/xxx,xvda,w']
vif = [ 'ip=192.168.178.1 fd00::1,bridge=xenbr0','ip=192.168.12.2
fd10:0:0:12::2,bridge=xenbr2', 'ip=192.168.14.1
fd00:0:0:14::1,bridge=xenbr1', 'ip=192.168.15.1,bridge=xenbr5' ]
pci = ['05:00.0,permissive=1']
on_reboot = 'restart'
on_crash = 'destroy'
lspci -v on Dom0 (only the pci WiFi adapter):
05:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac
Wireless Network Adapter
Flags: bus master, fast devsel, latency 0, IRQ 126
Memory at d0000000 (64-bit, non-prefetchable) [size=2M]
Expansion ROM at d0200000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/8 Maskable+ 64bit-
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Capabilities: [160] Device Serial Number 00-00-00-00-00-00-00-00
Kernel driver in use: pciback
Kernel modules: ath10k_pci
Side information:
-----------------
1. There are some recent entries in the Debian package linux-image-amd64
changelog which might be relevant, but they might also be completely
misleading, I can't tell:
- [amd64] iommu/amd: Fix pci device refcount leak in ppr_notifier()
- [amd64] iommu/amd: Fix ivrs_acpihid cmdline parsing code
- [amd64] iommu/amd: Add PCI segment support for
ivrs_[ioapic/hpet/acpihid] commands
- [amd64] iommu/amd: Fix ill-formed ivrs_ioapic, ivrs_hpet and
ivrs_acpihid options
- [amd64] iommu/amd: Add a length limitation for the ivrs_acpihid
command-line parameter
2. There has been recent traffic on xen-devel and kernel mailing lists
with regard to swiotlb. Again, I can't tell if it is related.
3. I did a memory test (one run), without errors. However, the system is
using ECC anyway, and as the issue is solved by changing the kernel
version, I don't think it is an memory (or any other faulty hardware) issue.
4. An intermittently occurring issue is that Xen sometimes stops
booting, with the last entries being
(XEN) HVM: SVM enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) alt table ffff82d040451290 -> ffff82d04045c342
(XEN)
A reboot usually solves this issue. This issue is not reproducible
reliably. I have not idea if it is related to the passthrough issue
described in this mail.
Reply to: