Bug#850425: mpt3sas "swiotlb buffer is full" problem only under Xen
On 10/03/2019 23:03, Andrew Cooper wrote:
> On 10/03/2019 21:35, Hans van Kranenburg wrote:
>> found -1 4.19.20-1
>> thanks
>>
>> Hi,
>>
>> Reviving a thing from Jan 2017 here. I don't have this thread in my
>> mailbox, so no inline quotes.
>>
>> I just installed some HP z820 workstation and rebooted it into Xen
>> 4.11.1+26-g87f51bf366-3 with linux 4.19.20-1 as dom0 kernel.
>>
>> During boot I'm greeted by a long list of...
>>
>> [ 14.518793] mpt3sas 0000:02:00.0: swiotlb buffer is full (sz: 65536
>> bytes)
>> [ 14.518899] mpt3sas 0000:02:00.0: swiotlb buffer is full
>> [ 14.518956] mpt3sas 0000:02:00.0: swiotlb buffer is full (sz: 65536
>> bytes)
>> [ 14.518988] sd 6:0:3:0: pci_map_sg failed: request for 786432 bytes!
>> [ 14.519081] mpt3sas 0000:02:00.0: swiotlb buffer is full
>> [ 14.519309] sd 6:0:1:0: pci_map_sg failed: request for 1310720 bytes!
>> [ 14.524611] mpt3sas 0000:02:00.0: swiotlb buffer is full (sz: 65536
>> bytes)
>> [ 14.527309] mpt3sas 0000:02:00.0: swiotlb buffer is full
>> [ 14.527405] sd 6:0:3:0: pci_map_sg failed: request for 786432 bytes!
>> [...]
>>
>> ...and some hangs here and there. This indeed did not happen when
>> booting just Linux, without Xen.
>>
>> Some searching brought me to this Debian bug. So, thanks for writing
>> down all kinds of research here already. Even if it's not fixed upstream
>> yet, this helps a lot. :-)
>>
>> Using dom0_mem=2GiB,max:4GiB instead of dom0_mem=2GiB,max:2GiB (which I
>> started with) makes the errors go away, so workaround confirmed.
>>
>> I can try any of the linked patches, but I see that in message 54,
>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=850425#54
>> Andrew says: "IIRC, they were essentially rejected,". Next message, Ian
>> asks "Do you have a reference ?", but I don't see any fup on that.
>>
>> I think I'm fine with this workaround.
>>
>> If someone will ever work on the upstream patches, then this is just to
>> let know that I might be able to help testing. However, I only have one
>> of this type of box and it's gonna be installed as server at some
>> non-profit organization without OOB access, replacing even older donated
>> hardware, so, it will be kinda limited... :)
>
> I think
> https://lists.xen.org/archives/html/xen-devel/2014-12/msg00699.html is
> the last attempt David made to upstream the fixes.
Attached is a rebase of the last part missing.
Should apply on top of 5.0 kernel, 4.20 should be okay, too. Earlier
kernels will miss some prerequisites.
Juergen
From: David Vrabel <david.vrabel@citrix.com>
Date: Mon, 11 Mar 2019 14:40:00 +0100
Subject: [PATCH] x86/xen: assume a 64-bit DMA mask is required
On a Xen PV guest the DMA addresses and physical addresses are not 1:1
(such as Xen PV guests) and the generic dma_get_required_mask() does
not return the correct mask (since it uses max_pfn).
Some device drivers (such as mptsas, mpt2sas) use
dma_get_required_mask() to set the device's DMA mask to allow them to
use only 32-bit DMA addresses in hardware structures. This results in
unnecessary use of the SWIOTLB if DMA addresses are more than 32-bits,
impacting performance significantly.
We could base the DMA mask on the maximum MFN but:
a) The hypercall op to get the maximum MFN (XENMEM_maximum_ram_page)
will truncate the result to an int in 32-bit guests.
b) Future uses of the IOMMU in Xen may map frames at bus addresses
above the end of RAM.
So, just assume a 64-bit DMA mask is always required.
Signed-off-by: David Vrabel <david.vrabel@citrix.com>
Reviewed-by: Juergen Gross <jgross@suse.com>
---
drivers/xen/swiotlb-xen.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
index bb7888429be6..75e6e440d982 100644
--- a/drivers/xen/swiotlb-xen.c
+++ b/drivers/xen/swiotlb-xen.c
@@ -680,6 +680,11 @@ xen_swiotlb_get_sgtable(struct device *dev, struct sg_table *sgt,
return dma_common_get_sgtable(dev, sgt, cpu_addr, handle, size, attrs);
}
+static u64 xen_swiotlb_get_required_mask(struct device *dev)
+{
+ return DMA_BIT_MASK(64);
+}
+
const struct dma_map_ops xen_swiotlb_dma_ops = {
.alloc = xen_swiotlb_alloc_coherent,
.free = xen_swiotlb_free_coherent,
@@ -694,4 +699,5 @@ const struct dma_map_ops xen_swiotlb_dma_ops = {
.dma_supported = xen_swiotlb_dma_supported,
.mmap = xen_swiotlb_dma_mmap,
.get_sgtable = xen_swiotlb_get_sgtable,
+ .get_required_mask = xen_swiotlb_get_required_mask,
};
Reply to: