Bug#1015871: Enabling PCI_P2PDMA for distro kernels?

[Logan Gunthorpe <logang@deltatee.com>]

On 2023-10-25 11:35, Bjorn Helgaas wrote:
> On Wed, Oct 25, 2023 at 07:11:26PM +0200, Lukas Wunner wrote:
>> On Wed, Oct 25, 2023 at 10:30:07AM -0600, Logan Gunthorpe wrote:
>>> In addition to the above, P2PDMA transfers are only allowed by the
>>> kernel for traffic that flows through certain host bridges that are
>>> known to work. For AMD, all modern CPUs are on this list, but for Intel,
>>> the list is very patchy.
>>
>> This has recently been brought up internally at Intel and nobody could
>> understand why there's a whitelist in the first place.  A long-time PCI
>> architect told me that Intel silicon validation has been testing P2PDMA
>> at least since the Lindenhurst days, i.e. since 2005.
>>
>> What's the reason for the whitelist?  Was there Intel hardware which
>> didn't support it or turned out to be broken?
>>
>> I imagine (but am not certain) that the feature might only be enabled
>> for server SKUs, is that the reason?
> 
> No, the reason is that the PCIe spec doesn't require routing of
> peer-to-peer transactions between Root Ports:
> https://git.kernel.org/linus/0f97da831026
> 
> I think there was a little discussion about adding a firmware
> interface to advertise this capability, but I guess nobody cared
> enough to advance it.

Yes, I remember someone advancing that in the PCI spec, but I don't know
that it got anywhere.

I definitely remember also testing Intel hardware several years ago
where P2PDMA "worked" but the performance was so awful there was no point.

I vaguely remember this not working on non-server machines in the past
(circa 2015). That's why we had to buy a Xeon. Though this was a long
time ago and my memory is fuzzy.

I'd love it if someone from Intel can give us a reasonable check on the
CPU that guarantees P2PDMA will work for everything that passes the
check (like AMD has done.) But in the absence of Intel telling us this
we can't easily make these assumptions.

Logan