Bug#1015871: Enabling PCI_P2PDMA for distro kernels?

["Deucher, Alexander" <Alexander.Deucher@amd.com>]

[Public]

> -----Original Message-----
> From: Logan Gunthorpe <logang@deltatee.com>
> Sent: Wednesday, October 25, 2023 12:30 PM
> To: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>; Bjorn Helgaas
> <bhelgaas@google.com>
> Cc: Simon Richter <sjr@debian.org>; 1015871@bugs.debian.org; linux-
> pci@vger.kernel.org; Deucher, Alexander <Alexander.Deucher@amd.com>;
> Krzysztof Wilczyński <kw@linux.com>; Emanuele Rocca <ema@debian.org>
> Subject: Re: Enabling PCI_P2PDMA for distro kernels?
>
>
>
> On 2023-10-25 00:19, Uwe Kleine-König wrote:
> > Hello,
> >
> > in https://bugs.debian.org/1015871 the Debian kernel team got a
> > request to enable PCI_P2PDMA. Given the description of the feature and
> > also the "If unsure, say N." I wonder if you consider it safe to
> > enable this option.
>
> I don't know. Not being a security expert, I'd say the attack surface exposed is
> fairly minimal. Most of what goes on is internal to the kernel. So the main risk
> is the same rough risk that goes with enabling any feature: there may be bugs.
>
> My opinion is that 'No' is recommended because the feature is still very
> nascent and advanced. Right now it enables two user visible niche
> features: p2p transfers in nvme-target between an NVMe device and an
> RDMA NIC and transferring buffers between two NVMe devices through the
> CMB via O_DIRECT. Both uses require an NVMe device with CMB memory,
> which is rare.
>
> Anyone using this option to do GPU P2PDMA transfers are certainly using out
> of tree (and likely proprietary) modules as the upstream kernel does not yet
> appear to support anything like that at this time. Thus it's not clear how such
> code is using the P2PDMA subsystem or what implications there may be.
>

AMD GPUs can use P2PDMA for resource sharing between GPUs using upstream kernels and mesa and also ROCm.  E.g., if you have multiple GPUs in a system you can render on one and display on the other without an extra trip through system memory.  This is common on laptops and desktops with multiple GPUs.  Enabling P2PDMA provides a nice perf boost on these systems due to reduced copies.  Or with ROCm, GPUs can directly access local memory on other GPUs.  It's also possible between at least AMD GPUs and some RDMA NICs.  There are also a lot of use cases for P2PDMA between devices and NVME devices, but due to differences in memory sharing APIs there is no simple path to move forward here.  I think it's something is a chicken and an egg problem for wider adoption.


> It's not commonly the case that using these features increases throughput as
> CMB memory is usually much slower than system memory. It's use makes
> more sense in smaller/cheaper boutique systems where the system memory
> or bus bandwidth to the CPU is limited. Typically with a PCIe switch involved.
>
> In addition to the above, P2PDMA transfers are only allowed by the kernel for
> traffic that flows through certain host bridges that are known to work. For
> AMD, all modern CPUs are on this list, but for Intel, the list is very patchy.
> When using a PCIe switch (also uncommon) this restriction is not present
> seeing the traffic can avoid the host bridge.

The older pre-Zen AMD CPUs support it too, but only for writes.

Alex

>
> Thus, my contention is anyone experimenting with this stuff ought to be
> capable of installing a custom kernel with the feature enabled.
>
> Logan