Re: Oracle S7-2 and ConnextX4-LX, ConnectX3 (VPI), and i40e
On 9/9/25 23:41, Tony Rodriguez wrote:
For those with an Oracle S7-2, but this may be valid for other Sparc
systems as well: (just a heads-up)
1) The "ConnectX-4 lx" NIC appears to work with Solaris 11.4 and Debian
12. However, on Debian 12, the "ConnectX-4 lx" will generate many
tainted kernel messages (related to mlx5 module). Tried latest
firmwares for this card, (from Dell and Nvidia), all have the same
issues. In my case, the tainted messages appear when configured to use
systemd-networkd and during system boot/reboot/poweroff.
2) As such, decided to downgrade and use "ConnectX-3 CX354A-FCBT
(VPI)". This particular card works with Solaris 11.4 and Debian 12. It
also works in Ethernet and Infiniband modes. Both Solaris 11.4 and
Debian 12 see this nic in either mode. It uses the mlx4 module and
appears to avoid generating tainted kernel messages.
Note(S):
2A) For infiniband verbs, the (rdma-core package) a.k.a. (infiniband-
diags package replacement) will need to add a specific build option to
create the ibverbs executables. I did so by building from source.
Might be helpful to modify the rdma-core build so it will automatically
generate all ibverbs such as ibstat, ibstatus, etc.
2B) opensm package has a dependency with infiniband-diags package.
However, the infiniband-diags package will not install (possibly because
of rdma-core). Guessing opensm package will require an update to use
rdma-core and possibly dropping infiniband-diags. Didn't attempt to
build opensm from scratch, so I can't say for sure yet. Unsure if
infiniband-diags is required by other packages at this point.
2C) On another note, the "ConnectX-3 CX314A-BCCT" will work with Debian
12 in Ethernet only mode, but it is not detected/recognized by Solaris
11.4. If Solaris 11.4, Debian 12, Ethernet, and Infiniband
compatibility are required, make sure to use the "ConnectX-3 CX354A-FCBT
(VPI)" NIC.
3) The built-in i40e 10GBE ports on the S7-2 generate the following
messages within Debian 12. It works and doesn't appear to be a major
issue. Just unsure if this unaligned access may impact performance on
some level? iperf3 didn't show any noticeable performance drop, but it
may be something to keep an eye on.
For Example:
i40e 0000:03:00.1: DCB init failed -16, disabled
Kernel unaligned access at TPC[101a7f28] i40e_config_netdev+0x1e8/0x2e0
[i40e]
Thank you for this very valuable information. I have an S7-2 being
shipped to me and I expect it before end of month. I look forward
to working on this and coordinating with you.
--
--
Dennis Clarke
RISC-V/SPARC/PPC/ARM/CISC
UNIX and Linux spoken
Reply to: