On Tue, 2015-08-11 at 10:38 +0900, Christian Balzer wrote:
> Hello Ben,
>
> thanks for the quick and detailed reply.
>
> On Mon, 10 Aug 2015 15:53:57 +0200 Ben Hutchings wrote:
>
> > Control: severity -1 important
> > Control: tag -1 upstream
> >
> > On Mon, 2015-08-10 at 13:52 +0900, Christian Balzer wrote:
> > [...]
> > > I'm also not seeing this on several other machines we use for Ceph
> > > with the current Jessie kernel, but to be fair they use slightly
> > > different (QDR, not FDR) ConnectX-3 HBAs.
> >
> > If SR-IOV is enabled on the adapter then the ports will always operate
> > in Ethernet mode as it's apparently not supported for IB. Perhaps SR
> > -IOV enabled on some adapters but not others?
> >
> I was wondering about that, but wasn't aware of the Ethernet only bit of
> SR-IOV.
> Anyway, the previous cluster and one blade of this new one have Mellanox
> firmware 2.30.8000, which doesn't offer the Flexboot Bios menu and thus
> have no SR-IOV configuration option at boot time.
>
> However the other blade (replacement mobo for a DoA one) in the new server
> does have firmware 2.33.5100 and the Flexboot menu and had SR-IOV enabled.
>
> Alas disabling it (and taking out the fake-install) did result in the same
> behavior, mlx4_en was auto-loaded before mlx4_ib.
[...]
> I added that "options mlx4_core port_type_array=1" (since there is only
> one port) to /etc/modprobe.d/local.conf, depmod -a, update-initramfs -u,
> but no joy.
> The mlx4_en module gets auto-loaded before the IB one as well with this
> setting.
[...]
There was a deliberate change in mlx4_core in Linux 3.15 to load
mlx4_en first if it finds any Ethernet port. But that is separate from
the decision of what types of port are configured.
What messages do the drivers log? (dmesg | grep mlx4)
What is the output of:
find /sys/bus/pci/drivers/mlx4_core/0*/ -name port_type | xargs grep -H .
Ben.
--
Ben Hutchings
Theory and practice are closer in theory than in practice.
- John Levine, moderator of comp.compilers
Attachment:
signature.asc
Description: This is a digitally signed message part