Hi Salvatore,
We are kind of lucky. My colleagues updated the kernel on one the hosts, rebooted it and the problem occurred right away. Currently the system is running, but it looks like all san paths are gone ("multipath -ll" shows nothing). I got the kernel log and a verbose output from multipath. If something other is required, please contact me.
Regards, Daniel
-----Ursprüngliche Nachricht-----
Von: Salvatore Bonaccorso <salvatore.bonaccorso@gmail.com> Im Auftrag von Salvatore Bonaccorso
Gesendet: Mittwoch, 7. August 2024 10:01
An: Ufer, Daniel <Daniel.Ufer@telekom.de>
Cc: itsd-ut@t-systems-mms.com; 1078030@bugs.debian.org
Betreff: Re: AW: Bug#1078030: lpfc: lost all san paths
Control: tags -1 - moreinfo
Hi Daniel,
On Wed, Aug 07, 2024 at 07:14:16AM +0000, Daniel.Ufer@telekom.de wrote:
> Hello Salvatore,
>
> thanks for looking into this issue. This issue appeared on 2 different
> systems, who are running on the same hardware and the same software.
> After the issue occurred on the first system, we realized a week later
> that the second server developed the same problem. We managed to
> reboot the second server before the postgresql service crashed.
> Unfortunately I have no further logs.
> The problem occurred out of the blue, there was no change involved.
> That's why we cannot reproduce the issue.
> We are going to update the kernel package in the next few days.
Thanks for quickly reporting back already.
So please du upgrade to the latest kernel, and then report back if you still encountered the issue. If you are able to collect kernel logs that would be helpful for the lpfc maintainers in case we have something to be forwarded.
Regards,
Salvatore
Attachment:
dmesg.log
Description: dmesg.log
1342.223856 | set open fds limit to 1048576/1048576
1342.223897 | loading /lib/multipath/libchecktur.so checker
1342.223955 | checker tur: message table size = 3
1342.223958 | loading /lib/multipath/libprioconst.so prioritizer
1342.224018 | _init_foreign: foreign library "nvme" is not enabled
1342.228296 | sdb: size = 1048576000
1342.228342 | sdb: vendor = HITACHI
1342.228349 | sdb: product = OPEN-V
1342.228357 | sdb: rev = 9301
1342.228739 | find_hwe: found 2 hwtable matches for HITACHI:OPEN-V:9301
1342.228742 | sdb: h:b:t:l = 0:0:0:1
1342.228862 | sdb: tgt_node_name = 0x50060e8021352700
1342.228864 | sdb: uid_attribute = ID_SERIAL (setting: multipath internal)
1342.228865 | sdb: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
1342.228967 | sdb: 65270 cyl, 255 heads, 63 sectors/track, start at 0
1342.228970 | sdb: vpd_vendor_id = 0 "undef" (setting: multipath internal)
1342.228976 | sdb: serial = 50640002
1342.228978 | sdb: detect_checker = yes (setting: multipath internal)
1342.229561 | sdb: path_checker = tur (setting: storage device autodetected)
1342.229563 | sdb: checker timeout = 30 s (setting: kernel sysfs)
1342.229638 | sdb: tur state = up
1342.229701 | sda: size = 1048576000
1342.229739 | sda: vendor = HITACHI
1342.229746 | sda: product = OPEN-V
1342.229752 | sda: rev = 9301
1342.230115 | find_hwe: found 2 hwtable matches for HITACHI:OPEN-V:9301
1342.230117 | sda: h:b:t:l = 0:0:1:1
1342.230235 | sda: tgt_node_name = 0x50060e802134a100
1342.230237 | sda: uid_attribute = ID_SERIAL (setting: multipath internal)
1342.230239 | sda: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
1342.230418 | sda: 65270 cyl, 255 heads, 63 sectors/track, start at 0
1342.230419 | sda: vpd_vendor_id = 0 "undef" (setting: multipath internal)
1342.230425 | sda: serial = 50640002
1342.230427 | sda: detect_checker = yes (setting: multipath internal)
1342.231049 | sda: path_checker = tur (setting: storage device autodetected)
1342.231051 | sda: checker timeout = 30 s (setting: kernel sysfs)
1342.231102 | sda: tur state = up
1342.231165 | sdc: size = 1048576000
1342.231205 | sdc: vendor = HITACHI
1342.231212 | sdc: product = OPEN-V
1342.231219 | sdc: rev = 9301
1342.231544 | find_hwe: found 2 hwtable matches for HITACHI:OPEN-V:9301
1342.231546 | sdc: h:b:t:l = 0:0:2:1
1342.231658 | sdc: tgt_node_name = 0x50060e8021352710
1342.231660 | sdc: uid_attribute = ID_SERIAL (setting: multipath internal)
1342.231661 | sdc: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
1342.231918 | sdc: 65270 cyl, 255 heads, 63 sectors/track, start at 0
1342.231920 | sdc: vpd_vendor_id = 0 "undef" (setting: multipath internal)
1342.231926 | sdc: serial = 50640002
1342.231927 | sdc: detect_checker = yes (setting: multipath internal)
1342.232514 | sdc: path_checker = tur (setting: storage device autodetected)
1342.232516 | sdc: checker timeout = 30 s (setting: kernel sysfs)
1342.232595 | sdc: tur state = up
1342.232656 | sdd: size = 1048576000
1342.232696 | sdd: vendor = HITACHI
1342.232703 | sdd: product = OPEN-V
1342.232710 | sdd: rev = 9301
1342.233068 | find_hwe: found 2 hwtable matches for HITACHI:OPEN-V:9301
1342.233069 | sdd: h:b:t:l = 0:0:3:1
1342.233181 | sdd: tgt_node_name = 0x50060e802134a110
1342.233182 | sdd: uid_attribute = ID_SERIAL (setting: multipath internal)
1342.233184 | sdd: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
1342.233404 | sdd: 65270 cyl, 255 heads, 63 sectors/track, start at 0
1342.233406 | sdd: vpd_vendor_id = 0 "undef" (setting: multipath internal)
1342.233411 | sdd: serial = 50640002
1342.233413 | sdd: detect_checker = yes (setting: multipath internal)
1342.233923 | sdd: path_checker = tur (setting: storage device autodetected)
1342.233925 | sdd: checker timeout = 30 s (setting: kernel sysfs)
1342.233997 | sdd: tur state = up
1342.234058 | sde: size = 1048576000
1342.234098 | sde: vendor = HITACHI
1342.234105 | sde: product = OPEN-V
1342.234111 | sde: rev = 9301
1342.234450 | find_hwe: found 2 hwtable matches for HITACHI:OPEN-V:9301
1342.234452 | sde: h:b:t:l = 1:0:0:1
1342.234573 | sde: tgt_node_name = 0x50060e802134a120
1342.234575 | sde: uid_attribute = ID_SERIAL (setting: multipath internal)
1342.234576 | sde: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
1342.234754 | sde: 65270 cyl, 255 heads, 63 sectors/track, start at 0
1342.234756 | sde: vpd_vendor_id = 0 "undef" (setting: multipath internal)
1342.234762 | sde: serial = 50640002
1342.234763 | sde: detect_checker = yes (setting: multipath internal)
1342.235247 | sde: path_checker = tur (setting: storage device autodetected)
1342.235249 | sde: checker timeout = 30 s (setting: kernel sysfs)
1342.235300 | sde: tur state = up
1342.235362 | sdf: size = 1048576000
1342.235402 | sdf: vendor = HITACHI
1342.235409 | sdf: product = OPEN-V
1342.235415 | sdf: rev = 9301
1342.235775 | find_hwe: found 2 hwtable matches for HITACHI:OPEN-V:9301
1342.235777 | sdf: h:b:t:l = 1:0:1:1
1342.235889 | sdf: tgt_node_name = 0x50060e8021352720
1342.235890 | sdf: uid_attribute = ID_SERIAL (setting: multipath internal)
1342.235892 | sdf: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
1342.236034 | sdf: 65270 cyl, 255 heads, 63 sectors/track, start at 0
1342.236036 | sdf: vpd_vendor_id = 0 "undef" (setting: multipath internal)
1342.236041 | sdf: serial = 50640002
1342.236043 | sdf: detect_checker = yes (setting: multipath internal)
1342.236539 | sdf: path_checker = tur (setting: storage device autodetected)
1342.236541 | sdf: checker timeout = 30 s (setting: kernel sysfs)
1342.236611 | sdf: tur state = up
1342.236672 | sdg: size = 1048576000
1342.236713 | sdg: vendor = HITACHI
1342.236719 | sdg: product = OPEN-V
1342.236725 | sdg: rev = 9301
1342.237047 | find_hwe: found 2 hwtable matches for HITACHI:OPEN-V:9301
1342.237049 | sdg: h:b:t:l = 1:0:2:1
1342.237166 | sdg: tgt_node_name = 0x50060e802134a130
1342.237167 | sdg: uid_attribute = ID_SERIAL (setting: multipath internal)
1342.237169 | sdg: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
1342.237342 | sdg: 65270 cyl, 255 heads, 63 sectors/track, start at 0
1342.237344 | sdg: vpd_vendor_id = 0 "undef" (setting: multipath internal)
1342.237350 | sdg: serial = 50640002
1342.237352 | sdg: detect_checker = yes (setting: multipath internal)
1342.237860 | sdg: path_checker = tur (setting: storage device autodetected)
1342.237862 | sdg: checker timeout = 30 s (setting: kernel sysfs)
1342.237921 | sdg: tur state = up
1342.237982 | sdh: size = 1048576000
1342.238022 | sdh: vendor = HITACHI
1342.238028 | sdh: product = OPEN-V
1342.238035 | sdh: rev = 9301
1342.238380 | find_hwe: found 2 hwtable matches for HITACHI:OPEN-V:9301
1342.238382 | sdh: h:b:t:l = 1:0:3:1
1342.238496 | sdh: tgt_node_name = 0x50060e8021352730
1342.238498 | sdh: uid_attribute = ID_SERIAL (setting: multipath internal)
1342.238499 | sdh: recheck_wwid = 1 (setting: multipath.conf defaults/devices section)
1342.238699 | sdh: 65270 cyl, 255 heads, 63 sectors/track, start at 0
1342.238701 | sdh: vpd_vendor_id = 0 "undef" (setting: multipath internal)
1342.238707 | sdh: serial = 50640002
1342.238708 | sdh: detect_checker = yes (setting: multipath internal)
1342.239308 | sdh: path_checker = tur (setting: storage device autodetected)
1342.239310 | sdh: checker timeout = 30 s (setting: kernel sysfs)
1342.239372 | sdh: tur state = up
1342.239421 | loop0: device node name blacklisted
1342.239444 | loop1: device node name blacklisted
1342.239467 | loop2: device node name blacklisted
1342.239487 | loop3: device node name blacklisted
1342.239508 | loop4: device node name blacklisted
1342.239530 | loop5: device node name blacklisted
1342.239551 | loop6: device node name blacklisted
1342.239572 | loop7: device node name blacklisted
1342.239604 | nvme0n1: size = 937571328
1342.239711 | dm-0: device node name blacklisted
1342.239732 | dm-1: device node name blacklisted
1342.239755 | dm-2: device node name blacklisted
1342.239774 | dm-3: device node name blacklisted
1342.239794 | dm-4: device node name blacklisted
1342.239814 | dm-5: device node name blacklisted
1342.242258 | multipath-tools v0.9.4 (12/19, 2022)
1342.242269 | libdevmapper version 1.02.185
1342.242326 | kernel device mapper v4.47.0
1342.242334 | DM multipath kernel driver v1.14.0
1342.242673 | unloading tur checker
1342.242692 | unloading const prioritizer
===== paths list =====
uuid hcil dev dev_t pri dm_st chk_st vend/prod/rev dev_st
0:0:0:1 sdb 8:16 -1 undef undef HITACHI,OPEN-V unknown
0:0:1:1 sda 8:0 -1 undef undef HITACHI,OPEN-V unknown
0:0:2:1 sdc 8:32 -1 undef undef HITACHI,OPEN-V unknown
0:0:3:1 sdd 8:48 -1 undef undef HITACHI,OPEN-V unknown
1:0:0:1 sde 8:64 -1 undef undef HITACHI,OPEN-V unknown
1:0:1:1 sdf 8:80 -1 undef undef HITACHI,OPEN-V unknown
1:0:2:1 sdg 8:96 -1 undef undef HITACHI,OPEN-V unknown
1:0:3:1 sdh 8:112 -1 undef undef HITACHI,OPEN-V unknown