[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Kernel Panics / Re: GRUB testers on SPARC needed

Hi Adrian,
for the sake of visibility, here the response to the kernel-trouble:

Am 17.05.2021 um 10:23 schrieb John Paul Adrian Glaubitz:
Installing on two SunFire v215 went reasonably well

/- (apart from recurring Kernel Panics with "Unable to handle kernel paging request in mna handler",
most often triggered on boot immediately after the systemd binfmt service tries to start. This seems
to have been mentioned in /2020/04/msg00020.html but never pinpointed and fixed?) -/
What kernel version are you running. There have actually been some fixes in this regard, in particular
this fix:

I'm using the latest version from the repositories:
5.10.0-6-sparc64-smp #1 SMP Debian 5.10.28-1 (2021-04-09) sparc64 GNU/Linux
The commit you mention seems to be in 5.12 and 5.13-rc2?
Is there a pre-built SMP-image for this or do I have to set up building myself?

Running on the v215 was a real nightmare yesterday. They will be stable for hours, but certain actions (in one occurence executing dmesg (!), apt-get installing nfs-common, some mounts systemd tries) crashed both of the boxes with various errors, most of them the dreaded line with "p Retrying the dpkg-reconfigure as well. After the systemd-unit for - I think - rpcbind was activated during package configuration, both boxes crashed about 4-6 times, I had to reset from OBP.
After a few tries, installation went through, then. Now I can mount nfs...
The hours I spent in the rescue mode of the current installation CD without any trouble made me suspect that non-SMP-kernels are "more stable". I'm currently running the SMP variant with "maxcpus=1", that seems stable so far... But as with any other sporadic issue, that's hard to tell with a way to reliably trigger the errors...

The worst offender so far seems to be xfs, though...
I initially installed both v215 with ext3 /boot and xfs for /.
I'm not sure if the problem is related, but xfs seems to frequently encounter
[   35.325122] XFS (md1): Metadata corruption detected at xfs_dinode_verify.part.0+0x358/0x6c0 [xfs], inode 0x402c4d0 dinode
[   35.469639] XFS (md1): Unmount and run xfs_repair
on both machines. xfs_repair doesn't do anything, though. Either, these inodes were the last ones written during kernel panics, or the underlying issue of the panics leads to checksum-mismatches in-memory? The latter seems more likely, because during dpkg-installs the following popped up a few times as well:
[  195.360257] XFS (md1): Corruption of in-memory data detected.  Shutting down filesystem
(after that, obviously, the system is unusable despite not panicking, as root is missing entirely...)

Some faults:
[  281.304119] WARNING: CPU: 1 PID: 11 at kernel/smp.c:633 smp_call_function_many_cond+0x3bc/0x3e0 [  281.418696] Modules linked in: ext4(E) crc16(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) ata_generic(E) tg3(E) libphy(E) ptp(E) ohci_pci(E) sg(E) pata_ali(E) ehci_pci(E) ohci_hcd(E) ehci_hcd(E) libata(E) pps_core(E) usbcore(E) usb_common(E) flash(E) drm(E) drm_panel_orientation_quirks(E) i2c_core(E) fuse(E) configfs(E) ip_tables(E) x_tables(E) autofs4(E) xfs(E) raid10(E) raid456(E) async_raid6_recov(E) async_memcpy(E) async_pq(E) raid6_pq(E) async_xor(E) xor(E) async_tx(E) libcrc32c(E) crc32c_generic(E) raid0(E) multipath(E) linear(E) raid1(E) md_mod(E) sd_mod(E) t10_pi(E) crc_t10dif(E) crct10dif_generic(E) crct10dif_common(E) mptsas(E) scsi_transport_sas(E) mptscsih(E) mptbase(E) scsi_mod(E) [  282.224447] CPU: 1 PID: 11 Comm: ksoftirqd/1 Tainted: G D     E     5.10.0-6-sparc64-smp #1 Debian 5.10.28-1
[  282.359710] Call Trace:
[  282.391788] [<000000000046c67c>] __warn+0xbc/0x120
[  282.454810] [<0000000000c450f8>] warn_slowpath_fmt+0x34/0x74
[  282.529285] [<0000000000517b5c>] smp_call_function_many_cond+0x3bc/0x3e0
[  282.617512] [<0000000000517be4>] smp_call_function+0x24/0x40
[  282.691989] [<0000000000441828>] smp_send_stop+0x28/0x120
[  282.763028] [<0000000000c44e84>] panic+0x110/0x350
[  282.826047] [<0000000000472ad0>] do_exit+0xad0/0xb20
[  282.891357] [<0000000000c43ab0>] die_if_kernel+0x1f4/0x260
[  282.963543] [<0000000000c5501c>] unhandled_fault+0x88/0xac
[  283.035728] [<0000000000c553c8>] do_sparc64_fault+0x388/0xa80
[  283.111354] [<0000000000407714>] sparc64_realfault_common+0x10/0x20
[  283.193850] [<00000000005a1e64>] __bpf_prog_put_rcu+0x24/0x60
[  283.269470] [<00000000004f5c20>] rcu_core+0x240/0x620
[  283.335926] [<00000000004f600c>] rcu_core_si+0xc/0x20
[  283.402383] [<0000000000c5602c>] __do_softirq+0x10c/0x3a0
[  283.473423] [<0000000000473b14>] run_ksoftirqd+0x34/0x60
[  283.543315] ---[ end trace 9f0a29fcdf85be47 ]---

[  124.914048] CPU[1]: Cheetah+ D-cache parity error at TPC[00000000005bc2b0]
[  125.004638] TPC<bpf_check+0x1cd0/0x32e0>
nfs-utils.service is a disabled or a static unit, not starting it.
[  125.528183] Kernel unaligned access at TPC[8ffba4] atomic64_sub_return+0x4/0x54
[  125.624591] Unable to handle kernel paging request in mna handler
[  125.624595]  at virtual address 6f430c861b2ffaab
[  125.765686] current->{active_,}mm->context = 00000000000000c1
[  125.841410] current->{active_,}mm->pgd = fff0000001b94000
[  125.912544]               \|/ ____ \|/
[  125.912544]               "@'/ .. \`@"
[  125.912544]               /_| \__/ |_\
[  125.912544]                  \__U_/
[  126.106299] systemd(1): Oops [#1]

Especially the "Unable to handle kernel paging request in mna handler" is interesting. It's nearly identical to the issue posted ~ a year ago, seemingly introduced somewhere around the Kernel 5.0-time. It's not always accompanied by the "Cheetah+ D-cache parity error at TPC[00000000005bc2b0]"-error. And while CPU L1-Cache parity errors "seem" like a hardware issue, I highly suspect it is not. Both machines show these errors sporadically (often without panicking!) and I found mention of these errors in other contexts... Also, both systems were highly stable in the past, exceeding 2-3 years of uptime, although on Solaris 10 :-(

On a not entirely unrelated note:
Are there any news on functioning netboot images? The last post I could find points to images from April '17 on your webspace, which were, according to the ML, not bootable because of the size.
At least I can't boot them either.
If there is no more recent version, I'll try to build something myself - are there any pointers on how to go about this? Minimal OS or the netinstaller in an .img would be preferred.

I think that would help in quick testing, as I have multiple other systems with Cheetah (UltraSPARC III, III CU and IIIi) I'd like to try provoking the panics on. Also, some older (UltraSPARC IIi and IIe+) systems are waiting for recent Debian :-)


- Robin

Reply to: