Re: rx2660 + debian
> On 2022/Apr/26, at 01:01, Sergei Trofimovich <slyich@gmail.com> wrote:
>
> On Tue, 26 Apr 2022 02:43:00 +0000
> Pedro Miguel Justo <pmsjt@texair.net> wrote:
>
>>> On 2022/Apr/25, at 14:27, Pedro Miguel Justo <pmsjt@texair.net> wrote:
>>>
>>>
>>>
>>>> On 2022/Apr/25, at 14:09, Sergei Trofimovich <slyich@gmail.com> wrote:
>>>>
>>>> On Mon, 25 Apr 2022 15:07:58 +0000
>>>> Pedro Miguel Justo <pmsjt@texair.net> wrote:
>>>>
>>>>>> On 2022/Apr/25, at 01:22, Pedro Miguel Justo <pmsjt@texair.net> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On 2022/Apr/25, at 01:14, Frank Scheiner <frank.scheiner@web.de> wrote:
>>>>>>>
>>>>>>> Hi guys,
>>>>>>>
>>>>>>> On 25.04.22 10:09, John Paul Adrian Glaubitz wrote:
>>>>>>>>> From what I can understand by the information in the bugcheck, this is somewhat related to a violation
>>>>>>>>> in parameter copy from user to kernel during some boot-time, crypto, self-test. Does that sound right?
>>>>>>>>> If that is the case, how would this be related to FW?
>>>>>>>>
>>>>>>>> I'm not claiming that it must be related to the firmware, I'm just saying that I don't see this problem
>>>>>>>> on my RX2660 at all and I have even reinstalled it recently with one of the latest firmware images
>>>>>>>> without having to pass any parameter to the command line.
>>>>>>>
>>>>>>> A difference between Adrian's rx2660 and Pedro's rx2660 is Montecito
>>>>>>> left and Montvale right.
>>>>>>>
>>>>>>> But could still be multiple other reasons we haven't looked at yet in
>>>>>>> detail:
>>>>>>>
>>>>>>> * amount of memory installed
>>>>>>> * SMT enabled or not
>>>>>>> * number of processor modules installed
>>>>>>>
>>>>>>> It might be possible for me to check on my rx2660s (one with Montvale
>>>>>>> and one with Montecito(s)) tomorrow. I will then also look at my other
>>>>>>> Itanium gear and gather relevant information.
>>>>>>>
>>>>>>
>>>>>> Yes, this sounds mode likely to me too.
>>>>>>
>>>>>> The crypto self-tests seem to be an innocent bystander here. I tried booting the most recent kernel with the option “cryptomgr.notests” and it went much farther. Alas it still failed with another buffer copy validation for a different caller altogether:
>>>>>>
>>>>>> [ 3.836466] [<a000000101353690>] usercopy_abort+0x120/0x130
>>>>>> [ 3.836466] sp=e0000001000cfdf0 bsp=e0000001000c9388
>>>>>> [ 3.836466] [<a0000001004c5660>] __check_object_size+0x3c0/0x420
>>>>>> [ 3.836466] sp=e0000001000cfe00 bsp=e0000001000c9350
>>>>>> [ 3.836466] [<a000000100570030>] sys_getcwd+0x250/0x420
>>>>>> [ 3.836466] sp=e0000001000cfe00 bsp=e0000001000c92c8
>>>>>> [ 3.836466] [<a00000010000c860>] ia64_ret_from_syscall+0x0/0x20
>>>>>> [ 3.836466] sp=e0000001000cfe30 bsp=e0000001000c92c8
>>>>>> [ 3.836466] [<a000000000040720>] ia64_ivt+0xffffffff00040720/0x400
>>>>>> [ 3.836466] sp=e0000001000d0000 bsp=e0000001000c92c8
>>>>>>
>>>>>> This suggests the bug might be in the logic validating these buffers against the allocations (heap, span, etc).
>>>>>>
>>>>>> I don’t know why hardened_usercopy=off is not being observed by the kernel. As a work-around I am copying myself a new kernel with CONFIG_HARDENED_USERCOPY disabled at the source.
>>
>> So, I finished compiling my kernel with CONFIG_HARDENED_USERCOPY disabled. Guess what:
>>
>> pmsjt@debian:~$ uname -a
>> Linux debian 5.17.3-rt17 #2 SMP Mon Apr 25 16:55:00 PDT 2022 ia64 GNU/Linux
>>
>> Yup, the system starts just fine with the most recent kernel. So, two things we can infer from this:
>> - Yes, usercopy validation appears to be broken. The contours of how broken it is are yet unknown but we’ll have to investigate to see what part of the validation is failing.
>
> My vague memory tells me that the breakage mechanics is the following:
>
> 1. [ok] mm/usercopy.c:check_kernel_text_object() gets called on various kernel addresses
> 2. [ok] ia64 address space has at least two widely used kernel addresses:
> - 0xe0000000_00000000-0xe000..ff-ffffffff - proper linear mapping, 0xe0000000_00000000 + phys
> (so called, RGN_KERNEL = 7)
> - 0xa0000001_00000000-0xa0000000_00ffffff - kernel ~linear mapping, 0xa0000001_00000000 + phys
> (so called, RGN_GATE = 5)
>
> Both are easy to translate to proper linear address, but the translations
> are not identical. Note that RGN_GATE has an offset.
>
> ia64's stack for some reason lives in RGN_KERNEL space (probably due to
> historical reasons before kernel switched to vmapped stacks on other arches).
> You can see these addresses in backtraces as
> sp=e0000001000cfdf0 bsp=e0000001000c9388
>
> ia64's kernel code on the other hand lives in RGN_GATE.
> You can see these addresses in backtraces as
> [<a000000101353690>] usercopy_abort+0x120/0x130
>
> Now, check_kernel_text_object() compares passed kernel address for 2 ranges:
> - RGN_GATE:
> unsigned long textlow = (unsigned long)_stext;
> unsigned long texthigh = (unsigned long)_etext;
> if (overlaps(ptr, n, textlow, texthigh))
> usercopy_abort("kernel text", NULL, to_user, ptr - textlow, n);
>
> I think this check is correct. There is no address translation.
>
> - RGN_KERNEL:
> textlow_linear = (unsigned long)lm_alias(textlow);
> texthigh_linear = (unsigned long)lm_alias(texthigh);
> if (overlaps(ptr, n, textlow_linear, texthigh_linear))
> usercopy_abort("linear kernel text", NULL, to_user,...
>
> I think this one is wrong. Looking at the definition of lm_alias()
> it can ge used only on RGN_KERNEL addresses (which looks useless,
> they are already in correct form) and literal symbol names:
>
> include/linux/mm.h:#define lm_alias(x) __va(__pa_symbol(x))
> include/linux/mm.h:#define __pa_symbol(x) __pa(RELOC_HIDE((unsigned long)(x), 0))
> arch/ia64/include/asm/page.h:#define __pa(x) ({ia64_va _v; _v.l = (long) (x); _v.f.reg = 0; _v.l;})
> #define __va(x) ({ia64_va _v; _v.l = (long) (x); _v.f.reg = -1; _v.p;})
>
> I think that means __pa_symbol(x) against local variables like textlow
> is completely broken.
>
> If lm_alias() works at all on ia64 maybe that is enough to unbreak usercoy:
>
> --- a/mm/usercopy.c
> +++ b/mm/usercopy.c
> @@ -133,13 +133,13 @@ static inline void check_kernel_text_object(const unsigned long ptr,
> * __pa() is not just the reverse of __va(). This can be detected
> * and checked:
> */
> - textlow_linear = (unsigned long)lm_alias(textlow);
> + textlow_linear = (unsigned long)lm_alias(_stext);
> /* No different mapping: we're done. */
> if (textlow_linear == textlow)
> return;
>
> /* Check the secondary mapping... */
> - texthigh_linear = (unsigned long)lm_alias(texthigh);
> + texthigh_linear = (unsigned long)lm_alias(_etext);
> if (overlaps(ptr, n, textlow_linear, texthigh_linear))
> usercopy_abort("linear kernel text", NULL, to_user,
> ptr - textlow_linear, n);
>
> If not here is an ia64-specific workaround:
>
> --- a/mm/usercopy.c
> +++ b/mm/usercopy.c
> @@ -133,13 +133,13 @@ static inline void check_kernel_text_object(const unsigned long ptr,
> * __pa() is not just the reverse of __va(). This can be detected
> * and checked:
> */
> - textlow_linear = (unsigned long)lm_alias(textlow);
> + textlow_linear = (unsigned long)ia64_imva(textlow);
> /* No different mapping: we're done. */
> if (textlow_linear == textlow)
> return;
>
> /* Check the secondary mapping... */
> - texthigh_linear = (unsigned long)lm_alias(texthigh);
> + texthigh_linear = (unsigned long)ia64_imva(texthigh);
> if (overlaps(ptr, n, textlow_linear, texthigh_linear))
> usercopy_abort("linear kernel text", NULL, to_user,
> ptr - textlow_linear, n);
>
> CAVEAT: I did not compile- or run- test it. Might need a bit of pointer/long casting around.
>
I don’t mind at all giving this a try. Sounds quite plausible.
Do you think the stacks (and RSE) will eventually have to move from RGN_KERNEL to vmapped the same as other archs to avoid issues like this and other to come?
>> - hardened_usercopy=off seems to be ignored by current kernels. When passing this option the system was still failing just the same.
>
> I wonder how you are passing that flag to the kernel. Just a grub parameter to kernel?
> Worth specifying it first and last to make sure if does not get eaten by intermediate
> flags (in case there is a parser bug).
>
>>>>>>
>>>>>
>>>>> Even with kernel "Linux debian 4.19.0-5-mckinley #1 SMP Debian 4.19.37-5 (2019-06-19) ia64 GNU/Linux"
>>>>>
>>>>> Things are still not 100%. After a few hours into building the kernel it started crashing also with usercopy validations but, this time, the other way around. And because it was the other way around, it led to process termination instead of full-blown bugcheck. This could be related or not. Coule very well be a different bug that happens to manifest itself round the same validation.
>>>>>
>>>>> CC [M] drivers/net/wireless/realtek/rtw88/rtw8822be.o
>>>>> LD [M] drivers/net/wireless/realtek/rtw88/rtw88_8822be.o
>>>>> CC [M] drivers/net/wireless/realtek/rtw88/rtw8822c.o
>>>>> Segmentation fault
>>>>> make[5]: *** [scripts/Makefile.build:293: drivers/net/wireless/realtek/rtw88/rtw8822c.o] Error 139
>>>>> make[5]: *** Deleting file 'drivers/net/wireless/realtek/rtw88/rtw8822c.o'
>>>>> make[4]: *** [scripts/Makefile.build:555: drivers/net/wireless/realtek/rtw88] Error 2
>>>>> make[3]: *** [scripts/Makefile.build:555: drivers/net/wireless/realtek] Error 2
>>>>> make[2]: *** [scripts/Makefile.build:555: drivers/net/wireless] Error 2
>>>>> make[1]: *** [scripts/Makefile.build:555: drivers/net] Error 2
>>>>> make: *** [Makefile:1855: drivers] Error 2
>>>>> pmsjt@debian:~/linux-source-5.17$ make
>>>>>
>>>>> Message from syslogd@debian at Apr 25 07:58:08 ...
>>>>> kernel:[23420.984012] usercopy: Kernel memory overwrite attempt detected to linear kernel text (offset 1916912, size 8)!
>>>>>
>>>>> Message from syslogd@debian at Apr 25 07:58:08 ...
>>>>> kernel:[23421.268009] usercopy: Kernel memory overwrite attempt detected to linear kernel text (offset 1818608, size 8)!
>>>>> HOSTCC scripts/sign-file
>>>>> CALL scripts/checksyscalls.sh
>>>>> <stdin>:1517:2: warning: #warning syscall clone3 not implemented [-Wcpp]
>>>>> CALL scripts/atomic/check-atomics.sh
>>>>> CHK include/generated/compile.h
>>>>> make[2]: *** [scripts/Makefile.build:294: arch/ia64/kernel/signal.o] Segmentation fault
>>>>>
>>>>> Message from syslogd@debian at Apr 25 07:58:11 ...
>>>>> kernel:[23423.626254] usercopy: Kernel memory overwrite attempt detected to linear kernel text (offset 1933296, size 8)!
>>>>> make[1]: *** [scripts/Makefile.build:555: arch/ia64/kernel] Error 2
>>>>> make: *** [Makefile:1855: arch/ia64] Error 2
>>>>
>>>
>>> Hi Sergei
>>>
>>>> In my understanding hardened_usercopy=on is completely broken on ia64
>>>> today. It can't run any userspace. Even init process would not survive
>>>> machine boot. At least that's what I experienced on rx3600.
>>>>
>>>> Thus I think if your system survives that much time I would guess
>>>> that you have hardened_usercopy=off in full effect at least at boot.
>>>>
>>>
>>> I want to make sure there is no confusion here. My system only ’survives’ this much when I am using the 4.19 kernel (even when the hardened_usercopy=off is not present). With kernels more recent than that the system will bugcheck very early on boot even if hardened_usercopy=off is present.
>>>
>>>> I would speculate it's some kind of memory corruption around
>>>> 'bypass_usercopy_checks' key.
>>>>
>>>> Worth adding a few printk()s to mm/usercopy.c into 'usercopy_abort()'
>>>> and into 'set_hardened_usercopy()' just to make sure 'bypass_usercopy_checks'
>>>> has expected 'true' setting at boot time and at crash time.
>>>
>>> Right - we definitively need more context about what is the root cause and characteristics of the bug. When the failure happens, is the (pointer, range) of the copy really out-of-whack, or is the validation code not making sense of the boundaries and over-actively failing.
>>>
>>>>
>>>> --
>>>>
>>>> Sergei
>>>
>>
>
>
> --
>
> Sergei
Reply to: