Well, I had a better run with my current kernel, it ran for 26 days before things started going haywire this time. Instead of just panicking, instead I kept getting random process deaths, some swap-related errors, and a few oops', which I've attached for posterity. I eventually had to reboot manually as it decided it was out of memory despite having 200MB+ of swap free. I killed all processes to get back into the system and then went down to single user and back again, but portmap just wouldn't respond and thus I had to reboot. Here's one of the swap errors: > swap_dup: Bad swap file entry 10013830 > VM: killing process smbd > swap_free: Bad swap file entry 10013830 ...and another... > swapper: page allocation failure. order:1, mode:0x20 > [__alloc_pages+847/952] __alloc_pages+0x34f/0x3b8 > [kmem_getpages+42/136] kmem_getpages+0x2a/0x88 > [cache_grow+120/368] cache_grow+0x78/0x170 > [cache_alloc_refill+444/488] cache_alloc_refill+0x1bc/0x1e8 > [pg0+73770987/1069900800] tcp_in_window+0x22b/0x4ec [ip_conntrack] > [__kmalloc+102/112] __kmalloc+0x66/0x70 > [pskb_expand_head+77/300] pskb_expand_head+0x4d/0x12c > [skb_checksum_help+176/204] skb_checksum_help+0xb0/0xcc > [pg0+73810190/1069900800] ip_nat_fn+0x10e/0x1c4 [iptable_nat] > [pg0+73810728/1069900800] ip_nat_local_fn+0x4c/0x78 [iptable_nat] > [dst_output+0/28] dst_output+0x0/0x1c > [dst_output+0/28] dst_output+0x0/0x1c > [nf_iterate+81/120] nf_iterate+0x51/0x78 > [dst_output+0/28] dst_output+0x0/0x1c > [dst_output+0/28] dst_output+0x0/0x1c > [nf_hook_slow+62/184] nf_hook_slow+0x3e/0xb8 > [dst_output+0/28] dst_output+0x0/0x1c > [ip_queue_xmit+523/1156] ip_queue_xmit+0x20b/0x484 > [dst_output+0/28] dst_output+0x0/0x1c > [pg0+73170952/1069900800] uhci_destroy_urb_priv+0x9c/0x11c [uhci_hcd] > [ip_local_deliver+63/532] ip_local_deliver+0x3f/0x214 > [pg0+73170952/1069900800] uhci_destroy_urb_priv+0x9c/0x11c [uhci_hcd] > [ip_local_deliver+183/532] ip_local_deliver+0xb7/0x214 > [ip_rcv+780/1152] ip_rcv+0x30c/0x480 > [tcp_cwnd_restart+31/244] tcp_cwnd_restart+0x1f/0xf4 > [tcp_transmit_skb+912/2024] tcp_transmit_skb+0x390/0x7e8 > [skb_clone+23/424] skb_clone+0x17/0x1a8 > [tcp_write_wakeup+275/504] tcp_write_wakeup+0x113/0x1f8 > [tcp_write_timer+0/172] tcp_write_timer+0x0/0xac > [tcp_send_probe0+11/276] tcp_send_probe0+0xb/0x114 > [tcp_write_timer+140/172] tcp_write_timer+0x8c/0xac > [run_timer_softirq+182/372] run_timer_softirq+0xb6/0x174 > [__do_softirq+63/144] __do_softirq+0x3f/0x90 > [do_softirq+41/44] do_softirq+0x29/0x2c > [do_IRQ+34/40] do_IRQ+0x22/0x28 > [common_interrupt+26/32] common_interrupt+0x1a/0x20 > [default_idle+0/44] default_idle+0x0/0x2c > [default_idle+41/44] default_idle+0x29/0x2c > [cpu_idle+16/72] cpu_idle+0x10/0x48 > [start_kernel+344/424] start_kernel+0x158/0x1a8 ...and one of the out of memory errors: > DMA free:448kB min:256kB low:320kB high:384kB active:0kB inactive:220kB present:16384kB pages_scanned:619 all_unreclaimable? no > lowmem_reserve[]: 0 48 48 > Normal free:952kB min:768kB low:960kB high:1152kB active:112kB inactive:192kB present:49152kB pages_scanned:181 all_unreclaimable? no > lowmem_reserve[]: 0 0 0 > HighMem free:0kB min:128kB low:160kB high:192kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no > lowmem_reserve[]: 0 0 0 > DMA: 0*4kB 4*8kB 16*16kB 3*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 448kB > Normal: 48*4kB 5*8kB 11*16kB 9*32kB 2*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 952kB > HighMem: empty > Swap cache: add 27568130, delete 27566294, find 5306420/10672086, race 277+979 > Free swap = 330788kB > Total swap = 570488kB > Out of Memory: Killed process 446 (ldapdns). Just in case someone can help me, I'll stick with this kernel as oopses are better than panics integrity-wise. - Jamie
ksymoops 2.4.11 on i486 2.6.12-1-386. Options used
-V (default)
-K (specified)
-L (specified)
-O (specified)
-m /boot/System.map-2.6.12-1-386 (specified)
Unable to handle kernel NULL pointer dereference at virtual address 00000010
c015e588
*pde = 00000000
Oops: 0000 [#1]
CPU: 0
EIP: 0060:[__d_path+80/296] Not tainted VLI
EFLAGS: 00010292 (2.6.12-1-386)
eax: c208dfe8 ebx: 00000000 ecx: c208dfe8 edx: 00000000
esi: c0ff4200 edi: c208dfe9 ebp: 080ff1e8 esp: c30fde84
ds: 007b es: 007b ss: 0068
Stack: 0000000a ffffffff c208dfe9 00000f09 c1270d14 c3ff4aa0 00000000 c0b495a0
c015e6aa c0e6d1ec 00000000 c1270d14 c3ff4aa0 c208d0df 00000f21 c208d0df
00000070 00000000 c0164a50 c0e6d1ec c0ff4200 c208d0df 00000f21 00000078
Call Trace:
Warning (Oops_read): Code line not seen, dumping what data is available
>>eax; c208dfe8 <pg0+1ce5fe8/3fc56400>
>>ecx; c208dfe8 <pg0+1ce5fe8/3fc56400>
>>esi; c0ff4200 <pg0+c4c200/3fc56400>
>>edi; c208dfe9 <pg0+1ce5fe9/3fc56400>
>>esp; c30fde84 <pg0+2d55e84/3fc56400>
Code: 0c 74 0a f6 45 04 10 0f 85 b2 00 00 00 8b 44 24 0c 85 c0 0f 8e 99 00 00 00 8d 41 ff c6 41 ff 2f 39 6c 24 2c 74 59 90 8b 5c 24 28 <39> 6b 10 74 63 8b 5d 0c 39 dd 74 5c 8d 74 26 00 8b 55 14 8b 44
Using defaults from ksymoops -t elf32-i386 -a i386
Code; ffffffd5 <__kernel_rt_sigreturn+1b95/????>
00000000 <_EIP>:
Code; ffffffd5 <__kernel_rt_sigreturn+1b95/????>
0: 0c 74 or $0x74,%al
Code; ffffffd7 <__kernel_rt_sigreturn+1b97/????>
2: 0a f6 or %dh,%dh
Code; ffffffd9 <__kernel_rt_sigreturn+1b99/????>
4: 45 inc %ebp
Code; ffffffda <__kernel_rt_sigreturn+1b9a/????>
5: 04 10 add $0x10,%al
Code; ffffffdc <__kernel_rt_sigreturn+1b9c/????>
7: 0f 85 b2 00 00 00 jne bf <_EIP+0xbf>
Code; ffffffe2 <__kernel_rt_sigreturn+1ba2/????>
d: 8b 44 24 0c mov 0xc(%esp),%eax
Code; ffffffe6 <__kernel_rt_sigreturn+1ba6/????>
11: 85 c0 test %eax,%eax
Code; ffffffe8 <__kernel_rt_sigreturn+1ba8/????>
13: 0f 8e 99 00 00 00 jle b2 <_EIP+0xb2>
Code; ffffffee <__kernel_rt_sigreturn+1bae/????>
19: 8d 41 ff lea 0xffffffff(%ecx),%eax
Code; fffffff1 <__kernel_rt_sigreturn+1bb1/????>
1c: c6 41 ff 2f movb $0x2f,0xffffffff(%ecx)
Code; fffffff5 <__kernel_rt_sigreturn+1bb5/????>
20: 39 6c 24 2c cmp %ebp,0x2c(%esp)
Code; fffffff9 <__kernel_rt_sigreturn+1bb9/????>
24: 74 59 je 7f <_EIP+0x7f>
Code; fffffffb <__kernel_rt_sigreturn+1bbb/????>
26: 90 nop
Code; fffffffc <__kernel_rt_sigreturn+1bbc/????>
27: 8b 5c 24 28 mov 0x28(%esp),%ebx
Code; 00000000 Before first symbol
2b: 39 6b 10 cmp %ebp,0x10(%ebx)
Code; 00000003 Before first symbol
2e: 74 63 je 93 <_EIP+0x93>
Code; 00000005 Before first symbol
30: 8b 5d 0c mov 0xc(%ebp),%ebx
Code; 00000008 Before first symbol
33: 39 dd cmp %ebx,%ebp
Code; 0000000a Before first symbol
35: 74 5c je 93 <_EIP+0x93>
Code; 0000000c Before first symbol
37: 8d 74 26 00 lea 0x0(%esi),%esi
Code; 00000010 Before first symbol
3b: 8b 55 14 mov 0x14(%ebp),%edx
Code; 00000013 Before first symbol
3e: 8b .byte 0x8b
Code; 00000014 Before first symbol
3f: 44 inc %esp
1 warning issued. Results may not be reliable.
ksymoops 2.4.11 on i486 2.6.12-1-386. Options used
-V (default)
-K (specified)
-L (specified)
-O (specified)
-m /boot/System.map-2.6.12-1-386 (specified)
Unable to handle kernel paging request at virtual address 6d657430
c015e588
*pde = 00000000
Oops: 0000 [#2]
CPU: 0
EIP: 0060:[__d_path+80/296] Not tainted VLI
EFLAGS: 00010a02 (2.6.12-1-386)
eax: c29fffe8 ebx: 6d657420 ecx: c29fffe8 edx: 6d657420
esi: c0ff4200 edi: c29fffe9 ebp: 61726f70 esp: c2dc1e84
ds: 007b es: 007b ss: 0068
Stack: 0000000a ffffffff c29fffe9 00000f09 c1270d14 c3ff4aa0 00000000 c0a7bbe0
c015e6aa c0e6d1ec 6d657420 c1270d14 c3ff4aa0 c29ff0df 00000f21 c29ff0df
00000070 00000000 c0164a50 c0e6d1ec c0ff4200 c29ff0df 00000f21 00000078
Call Trace:
Warning (Oops_read): Code line not seen, dumping what data is available
>>eax; c29fffe8 <pg0+2657fe8/3fc56400>
>>ecx; c29fffe8 <pg0+2657fe8/3fc56400>
>>esi; c0ff4200 <pg0+c4c200/3fc56400>
>>edi; c29fffe9 <pg0+2657fe9/3fc56400>
>>esp; c2dc1e84 <pg0+2a19e84/3fc56400>
Code: 0c 74 0a f6 45 04 10 0f 85 b2 00 00 00 8b 44 24 0c 85 c0 0f 8e 99 00 00 00 8d 41 ff c6 41 ff 2f 39 6c 24 2c 74 59 90 8b 5c 24 28 <39> 6b 10 74 63 8b 5d 0c 39 dd 74 5c 8d 74 26 00 8b 55 14 8b 44
Using defaults from ksymoops -t elf32-i386 -a i386
Code; ffffffd5 <__kernel_rt_sigreturn+1b95/????>
00000000 <_EIP>:
Code; ffffffd5 <__kernel_rt_sigreturn+1b95/????>
0: 0c 74 or $0x74,%al
Code; ffffffd7 <__kernel_rt_sigreturn+1b97/????>
2: 0a f6 or %dh,%dh
Code; ffffffd9 <__kernel_rt_sigreturn+1b99/????>
4: 45 inc %ebp
Code; ffffffda <__kernel_rt_sigreturn+1b9a/????>
5: 04 10 add $0x10,%al
Code; ffffffdc <__kernel_rt_sigreturn+1b9c/????>
7: 0f 85 b2 00 00 00 jne bf <_EIP+0xbf>
Code; ffffffe2 <__kernel_rt_sigreturn+1ba2/????>
d: 8b 44 24 0c mov 0xc(%esp),%eax
Code; ffffffe6 <__kernel_rt_sigreturn+1ba6/????>
11: 85 c0 test %eax,%eax
Code; ffffffe8 <__kernel_rt_sigreturn+1ba8/????>
13: 0f 8e 99 00 00 00 jle b2 <_EIP+0xb2>
Code; ffffffee <__kernel_rt_sigreturn+1bae/????>
19: 8d 41 ff lea 0xffffffff(%ecx),%eax
Code; fffffff1 <__kernel_rt_sigreturn+1bb1/????>
1c: c6 41 ff 2f movb $0x2f,0xffffffff(%ecx)
Code; fffffff5 <__kernel_rt_sigreturn+1bb5/????>
20: 39 6c 24 2c cmp %ebp,0x2c(%esp)
Code; fffffff9 <__kernel_rt_sigreturn+1bb9/????>
24: 74 59 je 7f <_EIP+0x7f>
Code; fffffffb <__kernel_rt_sigreturn+1bbb/????>
26: 90 nop
Code; fffffffc <__kernel_rt_sigreturn+1bbc/????>
27: 8b 5c 24 28 mov 0x28(%esp),%ebx
Code; 00000000 Before first symbol
2b: 39 6b 10 cmp %ebp,0x10(%ebx)
Code; 00000003 Before first symbol
2e: 74 63 je 93 <_EIP+0x93>
Code; 00000005 Before first symbol
30: 8b 5d 0c mov 0xc(%ebp),%ebx
Code; 00000008 Before first symbol
33: 39 dd cmp %ebx,%ebp
Code; 0000000a Before first symbol
35: 74 5c je 93 <_EIP+0x93>
Code; 0000000c Before first symbol
37: 8d 74 26 00 lea 0x0(%esi),%esi
Code; 00000010 Before first symbol
3b: 8b 55 14 mov 0x14(%ebp),%edx
Code; 00000013 Before first symbol
3e: 8b .byte 0x8b
Code; 00000014 Before first symbol
3f: 44 inc %esp
1 warning issued. Results may not be reliable.
ksymoops 2.4.11 on i486 2.6.12-1-386. Options used
-V (default)
-K (specified)
-L (specified)
-O (specified)
-m /boot/System.map-2.6.12-1-386 (specified)
<1>Unable to handle kernel paging request at virtual address 75206583
c015e588
*pde = 00000000
Oops: 0000 [#3]
CPU: 0
EIP: 0060:[__d_path+80/296] Not tainted VLI
EFLAGS: 00010a02 (2.6.12-1-386)
eax: c3c0efe8 ebx: 75206573 ecx: c3c0efe8 edx: 75206573
esi: c0ff4200 edi: c3c0efe9 ebp: 74616470 esp: c1cd5e84
ds: 007b es: 007b ss: 0068
Stack: 0000000a ffffffff c3c0efe9 00000f09 c1270d14 c3ff4aa0 00000000 c2d75c20
c015e6aa c0e6d1ec 75206573 c1270d14 c3ff4aa0 c3c0e0df 00000f21 c3c0e0df
00000070 00000000 c0164a50 c0e6d1ec c0ff4200 c3c0e0df 00000f21 00000078
Call Trace:
Warning (Oops_read): Code line not seen, dumping what data is available
>>eax; c3c0efe8 <pg0+3866fe8/3fc56400>
>>ecx; c3c0efe8 <pg0+3866fe8/3fc56400>
>>esi; c0ff4200 <pg0+c4c200/3fc56400>
>>edi; c3c0efe9 <pg0+3866fe9/3fc56400>
>>esp; c1cd5e84 <pg0+192de84/3fc56400>
Code: 0c 74 0a f6 45 04 10 0f 85 b2 00 00 00 8b 44 24 0c 85 c0 0f 8e 99 00 00 00 8d 41 ff c6 41 ff 2f 39 6c 24 2c 74 59 90 8b 5c 24 28 <39> 6b 10 74 63 8b 5d 0c 39 dd 74 5c 8d 74 26 00 8b 55 14 8b 44
Using defaults from ksymoops -t elf32-i386 -a i386
Code; ffffffd5 <__kernel_rt_sigreturn+1b95/????>
00000000 <_EIP>:
Code; ffffffd5 <__kernel_rt_sigreturn+1b95/????>
0: 0c 74 or $0x74,%al
Code; ffffffd7 <__kernel_rt_sigreturn+1b97/????>
2: 0a f6 or %dh,%dh
Code; ffffffd9 <__kernel_rt_sigreturn+1b99/????>
4: 45 inc %ebp
Code; ffffffda <__kernel_rt_sigreturn+1b9a/????>
5: 04 10 add $0x10,%al
Code; ffffffdc <__kernel_rt_sigreturn+1b9c/????>
7: 0f 85 b2 00 00 00 jne bf <_EIP+0xbf>
Code; ffffffe2 <__kernel_rt_sigreturn+1ba2/????>
d: 8b 44 24 0c mov 0xc(%esp),%eax
Code; ffffffe6 <__kernel_rt_sigreturn+1ba6/????>
11: 85 c0 test %eax,%eax
Code; ffffffe8 <__kernel_rt_sigreturn+1ba8/????>
13: 0f 8e 99 00 00 00 jle b2 <_EIP+0xb2>
Code; ffffffee <__kernel_rt_sigreturn+1bae/????>
19: 8d 41 ff lea 0xffffffff(%ecx),%eax
Code; fffffff1 <__kernel_rt_sigreturn+1bb1/????>
1c: c6 41 ff 2f movb $0x2f,0xffffffff(%ecx)
Code; fffffff5 <__kernel_rt_sigreturn+1bb5/????>
20: 39 6c 24 2c cmp %ebp,0x2c(%esp)
Code; fffffff9 <__kernel_rt_sigreturn+1bb9/????>
24: 74 59 je 7f <_EIP+0x7f>
Code; fffffffb <__kernel_rt_sigreturn+1bbb/????>
26: 90 nop
Code; fffffffc <__kernel_rt_sigreturn+1bbc/????>
27: 8b 5c 24 28 mov 0x28(%esp),%ebx
Code; 00000000 Before first symbol
2b: 39 6b 10 cmp %ebp,0x10(%ebx)
Code; 00000003 Before first symbol
2e: 74 63 je 93 <_EIP+0x93>
Code; 00000005 Before first symbol
30: 8b 5d 0c mov 0xc(%ebp),%ebx
Code; 00000008 Before first symbol
33: 39 dd cmp %ebx,%ebp
Code; 0000000a Before first symbol
35: 74 5c je 93 <_EIP+0x93>
Code; 0000000c Before first symbol
37: 8d 74 26 00 lea 0x0(%esi),%esi
Code; 00000010 Before first symbol
3b: 8b 55 14 mov 0x14(%ebp),%edx
Code; 00000013 Before first symbol
3e: 8b .byte 0x8b
Code; 00000014 Before first symbol
3f: 44 inc %esp
1 warning issued. Results may not be reliable.
ksymoops 2.4.11 on i486 2.6.12-1-386. Options used
-V (default)
-K (specified)
-L (specified)
-O (specified)
-m /boot/System.map-2.6.12-1-386 (specified)
Unable to handle kernel paging request at virtual address 24448920
c015e588
*pde = 00000000
Oops: 0000 [#4]
CPU: 0
EIP: 0060:[__d_path+80/296] Not tainted VLI
EFLAGS: 00010282 (2.6.12-1-386)
eax: c1841fe8 ebx: 24448910 ecx: c1841fe8 edx: 24448910
esi: c0ff4200 edi: c1841fe9 ebp: 04468b04 esp: c1331e84
ds: 007b es: 007b ss: 0068
Stack: 0000000a ffffffff c1841fe9 00000f09 c1270d14 c3ff4aa0 00000000 c0a7b5a0
c015e6aa c0e6d1ec 24448910 c1270d14 c3ff4aa0 c18410df 00000f21 c18410df
00000070 00000000 c0164a50 c0e6d1ec c0ff4200 c18410df 00000f21 00000078
Call Trace:
Warning (Oops_read): Code line not seen, dumping what data is available
>>eax; c1841fe8 <pg0+1499fe8/3fc56400>
>>ecx; c1841fe8 <pg0+1499fe8/3fc56400>
>>esi; c0ff4200 <pg0+c4c200/3fc56400>
>>edi; c1841fe9 <pg0+1499fe9/3fc56400>
>>esp; c1331e84 <pg0+f89e84/3fc56400>
Code: 0c 74 0a f6 45 04 10 0f 85 b2 00 00 00 8b 44 24 0c 85 c0 0f 8e 99 00 00 00 8d 41 ff c6 41 ff 2f 39 6c 24 2c 74 59 90 8b 5c 24 28 <39> 6b 10 74 63 8b 5d 0c 39 dd 74 5c 8d 74 26 00 8b 55 14 8b 44
Using defaults from ksymoops -t elf32-i386 -a i386
Code; ffffffd5 <__kernel_rt_sigreturn+1b95/????>
00000000 <_EIP>:
Code; ffffffd5 <__kernel_rt_sigreturn+1b95/????>
0: 0c 74 or $0x74,%al
Code; ffffffd7 <__kernel_rt_sigreturn+1b97/????>
2: 0a f6 or %dh,%dh
Code; ffffffd9 <__kernel_rt_sigreturn+1b99/????>
4: 45 inc %ebp
Code; ffffffda <__kernel_rt_sigreturn+1b9a/????>
5: 04 10 add $0x10,%al
Code; ffffffdc <__kernel_rt_sigreturn+1b9c/????>
7: 0f 85 b2 00 00 00 jne bf <_EIP+0xbf>
Code; ffffffe2 <__kernel_rt_sigreturn+1ba2/????>
d: 8b 44 24 0c mov 0xc(%esp),%eax
Code; ffffffe6 <__kernel_rt_sigreturn+1ba6/????>
11: 85 c0 test %eax,%eax
Code; ffffffe8 <__kernel_rt_sigreturn+1ba8/????>
13: 0f 8e 99 00 00 00 jle b2 <_EIP+0xb2>
Code; ffffffee <__kernel_rt_sigreturn+1bae/????>
19: 8d 41 ff lea 0xffffffff(%ecx),%eax
Code; fffffff1 <__kernel_rt_sigreturn+1bb1/????>
1c: c6 41 ff 2f movb $0x2f,0xffffffff(%ecx)
Code; fffffff5 <__kernel_rt_sigreturn+1bb5/????>
20: 39 6c 24 2c cmp %ebp,0x2c(%esp)
Code; fffffff9 <__kernel_rt_sigreturn+1bb9/????>
24: 74 59 je 7f <_EIP+0x7f>
Code; fffffffb <__kernel_rt_sigreturn+1bbb/????>
26: 90 nop
Code; fffffffc <__kernel_rt_sigreturn+1bbc/????>
27: 8b 5c 24 28 mov 0x28(%esp),%ebx
Code; 00000000 Before first symbol
2b: 39 6b 10 cmp %ebp,0x10(%ebx)
Code; 00000003 Before first symbol
2e: 74 63 je 93 <_EIP+0x93>
Code; 00000005 Before first symbol
30: 8b 5d 0c mov 0xc(%ebp),%ebx
Code; 00000008 Before first symbol
33: 39 dd cmp %ebx,%ebp
Code; 0000000a Before first symbol
35: 74 5c je 93 <_EIP+0x93>
Code; 0000000c Before first symbol
37: 8d 74 26 00 lea 0x0(%esi),%esi
Code; 00000010 Before first symbol
3b: 8b 55 14 mov 0x14(%ebp),%edx
Code; 00000013 Before first symbol
3e: 8b .byte 0x8b
Code; 00000014 Before first symbol
3f: 44 inc %esp
1 warning issued. Results may not be reliable.
Attachment:
signature.asc
Description: OpenPGP digital signature