I spent a few hours bisecting the kernel, unfortunately this didn't result in a "smoking gun". I relied upon the fact that I am able to reproduce the crash with recent Debian 4.2 kernels immediately upon login, however ISTR that during October the crash wouldn't happen immediately, therefore I may have issued a 'git bisect good' when it was actually bad. This makes me suspect that 4.2 introduced the regression in one commit, which was difficult to reproduce, then later another commit exacerbated the regression making it guaranteed to trigger upon login. At first I built several upstream stable kernel tarballs: 4.1.13, 4.2(.0), 4.2.1, etc. and then booted them. 4.1.13 does not crash, even after several days of uptime, whereas 4.2(.0) crashes immediately. I then cloned linux-stable.git, checked out v4.2, and began bisection following the instructions in the Debian kernel-handbook: $ git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git $ cd linux-stable $ git checkout v4.2 $ git bisect start $ git bisect bad $ git bisect good v4.1.13 Bisecting: a merge base must be tested [b953c0d234bc72e8489d3bf51a276c5c4ec85345] Linux 4.1 $ make localmodconfig $ scripts/config --disable DEBUG_INFO $ make -j8 deb-pkg # dpkg -i ../linux-image-4.1.0_4.1.0-1_amd64.deb # reboot Repeated for several hours and ended up with the following bisect log: $ git bisect log git bisect start # bad: [64291f7db5bd8150a74ad2036f1037e6a0428df2] Linux 4.2 git bisect bad 64291f7db5bd8150a74ad2036f1037e6a0428df2 # good: [1f2ce4a2e7aea3a2123b17aff62a80553df31e21] Linux 4.1.13 git bisect good 1f2ce4a2e7aea3a2123b17aff62a80553df31e21 # good: [b953c0d234bc72e8489d3bf51a276c5c4ec85345] Linux 4.1 git bisect good b953c0d234bc72e8489d3bf51a276c5c4ec85345 # bad: [c11d716218910c3aa2bac1bb641e6086ad649555] Merge tag 'armsoc-cleanup' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc git bisect bad c11d716218910c3aa2bac1bb641e6086ad649555 # good: [8a8c35fadfaf55629a37ef1a8ead1b8fb32581d2] mm: kmemleak_alloc_percpu() should follow the gfp from per_alloc() git bisect good 8a8c35fadfaf55629a37ef1a8ead1b8fb32581d2 # bad: [14738e03312ff1137109d68bcbf103c738af0f4a] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input git bisect bad 14738e03312ff1137109d68bcbf103c738af0f4a # bad: [e80ca0139929a37a8d2b7fc90625aa5107066f57] sfc: add "port_" prefix to MAC stats git bisect bad e80ca0139929a37a8d2b7fc90625aa5107066f57 # bad: [b562fc3713d807a8ae66aeabe18fd9eb3f5e7f48] cxgb4: Optimize and cleanup setup memory window code git bisect bad b562fc3713d807a8ae66aeabe18fd9eb3f5e7f48 # good: [9451980a6646ed487efce04a9df28f450935683e] net: Use cached copy of pfmemalloc to avoid accessing page git bisect good 9451980a6646ed487efce04a9df28f450935683e # good: [0c58a2db91747c841d042b1d56615fb1eaf138c7] netns: fix unbalanced spin_lock on error git bisect good 0c58a2db91747c841d042b1d56615fb1eaf138c7 # good: [0bc4c07046de5ce2a2f25ef2192b6f5878c80f83] Merge git://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf-next git bisect good 0bc4c07046de5ce2a2f25ef2192b6f5878c80f83 # good: [3f55b7ed5e5d4aa7291e3a1e2f7224eeba5810ba] Merge branch 'ebpf-tail-call' git bisect good 3f55b7ed5e5d4aa7291e3a1e2f7224eeba5810ba # good: [cfc77c2fbadf5b806fea2e35738c7437fc62f522] sfc: save old MAC address in case sriov_mac_address_changed fails git bisect good cfc77c2fbadf5b806fea2e35738c7437fc62f522 # good: [b9af90499a9ac3b42f364216ec65e9c096c3a076] sfc: Add ndo_get_vf_config() function for EF10 git bisect good b9af90499a9ac3b42f364216ec65e9c096c3a076 # good: [4392dc6900618c1d5137a3cd43805d746a2c563a] sfc: add ndo_set_vf_link_state() function for EF10 git bisect good 4392dc6900618c1d5137a3cd43805d746a2c563a # good: [910c8789a777c0a18cd997765de291b49bd26906] sfc: set the MAC address using MC_CMD_VADAPTOR_SET_MAC git bisect good 910c8789a777c0a18cd997765de291b49bd26906 # good: [4e7b3be4064249140ab34361a9df4897b27e3648] Merge branch 'sfc-next' git bisect good 4e7b3be4064249140ab34361a9df4897b27e3648 # first bad commit: [b562fc3713d807a8ae66aeabe18fd9eb3f5e7f48] cxgb4: Optimize and cleanup setup memory window code The last several bisections were bogus, "sfc" driver wasn't enabled, "make deb-pkg" didn't actually build any changes. Blaming "cxgb4" is silly because I don't have one of those. However earlier during the bisections something did catch my eye: I ran 'git bisect visualize' while waiting for the kernel to build, noticed a patch series prefixed with "xfrm" (d7a32b6..8faf491) and each time thought to myself "oh this build is going to crash" and was correct each time ;-) I suppose I'll restart bisection at last 'bad' and let the kernels run for a day before issueing 'git bisect good'. -- Gerald Turner <gturner@unzane.com> Encrypted mail preferred! OpenPGP: 4096R / CA89 B27A 30FA 66C5 1B80 3858 EC94 2276 FDB8 716D
Attachment:
signature.asc
Description: PGP signature