Bug#666021: separate ceph bug
Upon further investigation I am concluding, that the following is *NOT*
the same bug as reported here originally:
On Tue, 31 May 2016, Tomas Pospisek wrote:
[...] failures that would be coming from the
rbd/ceph driver and would look like this:
May 27 03:46:32 vil kernel: kworker/4:1: page allocation failure: order:1, mode:0x204020
May 27 03:46:32 vil kernel: CPU: 4 PID: 426028 Comm: kworker/4:1 Not tainted 3.16.0-4-amd64 #1 Debian 3.16.7-ckt25-2
May 27 03:46:32 vil kernel: Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 1106 10/17/2011
May 27 03:46:32 vil kernel: Workqueue: rbd2 rbd_request_workfn [rbd]
May 27 03:46:32 vil kernel: 0000000000000000 ffffffff8150e835 0000000000204020 ffff880233303a80
May 27 03:46:32 vil kernel: ffffffff81142d3f 0000000000000000 0000000000000000 ffffffff00000002
May 27 03:46:32 vil kernel: 000000013fdede00 ffff88043fdeec00 0000000000000046 0000000000000001
May 27 03:46:32 vil kernel: Call Trace:
May 27 03:46:32 vil kernel: [<ffffffff8150e835>] ? dump_stack+0x5d/0x78
May 27 03:46:32 vil kernel: [<ffffffff81142d3f>] ? warn_alloc_failed+0xdf/0x130
May 27 03:46:32 vil kernel: [<ffffffff81146fea>] ? __alloc_pages_nodemask+0x8ca/0xb30
May 27 03:46:32 vil kernel: [<ffffffff8118d08b>] ? kmem_getpages+0x5b/0x110
May 27 03:46:32 vil kernel: [<ffffffff8118e6bf>] ? fallback_alloc+0x1cf/0x210
May 27 03:46:32 vil kernel: [<ffffffff8118f2a0>] ? kmem_cache_alloc+0x1f0/0x450
May 27 03:46:32 vil kernel: [<ffffffffa063d0f3>] ? ceph_osdc_alloc_request+0x53/0x2f0 [libceph]
May 27 03:46:32 vil kernel: [<ffffffffa05ba33f>] ? rbd_osd_req_create.isra.25+0x6f/0x170 [rbd]
May 27 03:46:32 vil kernel: [<ffffffffa05bafb6>] ? rbd_img_request_fill+0x2b6/0x910 [rbd]
May 27 03:46:32 vil kernel: [<ffffffffa05be16b>] ? rbd_request_workfn+0x24b/0x390 [rbd]
May 27 03:46:32 vil kernel: [<ffffffff810817c2>] ? process_one_work+0x172/0x420
May 27 03:46:32 vil kernel: [<ffffffff81081e53>] ? worker_thread+0x113/0x4f0
May 27 03:46:32 vil kernel: [<ffffffff81510d61>] ? __schedule+0x2b1/0x700
May 27 03:46:32 vil kernel: [<ffffffff81081d40>] ? rescuer_thread+0x2d0/0x2d0
May 27 03:46:32 vil kernel: [<ffffffff8108809d>] ? kthread+0xbd/0xe0
May 27 03:46:32 vil kernel: [<ffffffff81087fe0>] ? kthread_create_on_node+0x180/0x180
May 27 03:46:32 vil kernel: [<ffffffff81514958>] ? ret_from_fork+0x58/0x90
May 27 03:46:32 vil kernel: [<ffffffff81087fe0>] ? kthread_create_on_node+0x180/0x180
[...]
Note that the original reason why I ended up in this bug report here was,
that some *VM*'s filesystem would be mounted read-only for no apparent
reason.
The course of events that would lead up to that symptom was apparently:
* some app makes accesses the file system
* which makes ext4 do something
* which makes the in kernel jbd2 journaling daemon/thread do an operation
* which is passed down to the rdb block device
* which makes rbd do something over the network (?)
* which triggers rbd to allocate memory
* which fails
* so rbd reports the error back to jbd2
* jbd2 reports back to ext4
* ext4 remounts the file system read only
This is from our logs:
May 27 03:57:07 vil kernel: rbd: rbd2: write 1000 at 1883c1000 result -12
May 27 03:57:07 vil kernel: end_request: I/O error, dev rbd2, sector 12852744
May 27 03:57:07 vil kernel: Buffer I/O error on device rbd2, logical block 1606593
May 27 03:57:07 vil kernel: lost page write due to I/O error on rbd2
May 27 03:57:07 vil kernel: JBD2: Error -5 detected when updating journal superblock for rbd2-8.
May 27 03:57:31 vil kernel: EXT4-fs error (device rbd2): ext4_journal_check_start:56: Detected aborted journal
May 27 03:57:31 vil kernel: EXT4-fs (rbd2): Remounting filesystem read-only
May 27 03:57:31 vil kernel: EXT4-fs (rbd2): ext4_writepages: jbd2_start: 13312 pages, ino 796517; err -30
Instead this looks very much like this bug here [1].
As of today the bug reported in [1] is *NOT* fixed in the jessie kernel,
the patch [2] mentioned in [1] which has been backported to the 3.18.26
kernel does *not* apply cleanly to the 3.16 kernel in jessie, however it
*is* contained in the jessie-backports 4.5 linux-kernel package.
Note that there have been additional fixes [3] to ceph/rdb included in the
3.18 stable kernel series and very probably to latest upstream linux
kernels.
This email is in part intended as documentation in case other Debian
server admins are seeing the same problem and are looking for a solution.
Thanks,
*t
[1] http://tracker.ceph.com/issues/13712
[2] http://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/?h=linux-3.18.y&id=ebf6b5329105efecf9b9dfc45d486b97cecbf86e
[3] http://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/log/?h=linux-3.18.y&qt=author&q=Ilya+Dryomov
Reply to: