Pageout not succeeding

[Michael Kelly <mike@weatherwax.co.uk>]

I've been looking at why the page out to the default pager isn't 
currently working. It was a simple code reversion to restore this 
functionality (see patch to vm_page.c below). The current code only 
calls vm_object_pager_create() when double paging so an internal object 
does not get initialised, has no pager and the later call to 
memory_object_data_return() fails.

There is also however a comment within that area of the code describing 
an important special case of deadlock resulting from the double paging. 
I have been attempting to see if I could construct a simple test bed to 
reproduce:

    * There is one important special case: the default pager can
     * back external memory objects. When receiving the first
     * pageout request, where the page is no longer present, a
     * fault could occur, during which the map would be locked.
     * This fault would cause a new paging request to the default
     * pager. Receiving that request would deadlock when trying to
     * lock the map again. Instead, the page isn't double paged
     * and vm_pageout_setup wires the page down, trusting the
     * default pager as for internal pages.
     */

My test bed consisted of a 1.5Gb single processor virtual machine. A 
process continuously reads and writes to random pages within a 2GB 
memory mapped file to drive the external memory object code (my file was 
on a partition with --no-sync ext2fs). At regular intervals I would 
start a sufficiently large process to consume 500M of memory, sleep for 
a while, then terminate so as to drive the page out code.

I was unfortunately not able to generate the deadlock described by the 
above comment. I was also unable to see a code path that matched the 
locking described. My limited knowledge of the mach kernel is probably 
limiting me here and I'd be grateful if someone could clear my 
understanding.

I was however able to repeatedly bring the kernel to a deadlock 
situation very quickly at times when the pageout thread was active. In 
all cases the underlying cause was several threads awaiting a memory 
page to allow kmem_cache_alloc() to return memory for the many aspects 
of ipc, threads, pmap and so on. One of the pager tasks (mach-defpager 
and ext2fs) would also be stuck waiting for memory for page in to progress.

Description of my patch set:

1) Remove testing double_paging to restore the object pager creation.

2) There is code within vm_map that grants interim vm_privilege for 
memory allocation and I wondered whether the same would be appropriate 
for kmem code too (kern/slab.c). No incidence of deadlock has occurred 
during many hours running my test bed since I added the same 
vm_privilege functionality to kmem_pagealloc_physmem with about 20 
million double paging occurrences in multiple tests.

3) There is a case within vm_page_seg_evict() where 'reclaim' could be 
accessed uninitialised. This would occur for example if the first page 
pulled for eviction was inactive and referenced and the 2nd attempt at 
pulling a page returned NULL. This must be extremely rare because in 
such a case vm_page_free would be called with a NULL page which I'm 
supposing would result in assertions. It's a simple alteration though so 
included here.

4) The final suggestion contained in the attached patch is to allow 
memory allocation within a memory segment that would normally be paused 
(vm_page_alloc_paused==1) but actually has nr_free_pages high enough to 
permit the allocation within that segment. In such cases the pausing was 
triggered by a different memory segment.

It seems from a number of comments throughout the mach code that there 
is a long running worry about memory being sufficiently available in 
order to allow page out to operate and release memory. Has there ever 
been any tentative proposals for how to address this issue with more 
certainty?

Apologies if some or all of these suggestions are inappropriate, 
previously discounted or simply wrong.

Regards,

Mike.

diff -r git/gnumach/kern/slab.c gnumach/kern/slab.c
368a369
>     thread_t thread = current_thread();
373a375,377
>     if (thread)
>         thread->vm_privilege++;
>
374a379,381
>
>     if (thread)
>         thread->vm_privilege--;

diff -r git/gnumach/vm/vm_page.c gnumach/vm/vm_page.c
371,374c371
<     if (vm_page_alloc_paused && current_thread()
<         && !current_thread()->vm_privilege) {
<         return NULL;
<     } else if (seg->nr_free_pages <= seg->low_free_pages) {
---
>     if (seg->nr_free_pages <= seg->low_free_pages) {
1138a1136
>     object = NULL;
1213c1211
<     if (double_paging && !object->pager_initialized) {
---
>     if (!object->pager_initialized) {
1217c1215
<     if (double_paging && !object->pager_initialized) {
---
>     if (!object->pager_initialized) {
1221c1219
<     if (double_paging && !object->pager_initialized) {
---
>     if (!object->pager_initialized) {