Thread $task61.1 receives the message to handle the SIGALRM signal for $task61. Signal handling requires the main thread to be aborted by calling thread_abort($task61.0). Thread $task61.1 then enters thread_sleep() awaiting thread $task61.0 to become suspended. Thread $task61.1 has a state of TH_WAIT|TH_RUN at this time.
The stressor program ($task60) reaches its "limit of patience" and sends SIGKILL to the worker ($task61). SIGKILL is handled specially and so $task60 calls task_terminate($task61) directly. An early phase of this calls task_hold_locked() which calls thread_hold() on each thread in $task61. This is when $task61.1 (whilst sleeping) changes to TH_WAIT|TH_SUSP before it emerges from thread_sleep().
When $task61.1 is woken up by thread_wakeup_prim() it matches the case where it is "Either already running, or suspended" so the code only removes the TH_WAIT state and the thread is not set running. It remains in this state as there are no further triggers to set it running again to tidy its state before termination. The most relevant state that cannot be tidied is the remaining reference to thread $task61.0.
Task $task60 is now spinning away with continuous attempts to terminate the first thread in the list ($task61.0) but that thread never terminates because of the reference held by $task61.1. Both $task60 and $task61 are now permanently stuck in their current states.
My suggestion is to alter the code within task_terminate() to call thread_force_terminate() for each thread in the list rather than just the head of the list. Then repeat the iteration through the remaining threads in the list until there are no threads remaining. This is compared to the existing code which relies on the threads being able to be terminated in sequence which might not always be possible (if my analysis above is correct).
I've attached my proposal for review. The solution appears clumsy but I could not see a more trivial method of traversing the list safely with the locks necessarily released for the call to thread_force_terminate(). I wouldn't be at all surprised if what I have proposed is not safe but I'd welcome any feedback to improve what I have or to find error in my analysis. I've included the original code '#if 0' for more simple comparison rather than a diff at this stage. In any case, even if my suggestion finds favour, I'd imagine this alteration would want much scrutiny before being released into what is a critical part of gnumach.
I have tested this patch with my stress-ng test case and it has not failed for more than 90 minutes now which is some kind of record. It usually fails within 20 minutes and often less than that.
Regards, Mike. On 23/07/2025 20:13, Michael Kelly wrote:
Some additional context for consideration. The thread 0xf60f9170 has a reference count of 1 so presumably other than during repetitions of the while loop in $task60.0 the only reference is held by $task61.1. That thread is sleeping waiting for TH_EV_WAKE_ACTIVE on thread 0xf60f9170. That wakeup event presumably never arrives. Is that down to the task it is associated with being terminated?On 22/07/2025 20:14, Michael Kelly wrote:Hi All,I've been experimenting with stress-ng for some time to stress test my hurd virtual machine. This has already exposed a few problems but here is another. Sorry, for the long explanation, but it might be necessary to make sense of the problem. The scenario under test goes something like:1) Top level supervisory process 'stress-ng' begins execution2) It forks N times, one per stressor under test (in my case 64 times). Call these processes 'stressor'.3) The particular tests I am running are stress-vm and stress-mmap. In these tests each of the stressor processes forks again so that it can be supervised and restart the test should it run out of resources. Call these processes 'worker'.4) Each stressor sets a timeout using alarm() and then waits for the worker to terminate by calling waitpid().5) The stressor SIGALRM handler sets a variable tested occasionally within the worker. If the worker tests that variable quickly then it exits normally. If it does not, then the stressor sends a series of signals SIGALRM (4 times), SIGTERM then finally SIGKILL with a short time gap between them.The test scenario I set up uses all the vm's real memory and a certain portion of swap. Consequently when the timeout expires, many of the processes are paged out and they do not respond quickly which means that many workers receive all 6 signals. Occasionally, one of the stressor processes gets stuck within this while loop within task_terminate ($task60.0):while (!queue_empty(list)) {thread = (thread_t) queue_first(list); /* thread is 0xf60f9170 and is within the worker process */...... thread_force_terminate(thread); ...... }thread_force_terminate(thread) calls thread_halt(thread, TRUE) and in this instance does very little as the the thread is already halted and it simply increases the thread suspend_count (currently standing at 0x64c0fc8e !). The thread is not removed from the list and it is repeatedly processed in the loop.The thread 0xf60f9170 is in $task61 (the worker) and is the main thread which does all the stress testing. Examining its state suggests it is already halted with a state of 0x112 (TH_SUSP|TH_HALTED|TH_SWAPPED).All stack traces are attached and are annotated with extra context.I'm trying to make sense of the thread code but as it's rather complex I thought it might save time by asking if anyone had any input to make. In particular what do I need to look at or consider to determine why the state has ended this way? Better yet someone might immediately see the cause of the problem. I have a virtual machine snapshot of this moment saved so I can easily relay any additional information required.There is a 2nd thread ($task60.1) in the stressor process which is also looping but I think that is just stuck waiting for the task_terminate() to complete. (This 2nd thread is processing a secondary timeout setup by the stressor using alarm(1) but I don't think that is necessarily relevant).None of the threads in $task61 appear to be active based on their 'last updated' time reported by the kernel debugger.Any ideas?
#if 0
while (!queue_empty(list)) {
thread = (thread_t) queue_first(list);
thread_reference(thread);
task_unlock(task);
thread_force_terminate(thread);
thread_deallocate(thread);
thread_block(thread_no_continuation);
task_lock(task);
}
#else
while (!queue_empty(list)) {
thread = (thread_t) queue_first(list);
thread_reference(thread);
do
{
thread_t next = (thread_t) queue_next(&thread->thread_list);
if (!queue_end(list, (queue_entry_t) next))
thread_reference(next);
task_unlock(task);
thread_force_terminate(thread);
thread_deallocate(thread);
thread_block(thread_no_continuation);
thread = next;
task_lock(task);
}
while (!queue_end(list, (queue_entry_t) thread));
}
#endif