Bug#897398: gridengine-exec: qlogin fails when prolog is used
Package: gridengine-exec
Version: 8.1.9+dfsg-4
Severity: important
Tags: upstream patch
Dear Maintainer,
We have an existing queue environment based on GridEngine and after upgrading
our grid to Debian 9 "Stretch", attempts to "qlogin" would fail after the qlogin
was assigned to a node. We have GridEngine configured to send the admins mail
when this happens and I'll give an example email below that shows how it fails.
It fails while trying to call seteuid(), I'll explain more below.
This happens just after running the prolog script (note on the meaning of
prolog: it's an administrator-specifiable script that runs just before jobs are
started:
qconf -sq all.q | grep prolog
prolog /var/lib/gridengine/default/common/prolog.sh
and note: I am not 100% sure that it even matters that we use a prolog, but
I think it might be relevant and I suspect it might help explain why this
problem hasn't happened to others).
Anyway the real error seems to be of this type:
"Cannot reset euid yshao due to Operation not permitted"
The issue seems to be that the GridEngine code uses seteuid() to set the
effective userid to root and then back to the job-submitter, and then directly
back to 'sgeadmin' without going via root, and it seems there is some kind of
constraint that you can't use seteuid() to change from one regular user to
another, you have to go via root. "man seteuid()" is vague on this (speaking
only of privilege without defining it precisely), but comments in the GridEngine
code indicate there is such a constraint, and that code goes out of its way to
first seteuid() to root before changing to the job-submitter, which is odd
because when changing back from the job-submitter to sgeadmin it doesn't go via
root. The intended properties of Linux in this regard, in terms of when exactly
you're allowed to call seteuid() with a certain value, is very complicated and
although I've read all kinds of man pages I've never found an unambiguous
description.
Anyway, when I downloaded the source package, changed the code of shepherd.c to
change user back via root, compiled, and replaced the 'sge_shepherd' binary on
all our exec nodes, the problem disappeared.
For the key code changes see here
https://github.com/gridengine/gridengine/issues/10
where I have posted the part of the original code that fails and a diff, but
note, I also made other openssl-version-related changes, not shown, to fix
compilation errors. The real mystery to me is why the behavior of seteuid()
would have changed, since it seems a super stable part of Linux-- or if it
didn't change, what else might have changed from Debian 8 to 9 so that this part
of the code suddenly stopped working-- and why they would have gone to the
trouble in the original code of changing users with root in between if it wasn't
even necessary in practice at the time. Another mysterious observation: this
problem only happened on nodes on which we installed Debian 9 from scratch, and
not on those on which we did an incremental upgrade from Debian 8, even though
the nodes had (or at least were supposed to have) the same configuration and
packages.
Also there is a separate issue that I'd appreciate a response on: does anyone
maintain the GridEngine code? Who in the Debian organization at least maintains
the package? I'm considering persuading some other GridEngine users to help me
maintain a github repo for GridEngine. (I'm the leader of a large open source
project "kaldi" already, so I'm familiar with the process).
====
email from GridEngine is below; search for 'euid' for the first error
Job 8887005 caused action: none
User = yshao
Queue = all.q@c08.clsp.jhu.edu
Start Time = <unknown>
End Time = <unknown>
failed in prolog: 04/30/2018 21:52:23 [115:2598]: exit_status of prolog = 143
Shepherd trace:
04/30/2018 21:52:23 [0:2598]: shepherd called with uid = 0, euid = 0
04/30/2018 21:52:23 [0:2598]: qlogin_daemon = builtin
04/30/2018 21:52:23 [115:2598]: starting up 8.1.9
04/30/2018 21:52:23 [115:2598]: setpgid(2598, 2598) returned 0
04/30/2018 21:52:23 [115:2598]: do_core_binding: "binding" parameter not found in config file
04/30/2018 21:52:23 [115:2598]: calling fork_pty()
04/30/2018 21:52:23 [115:2598]: parent: forked "prolog" with pid 2600
04/30/2018 21:52:23 [115:2598]: using signal delivery delay of 120 seconds
04/30/2018 21:52:23 [115:2598]: parent: prolog-pid: 2600
04/30/2018 21:52:23 [61161:2598]: Cannot reset euid yshao due to Operation not permitted
04/30/2018 21:52:23 [115:2600]: child: starting son(prolog, /var/lib/gridengine/default/common/prolog.sh, 0, 10000);
04/30/2018 21:52:23 [61161:2598]: now sending signal TERM to pid -2600
04/30/2018 21:52:23 [61161:2598]: Cannot reset euid yshao due to Operation not permitted
04/30/2018 21:52:23 [61161:2598]: now sending signal TERM to pid -2600
04/30/2018 21:52:23 [115:2598]: Poll received POLLHUP (Hang up). Unregister the FD.
04/30/2018 21:52:23 [115:2598]: wait3 returned 2600 (status: 15; WIFSIGNALED: 1, WIFEXITED: 0, WEXITSTATUS: 0)
04/30/2018 21:52:23 [115:2598]: prolog exited with exit status 0
04/30/2018 21:52:23 [115:2598]: reaped "prolog" with pid 2600
04/30/2018 21:52:23 [115:2598]: prolog exited due to signal
04/30/2018 21:52:23 [115:2598]: prolog signaled: 15
04/30/2018 21:52:23 [115:2598]: exit_status of prolog = 143
04/30/2018 21:52:23 [115:2598]: no epilog script to start
04/30/2018 21:52:23 [115:2598]: writing exit status to qrsh: 0
04/30/2018 21:52:23 [115:2598]: sending UNREGISTER_CTRL_MSG with exit_status = "0"
04/30/2018 21:52:23 [115:2598]: sending to host: <null>
04/30/2018 21:52:23 [115:2598]: comm_write_message returned: can't find handle
04/30/2018 21:52:23 [115:2598]: close_parent_loop: comm_write_message() returned 0 instead of 1!!!
04/30/2018 21:52:23 [115:2598]: waiting for UNREGISTER_RESPONSE_CTRL_MSG
04/30/2018 21:52:23 [115:2598]: No connection or problem while waiting for message: 1
04/30/2018 21:52:23 [115:2598]: parent: cl_com_ignore_timeouts
04/30/2018 21:52:23 [115:2598]: parent: error in comm_cleanup_lib(): 3
04/30/2018 21:52:23 [115:2598]: parent: leaving close_parent_loop()
Shepherd error:
04/30/2018 21:52:23 [115:2598]: exit_status of prolog = 143
Shepherd pe_hostfile:
c08.clsp.jhu.edu 1 all.q@c08.clsp.jhu.edu UNDEFINED
===
It probably doesn't matter, but here is our prolog script:
#!/bin/bash
function test_ok {
if [ ! -z "$JOB_SCRIPT" ] && [ "$JOB_SCRIPT" != QLOGIN ] && [ "$JOB_SCRIPT" != QRLOGIN ]; then
if [ ! -f "$JOB_SCRIPT" ]; then
echo "$0: warning: no such file $JOB_SCRIPT, will wait" 1>&2
return 1;
fi
fi
if [ ! -z "$SGE_STDERR_PATH" ]; then
if [ ! -d "`dirname $SGE_STDERR_PATH`" ]; then
echo "$0: warning: no such directory $JOB_SCRIPT, will wait." 1>&2
return 1;
fi
fi
return 0;
}
if ! test_ok; then
sleep 2;
if ! test_ok; then
sleep 4;
if ! test_ok; then
sleep 8;
fi
fi
fi
exit 0;
-- System Information:
Debian Release: 9.4
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 4.9.0-6-amd64 (SMP w/24 CPU cores)
Locale: LANG=en_US, LC_CTYPE=en_US (charmap=ANSI_X3.4-1968) (ignored: LC_ALL set to C), LANGUAGE=en_US (charmap=ANSI_X3.4-1968) (ignored: LC_ALL set to C)
Shell: /bin/sh linked to /bin/bash
Init: systemd (via /run/systemd/system)
Versions of packages gridengine-exec depends on:
ii csh [c-shell] 20110502-2.2+b1
ii debconf [debconf-2.0] 1.5.61
ii gridengine-client 8.1.9+dfsg-4
ii gridengine-common 8.1.9+dfsg-4
ii libc6 2.24-11+deb9u3
ii libhwloc5 1.11.5-1
ii libjemalloc1 3.6.0-9.1
ii libmunge2 0.5.12-1+b1
ii libssl1.0.2 1.0.2l-2+deb9u3
ii lsb-base 9.20161125
ii procps 2:3.3.12-3
ii tcsh [c-shell] 6.20.00-7
gridengine-exec recommends no packages.
Versions of packages gridengine-exec suggests:
ii openssh-server 1:7.4p1-10+deb9u3
-- debconf information:
* shared/gridenginemaster: deb8qmaster.clsp.jhu.edu
* shared/gridengineconfig: true
* shared/gridenginecell: default
Reply to: