--- Begin Message ---
Package: release.debian.org
Severity: normal
User: release.debian.org@packages.debian.org
Usertags: unblock
Please unblock package oar
I would like to upload a new version of oar fixing 3 bugs (#772564, #773790 and
#775957) that I will explain below and fixing the Vcs-Git (adding branch
information). All 3 bugs are fixed by debian/patches/ filesr. So, if you really
want to allow only part of them to be fixed, it will be easy to do so.
All of them are selected backports from upstream development (where bugs have
been initially reported) and have been tested in production on some systems.
#772564
=======
This bug is a typo that makes the current version create a
/var/run/.pid file instead of /var/run/oar-node.pid
The debian/patches/ file is this one:
$ cat debian/patches/001-fix_oar-node_pid_file
Description: fix oar-node pid file
set oar-node pid file location to /var/run/oar-node_ssh.pid
Author: Pierre Neyron <pierre.neyron@free.fr>
Origin: upstream
Bug-Debian: #772564
Applied-Upstream: a91fa8651963c5a889f6885a4e36a1d7479
Index: oar/setup/init.d/oar-node.in
===================================================================
--- oar.orig/setup/init.d/oar-node.in
+++ oar/setup/init.d/oar-node.in
@@ -22,7 +22,7 @@ PATH=%%SBINDIR%%:%%BINDIR%%:/sbin:/bin:/
NAME=oar-node
DESC="OAR node"
OAR_SSHD_CONF="%%OARCONFDIR%%/sshd_config"
-PIDFILE="%%RUNDIR%%/$NAME_sshd.pid"
+PIDFILE="%%RUNDIR%%/${NAME}_sshd.pid"
SSHD_OPTS="-f $OAR_SSHD_CONF -o PidFile=$PIDFILE"
SELINUXENABLED_CMD=$(which selinuxenabled 2>/dev/null)
NOLSB=
#773790
=======
This bug reports an issue with OAR 2.5.4-1 preventing to function with MySQL
as backend database. The proposed patch comes from upstream and is quite simple
and also fixes an issue with PostgreSQL. All those issues are regressions in
OAR that are fixed here.
The debian/patches/ file is this one:
$ cat debian/patches/002-replace-truncate-requests-by-delete
Description: fix errors with the TRUNCATE request
The TRUNCATE SQL request is incompatible with MySQL an causes deadlock with PostgreSQL. Use DELETE instead.
Author: Nicolas Capit <nicolas.capit@imag.fr>
Origin: upstream
Bug: https://github.com/oar-team/oar/issues/11
Bug-Debian: #773790
Applied-Upstream: 3e2d9d88d3ae62f8308c848bcfde125f29f9c8d3
Index: oar/sources/core/common-libs/lib/OAR/IO.pm
===================================================================
--- oar.orig/sources/core/common-libs/lib/OAR/IO.pm
+++ oar/sources/core/common-libs/lib/OAR/IO.pm
@@ -6366,12 +6366,9 @@ sub update_gantt_visualization($){
lock_table($dbh, ["gantt_jobs_predictions_visu","gantt_jobs_resources_visu","gantt_jobs_predictions","gantt_jobs_resources"]);
-# $dbh->do("DELETE FROM gantt_jobs_predictions_visu");
-# $dbh->do("DELETE FROM gantt_jobs_resources_visu");
+ $dbh->do("DELETE FROM gantt_jobs_predictions_visu");
+ $dbh->do("DELETE FROM gantt_jobs_resources_visu");
## $dbh->do("OPTIMIZE TABLE ganttJobsResources_visu, ganttJobsPredictions_visu");
- $dbh->do("TRUNCATE TABLE gantt_jobs_predictions_visu");
- $dbh->do("TRUNCATE TABLE gantt_jobs_resources_visu");
-
$dbh->do("INSERT INTO gantt_jobs_predictions_visu
SELECT *
FROM gantt_jobs_predictions
@@ -6482,8 +6479,8 @@ sub gantt_flush_tables($$$){
$sql
");
}else{
- $dbh->do("TRUNCATE TABLE gantt_jobs_resources");
- $dbh->do("TRUNCATE TABLE gantt_jobs_predictions");
+ $dbh->do("DELETE FROM gantt_jobs_predictions");
+ $dbh->do("DELETE FROM gantt_jobs_resources");
}
}
#775957
=======
The last one is longer. OAR is a HPC batch scheduler, it's role is to
schedule and run jobs on a HPC cluster. One can define moldable jobs, ie jobs
with several duration/resources specifications instead of just one, and OAR's
scheduler must choose one. If a moldable job is submitted with the current
OAR version in Jessie, the scheduling of later advance reservation jobs will be
broken, preventing OAR to function. This is likely to happen in any
installation of OAR.
The patch comes from upstream and has been tested by upstream. Contrary to
the two previous one, it is not obvious to understand this patch without
really looking at the whole OAR sources.
As the patch is bigger, it is attached to this bug report (instead of inlined).
The whole diff is put here (but I removed the three debian/patches/ files
I already talk about):
$ debdiff oar_2.5.4-1.dsc oar_2.5.4-2.dsc
diff: /tmp/VURIF7uC1A/oar-2.5.4/docs/documentation/doc_drmaa1.rst: Aucun fichier ou dossier de ce type
diff: /tmp/3TgJBvs3_D/oar-2.5.4/docs/documentation/doc_drmaa1.rst: Aucun fichier ou dossier de ce type
diff -Nru oar-2.5.4/debian/changelog oar-2.5.4/debian/changelog
--- oar-2.5.4/debian/changelog 2014-10-24 21:57:53.000000000 +0200
+++ oar-2.5.4/debian/changelog 2015-02-13 11:07:39.000000000 +0100
@@ -1,3 +1,13 @@
+oar (2.5.4-2) unstable; urgency=medium
+
+ [ Pierre Neyron ]
+ * Fix oar-node pid file location (Closes: #772564)
+ * Fix errors with the TRUNCATE SQL request which is incompatible with MySQL
+ and causes deadlocks with PostgreSQL (Closes: #773790)
+ * Fix errors with advance reservation when moldable jobs are used (Closes: #775957)
+
+ -- Vincent Danjean <vdanjean@debian.org> Fri, 13 Feb 2015 11:03:53 +0100
+
oar (2.5.4-1) unstable; urgency=medium
* New upstream release. No change from rc12
diff -Nru oar-2.5.4/debian/control oar-2.5.4/debian/control
--- oar-2.5.4/debian/control 2014-10-24 21:57:53.000000000 +0200
+++ oar-2.5.4/debian/control 2015-02-13 11:07:39.000000000 +0100
@@ -6,7 +6,7 @@
Vincent Danjean <vdanjean@debian.org>
Build-Depends: debhelper (>= 9~), po-debconf, python-docutils, perl
Standards-Version: 3.9.6
-Vcs-Git: git://scm.gforge.inria.fr/oar/oar.git/
+Vcs-Git: git://scm.gforge.inria.fr/oar/oar.git/ -b debian.org
Homepage: http://oar.imag.fr/
Package: liboar-perl
diff -Nru oar-2.5.4/debian/patches/001-fix_oar-node_pid_file oar-2.5.4/debian/patches/001-fix_oar-node_pid_file
[See above]
diff -Nru oar-2.5.4/debian/patches/002-replace-truncate-requests-by-delete oar-2.5.4/debian/patches/002-replace-truncate-requests-by-delete
[See above]
diff -Nru oar-2.5.4/debian/patches/003-fix-advance-reservation-vs-moldable-jobs oar-2.5.4/debian/patches/003-fix-advance-reservation-vs-moldable-jobs
[See attachment]
diff -Nru oar-2.5.4/debian/patches/series oar-2.5.4/debian/patches/series
--- oar-2.5.4/debian/patches/series 2014-10-24 21:57:53.000000000 +0200
+++ oar-2.5.4/debian/patches/series 2015-02-13 11:07:39.000000000 +0100
@@ -0,0 +1,3 @@
+001-fix_oar-node_pid_file
+002-replace-truncate-requests-by-delete
+003-fix-advance-reservation-vs-moldable-jobs
There are other bugs upstream fixed but they are less important (and some
are also too much intrusive), so they will be dealt with backports once
Jessie will be released.
The fixes proposed here are the fixes we would really like to see in
Jessie (ie for all oar users by default)
Regards,
Vincent
unblock oar/2.5.4-2
-- System Information:
Debian Release: 8.0
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'unstable'), (500, 'testing'), (500, 'stable'), (500, 'oldstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386, armel, mipsel
Kernel: Linux 3.18.0-trunk-amd64 (SMP w/8 CPU cores)
Locale: LANG=fr_FR.utf8, LC_CTYPE=fr_FR.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
Description: fix advance reservation versus moldable jobs bug
Errors occurs with advance reservation when moldable jobs are used
Author: Nicolas Capit <nicolas.capit@imag.fr>
Origin: upstream
Bug: https://github.com/oar-team/oar/issues/14
Bug-Debian: #775957
Applied-Upstream: 86e3aac682b1140868081238032f1f56208db638
Index: oar/sources/core/common-libs/lib/OAR/IO.pm
===================================================================
--- oar.orig/sources/core/common-libs/lib/OAR/IO.pm
+++ oar/sources/core/common-libs/lib/OAR/IO.pm
@@ -6424,7 +6424,7 @@ sub get_gantt_visu_date($){
sub get_waiting_reservations_already_scheduled($){
my $dbh = shift;
- my $sth = $dbh->prepare(" SELECT moldable_job_descriptions.moldable_job_id, gantt_jobs_predictions.start_time, gantt_jobs_resources.resource_id, moldable_job_descriptions.moldable_walltime
+ my $sth = $dbh->prepare(" SELECT moldable_job_descriptions.moldable_job_id, gantt_jobs_predictions.start_time, gantt_jobs_resources.resource_id, moldable_job_descriptions.moldable_walltime, moldable_job_descriptions.moldable_id
FROM jobs, moldable_job_descriptions, gantt_jobs_predictions, gantt_jobs_resources
WHERE
(jobs.state = \'Waiting\'
@@ -6440,6 +6440,7 @@ sub get_waiting_reservations_already_sch
push(@{$res->{$ref[0]}->{resources}}, $ref[2]);
$res->{$ref[0]}->{start_time} = $ref[1];
$res->{$ref[0]}->{walltime} = $ref[3];
+ $res->{$ref[0]}->{moldable_id} = $ref[4];
}
$sth->finish();
return($res);
@@ -6467,9 +6468,12 @@ sub gantt_flush_tables($$$){
}
my $sql = "\'1\'";
- my @jobs_to_keep = keys(%{$reservations_to_keep});
- if ($#jobs_to_keep >= 0){
- $sql = "moldable_job_id NOT IN (".join(',',@jobs_to_keep).")";
+ my @moldable_jobs_to_keep;
+ foreach my $i (keys(%{$reservations_to_keep})){
+ push(@moldable_jobs_to_keep, $reservations_to_keep->{$i}->{moldable_id});
+ }
+ if ($#moldable_jobs_to_keep >= 0){
+ $sql = "moldable_job_id NOT IN (".join(',',@moldable_jobs_to_keep).")";
$dbh->do(" DELETE FROM gantt_jobs_predictions
WHERE
$sql
Index: oar/sources/core/common-libs/lib/OAR/Schedulers/ResourceTree.pm
===================================================================
--- oar.orig/sources/core/common-libs/lib/OAR/Schedulers/ResourceTree.pm
+++ oar/sources/core/common-libs/lib/OAR/Schedulers/ResourceTree.pm
@@ -400,7 +400,7 @@ sub get_tree_leafs($){
if (is_node_a_leaf($current_node) == 1){
#push(@result, $node_name_pile[0]);
push(@result, $current_node);
- #print("Leaf: ".get_current_resource_value($current_node)."\n");
+ #print("Leaf: ".get_current_resource_name($current_node)." = ".get_current_resource_value($current_node)."\n");
}
# Look at brothers
while(defined($current_node) and (!defined(get_next_brother($current_node)))){
@@ -442,7 +442,7 @@ sub get_tree_leafs_vec($){
if (is_node_a_leaf($current_node) == 1){
vec($result_leafs_vec, get_current_resource_value($current_node), 1) = 1;
$result_leafs_hash{get_current_resource_value($current_node)} = $current_node;
- #print("Leaf: ".get_current_resource_value($current_node)."\n");
+ #print("Leaf: ".get_current_resource_name($current_node)." = ".get_current_resource_value($current_node)."\n");
}
# Look at brothers
while($current_node and (!get_next_brother($current_node))){
Index: oar/sources/core/modules/scheduler/oar_meta_sched
===================================================================
--- oar.orig/sources/core/modules/scheduler/oar_meta_sched
+++ oar/sources/core/modules/scheduler/oar_meta_sched
@@ -159,6 +159,7 @@ foreach my $i (keys(%{$reservation_alrea
my $timesharing_name = "";
my $vec = '';
my $types = OAR::IO::get_job_types_hash($base,$i);
+ my $job_there = OAR::IO::get_job($base,$i);
foreach my $r (@{$reservation_already_there->{$i}->{resources}}){
vec($vec, $r, 1) = 1;
}
@@ -179,7 +180,7 @@ foreach my $i (keys(%{$reservation_alrea
OAR::Schedulers::GanttHoleStorage::new_with_1_hole($Max_resources, $Minimum_hole_time, $reservation_already_there->{$i}->{start_time}, $reservation_already_there->{$i}->{walltime} + $Security_time_overhead , $vec, $All_resource_list_vec);
} else {
($set_placeholder_name, $use_placeholder_name, $timesharing_user, $timesharing_name) =
- OAR::Schedulers::GanttHoleStorage::manage_gantt_for_timesharing_and_placeholder($Gantt, $reservation_already_there->{$i}->{job_user}, $reservation_already_there->{$i}->{job_name}, $types, $inner_id, "[MetaSched] [$i]");
+ OAR::Schedulers::GanttHoleStorage::manage_gantt_for_timesharing_and_placeholder($Gantt, $job_there->{job_user}, $job_there->{job_name}, $types, $inner_id, "[MetaSched] [$i]");
}
#Fill all other gantts
@@ -262,6 +263,8 @@ oar_debug("[MetaSched] End processing of
oar_debug("[MetaSched] Begin processing of waiting reservations (accepted reservations which do not have assigned resources yet)\n");
my @Rjobs = OAR::IO::get_waiting_reservation_jobs($base);
foreach my $job (@Rjobs){
+ next if (defined($reservation_already_there->{$job->{job_id}}));
+
my $container_id = 0;
my $inner_id = 0;
my $set_placeholder_name = "";
@@ -274,7 +277,6 @@ foreach my $job (@Rjobs){
my $job_descriptions = OAR::IO::get_resources_data_structure_current_job($base,$job->{job_id});
# For reservation we take the first moldable job
my $moldable = $job_descriptions->[0];
- next if (defined($reservation_already_there->{$moldable->[2]}));
# Get the list of resources where the reservation will be able to be launched
my ($available_resources_vector, undef) = OAR::IO::get_resource_ids_in_state($base,"Alive");
@@ -335,6 +337,7 @@ foreach my $job (@Rjobs){
}
my $tmp_tree;
$tmp_tree = OAR::IO::get_possible_wanted_resources($base_ro,$available_resources_vector,$resource_id_used_list_vector,\@dead_resources,"$job_properties AND $tmp_properties", $m->{resources}, "".$Order_part);
+ $tmp_tree = OAR::Schedulers::ResourceTree::delete_tree_nodes_with_not_enough_resources($tmp_tree);
$tmp_tree = OAR::Schedulers::ResourceTree::delete_unnecessary_subtrees($tmp_tree);
push(@tree_list, $tmp_tree);
my @leafs = OAR::Schedulers::ResourceTree::get_tree_leafs($tmp_tree);
@@ -581,6 +584,7 @@ sub check_reservation_jobs($$){
$tmp_properties = $m->{property};
}
my $tmp_tree = OAR::IO::get_possible_wanted_resources($base_ro,$available_resources_vector,undef,\@dead_resources,"$job_properties AND $tmp_properties", $m->{resources}, $order_part);
+ $tmp_tree = OAR::Schedulers::ResourceTree::delete_tree_nodes_with_not_enough_resources($tmp_tree);
push(@tree_list, $tmp_tree);
}
my @hole = OAR::Schedulers::GanttHoleStorage::find_first_hole($Gantt->{$inner_id}->{$use_placeholder_name}->{$timesharing_user}->{$timesharing_name},$job->{start_time}, $duration + $Security_time_overhead, \@tree_list, 30);
Index: oar/sources/core/modules/scheduler/oar_sched_gantt_with_timesharing
===================================================================
--- oar.orig/sources/core/modules/scheduler/oar_sched_gantt_with_timesharing
+++ oar/sources/core/modules/scheduler/oar_sched_gantt_with_timesharing
@@ -349,6 +349,7 @@ while (($job_index <= $#jobs) and ((time
$tmp_properties = $m->{property};
}
my $tmp_tree = OAR::IO::get_possible_wanted_resources($base_ro,$alive_resources_vector,undef,\@Dead_resources,"$job_properties AND $tmp_properties", $m->{resources}, $Order_part);
+ $tmp_tree = OAR::Schedulers::ResourceTree::delete_tree_nodes_with_not_enough_resources($tmp_tree);
push(@tree_list, $tmp_tree);
#my @leafs = OAR::Schedulers::ResourceTree::get_tree_leafs($tmp_tree);
#foreach my $l (@leafs){
Index: oar/sources/core/modules/scheduler/oar_sched_gantt_with_timesharing_and_fairsharing
===================================================================
--- oar.orig/sources/core/modules/scheduler/oar_sched_gantt_with_timesharing_and_fairsharing
+++ oar/sources/core/modules/scheduler/oar_sched_gantt_with_timesharing_and_fairsharing
@@ -469,6 +469,7 @@ while (($job_index <= $#jobs) and ((time
$tmp_properties = $m->{property};
}
my $tmp_tree = OAR::IO::get_possible_wanted_resources($base_ro,$alive_resources_vector,undef,\@Dead_resources,"$job_properties AND $tmp_properties", $m->{resources}, $Order_part);
+ $tmp_tree = OAR::Schedulers::ResourceTree::delete_tree_nodes_with_not_enough_resources($tmp_tree);
push(@tree_list, $tmp_tree);
#my @leafs = OAR::Schedulers::ResourceTree::get_tree_leafs($tmp_tree);
#foreach my $l (@leafs){
Index: oar/sources/core/modules/scheduler/oar_sched_gantt_with_timesharing_and_fairsharing_and_placeholder
===================================================================
--- oar.orig/sources/core/modules/scheduler/oar_sched_gantt_with_timesharing_and_fairsharing_and_placeholder
+++ oar/sources/core/modules/scheduler/oar_sched_gantt_with_timesharing_and_fairsharing_and_placeholder
@@ -412,6 +412,7 @@ while (($job_index <= $#jobs) and ((time
$tmp_properties = $m->{property};
}
my $tmp_tree = OAR::IO::get_possible_wanted_resources($base_ro,$alive_resources_vector,undef,\@Dead_resources,"$job_properties AND $tmp_properties", $m->{resources}, $Order_part);
+ $tmp_tree = OAR::Schedulers::ResourceTree::delete_tree_nodes_with_not_enough_resources($tmp_tree);
push(@tree_list, $tmp_tree);
}
my $gantt_timeout = ($timeout - (time() - $initial_time)) / 4;
Index: oar/sources/core/modules/scheduler/oar_sched_gantt_with_timesharing_and_placeholder
===================================================================
--- oar.orig/sources/core/modules/scheduler/oar_sched_gantt_with_timesharing_and_placeholder
+++ oar/sources/core/modules/scheduler/oar_sched_gantt_with_timesharing_and_placeholder
@@ -295,6 +295,7 @@ while (($job_index <= $#jobs) and ((time
$tmp_properties = $m->{property};
}
my $tmp_tree = OAR::IO::get_possible_wanted_resources($base_ro,$alive_resources_vector,undef,\@Dead_resources,"$job_properties AND $tmp_properties", $m->{resources}, $Order_part);
+ $tmp_tree = OAR::Schedulers::ResourceTree::delete_tree_nodes_with_not_enough_resources($tmp_tree);
push(@tree_list, $tmp_tree);
}
my $gantt_timeout = ($timeout - (time() - $initial_time)) / 4;
--- End Message ---