[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: #518696 ITP: parallel -- build and execute command lines from standard input in parallel]

On Mon, Mar 09, 2009 at 10:57:57PM +0100, Samuel Thibault wrote:
> I thought at first "it's not particularly convenient", then "well, so
> what".  Now I'm thinking "Mmm, but people won't know they should do it
> and blame xargs for being broken".  Also annotate-output is not enough
> when programs e.g. output Packages entries, which not only should be
> line-atomic, but also paragraph-atomic...

Below is what I had in mind when I mentioned adapting annotate-output to
a different "atomic-output" script.  This script is usefull not just
with "xargs -P", but also with "make -j" and with standard background
jobs (shell & operator), all of which produce mixed output.

Similarly, about matching the number of parallel jobs with the number of
processors/cores, we can write a script "ncpus" which returns the number
of processors/cores/hyper-threads.  You can use the ncpus script with
xargs, with make, or with my new project mdm (mdm.berlios.de)...

I consider separating these concerns (output management, processor
thread detection) into small, separate, and reusable scripts a cleaner
solution.  Of course, doing it this way requires some user education, so
a few manpage updates (for example, adding atomic-output and ncpus to
the SEE ALSO section of xargs) may be in order.


#! /bin/bash
# Display stdout and stderr output after program termination
# Adapted from annotate-output by Chuan-kai Lin
# Original annotate-output author info and copyright notice as follows

# this script was downloaded from:
# http://jeroen.a-eskwadraat.nl/sw/annotate 
# and is part of devscripts 2.10.46

# Executes a program annotating the output linewise with time and stream
# Version 1.2

# Copyright 2003, 2004 Jeroen van Wolffelaar <jeroen@wolffelaar.nl>

# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; version 2 of the License
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA

OUT=`mktemp /tmp/atomic.XXXXXX` || exit 1
ERR=`mktemp /tmp/atomic.XXXXXX` || exit 1

echo "------ `date +%H:%M:%S` Started $@" > $ERR
echo "------ STDERR" >> $ERR
echo "------ STDOUT" >> $OUT
"$@" >> $OUT 2>> $ERR ; EXIT=$?

cat $ERR
cat $OUT
echo "------ `date +%H:%M:%S` Finished with exitcode $EXIT"
rm -f $OUT $ERR

exit $EXIT

Chuan-kai Lin

Reply to: