Re: #518696 ITP: parallel -- build and execute command lines from standard input in parallel]
On Mon, Mar 09, 2009 at 10:57:57PM +0100, Samuel Thibault wrote:
> I thought at first "it's not particularly convenient", then "well, so
> what". Now I'm thinking "Mmm, but people won't know they should do it
> and blame xargs for being broken". Also annotate-output is not enough
> when programs e.g. output Packages entries, which not only should be
> line-atomic, but also paragraph-atomic...
Below is what I had in mind when I mentioned adapting annotate-output to
a different "atomic-output" script. This script is usefull not just
with "xargs -P", but also with "make -j" and with standard background
jobs (shell & operator), all of which produce mixed output.
Similarly, about matching the number of parallel jobs with the number of
processors/cores, we can write a script "ncpus" which returns the number
of processors/cores/hyper-threads. You can use the ncpus script with
xargs, with make, or with my new project mdm (mdm.berlios.de)...
I consider separating these concerns (output management, processor
thread detection) into small, separate, and reusable scripts a cleaner
solution. Of course, doing it this way requires some user education, so
a few manpage updates (for example, adding atomic-output and ncpus to
the SEE ALSO section of xargs) may be in order.
----------
#! /bin/bash
# Display stdout and stderr output after program termination
# Adapted from annotate-output by Chuan-kai Lin
# Original annotate-output author info and copyright notice as follows
# this script was downloaded from:
# http://jeroen.a-eskwadraat.nl/sw/annotate
# and is part of devscripts 2.10.46
# Executes a program annotating the output linewise with time and stream
# Version 1.2
# Copyright 2003, 2004 Jeroen van Wolffelaar <jeroen@wolffelaar.nl>
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; version 2 of the License
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
OUT=`mktemp /tmp/atomic.XXXXXX` || exit 1
ERR=`mktemp /tmp/atomic.XXXXXX` || exit 1
echo "------ `date +%H:%M:%S` Started $@" > $ERR
echo "------ STDERR" >> $ERR
echo "------ STDOUT" >> $OUT
"$@" >> $OUT 2>> $ERR ; EXIT=$?
cat $ERR
cat $OUT
echo "------ `date +%H:%M:%S` Finished with exitcode $EXIT"
rm -f $OUT $ERR
exit $EXIT
--
Chuan-kai Lin
http://web.cecs.pdx.edu/~cklin/
Reply to: