On Jul 27, 2011 3:44 AM, "Ivan Shmakov" <ivan@gray.siamics.net> wrote:
>
> >>>>> shawn wilson <ag4ve.us@gmail.com> writes:
>
> […]
>
> > Might I recommend a plethora of free database solutions available to
> > you? I see this type of question quite frequently and am stunned to
> > see such a thing here (or on other mailing lists for platforms just a
> > poorly suited for such data). I'm not sure what your data is from or
> > what it is for but I almost guarantee you that it should be in a
> > database. Either that or open LibraOffice calc and do the one off
> > data task in a few minutes and be done with it.
>
> While I've little to say about using a database for this case,
> I'd strongly recommend /against/ using any office-like solutions
> for data processing, as these are generally overweight and
> rarely convenient when it comes to writing scripts.
>
The point of writing a script / program is to have a reusable process. If you don't need that, pointing and clicking your way through will generally give you a better bang for your buck (as a perl person, I don't like admitting this and am generally not the one pointing and clicking but am pretty sure this is the case).
> > Either way, there's generally a right tool for the job and I highly
> > doubt that bash, cut, tr, grep, sed, awk, etc are the hammers you
> > really want to use for this nail.
>
> Actually, I've seen these tools (along with, say, Gnuplot and
> GNU M4) being successfully used to process various scientific
> data. (Not to mention that I've a first-hand experience with
> this just as well.)
>
> It may be not a perfect solution, but I'm quite certain it's one
> of the best currently available ones.
>
Never used m4 so can't comment specifically. However, I'd look at some of the bio perl modules if this was the type of data I was looking at. Either way, learning dozens of tools to manipulate lots of data is quite time consuming, prone to failure, and quite frankly senseless. I know someone that uses tcl to process data at the national institute of health in the us to process medical data and someone else that uses python at the same institution to process another type of medical data. I don't know anyone that uses a shell to process large amounts of any data.