Re: md5sum input/output errors with large files

[Almut Behrens <almut_behrens@gmx.net>]

On Fri, May 13, 2005 at 11:35:22PM -0400, Roberto C. Sanchez wrote:
> Pollywog wrote:
> > When I do a md5sum check on large files (500MB or larger), I get this sort of 
> > error:
> > 
> > pollywog@slider:~/RO$ md5sum RO_Beta_v3.2_Full.zip
> > error processing RO_Beta_v3.2_Full.zip: failed in buffer_read(fd): mdfile: 
> > Input/output error
> > 
> > Does anyone know what is going on?
> 
> It looks like the buffer_read() call is failing :-)

yes, I guess it's just what it says: the underlying hardware has a
problem reading the data...
Can you copy the file (just to check whether some other read access on
the file does fail, too)?

> Seriously, you are probably running out of memory/swap.
> Try inreasing them, or using split(1) on the file and
> sum the smaller chunks.

I don't think memory matters much here -- due to the nature of the MD5
algorithm (it's 'incrementally' computed), you won't need to hold the
whole file in memory -- if that's what was meant.  (Actually, I've
successfully computed md5sums of files definitely larger than all
virtual memory available on the respective machine...)

Almut