On 11/30/2013 03:50 AM, Albretch Mueller wrote: > md5, sha-*, ... sums are way too slow when you need to hash large > numbers of large files What processor are you using and from where do you read the files? If you read them from a normal HDD (or even most SSDs) on a modern processor md5sum will not cost you that much: ~$ cat /dev/zero | pv | sha256sum > /dev/null ^C16MB 0:00:06 [ 132MB/s] ~$ cat /dev/zero | pv | md5sum > /dev/null ^C69GB 0:00:06 [ 466MB/s] ~$ cat /dev/zero | pv | crc32 /dev/stdin ^C73GB 0:00:16 [ 628MB/s] ~$ cat /dev/zero | pv > /dev/null ^C.5GB 0:00:08 [3.41GB/s] 466MB/s is pretty fast IMHO and should be acceptable for large files (depending on what you mean with large files of course) as well -- no normal HDD is that fast. If it is not fast enough you could also run as many md5sum instances in parallel as you have processor cores available as you mention "large numbers" of files. Just my point of view and HTH Linux-Fan -- http://masysma.ohost.de/
Attachment:
signature.asc
Description: OpenPGP digital signature