On 2020-05-14 03:47, Albretch Mueller wrote:
The thing is that I have to call, say sha256sum, on millions of files Probably debian admin people dealing with packaging have to deal with the same kinds of issues. lbrtchx
The need to checksum files is common; it is a good test case for trying out different computing paradigms and/or programming languages.
As other people have mentioned, using find(1) or xargs(1) from the command line to invoke sha256sum(1) is one possibility. All of these tools are mature and should produce predictable results. When used correctly, their performance is good. For ad-hoc tasks, this is how its done.
If you find that you need to parameterize the invocation, such as to use one set of arguments and/or options for one set of files and another set for other files, you can cut and paste an example invocation into a text file, parameterize it with variables, and add code to make the file into a script. I would start with a Bourne shell script. Of course, there are many other scripting languages to choose from; pick your favorite.
Even if you do not need parameterization, typing './myscript' requires fewer keystrokes and less mental effort than recalling a find(1) incantation over and over again. And, it provides consistency. These considerations are important when you are brain fried and heading for log off, or crawling through the files months later.
As you plan to perform the SHA256 computation a great many times, you should consider the cost of Unix process creation and tear-down -- e.g. CPU cycles (time) and memory usage. If you write a program that computes many checksums per process, it will have less overhead and should finish in less time than a program that creates one process per input file. Benchmarking will tell.
The above is related to the desired output format. Obvious choices include one checksum file for all input files vs. one checksum file per input file. The plus sign in the '-exec command {} +' option to find(1) facilitates the former, and should be efficient.
Also, where to put the output file(s) -- in the current working directory, within the input tree, within a parallel tree, or someplace else? One output file for everything is easiest, but my archive and image scripts checksum the input files individually and touch(1) the checksum file modification times to match.
Another consideration is concurrency. If you have a multi-core processor and implement a solution that puts two or more cores to work at the same time, a concurrent program should finish sooner than a sequential program. Again, benchmarking.
I find that Bourne shell scripts are comfortable only up to a certain level of complexity. Above that, I use Perl. That said, Go would be well suited to this task and should be faster. Then there is C, assembly, and/or hardware acceleration.
David