Re: Growisofs input cache (get away from dd?)

["Thomas Schmitt" <scdbackup@gmx.net>]

Hi,

> You are still thinking about benefit to the image generation application
> rather than the overall system.

I have both in mind. My app shall not run poorly and
the system shall stay usable.

> The reads are small and scattered, but they
> are single use i/o, meaning that it is highly unlikely that the data will be
> used before it passes out of the cache.

I thought rather of something like read-ahead
caching. My impression when backing up small
files is that first the disk rattles like crazy
but soon it becomes calm while the burner is
still getting data at full speed.
I blame that effect on the disk cache which can
become several 100 MB, i confess.

> use readahead buffering and
> therefore will have the data to burn before it's needed.

We use fifos to smoothen the output stream.
That does not help with small files from disk
because the ISO formatter first has to collect
them and transform them into a stream.

The "read-ahead" i think of is assumed in the
file system driver and i believe it reads more
than one disk block with a single read transaction.
O_DIRECT is supposed only to read what the app
demands and will not cache any surplus which it
got due to block aligning.

Ok. We have two read processes:
- The (ISO) formatter where i doubt O_DIRECT is
  beneficial.
- The burn program where i mostly wonder whether
  it is worth to introduce a non-usual i/o gesture.
  I assume it will not be of harm in most cases
  although this discussion started because David
  Weisgerber found one case where it was suboptimal.


> Burning is a slow process relative to creating the image,

Not necessarily. The said small files can bring
disk performance under 8x DVD speed. (If i burn
the image on the fly.)


> That depends on how large your machine is. ;-) If you have 64GB a Blu-Ray
> will fit.

It is not about fitting all the media into RAM.
It is about the time span which other processes
have to renew the claim of their own RAM cache.
Any process which is agile enough can keep it.
I believe this effect eats up most of the theoretical
benefit of O_DIRECT streaming on my systems.
(I see no hampering with non-O_DIRECT so i can
 hardly expect big relief if i enable it.)


David and Andy, i believe that there are reasons
to use O_DIRECT on the input side of the burner
fifo.
But i never stumbled over them. Not on normal desktop
Linux systems.
My typical i/o problems - in burn app as in other
concurrent processes - is when the disk goes wild
during system backups in /usr. Then my X Window
curser begins to move in hops and my system monitors
cannot get their samples every second.
As long as larger files (>= 100 kB) are to be backuped
everyting runs smooth:
One burner burning, one drive checkreading (via cached i/o),
one or two dowloads streaming on disk, ... no problems.
No kernel compiling here, i have to confess.

Maybe that's the reason for our differences in
perception: my use cases for large data are mostly
sequential. Only the small file collector is truely
random access.


> You could try:
> 1 - drop cache (echo 1 >/proc/sys/vm/drop_caches)
> 2 - start a kernel compile
> 3 - start building a 4GB or larger ISO image.
> Then repeat with O_DIRECT. Use of iostat may give you additional data.

Step 1 is obscure to me. I assume it gives up
all cached data but allows to build up new
caches during the test. Right ?

Is step 3 intended as disk to disk operation ?
(or as disk to burner on the fly ?)


I have O_DIRECT tests on my todo list now.
I'll see.


Have a nice day :)

Thomas