Re: /tmp as tmpfs and consequence for imaging software
On Sun, Nov 13, 2011 at 10:06 PM, Tollef Fog Heen <tfheen@err.no> wrote:
> ]] bastien ROUCARIES
>
> Hi,
>
> please don't cc me on mailing list posts. It's rude and against Debian
> list policy, even more so when I've set mail-followup-to.
>
> | Le Sunday 13 November 2011 16:24:18, Tollef Fog Heen a écrit :
> | > ]] Bastien ROUCARIES
> | >
> | > Hi,
> | >
> | > | No it is not true. Science and imaging software are better to use true
> | > | disk baked file. For instance, if I want ot invert a big matrix they
> | > | are pretty good algorithm that force only some part of the file to be
> | > | keep on disk. They known better than kernel when to put somepart on
> | > | the data on the slow disk.
> | >
> | > I doubt the authors of said software is better at writing VMs than the
> | > kernel authors are. Even if they are as good, the software isn't in a
> | > position to know the needs ot the whole system, just what the particular
> | > application needs.
> |
> | Yes they are particularly for inversion of matrix bigger than memory. And moviemaker software work on file bigger than memory.
> | Moreover it is not a problem of VM it is a problem of:
> | - information: kernel lack information on the structure of the problem: how can the kernel known that your piece of data is
> | positive definite (ftp.numerical.rl.ac.uk/pub/talks/jas.berkeley.28III2007.pdf)
>
> Trying to write a VM in userspace (which the abovementioned PDF
> describes) quickly runs into the problem of the kernel swapping out
> pages underneath your feet. The easiest way to solve this is to not try
> to manage it explicitly, but just allocating lots of memory or mmap-ing
> a big file and letting the kernel do the paging in and out.
No it does not work like you said. We know the matrix structure, not
the kernel. We map and unmap manually. Doing as you said is
inneficient and trash a lot cache and so on.
> [...]
>
> | > | Using tmpfs under /tmpfs you break assumptions on the life expendancy
> | > | of memory object.
> | >
> | > How is this broken? It's not like /tmp is magically cleaned more often
> | > just because it's a tmpfs.
> |
> | No but offlining is not made by application request but by kernel. It
> | is fine in lot of case but not for all the case.
>
> What do you mean by «offlining»?
But some part of the matrix on the disk.
>
> | > | And you slow down this kind of software (that work perfecly for 40
> | > | years).
> | >
> | > If the architecture of a piece of software is the same as it was 40
> | > years ago, it's not particularly well adapted to today's machines, since
> | > it won't know how to take advantage of virtual memory, multiple cores,
> | > etc.
> |
> | Not sure for instance see http://www.tau.ac.il/~stoledo/taucs/ or high
> | performance eigen solver have not been updated since 15 years. But
> | work well and are efficient.
>
> «September 4, 2003: Version 2.2! Multithreading using Cilk.»
>
> 2003 is not 15 years ago. Also, 15 years is not 40 years. ;-)
>
> Also, it talks about «tens of gigabytes» as if that's a huge
> system. It's not, it's a pretty small system; you can get machines today
> that has 1T of memory in a single chassis and that's before you start
> hitting the SSDs. The PDF you referenced talks about doing test runs on
> a machine that's about as capable as the three-year old laptop I'm
> writing this on, which doesn't seem to be relevant at all to how those
> problems ought to be solved.
We have also system with 1T but I run this kind of software on my own
laptop on small scale. And I need to perform inversion of 20G sparse
matrix with only 6GB of memory. I care about the 1T case with
infiniband and with my laptop doing some work.
> | Moreover the tmpfs solution has no quota like on disk file and this could lead to dos (see
> | http://0pointer.de/blog/projects/plumbers-wishlist-2.html for a solution and they are recent post on lkml).
>
> Sure, that's a valid criticism of tmpfs. It's something that's
> relatively easy to fix, though.
> | The problem is that you propose band aid and not long term
> | solution. Apps known better when kernel theirs need.
>
> I haven't proposed any solution, and if apps need to know whether their
> non-permanent working data is in physical memory or in a memory mapped
> file on a physical disk, something is broken somewhere. The right fix
> is to fix that, not introduce workarounds in other parts of the system.
Memory is used as a cache. This is not broken. This a valid use. I
dislike flame war like this but sometime the kernel is the good place
to do optimisation.I need to store more data than the main memory and
in this case out of core facility have only a penality of 15-20% for
huge problem. If you want to implement partitionning in the kernel you
are welcome to push to linus.
Bastien
Reply to: