Large filesystems

[Maitland Bottoms <bottoms@debian.org>]

Hi,

It's not too painful to start playing with large files now, but until
a clear policy develops it will take a little hands-on effort beyond
what is pre packaged.

Here's a micro-HOWTO on deploying a Debian Large File system:

1. Start with a woody system.

2. You'll need to build your own new 2.4 style linux kernel. (Maybe
   the large file patches are there for 2.2 series, but that's not the
   route I took.)

   Use make-kpkg (kernel-package) tool, and build both kernel-image
   and kernel-headers targets.

   Install both your new kernel-image... and
   kernel-headers... packages.

3. Now that you're running woody with a kernel capable of using large
   files, your need your own glibc.

   apt-get --build source glibc

   The nice thing is that the fine packaging of libc for Debian will
   detect the bleeding-edge kernel headers you have intalled, and will
   enable large file support for you.

   Building libc is resource intensive, but if you're going the large
   file route, you've probably go a system like the one I use, with an
   empty 300GB partition, dual 700 MHz Pentium IIIs, and a Gig of ram.

4. Build your own fileutils package so that those tools (ls and dd)
   work, and you can cope with moving those large files around.

5. Doing this probably isn't a complete step to large-file
   handling. I'd expect some applications to balk unless you recompile
   them yourself. But, again if you're like me, you've got your own
   source code to process those big files, 

   This does the right thing on both a Solaris 2.6 box and the Debian
   box: 
	fd = open(argv[1],O_RDWR|O_LARGEFILE|O_CREAT,0644);
   that program then goes on to write really big files.

I am having really good luck running 2.4.0-test12 kernel, and
using NFSv3 even (mount -o vers=3 server:/data /data) to go mucking
about in 5 to 10 GB files.

I am still waiting for the day that XFS or Reiserfs will play well
with NFS. Soon it seems...

-Maitland

>>>>> "Zdenek" == Zdenek Moravek <moravek@quantum.karlov.mff.cuni.cz> writes:

    Zdenek>  We need to create and use files whose size exceeds
    Zdenek>  2GB. Is there any
    Zdenek> support for large filesytems on the developres level in
    Zdenek> the new release of Debian so that one may construct
    Zdenek> applications working on these files,
    Zdenek>                                                     Zdenek