[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: >2Gb files



on Tue, Apr 10, 2001 at 06:06:36PM +0100, david.jackson@ogt.co.uk (david.jackson@ogt.co.uk) wrote:
> Hi, I'm looking for hints on using > 2GB files....
> 
> I'm using a 2.4.3 kernel-image compiled from debian kernel-source
> package. After installing the corresponding kernel-headers deb I
> created, I have recompiled glibc (2.2.2-4) with a
> #apt-get --build source glibc 
> then installed the created debs...
> #dpkg -i ./libc6-dev_2.2.2-4_i386.deb ./libc6_2.2.2-4_i386.deb  ./locales_2.2.2-4_all.deb
> I then build fileutils,
> #apt-get -t testing --build source fileutils
> and install them,
> #dpkg -i ./fileutils_4.0.43-1_i386.deb
> 
> Unfortunately a
> #/bin/dd if=/dev/zero of=file2 bs=1024 count=3553600
> gives me a
> "File size limit exceeded"
> 
> I've tried this in ext2fs and reiserfs and have tried a --build
> source and install of their respective progs packages, recreation
> of filesystems but I still have the 2GB limit. 
> 
> Any ideas? What I should be checking?

The following is pulled from VA's Knowledgebase.  This is currently
offline, but kick me periodically and I'll post a URL when it's
available.

The short answer is that there is no easy fix for large file support
under GNU/Linux.  As is often the case, I've no direct experience
attempting to implement any of this and would appreciate informed
opinion on either side of the debate.

------------------------------------------------------------------------
Copyright (c) VA Linux, Inc.
------------------------------------------------------------------------

Q:  Is there any way around the 2GB file-size limit in Linux?  Are there
    any stable patches to fix it?

A:   Short answer:  In a practical sense, no.  The 2GB limit is deeply
    embedded in the versions of Linux for 32-bit CPUs: in GNU libc, in the
    Linux kernel's filesystem drivers and VFS layer, in the fundamental
    design of some filesystems, and in the function calls used in the
    compiled applications and utilities furnished in (and for) typical Linux
    systems.  There is thus currently no VA-supported configuration.  Long
    answer:  All Linux ports for 32-bit CPUs (i.e., all architectures other
    than Alpha and IA-64) use 32-bit integers for file access and locking,
    yielding a maximum size of 2^31 - 1 = 2GB.  Creating a workaround for
    this situation without breaking existing code is an obstacle for all
    32-bit Unixes[1], whose creators met to design a standard 64-bit
    file-access programming interface, at a meeting sponsored by X/Open,
    called the Large File Summit (LFS)[2].  The resulting large-file
    standard and related system calls are likewise called LFS.

    To implement LFS (and remove the 2GB limit) on any of the Linux system
    loads VA Linux has used as of March 2001, you would have to:

    1.  Install glibc v. 2.2 or later, compiled against kernel headers
	from kernel 2.4.0test7 or later.  At this date, all software in
	VA's distributions are compiled using glibc 2.1.x and kernel
	2.2.x.  glibc 2.2 is considered experimental.

    2.  Install kernel 2.4.0test7 or later, or recent 2.2.x kernels with
	unofficial LFS patches, such as VA Linux's 2.2.18 kernel
	(ftp://ftp.valinux.com/pub/kernel/2.2.18pre11-va2.0/), or Red
	Hat Software's "enterprise" kernel.  Earlier versions omitted
	the necessary routines in the middle kernel layers (e.g., VFS)
	and improvements to the filesystem drivers.  A number of build
	problems have been reported with experimental kernel 2.2.x LFS
	efforts, including incompatibility with the much-requested NFS
	v. 3 and ext3 patches -- but VA Linux's 2.2.18 kernel
	nonetheless does include LFS, ext3, LVM, and David Higgen and
	H.J. Lu's NFSv3 patches.  (That kernel, however, may or may not
	include the necessary filesystem-driver improvements discussed
	below.  That aspect of the code is untested.)

    3.  Use suitable filesystems.  These include ext2, ReiserFS v. 3.6.x
	and up (available as a third-party patch), IBM's JFS, SGI's XFS,
	ext3 (a driver for which, it should be noted, is not yet
	included in kernel 2.4.0), and very recent NFSv3 client drivers
	(http://nfs.sourceforge.net/).  Problematic filesystems include
	NFS v.  2, earlier ReiserFS, AFS, coda, Intermezzo, Minix, UFS,
	SCO/SysV, msdos/umsdos/vfat, smbfs, and NCPfs.

	The new maximum size of an individual file is 8 terabytes on
	ext2 or ReiserFS filesystems, on LFS-compatible hosts.  For
	other suitable filesystems, a different limit applies in each,
	but reported values are all quite large.

    4.  Revise and recompile all programs and utilities that will
	manipulate large files, to use 64-bit file-access and
	file-locking system calls, instead of 32-bit ones.  Since you
	will not have access to source code for proprietary applications
	you may use (such as Oracle), this fix may be effectively
	impossible for them, and very daunting to carry out on the large
	number of relevant open-source userspace programs.

    Please be aware that, even if you implement these fixes without
    error, and even if the new 64-bit functions work perfectly, you are
    likely to encounter performance losses and greater memory usage,
    when using the revised and recompiled code.

    Since implementing the above-described LFS regime isn't very
    practical in most cases, sites having problems with excessively
    large files should generally find other ways around the problem:
    Large ongoing records can be written to multiple files branched off
    periodically (e.g., daily logfiles), or can be back-ended into one
    of the SQL databases, such as Oracle, that handle (work around)
    filesystem limits within their table-storage mechanisms.

[1]  I have heard it alleged that some 32-bit Unixes implemented fixes
different from those recommended by the Large File Summit, in some cases
earlier.

[2]  The X/Open Large File Summit's standards are documented here:
http://ftp.sas.com/standards/large.file/

------------------------------------------------------------------------

-- 
Karsten M. Self <kmself@ix.netcom.com>    http://kmself.home.netcom.com/
 What part of "Gestalt" don't you understand?       There is no K5 cabal
  http://gestalt-system.sourceforge.net/         http://www.kuro5hin.org

Attachment: pgp0ETJkdAmTX.pgp
Description: PGP signature


Reply to: