Re: Making a 2>GB sized file.
Peter Cordes wrote:
>On Wed, Aug 27, 2003 at 04:26:56AM +0900, Kyungwon Chun wrote:
>
>
>>I made my new cluster using Sarge. The problem is that I can not treat
>>a file bigger than 2 GB. I'm trying to make a file on NFS mounted
>>directory, using mpich and hdf5. The error message is
>>
>>p15_4159: p4_error: : 1
>>File locking failed in ADIOI_Set_lock. If the file system is NFS, you
>>need to use NFS version 3 and mount the directory with the 'noac' option
>>(no attribute caching).
>>
>>But, I'm mounted the directory using version 3 NFS with no attribute
>>cache. (I found that the locking function is not work properly with the
>>nfs-common package of Sarge. So, I used that one from the Sid by
>>compiling from the source package.) I also tried the suggestion of HDF5
>>installation manual i.e.
>>adding the following compiler option when building the mpich package.
>>
>>-cflags="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64"
>>
>>
>>But, It seems that still not working. Is there any suggestion?
>>
>>
>
> Can you try on an ext3 (or anything that isn't NFS) to see if the problem
>is due to NFS? I just created a > 2GB file on ext3, over NFS, using
>dd if=/dev/zero of=bigfile bs=1024k count=2200
>I'm appending to it with cat, and it's now up to 3.4GB. Big files don't
>seem to be a problem for NFS on Linux. I'm using Linux 2.4.22 on the client
>and server, with the NFS kernel server. Maybe your problem is that lockd
>isn't running on the server, or something like that? Anyway, I don't think
>the problem is just because of large files.
>
>
>
I did the same test on my NFS mounted directory (dd if=/dev/zero
of=bigfile bs=1024k count=2200). I could make 2.2GB file with this
method without any problem. I also think that big files don't seem to be
a problem for NFS on Linux. But, If I try to make >2GB file with MPICH,
It make a problem. I also, check that the lockd daemon in running on the
server. The other possible cause of the problem is HDF5 library. But,
HDF5 library also works fine on a host filesystem. I checked this using
the test programs in the source package and my own one. Now, I think the
problem originated from MPICH package.
The information of my system follows :
SERVER
$ uname -a
Linux magi 2.4.21-4-686 #1 Sat Aug 2 23:27:25 EST 2003 i686 GNU/Linux
$ rpcinfo -p
program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100004 2 udp 978 ypserv
100004 1 udp 978 ypserv
100004 2 tcp 981 ypserv
100004 1 tcp 981 ypserv
100009 1 udp 980 yppasswdd
600100069 1 udp 983
600100069 1 tcp 985
100007 2 udp 990 ypbind
100007 1 udp 990 ypbind
100007 2 tcp 993 ypbind
100007 1 tcp 993 ypbind
391002 1 tcp 32768 sgi_fam
391002 2 tcp 32768 sgi_fam
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100021 1 udp 32950 nlockmgr
100021 3 udp 32950 nlockmgr
100021 4 udp 32950 nlockmgr
100021 1 tcp 38192 nlockmgr
100021 3 tcp 38192 nlockmgr
100021 4 tcp 38192 nlockmgr
100005 1 udp 893 mountd
100005 1 tcp 896 mountd
100005 2 udp 893 mountd
100005 2 tcp 896 mountd
100005 3 udp 893 mountd
100005 3 tcp 896 mountd
100024 1 udp 913 status
100024 1 tcp 916 status
$ /usr/sbin/nfsstat
Server rpc stats:
calls badcalls badauth badclnt xdrcall
10428820 0 0 0 0
Server nfs v2:
null getattr setattr root lookup readlink
0 0% 2943386 74% 0 0% 0 0% 779826 19% 6113 0%
read wrcache write create remove rename
163391 4% 0 0% 0 0% 0 0% 0 0% 0 0%
link symlink mkdir rmdir readdir fsstat
0 0% 0 0% 0 0% 0 0% 40481 1% 66 0%
Server nfs v3:
null getattr setattr lookup access readlink
20 0% 1536536 23% 43233 0% 128242 1% 1017021 15% 5864 0%
read write create mkdir symlink mknod
108858 1% 3469747 53% 33793 0% 105 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
32027 0% 43 0% 1003 0% 133 0% 2277 0% 0 0%
fsstat fsinfo pathconf commit
839 0% 839 0% 0 0% 114977 1%
CLIENT
$ uname -a
Linux eva1 2.4.21 #1 Sun Aug 24 15:43:58 KST 2003 i686 GNU/Linux
$ rpcinfo -p
program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100021 1 udp 32768 nlockmgr
100021 3 udp 32768 nlockmgr
100021 4 udp 32768 nlockmgr
100007 2 udp 813 ypbind
100007 1 udp 813 ypbind
100007 2 tcp 816 ypbind
100007 1 tcp 816 ypbind
100024 1 udp 842 status
100024 1 tcp 845 status
$ /usr/sbin/nfsstat
Warning: /proc/net/rpc/nfsd: No such file or directory
Server rpc stats:
calls badcalls badauth badclnt xdrcall
0 0 0 0 0
Server nfs v2:
null getattr setattr root lookup readlink
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
read wrcache write create remove rename
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
link symlink mkdir rmdir readdir fsstat
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
Client rpc stats:
calls retrans authrefrsh
1607040 10298 0
Client nfs v2:
null getattr setattr root lookup readlink
0 0% 166851 77% 0 0% 0 0% 41090 19% 234 0%
read wrcache write create remove rename
5564 2% 0 0% 0 0% 0 0% 0 0% 0 0%
link symlink mkdir rmdir readdir fsstat
0 0% 0 0% 0 0% 0 0% 1631 0% 1 0%
Client nfs v3:
null getattr setattr lookup access readlink
0 0% 64453 4% 2169 0% 4554 0% 53622 3% 79 0%
read write create mkdir symlink mknod
2270 0% 1257097 90% 1516 0% 2 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
1443 0% 0 0% 47 0% 6 0% 61 0% 0 0%
fsstat fsinfo pathconf commit
90 0% 90 0% 0 0% 4170 0%
$ mount
rootfs on / type rootfs (rw)
/dev/root on / type nfs
(ro,v2,rsize=4096,wsize=4096,hard,udp,nolock,addr=192.168.0.1)
none on /dev type devfs (rw)
none on /proc type proc (rw)
192.168.0.1://var/lib/diskless/default/192.168.0.11/etc on /etc type nfs
(rw,v3,rsize=8192,wsize=8192,hard,udp,nolock,addr=192.168.0.1)
192.168.0.1:/var/lib/diskless/default/192.168.0.11/etc on /etc type nfs
(rw,v3,rsize=8192,wsize=8192,hard,udp,nolock,addr=192.168.0.1)
192.168.0.1:/var/lib/diskless/default/192.168.0.11/var on /var type nfs
(rw,v3,rsize=8192,wsize=8192,hard,udp,nolock,addr=192.168.0.1)
192.168.0.1:/var/lib/diskless/default/192.168.0.11/tmp on /tmp type nfs
(rw,v3,rsize=8192,wsize=8192,hard,udp,nolock,addr=192.168.0.1)
192.168.0.1:/var/lib/diskless/default/root/var/lib/dpkg on /var/lib/dpkg
type nfs (ro,v3,rsize=8192,wsize=8192,hard,udp,nolock,addr=192.168.0.1)
192.168.0.1:/home on /home type nfs
(rw,sync,v3,rsize=8192,wsize=8192,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,udp,noac,lock,addr=192.168.0.1)
Reply to: