Re: reading an empty directory after reboot is very slow
On 2015-04-13 16:28:27 -0600, Bob Proulx wrote:
> Vincent Lefevre wrote:
> > Loïc Grenié wrote:
> > > Vincent Lefevre wrote:
> > > > The problem is that this operation is (always?) very slow: something
> > > > like 100 seconds (1 minute and 40 seconds). It has been reproducible
> > > > for several months. The logs show nothing during this operation.
> > > >
> > > > Any idea?
>
> Of course immediately after a reboot the file system buffer cache is
> completely clean. You can avoid the reboot and recreate this
> situation by this:
>
> # echo 3 > /proc/sys/vm/drop_caches
>
> See this for the documentation:
>
> https://www.kernel.org/doc/Documentation/sysctl/vm.txt
Thanks for the reference (also available locally in linux-doc-*
packages).
> That would certainly be less disruptive to recreate the situation than
> rebooting the entire system.
>
> > > Maybe the directory is very large (even though its empty). Try
> > >
> > > ls -ld tmp.
> > >
> > > and see if the file "tmp" is large.
> >
> > Thanks! I didn't know that (I thought that the file system would
> > automatically "optimize" directories when files are removed, so
> > I've never looked closely at their size). Indeed:
> >
> > ypig:~/eftests> ls -ld tmp
> > drwxr-xr-x 2 vlefevre vlefevre 29655040 2015-04-13 15:25:55 tmp/
>
> What type of file system is it? Does it have dir_index? Both ext3
> and ext4 support it. Others are similar.
>
> # tune2fs -l /dev/mapper/v1-var | grep --color Filesystem.features
> Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
This is ext3 with dir_index:
# tune2fs -l /dev/sda1 | grep Filesystem.features
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
And for the FS creation:
Filesystem created: Mon Jan 4 16:26:16 2010
My machine is old, but I've never changed anything concerning the
file system.
> Without dir_index an ext filesystem with large directories is slow due
> to the linear nature of directories. But with dir_index it should be
> using a B-tree data structure and should be much faster.
So, why is it slow?
I also notice slowness with a large maildir directory:
drwx------ 2 vlefevre vlefevre 8409088 2015-03-24 14:04:33 Mail/oldarc/cur/
In this one, the files are real (145400 files), but I have a Perl
script that basically reads the headers and it takes a lot of time
(several dozens of minutes) after a reboot or dropping the caches
as you suggested above. With a second run of this script, it just
takes 8 seconds.
--
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)
Reply to: