Re: A question about deleting a big file structure from a big disk in Jessie: Why does this work? I'm really worried.
On 20150402_1803-0600, Bob Proulx wrote:
> Paul E Condon wrote:
> > For several years I have been making daily backups of my four Debian
> > computers using Rsync and a small script of my own devising. The data
> > has been accumulating on an external USB drive in a partition with the
> >...
> > I'm worried about what I found. I want to interest someone who has far
> > more knowledge about how the kernel actually works internally to look
> > into this. I done other experiments more complicated to report, I can't
> > find anything comforting about this situation. If you think it's OK,
> > you probably don't understand, IMHO.
>
> I often have problems with USB mounted file systems. I believe the
> cheap nature of the USB hardware all around to be the major
> contributor. I do use USB for "a big floppy" all of the time. But
> whenever I keep a USB disk mounted for a long time it has always
> failed after a while. A month. Six months. I find the USB file
> system subsystem unreliable. I would never trust it for critical data
> such as backups. I think you are seeing the same unreliable mounted
> USB disk problems that I have seen for a long time.
>
> If you remove the disk from your USB container and mount it directly
> with its native SATA (or IDE) connector then you will find that it is
> as reliable as the rest of the native storage. I blame the cheap USB
> controller electronics. Although perhaps the kernel driver is also to
> blame in there too.
>
> [On the converse I find USB network adapters and USB sound cards to
> have been rock solid. Meaning that while I avoid USB disks I actively
> use USB networking on several machines to add additional NICs. I am
> planning another site using additional USB NICs. It is probably
> hardware dependent but they have been working great for me regardless
> of seeing the opposite for disks. And I have three sites using USB
> sound cards very robustly. Since I disparaged USB disk I felt I
> needed to clarify that it was only disk and not other USB.]
>
> > I found two other ways to delay the crash:
> > 1) using nice as in: ' nice -n 19 find -depth -print -delete'
> > (this, I think, slows down the main running job in relation to the
> > running of the kernel.)
>
> Read the man page for "ionice". You might consider using it instead
> of nice. Nice works with cpu usage. But ionice works with I/O usage
> and is directly what you are fighting. You might try:
>
> ionice -c 3 find -depth -print -delete
Thanks for this suggestion.
>
> If I am deleting an entire file system I will usually simply mkfs on
> top and reset it to empty that way. On a file system with millions of
> files that will be much faster than deleting them individually.
> Obviously only works if it is the entire file system.
>
> Bob
About long term mounts of USB being a problem: On the same computer
there is mounted a USB labeled 'sgt1' which is a 1 TB external USB. It
is the disk that is currently collecting daily backups using the same
script as was mentioned in my first email. Day in, day out, if that
computer is on, cron takes a full backup a little before 630 AM and
sends an email to me if there is anything written to file descriptor
2. There is nothing, unless investigation reveals a cause, like I
unplugged the power supply while vacuuming the area and forgot to plug
it back in.
I know you have had a different experience. I have been running this
script unchanged since before 2009 and only since I have been running
Jessie and running copies of massive file structure have I been having
problems. The script uses the Rsync option --link-dest=DIR and really
a very small amount of data is actually transferred in each daily
run. Rsync regularly reports Speedup of over 1000 times. I had been
making quite a lot of progress on organizing these file in a more
useful way, until I made the decision to move to Jessie, for reasons
that I am sure will turn out to be justified. For me, every failure
of an external USB HD turned out, after the fact to be attributable to
my not already knowing something like the transition to larger sector
size. I've learned a lot. And I don't want to imply that 'sgt1' has
been in service since 2009, just that one HD was connected to that
computer with its spindle motor running, and always responding wnen
crontab called for it.
So I'm skeptical. You may be right. But we are mostly all running Jessie,
now, and will all be running Jessie soon.
Thanks,
--
Paul E Condon
pecondon@mesanetworks.net
Reply to: