Re: A question about deleting a big file structure from a big disk in Jessie: Why does this work? I'm really worried.

[Paul E Condon <pecondon@mesanetworks.net>]

On 20150402_2135-0500, David Wright wrote:
> Quoting Paul E Condon (pecondon@mesanetworks.net):
> [...]
> > Some time ago I decided to a make a copy of these data,
> > so I would have more than one copy.
> [...]
> Is the copying between a USB disk and an internal, or between two
> partitions on the same USB disk, or between two USB disks? (Ranked in
> decreasing reliability in my own experience.)
> 
> > When I tried, the job would always crash well before completion.
> 
> What are the symptoms of a crash? (Hang, segfault, write-failure
> as readonly, etc)
                      Switch of target disk to read-only mount.
> 
> [...]
> > But in both cases the
> > deletion failed because 'gfx2' has been remounted read-only, which
> > makes it impossible to update the target directory tree.
> 
> Do you watch /var/log/kern.log which this is going on. I find that
> quite useful. For example, messages like
> usb 1-8: reset high-speed USB device number 5 using ehci_hcd
> are accompanied by a pause of anything up to a minute in file
> transfer. I get these quite frequently if I do massive copies between
> two USB disks, so I now stage such copies through the internal disk.

Thanks. I do watch my own capture of the file descriptors 1 and 2 into
a file in /var/pec/ (sub-dir name, pec, is my initials). This will be
a useful addition, I'm sure.

In my system, most of my hypothesizing is from observing coincident
changes in two or more of the nine (soon to be ten) windows that I
monitor on system 'big'

> 
> I'm not so unlucky as Bob appears to be (he says, touching wood), but

I think Bob came to his conclusion during a previous period of
instability in Debian, but rather than start an argument that can only
degenerate trying to score debating point, I want to gather more
date. Bob has already helped me by making a truly useful suggestion,
for which I thank him.

It's getting late here. I won't get anything useful done tonight.
I'll just start making mistakes, if I start something new now.

May you both have a good night,
Paul

> I do get occasional I/O errors on USB transfers, which can make the
> disk readonly, but sometimes make it disappear altogether (ie it
> gets unmounted, not remounted).

All of my file systems are journaled. Did you notice a delay in remount
as the journal was replayed?

> > I have not tried it, but from my investigation I'm sure that a
> > massive delete of some obsolete file structure from the HD that
> > was /dev/sda1 during Debian install would trigger a remount-ro,
> > which surely would lead to a system crash in short order.
> 
> You get streams of error messages (like when the disk fills up)
> but it shouldn't actually crash.
> 
> (OTOH when you get a kernel error, there can be circumstances where
> the system will panic and *not* sync/write to the disk because to do
> so could cause corruption.)

So it helps to know about data that can be ignored for a legitimate
reason.

> [...]
> > I'm worried about what I found. I want to interest someone who has far
> > more knowledge about how the kernel actually works internally to look
> > into this. I done other experiments more complicated to report, I can't
> > find anything comforting about this situation. If you think it's OK,
> > you probably don't understand, IMHO.
> 
> My prejudices, based on no more than observations of my system, make me,
> like Bob, suspect the interface rather than the kernel. My wife,
> running windows, sees similar external symptoms (pauses, errors),
> though neither of us would know how to observe them in like manner to
> linux.

I like to play the scientist in my old age. All theories begin with curiosity
about a random observation. A classic case is the observation of a pocket
watch lieing on the path during a walk in a park. In our case of hypotheses
about the design of the kernel, it is unacceptable to me to invoke the idea
of a universal creator, and perfectly OK to contemplate the possibility of
a flaw in the design.

> Just in passing, if clamav wakes up and spots the USB drive, file

clamav is terra incognita to me. 

> transfers can stop for 10 or 15 minutes; the USB disk heads will still
> be very active. I keep an xterm running top so I can spot that (and
> other cpu-guzzlers like xulrunner).
> 
> Oh, and to David C, this happens irrespective of wheezy or jessie
> (for me).
> 
> Cheers,
> David.

Cheers, 

-- 
Paul E Condon           
pecondon@mesanetworks.net