Re: MDADM RAID1 of external USB 3.0 Drives
Linux-Fan <Ma_Sys.ma@web.de> writes:
> On 09/22/2014 03:23 AM, lee wrote:
>> Linux-Fan <Ma_Sys.ma@web.de> writes:
>>> On 09/21/2014 08:41 PM, lee wrote:
>>>> Linux-Fan <Ma_Sys.ma@web.de> writes:
>>>>>> On 09/20/2014 04:55 PM, lee wrote:
>>
>> I've seen the smart info show incredible numbers for the hours and for
>> the temperature. Hence I can only guess which of the values are true
>> and which aren't, so I'm simply ignoring them. And I never bothered to
>> try to relate them to a disk failure. When a disk has failed, it has
>> failed and what the smart info says is irrelevant.
>
> I always at least try to read/interpret the SMART data. I consider it
> valuable information, although it is sometimes difficult to interpret.
> (Especially Seagate's Raw Read Error Rate and some other attributes).
How do you know which of the numbers are correct and which aren't?
>>>> You might be better off without this RAID and backups to the second disk
>>>> with rsync instead.
>>>
>>> Also a good idea, but I had a drive failure /once/ (the only SSD I ever
>>> bought) and although the system was backed up and restored, it still
>>> took five hours to restore it to a correctly working state.
>>
>> By using two disks attached via unreliable connections in a RAID1, you
>> have more potential points of (unexpected) total failure in your setup
>> than you would have if you were using one disk while working and the
>> other one for backups.
>>
>> For all you know, you can have a problem with the USB connections that
>> leads to data on both disks being damaged at the same time or close
>> enough in time as to make (some of) the data unrecoverable. This is
>> particularly critical for instances when the RAID needs to be rebuilt,
>> especially when rebuilding it while you're working with the data.
>>
>> You are using two disks and two unreliable connections at the same time
>> because of the RAID. That increases your chances of a connection going
>> bad *and* of a disk failure: A disk which is just sitting there, like a
>> backup disk, is pretty unlikely to go bad while it sits. A non-existent
>> connection cannot go bad at all.
>>
>> When you don't use RAID but a backup, you are very likely to only lose
>> the changes you have made since the last backup in case a disk fails.
>> When you use RAID, you may lose all the data.
>
> I did not know USB was that unreliable -- I am probably going to
> implement these suggestions soon.
Above doesn't only apply to USB. You could have bad cabling with SATA
disks and increase your chances to lose data through using RAID with
that as well.
For the last couple days, I've been plagued by an USB mouse failing. It
would work for while and then become unresponsive. Plug it into another
USB port and it works again for some time. In between, the mouse would
go weird and randomly mark text in an xterm. Use another mouse and it
shows the same symptoms. Use another computer (disks switched over) and
the mouse seems to work, at least for a while.
I've never seen anything like that before and never had problems with
PS/2 connections. So how reliable is USB?
>> This odd RAID setup you have doesn't even save you time in case a disk
>> fails. Are you really sure that you would want to rebuild the RAID when
>> a disk has gone bad? I surely wouldn't do it; I would make a copy
>> before attempting a rebuild, and that takes time.
>>
>> You're merely throwing dice with this and are increasing your chances to
>> lose data. What you achieve is disadvantages for no advantages at all.
>
> I am no longer sure of the stability of my solution. I am surely also
> going to try the "one connected, one for backup" variant as it would
> simplify the setup and increase stability.
Take it as a learning experience :)
>>> The failure itself was not the problem -- it was just, that it was
>>> completely unexpected.
>>
>> There are no unexpected disk failures. Disk do fail, the only question
>> is when.
>
> I have not been using this technology long enough to come to this
> conclusion. But in the end, all devices will fail at some point.
Then you cannot experience unexpected failures of devices.
>> With the amount of data we store nowadays, classic RAID is even more or
>> less obsolete because it doesn't provide sufficient protection against
>> data loss or corruption. File systems like ZFS seem to be much better
>> in that regard. You also should have ECC RAM.
>
> The data consistency is truly an issue. Still, I do not trust ZFS on
> Linux or experimental Btrfs more than MDADM + scrubbing once per month.
Btrfs seems to have made some advances, and they have declared that the
format it uses to store data is now unlikely to change. I used it for
backups and it didn't give me any problems. I'm considering to actually
use it.
> Either of these "advanced" technologies add additional complexity which
> I have tried to avoid so far. I did not expect that USB would prove such
> an additional complexity.
RAID isn't exactly non-complex. And what's more complex: RAID with LVM
with ext4 or btrfs without LVM and without RAID because it has both
built-in? You can eliminate a hardware RAID controller which can fail
and has the disadvantage that you need a compatible controller to access
your data when it does. You can also eliminate LVM.
What's more reliable?
> ECC RAM is already there.
>
>>> The instability lays in a single point: Sometimes, upon system startup,
>>> the drive is not recognized. There has not been a single loss of
>>> connection while the system was running.
>>
>> not yet
>>
>> What if your cat gets (or you get) entangled in the USB cables, falls
>> off the table and thereby unplugs the disks?
>
> Of course, I have taken action to prevent the disks from being
> accidentally disconnected. Also, there are no pets here which could get
> in the way.
So you're trying pretty hard to experience an unexpected failure? ;)
> In the currently running system, all USB works as reliable as I expect
> it: Devices never lose connection and all work with reasonable latency
> (for me). As the external storage is not accessed very often (I only use
> it for a lot of big files which would otherwise need to be deleted and
> additional VMs) the disks sometimes make a silent "click" when they are
> accessed again.
Try to enable power management for the USB controllers with powertop and
see what happens. Have a backup ready before you do so.
Using the disks for VMs? That's all the more reason to have a server?
>> Like USB or not, USB requires polling. What if your CPU is busy and
>> doesn't have time to poll the data from the disks or has some hickup
>> that leads to losing or corrupting data on the USB connection? Even USB
>> keyboards are way too laggy for me.
>
> I have not experienced any keyboard lag yet and I did not know that USB
> required polling. The quad core is rarely completely occupied, which is
> probably why I never experienced such problems yet.
When you play fast games and are used to PS/2 keyboards and then try an
USB keyboard, you'll notice. They even manufacture USB keyboards which
supposedly use a higher polling frequency to reduce the lag. How silly
is that?! Why not just use PS/2?
Remember the above mouse problem: These computers are awfully slow, so
perhaps they forget about the mouse. Now put some good load on yours
and see what happens to your USB disks ...
>> AFAIK that's not a rebuild, it's some sort of integrity check, usually
>> done once a week now. It used to be done once a month --- what does
>> that tell you?
>
> As far as I can tell, this integrity check is once a month. The "one
> week" refers to the "one failed boot per week" as a result of not
> recognizing the drive.
It runs once a week on Fedora. Perhaps once a month isn't enough?
>> And this time wouldn't be spent better on shutting down a server?
>
> Actually, looking at it from now with some hours already spent on
> configuring this setup, it would have made more sense to use a server
> from the very beginning. However, I did not think so when I bought the
> disks: I thought it would not matter which way the drives were
> connected
That's what you are supposed to think. Manufacturers want to sell you
all kinds of USB devices and want to omit PS/2 ports on mainboards to
make more money. So they find ways to make USB appear as an advantage
and don't want anyone to discover the disadvantages.
When you buy USB disks, someone makes money on the enclosure and the
power supply and the increased power consumption through the
inefficiency of the power supply and later by recycling or dumping all
these additional parts --- which they wouldn't if you bought internal
disks instead ...
> and assumed I could get a more reliable system if it did /not/ involve
> another machine (I thought a server would only be necessary if I wanted
> other systems to access the data as well). It turned out I was wrong
> about that.
Nothing prevents you from making the files available to other machines
without a server. And a server is another machine that can fail. You
can't really win, I guess.
>>>>> On the other hand, I have learned my lesson and will not rely on USB
>>>>> disks for "permantently attached storage" again /in the future/.
>>>>
>>>> USB isn't even suited for temporarily attached storage.
>>>
>>> If I had to backup a medium amount of data, I would (still) save it to
>>> an external USB HDD -- why is this such a bad idea?
>>
>> see above
>>
>> It's also a bad idea because USB disks are awfully slow and because you
>
> Compared to what is already in the system (two 160 GB disks and two 500
> GB disks) the "new" USB disks are actually slightly faster. Good 10k rpm
> or 15k rpm disks will of course be much faster, but I do not have any.
Hm. Why don't you remove those USB disks from their enclosures and use
them to replace the disks you have built-in?
That would reduce the number of disks you're using, you'd have faster
disks, you wouldn't need a server and you'd have no potential problems
with USB connections. You could even swap the 500GB disks into the
enclosures and use them for backups.
>> What FS do you use on your USB disks? And how do you read your software
>> RAID when you plug your disks into your "average Windows machine"?
>
> Usually, I place a live USB stick and three live disks (one 32 bit CD,
> one 32 bit DVD and one 64 bit CD) to be able to run Linux under most
> circumstances.
You're pretty well prepared in that regard :)
> Otherwise, I backup important data (which is luckily not that much) to
> 16 GB CF cards formatted with FAT32 (I have taken measures to keep
> UNIX file permissions and special files like FIFOs etc. intact).
CF cards? FAT32? Do you have a good idea of how reliable this is?
People taking pictures on such cards don't seem to be too happy with
their reliability. And FAT was designed for floppy disks which could
store 180kB or so. Do you know how quick FAT is with deleting data when
you check a file system and how easily the FS can get damaged?
My experience with FAT is that it is bound to fail sooner than later. I
never used a CF card, only SD cards, though not much. I haven't seen an
SD card failing yet.
> In the worst case scenario, the RAID can not be read on Windows, but
> being able to run a live system I could access all the data (which I
> could not do with tapes because of the missing device to read them).
That's sure an advantage when you use USB disks.
--
Knowledge is volatile and fluid. Software is power.
Reply to: