Re: Some help with dd backing up into an iso
Thomas Schmitt:
> Hi,
>
> GiaThnYgeia wrote:
>> 2.1 Block by block, [...] erased data on
>> an empty block can be recovered because they are not zero. Correct?
>
> If not the filesystem overwrote the content of deleted files
> the problem will still be to find the content you are interested in.
Good to know, but no I am not looking for deleted data.
>> 2.2 If a block is zeroed it can't be unzeroed?
>
> Not by normal means. You have to be aware that especially solid state
> disks like USB keys have an own physical block management which might
> delay the overwriting further.
So each manufacturer may have a different internal system but the output
is standardized. So we don't really know what goes on in there, right?
I had gone down the isle of trying to read chips and modifying them (1
to break printer out of ink pseudo blocks - 2 to look into internal
ignition/injection maps for vehicle modification) but I gave up early
for not been able to locate proper instruments. One of the reasons that
I got back on computing and away from windowz.
>> So 4gb full of data in 1 partition with 4gb
>> of empty but encrypted partition is treated the same as 4gb full of
>> data, Correct?
>
> Well, an encrypted partition should be much less compressable than an
> empty unencrypted one. Encryption shall camouflage the true content.
> So it can hardly represent all zeros as a similarly redundant byte set.
Aaahh.. so part of the trick is filling in fake data so you can't tell
the real ones, I get it. I think!
>>> dd if=/dev/sdb | bzip2 >usbfilename.img
>> So the of=usbfilename is replaced by the | bzip2?
>
> Yes. If dd has no of= argument then it writes its data to standard
> output, which is normally your terminal window.
The matrix :) The real thing.
> But "|" establishes a pipe. It connects standard output of dd with
> standard input of program bzip2. (That standard input would normally
> be your keyboard and its Enter key.)
> Since bzip2 gets no file name argument it reads from its standard input
> and writes the compression result to its standard output.
> But the ">" redirects the standard output of bzip2 to the data file
> usbfilename.img.
> So you do not get printed a lot of text salad on your terminal but
> there rather emerges a data file with compressed content.
OK, perfect sense.
> This connecting and redirecting of output is done by the shell, not by
> the programs dd and bzip2.
So it is like combining various processes and their inputs and outputs
into one, which is the final product the user needs.
>>> When you put it back on the USB stick, you need to uncompress:
>> I will report back ... I'm willing to try this on my 1.8gb system
>
> Be careful not to spoil irrepairable data.
Well at this point it is all experimentation so I know how to do it
right when I need to.
>> I have yet to see anything been written
>> in the swap area.
>
> Maybe your computer has lots of RAM or the swap is not in use ?
> (What does shell command "free" report ? Is ther a line starting with
> "Swap:" and giving three numbers ?)
I didn't know of this command, all I could see was the partition always
being empty. I suppose things get written and deleted (swapped) only
when RAmemory runs out. And I thought only ms-win did such silly things.
>> maybe due to systemd it is no longer being used?
>
> Systemd is a convenient suspect for everything. But i doubt that it
> can make swap space obsolete when the RAM does not suffice.
$free
total used free shared buff/cache
available
Mem: 3884232 1596976 302428 135400 1984828
1864344
Swap: 0 0 0
Tried it on a stressed out 32bit VBox system and it was not 0 0 0 but
I had only given it 1gB of RAM.
>> 1M used to be big!
>
> Yeah ... Love, 36 bit, and punched cards ...
Before that in a machine shop CNC machine I wrote code into a paper tape
with 5 columns of holes. Like a 70s Telex machine
In the mid-90s I took a turn to work OUT with hands and tools and no
digits. Now I am back at easy comfortable air/conditioned life, red-hat
is still around, and multi-processing things and satas and all kinds of
crazy stuff I need to catch up with.
>> I am willing to bet that this dd goes back to backing up hard drives
>
> Its origin is in IBM's Job Control Language. From there it came to early
> Unix when there were still unused combinations of two letters. "cp", "ls",
> "dd", "cc", "ld" ... "cat" is of course an example of wastefulness.
I never touched any VMs ... the machine owners were DEC customers, and
we did alot of work on VT100s with real hard keys.
>> Etcher (https://etcher.io/) after finishing
>> went back and "verified" the whole thing,
>
> A good idea to do so.
So my headache is to create a proper image and in an efficient way,
restoring it seems easy.
>> I suspect it is a fork of xorriso
>
> No. xorriso packs up files as inhabitants of an ISO 9660 filesystem.
> There are no forks known.
Yes, I thought the problem with dd was that I didn't know how to tell it
to use this ISO9660 that I thought was the problem of not being able to
read the img as an archive. In other isos I've seen you can see the
filesystem partition. So I started studying xorriso but I got lost with
all the different options that had little meaning to me.
>> had too many gaps in knowledge to understand how to use its options.
>
> If you tell me the path to the mount point of the data partition i will
> modify my previous example to that address.
depending on the position or how many sticks I have on it is usually
dev/sdb or dev/sdc As filemanagers like pcmanfm do not reveal the right
partition name only the label, I use gparted to make sure I have the
right one.
>>> While the filesystem is mounted but fewly busy, you let a program read
>>> the files and pack them up in that program's archive format.
>
>> I may be unclear on this, my system was running on sda backing up a
>> different system on sdb, it wasn't archiving itself.
>
> That's very wise. Making a dd copy of an active system disk will at best
> cause the symptoms of a heavy system power failure when you restore the
> system to a disk and then try to start it up.
At best I suppose it would be like trying to start something that is
already running.
> Backup programs usually run on mounted filesystems and thus are prone
> to recording inconsistent file states if files change while the backup
> is run. This danger can be avoided entirely if the filesystem is mounted
> read-only.
Since you have been willing to teach at the elementary level I'll shoot
for more. Once you install a system like debian, does the information
in the hidden part of the disk ever change, of can I just copy the file
system partition as a backup and replace it if it breaks?
>>> Assumed you have your filesystems mounted as
>>> /mnt/usb_part1
>>> /mnt/usb_part2
>
>> But there is this gray area called usb_part(no number) which seems to
>> have some goodies that make parts 1 and 2 sing.
>
> Now i know that one of them is swap and thus not mountable.
> But the other one is supposed to be mounted in your overall filesystem.
> Either by an automounter (e.g. systemd-udev) or by a mount command issued
> by the superuser (i.e. you with your sudo hat on).
This systemd-udev is not within the filesystem partition, it is outside
somewhere in the beggining of the disk, which is what "mounts" the file
system and gets it running. Correct? What is the boot flag for and is it
necessary on linux? Or is the boot flag only needed when there is
nothing else telling bios to start that specific partition and run?
> You will have to find out that path to the root of your partition's
> filesystem. Then you can backup it.
OK, let's say swap is sdb1 and the filesystem is sdb2 (for this example
there is no other partition). There are about 2mb in the beggining of
the sdb then 2 partitions. This is an actual working installation not a
live system. With the exception of having to reinstall sound drivers I
got it working on 2 different but similar systems. I was surprised it
did, but it does. This may help me from having to maintain two separate
but parallel systems and just have one portable one and just use the hd
for just data files of work that I do.
>> I gave up trying to understand all the tsorriso options,
>
> Nobody is supposed to. Just collect in a shell script what you need and
> what you learn from the internet, man page examples, or me.
:) I should have looked at the name of the maintainer up before I open
my big mouth :) Nice to meet you mr Libburnia :)
>> And that is how a network of dummy terminals that always boot up a fresh
>> installation of linux works. No magic?
>
> My xorriso example (now to be done with only one partition) is for making
> a backup of the partition filesystem. You can access the files by mounting
> the ISO filesystem or by using one of the archivers which can read ISO 9660.
> (Mounting is a feature of the operating system.)
Many of the examples I found were specific to partitions and not a whole
bootable disk. Unless one studies filesystems it is hard to understand
why this 9660 is important. What I understood is its limitation to long
and complex filenames. So if I was to back up something with huge
filenames I suspect it may run into problems of altering them and not
being able to restore them correctly. Have I made stew out of what I read?
> The magic gate for booting is in the boot sectors which let the firmware
> start the very magic boot loader which then starts the incredibly magic
> operating system.
> I only know about boot sectors. The rest is the job of e.g. debian-cd which
> prepares the files of boot loader and operating system, and the job of
> the boot loader which has to deal with the hardware on which it wakes up.
I always wondered why a machine with no-hd and a cd drive, would have to
go on and run on an error before you can press the button, open the
door,insert the cd, and have to restart it. I think at some point that
changed into an error (no disk found, insert new disk and press enter)
or it was variation of bios setups.
My specific question with this is how often in a debian system do things
in the boot sector change, if ever. I can't believe that if you have a
Debian2 and you change the source to jessie the bootloader stays the
same of it is the same with entering a fresh install of 8.7.1
>> You have a teaching talent...
>
> Despite i am the one who wrote the man page of xorriso ?
Well, ... I guess it is different when writing a manual that you think
only developers and sys-admins will ever look at, who understand all
those funky terms, than when you have to explain it to a guitarist that
only cares about backing up his composition, recordings, midi files into
a stick and transport it to the studio. If the studio only uses some
commercial expensive apple stuff it helps having a file-system with you
to edit the files while on the run.
>> Most of them say: ZZZ is a tool for [...]
>
> It is quite difficult to describe to users something that you know on
> source code level.
This is a trend with any tech/science field today. Only experts of the
same specialty can understand each other. If you go back and read the
notes of technicians (applied physicists before engineering was a
science) of around WW1 era, things that may even seem advanced and
unpatented yet are written in language that can be understood by a wide
background. Today if a biologist is working on human skin tissue and
another is working on eye-cells they understand nothing of each other's
work. So science is totally useless to everyone but the employers of
scientists. What you are admitting here is being a bi-product of that
same system. And we need to change this, otherwise your great xorriso
code may only be useful to some SONY/SAMSUNG who may manufacture a tiny
little backup machine with closed up code to sell to people not like me,
who actually do have the money to buy it.
>> Imagine if English wasn't your native language!
>
> This might be another part of the problem. I'm german.
I meant for those who read the manuals generally for linux and apart
from the terminology their skills in English are limited as well.
> Have a nice day :)
> Thomas
You too, maybe I can take a more careful look at the site and help with
the descriptions and instructions from a different perspective. I am
trying to make sense out of all this stuff so I can create instructions
for another language.
--
"The most violent element in society is ignorance" rEG
Reply to: