Re: old vs new system uid/gid

Thank you, Bob. As always you are very helpful. Followups below.

On Tue, May 7, 2013 at 10:30 PM, Bob Proulx <bob@proulx.com> wrote:

Ross Boylan wrote:
> It would be very handy if, while creating a new system, I could
> replicate the uid and gid's of an old one--by which I mean replicate
> the mapping of numbers to names. I want to restore the old system,
> and my backup system (bacula) uses numeric uid/gid only. My backups
> have all the customized files, including /etc, but don't have
> standard ones, in particular /usr.

Yes. I often need to do this too.

> Any suggestions how to go about this? I can think of 3 general
> strategies, none without problems.
>
> 1. Copy passwd and related files.
> Debootstrap the new system and then copy the old passwd, group and
> shadow files over the exiisting ones.

Yes. Do just what you suggest. That is the right way. But only if
you are going from the same version to the same version. If you are
crossing the releases then sort the files by uid first and then just
copy the user accounts over.

I need the system accounts to line up too; starting from the old release on the dead system sounds like the thing to do.

> I suppose it would be safer to have a script read the old files and
> do adduser/addgroup as appropriate, since that would assure the home
> directories existed (and maybe do other stuff I'm not aware of).

Safer how? It seems much more complicated. Because you would need to
specify all of the parameters as options to adduser. Complexity leads
to the higher possibility of errors. Therefore I think it is more
dangerous.

If I don't adduser then home directories won't be created, even though /etc/passwd will refer to them. I figured that would lead to problems. I presumed adduser does other stuff too, which would not be done (aside from groups and passwords) if I skipped it. Your later remarks indicate there may not be much more adduser does. Actually, some of the skeleton files it usually copies may be inappropriate for system accounts.

> I'm not sure if creating accounts before installation of the
> packages that usually create the accounts will screw up the proper
> setup of the packages and accounts.

No. Most maintainers are fearful of removing accounts. I don't know
why but they are. Therefore if you remove a package that created a
user it will most of the time leave that user around in the passwd,
group, shadow and gshadow files. Then you might install it again.
That is actually being tested for to work correctly using piuparts
these days and should always work.

Therefore if the account exists it is okay to install and it will use
the existing account. Which is to say that if you have already
created the accounts by importing the old password file then package
installation will always be okay too.

Excellent.

If you are restoring the same version as you are copying the passwd,
group, shadow, gshadow files from then no problem. If you are copying
from an older version then it is possible that the base-passwd package
has changed some of the base system accounts. In that case after
doing the file copy read the manpage for update-passwd and run it and
have it verify and possibly update your system accounts.

Thanks; I wasn't aware of update-passwd.

> A problem with this strategy is that the installation procedure
> creates some accounts early on, and I'm not sure I can intervene
> before that point. The new system* has libuuid with uid 100; ssh is
> 101; the old system had ssh at 100 and no libuuid at all.

Yes. A very small number of packages with added accounts are
installed. You can minimize the number of those packages. For
example you don't need to install ssh and that avoids that issue.

And the new system is wheezy; with luck if I recreate a lenny it shouldn't have libuuid since it didn't have it before.

For the very small number of packages and their system accounts just
go ahead and use the old numbers and then chown the files on disk to
match the new numbering.

This is referring to restored files, right? I.e., restore them, which will give them the old ids. Then chown/chgrp to match the new system.

It is okay to do this. Mostly those system
accounts won't have any files on disk. Which is the entire point of
giving them an entire uid:gid to themselves, so that they won't have
any access to any other file on disk.

You can always manually merge the files. This isn't that hard. They
are simple text files. The format is designed to be obvious and human
editable. There isn't anything magic about using adduser that you
can't do by editing those files by hand. And if it is broken it isn't
that big of a deal to figure it out and fix it.

> 2. Restore old system and upgrade it.
> Restorethe old system into a VM and then upgrade. It would be a big
> upgrade, Lenny->wheezy (needs to be lenny->squeeze->wheezy to be
> supported) and 32 to 64 bit (which is not officially supported at
> all, though there's stuff around the net).

If you are sourcing passwd, group, shadow, gshadow from a very old
system then you probably shouldn't just copy it into place.

I think you mean if I restore the files from the old system (Lenny) onto a new one (Wheezy), be careful. I was actually thinking of trying to virtually reconstruct the old machine, including the fact that it was running Lenny.

Probably
okay.

meaning I might get away with just copying?

But perhaps a lot of differences in the system accounts and you
would need to walk through each releases update-passwd script.

Instead then I would merge the files. I would sort the file by uid to

instead of what? Are you thinking of retrieving the version-specific update-passwds without actually putting the whole system through the upgrades.? That's not what I meant.

make things easier.

sort -t: -k 3,3n -k4,4n /etc/passwd

Then all of the system accounts with small uids are at the top. All
of the user accounts with large uids are at the bottom. Copy just the
user accounts from the old to the new. Then repeat with group, shadow
and gshadow.

I think we're talking past each other. When I said "restore the old system into a VM and upgrade it" I meant that inside the VM I would be running the entire old system, not just borrowing some of its files. So the VM would start as 32 bit lenny with all my old files. I'd then upgrade to squeeze and then wheezy. Somewhere along the way I'd need to switch to 64 bit.

> I think this needs to be done in a VM not a chroot, since the latter
> will get its uid/gid info from the host system.

A chroot does *not* get uid/gid from the host system.

Thanks for the correction. I figured since the hostname in the chroot comes from the host (I think I've been told) that users and groups had a similar story.

A chroot has
its own /etc/passwd, group, shadow, gshadow files. This is
independent of the host system. If you enter a chroot shell and run a
'ps -ef' you might see the wrong names. Which is actually one of the
limitations of chroots. The files are separate but the process space
is shared with the host system.

Therefore one _should_ keep the passwd et al files in sync with the
host system. Because otherwise processes might run as the uid of a
different process between the host and the chroot.

Just debootstrapping a chroot and having the passwd file there be
different from the host shouldn't hurt anything because at that point
they are just files on disk. But if you run a deamon there and it has
different uids then the safety net of enforcing that they can't write
to each others files and memory is removed. Let's say bind or postfix
in a chroot has the same uid as ntp on the underlying host. One could
write the other's files. So we normally try to sync them together.
If you look at bind and postfix that run in a chroot you find that the
/etc/init.d script will sync the important files into their private
chroots just before they start.

But that only works because we actively try to make that happen. It
doesn't happen by default. And the general risk is low because
normally most people don't run daemons from a chroot. So in order to
trigger a case it takes a number of unusual things to converge.

> This also has some uid syncronization issues since I must recreate
> the system before putting the files onto it; these should be less
> acute since a variant of 1 should work more cleanly.

I didn't understand this part.

Old dead system was Lenny. So I bootstrap a new Lenny and install the packages that were present in the old system. To actually get the same ids that I had before I must either install packages in the same order I did before or setup the necessary id mappings (in /etc/passwdd et al) so that as I install the packages they get the right ids.

Then the remark about this working more cleanly means doing an old Lenny on top of a newly bootstrapped Lenny is likely to match accounts more smoothly than old Lenny on top of newly bootstrapped wheezy. Which I think is what you have been saying too.

> 3. Clean up afterwards.
>
> Restore the old files with the old ids, install the necessary
> packages to create all relevant accounts, and then make a script to
> swizzle the old ids to new ones.

Can be done. I have done this when I didn't plan ahead enough and
then needed to "save" the problem without going back and restarting it
all. I have needed to do this as recently as last week on another
server because other people did things like this without thinking
about the consequences. It is a little tedious but you can manually
clean up and fix these files. It isn't fun to do. It isn't the end
of the world. But easier to avoid it.

> Since I use VMs I should probably be using LDAP, but I think that's
> best left for later.

Using or not using LDAP for accounts I see as completely orthogonal to
this question. (Same for using mysql too.)

If a VM gets its account info from LDAP then it will use the same UIDs as elsewhere on the host and other VMs. So doesn't it provide a natural way to ensure the ids match, even if the virtual systems are different releases?

> (*) I expect I will need to recreate this new system; it is
> temporary but has the 64 bit wheezy environment I'm targetting. The
> ids on the temporary system are already significantly different from
> those of the old one. The old system and disk died (32 bit lenny).
> I have backups but not complete disk images; e.g., I didn't back up
> /usr.

If it were me I would run through the debian-isntaller and install the
absolute minimum system possible.

Lenny or wheezy?

Then merge the password, group,
shadow, gshadow files. Then install everything that you want.

Just last month there was a related thread of discussion where Thilo
Six and myself had a good talk about related issues. The thread
starts here.

http://lists.debian.org/debian-user/2013/04/msg00240.html

It isn't completely this same topic but it is in the same area. I
think it adds to the question you are asking here. And note that we
didn't completely solve it there. We just understood it better.
Hopefully it will be of use to you too.

Bob

Thanks for the tips, references and advice.

Ross