Re: [RFC] OpenLDAP automatic upgrade
Torsten Landschoff <torsten@debian.org> writes:
>> Some issues to consider for the 2.1 to 2.2 process:
>>
>> 1) OpenLDAP 2.1 generally used BDB 4.1. OpenLDAP 2.2 should only be used
>> with BDB 4.2.52+patches when using bdb or hdb as the backend (Ignore the
>> documentation with OpenLDAP 2.2.23 stating it requires BDB 4.3, that
>> statement is incorrect).
> Yes, I accidently built it with BDB 4.2 and it worked. But - what is the
> problem with 4.3? And what patches will be needed for 4.2.52 - it's not
> like I can go and change the libdb4.2 packages, I'll have to ask the
> maintainer for that favour.
The patches for BDB 4.2.52 are freely available from Sleepycat. They are
required to be in place if you want a stable BDB 4.2.52 distribution. I
would be very surprised if the package maintainer hadn't already included
the patches in their build. I also post links to the patches from my
website on building OpenLDAP:
http://www.stanford.edu/services/directory/openldap/configuration/
specifically in this case:
http://www.stanford.edu/services/directory/openldap/configuration/bdb-build-42.html
You can ignore the custom patch about transactions, as that is not an
official sleepycat patch, but one written by Howard Chu, one of the
primary OpenLDAP developers.
As for BDB 4.3:
You cannot use BDB 4.3 to load databases past a few million entries into
OpenLDAP. The way BDB handles logs changed enormously between BDB 4.2 and
BDB 4.3, and its log management is not stable, often running out of
space. In addition, numerous users have written the OpenLDAP list
complaining of issues they've hit using BDB 4.3 with OpenLDAP 2.2. The
solution was for them to move back to BDB 4.2.52+patches. This became
such a problem that with the OpenLDAP 2.2.24 release (done today), the
README file was updated to say:
SLAPD:
BDB backend requires (latest) Sleepycat Berkeley DB 4.2
I've been running OpenLDAP 2.2 with BDB 4.2.52+patches since OpenLDAP 2.2.6 was
released, and any issues I've had were with bugs in OpenLDAP, not with
BDB. I also ran benchmarks against BDB 4.3 based OpenLDAP 2.2 servers,
and they performed no better (and slightly worse) than BDB 4.2 based
OpenLDAP 2.2 servers.
In short, I cannot find a single reason to run OpenLDAP against BDB 4.3,
and even the current OpenLDAP release notes that BDB 4.2 is required. I
can find many reasons to not use BDB 4.3.
>> 2) Several changes were made to how ACL's were handled between OpenLDAP
>> 2.1 and OpenLDAP 2.2. You cannot simply upgrade the database and expect
>> everything to work. The ACL's must also be reviewed and correctly
>> updated.
> Surely. For simple installations it might work without that though and
> not every slapd package is installed to run the Stanford directory
> server. :)
I doubt that it will work for even simple installations. The very syntax
of many of the ACL declarations were changed. I had to go through and
update every single ACL in my ACL file for it to work right. ;)
>> 3) OpenLDAP bdb/hdb based databases are highly dependent on a well
>> configured DB_CONFIG file for the particular database in question. Simply
>> doing a slapcat/slapadd is really not sufficient for ensuring that the
>> resulting directory server will run well. A poorly configured
>> DB_CONFIG file can lead to the slapadd process taking hours or days. In
>> addition, there are two flags that should be set in the DB_CONFIG file
>> before the slapadd takes place, that should then be commented out once the
>> add is completed.
> Erm, the upgrading process is already really complex, and I fear I don't
> feel like implementing this in maintainer scripts. Which options are you
> thinking of, BTW? And the documentation for DB_CONFIG is really lacking
> as well, let's hope somebody wakes up the sleepy cats ;)
The two flags I am thinking of for BDB 4.2 are:
# Just use these settings when doing slapadd...
#set_flags DB_TXN_NOSYNC
#set_flags DB_TXN_NOT_DURABLE
They turn off checks that aren't necessary when performing a slapadd.
I also advise reading:
http://www.stanford.edu/services/directory/openldap/configuration/bdb-config-42.html
http://www.openldap.org/faq/data/cache/1075.html
> I wonder why the OpenLDAP server can not tune most of the BDB parameters
> itself during runtime. It has by far more information at its hands than
> the Debian maintainer scripts have.
If you really want to have this conversation, I suggest talking to Howard
Chu (Howard Chu <hyc@highlandsun.com>). He's the author of back-bdb and
back-hdb (The OL development team tends to refer to back-hdb as Howard's
DB. :P).
>> 4) If someone is using ldbm as their backend database, I'd throw a warning
>> noting that bdb/hdb are the preferred backends of OpenLDAP. ldbm is rife
>> with issues such as not catching data inconsistencies, leading to poor
>> directory performance and potential data loss.
> Where is the difference to bdb? ;-)
http://www.openldap.org/faq/data/cache/756.html
Pay close attention to:
"back-bdb uses BDB's most advanced features to offer transactional
consistency, durability, fine-grained locking, and other features that
offer improved concurrency, reliability, and useability."
"With back-ldbm, there is no fine-grain database locking. This means write
operations are serialized. And while multiple read operations may be
performed concurrently, they cannot be performed concurrently with any
write operation. Additionally, LDBM databases cannot be accessed by only
one program at a time (generally at the file level). (While one may be
able to bypass the locking mechanism, you will likely corrupt the database
(and/or obtain bogus information).)"
"back-ldbm stores entries in LDIF format, and back-bdb stores them in a
binary format that is 3-4 times faster to read and write. back-ldbm's
index management is reminiscent of filesystem inodes, with direct blocks
and indirect blocks, and individual index blocks are malloc'd and free'd
on demand. back-bdb's index management is much simpler, and blocks are
malloc'd and free'd much less frequently, which again yields better performance."
As an individual who dealt with an ldbm based directory, I can attest to
the superiority of back-bdb.
>> 5) I don't know what version of OpenLDAP 2.2 that debian is considering
>> for inclusion, but I wouldn't go with anything less than OpenLDAP 2.2.23
>> based on my experiences.
> That's what I am working on.
Okay. 2.2.24 was released. You might examine the changes file to see if
you want to use it, or pull in patches that it incorporated.
>> Essentially, the process of upgrading a directory service to OpenLDAP 2.2
>> is generally something that should be done by the person who runs the
>> service. There are a host of issues that must be addressed to properly
>> upgrade.
> That's for sure but I want to be able to do automatic upgrades for the
> simple cases. And at least help the admin by dumping the directory
> before starting the upgrade and taking care of the old database files in
> case he decides to downgrade again later :)
I don't think there is a simple case. ;)
--Quanah
--
Quanah Gibson-Mount
Principal Software Developer
ITSS/Shared Services
Stanford University
GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
Reply to: