Re: apache BASIC authentication w/large userbase

[Craig Sanders <cas@taz.net.au>]

On Wed, Apr 03, 2002 at 06:35:22PM -0500, Jeff S Wheeler wrote:
> I have a customer who requires BASIC authentication for their site.
> They have a fair amount of traffic as well as a very quickly growing
> userbase.  They were on mod_auth_mysql before, but with hundreds of
> apache children that is not very practical.
>
> [...]
>
> The userbase is presently around 100K and growing 5K/day or so.  They
> were having things go so slowly that users could not login.  

my rule of thumb is:

any site that requires <1000 username:password pairs uses AuthUserFile
and plain text .htpasswd files.  any larger site uses AuthDBUserFile,
with username:password pairs in a hashed db (which is generated from the
plain text file).  a hashed db is ideally suited to this task, it's a
simple key/value (i.e. username/password) fast, indexed lookup.

using AuthDBUserFile is a lot faster, and a lot less overhead (memory,
file handles, etc) than the mysql or pgsql authentication modules.

apache comes with a program called dbmmanage which can be used to manage
hashed db files.  see the man page for more details.  it's pretty slow,
though, because it's a general purpose tool.  if all you need to do is
convert a plain text .htpasswd file into a corresponding .db file then a
5-10 line perl script could do the job many times faster.

e.g. something like:

#! /usr/bin/perl

use DB_File;
$filename="passwd.db";

# create the .db in a temporary file and rename it when it's done.
# rename is an atomic operation.
tie %passwd, 'DB_File', "$filename.tmp", O_RDWR|O_CREAT, 0644, $DB_HASH ;

while (<>) {
    chomp ;
    ($key,$value) = split /:/;
    $passwd{$key} = $value;
};

# untie the handle, close the file and flush all records to disk.
untie %passwd;

# move the .db file into place.  
rename "$filename.tmp", $filename;



on a busy P3-450 webserver, this script takes about 14 seconds to
convert a .htpasswd file with 35,000 entries into a hashed db file.
apache's dbmmanage takes over 90 seconds to do the same job.


craig

-- 
craig sanders <cas@taz.net.au>

Fabricati Diem, PVNC.
 -- motto of the Ankh-Morpork City Watch


-- 
To UNSUBSCRIBE, email to debian-isp-request@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org