Re: slapadd speed
On Fri, 28 Feb 2003 16:30, Emile van Bergen wrote:
> > database ldbm
> > cachesize 100000
> > dbcachesize 10000000
>
> I suspect the law of diminishing returns here. Cachesize (in entries) is
> a factor 100 above slapd's default, the same for dbcachesize. I have no
> idea what their in-memory data structures look like, but they may be
> inefficient for larger sizes.
>
> You may also be suffering from excessive lock contention if you're
> pounding it on an SMP box. OpenLDAP isn't known to handle that
> particularly well.
I don't think that slapadd has any concurrancy issues as it only has one
process.
> Try setting threads and concurrency to 1, and enable dbnolocking.
>
> Lastly, even though the CPU time indicates no I/O seek latency problems,
> you may want to experiment enabling dbnosync as well. It can make a
> *huge* write throughput difference if you trust your backend storage
> enough.
Here's the initial time for a load (NB it's a bit longer than before because I
am now doing sub-string indexing on the uid attribute):
real 3m25.101s
user 2m47.120s
sys 0m23.950s
Here's the time with dbnolocking and dbnosync:
real 3m25.184s
user 2m47.750s
sys 0m23.980s
So I removed them as they obviously did no good.
Now I tried the default cachesize and dbcachesize values which isn't good:
real 3m55.394s
user 2m54.050s
sys 0m50.570s
default cachesize and the dbcachesize of 10M which shows that a large
cachesize is bad:
real 3m19.413s
user 2m46.840s
sys 0m23.650s
default cachesize and a dbcachesize of 40M, increasing the dbcachesize helps:
real 2m56.051s
user 2m39.360s
sys 0m6.940s
cachesize of 300 and a dbcachesize of 40M, so a really small cachesize doesn't
help:
real 2m56.204s
user 2m40.300s
sys 0m6.160s
default cachesize and a dbcachesize of 100M, we're getting towards the law of
diminishing returns but I've got plenty of RAM:
real 2m50.340s
user 2m38.430s
sys 0m5.020s
default cachesize and dbcachesize of 500M:
real 2m47.355s
user 2m36.690s
sys 0m4.070s
I doubt that I would get any benefit from specifying a dbcachesize >500M.
Also I think it would be unwise as 500M is 13% of the RAM in the machine and
with future growth that risks having slapadd starting to thrash.
Thanks for the suggestions, this resulted in a performance increase of almost
20%!
The speed still sucks though, I think it should be at least twice as fast.
I'll have to try out OpenLDAP 2.1.x with BDB.
--
http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/ Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/ My home page
Reply to: