XFree86 Memory usage

[Leo Spalteholz <leo@thewoodpecker.ca>]

I've noticed recently that if I keep my computer up for a while 
(2 days or so) the swap fills up until its all used (~360MB). 
top shows that XFree86 is using 441MB.  Now I realize that 64MB 
of that would be my video ram but 377MB is still huge.
I know that linux tries to use all available ram but isn't that 
limited to physical ram?

Or does XFree86 have a major memory leak somewhere?  I am running 
testing with no 3rd party packages installed.

Thanks,
Leo