Re: NUMA setup on Tyan s2895
On Wed, Sep 28, 2005 at 06:12:13PM +0200, Lionel Elie Mamane wrote:
> On Wed, Sep 28, 2005 at 10:28:17AM -0400, Lennart Sorensen wrote:
> > On Wed, Sep 28, 2005 at 10:51:57AM +0200, Lionel Elie Mamane wrote:
>
> >> Still with my Tyan s2895, I have two CPUs and 4GB, made of 8*512MB
> >> DIMM modules, of RAM. The motherboard's documentation clearly shows
> >> that four memory slots (two banks) are attached to each CPU. See the
> >> block diagram on page 2 of
> >> ftp://ftp.tyan.com/datasheets/d_s2895_100.pdf .
>
> >> What I find strange is the NUMA setup Linux reports:
>
> >> Scanning NUMA topology in Northbridge 24
> >> Number of nodes 2
> >> Node 0 MemBase 0000000000000000 Limit 000000007fffffff
> >> Node 1 MemBase 0000000080000000 Limit 000000015fffffff
>
> > So physically the memory appears to be mapped 2G at base 0 and 2G at
> > base 2G.
>
> Nah, more than that on node 1: from 2GiB to 5.5GiB. 15fffffff is
> 5.5Gi - 1.
Ah so it has memory starting at 2G (matches reg03 below) and all the way
to 5.5G (not contiguous however) which matches the 1.5G starting at 4G
(reg01 and reg02 below). So yes memory on node1 starts at 2G and
extends to 5.5G it just happens to have a hole from 2.5 to 4G in the
middle.
> >> node 1 shift 24 addr ff000000 conflict 0
> >> Using node hash shift of 25
>
> You know what these two lines mean?
No I don't.
> >> master@capsaicin:~$ cat /proc/mtrr
> >> reg00: base=0x00000000 ( 0MB), size=2048MB: write-back, count=1
> >> reg01: base=0x100000000 (4096MB), size=1024MB: write-back, count=1
> >> reg02: base=0x140000000 (5120MB), size= 512MB: write-back, count=1
> >> reg03: base=0x80000000 (2048MB), size= 512MB: write-back, count=1
>
> > So that looks same. 2G at base 0, another 512M at base 2G, 1G at base
> > 4G and another 512M at base 5G. So 2.5G from 0 to 2.5G, a hole of 1.5G
> > then another 1.5G starting at 4G.
>
> > Maybe at this point the kernel has remapped memory to better places
> > than the bios had.
>
> Can it do that? I thought only the bios could decide of these
> remappings.
I think the kernel can change any register it wants if it knows what
they are. The BIOS is just software with knowledge of the hardware
after all.
> > What does 'free' show you have for memory?
>
> 4029228 kiB = 3934MiB
That to me looks like all the 4G are accounted for. Might be some ram
reserved for the IOMMU and such (usually 64M if I remember correctly)
and the kernel uses a bit. Not sure. Looks like the total is slightly
short of 4096M. I wish I had even 1G of ram myself. :)
Len Sorensen
Reply to: