Re: multihomed server with ipv6

[oneman@onemanifest.net]

> On 13 dec. 2014, at 23:35, Pascal Hambourg <pascal@plouf.fr.eu.org> wrote:
> 
> Hello,
> 
> lists@onemanifest.net a écrit :
>> 
>> I've setup static adressing for both NICs for ipv4 and ipv6 like this:
>> 
>>  # The primary LAN network interface
>>  allow-hotplug eth0
>>  iface eth0 inet static
>>    address 192.168.1.xx
>>    netmask 255.255.255.0
>>    gateway 192.168.1.1
>> 
>>  iface eth0 inet6 static
>>    address 2001:xxxx:xxxx:3::20
>>    netmask 64
>>    gateway 2001:xxxx:xxxx:3::1
>> 
>>  # The secondary DMZ network interface
>>  allow-hotplug eth1
>>  iface eth1 inet static
>>    address 192.168.0.xx
>>    netmask 255.255.255.0
>>    gateway 192.168.0.1
>> 
>>  iface eth1 inet6 static
>>    address 2001:xxxx:xxxx:2::20
>>    netmask 64
>>    gateway 2001:xxxx:xxxx:2::1
>> 
>> The NICs are configured fine, but only _one_ ipv6 gets assigned. After a reboot either
>> eth0 or eth1 gets it's ipv6 assigned, never both.
> 
> I could reproduce it on a test machine. This looks like a tricky
> side-effect of multiple IPv4 default gateways.
> 
> Here is what happens :
> One interface is configured first. Since you used "allow-hotplug"
> insteald of "auto", the order may vary, depending on the discovery order
> by udev. When the second interface is configured for IPv4, ifup adds the
> address, tries to add the default route and gets an error because a
> default route already exists. Then it stops trying to configure the
> interface, including the IPv6 configuration.
> 
> Bottom line : don't define a gateway on multiple interfaces unless you
> are sure that only one interface is active at a time. It usually won't
> work the way you expect. "Default" means no other choice. If there are
> two active gateways then you still have to make a choice. Thus defining
> two active default gateways makes no sense.

I get it now, two defaults is a bit silly yes :-) You explanation is very clear. 

- I’ll use ‘auto’ to configure the interfaces, hot plug is not needed for this server.
- Since this server will be used for running openvz containers, I’ll probably have to create explicit routes for each (virtual) network to the subnets the NICs are connected to and define an explicit gateway for each subnet?

However, while doing some more research on this matter, it occurred to me that I can make life a lot easier on myself by using bridges. (Sorry if I sound very n00b-isch here) I think I could do the following:
- configure only the LAN NIC on the host, so I get admin access to the host from the LAN
- create a bridge on each NIC
- connect the containers to the appropriate bridge for either LAN or DMZ, setting up networking on each container independently.

This way the host is neatly isolated from the DMZ and the containers and the setup is far simpler. 

I’ll just dive into this and report back when I’m stuck. 

> 
>> Also, I noticed that if I ping6 a host is subnet A from a NIC in
>> subnet B, the servers ip6 in subnet A is used as source ip, even if an
>> ipv6 in subnet B is available on that NIC. Does this mean that the
>> return traffic is routed over subnet A in stead of B? This is not
>> desirable since I'd like the traffic between both subnets to be
>> fire-walled by the router.
> 
> I tried hard and failed to understand what you mean. Please describe
> more precisely what are the host, NIC, subnets, servers, router you're
> talking about, how they relate to the above, the commands you run and
> the results you observe.

My network is like this:
                                                other
                                                hosts
                                                 ||
            |NIC 2001:xxxx:xxxx:3::1/64| == LAN subnet == |NIC eth0 2001:xxxx:xxxx:3::20/64|
internet == |          router          |                  |            Server              |
            |NIC 2001:xxxx:xxxx:2::1/64| == DMZ subnet == |NIC eth1 2001:xxxx:xxxx:2::20/64|
                                                 ||
                                                other
                                                hosts

Configured with my half baked setup as described above, including adding the missing ipv6 due to the failure of ifup.

Then I did a ping6 to a host in LAN subnet, from the DMZ NIC on the server. I saw that the ping command reported using the LAN ip6 (2001:xxxx:xxxx:3::20/64) as a source ip, instead of the DMZ one (2001:xxxx:xxxx:2::20/64) as I expected it to do.

Hope this makes it clear.

thanks