[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: ifupdown-bonding package

Loïc Minier wrote:

On Tue, Oct 18, 2005, Jerome Martin wrote:
Well, I am a bit surprised by the way you see it. Could you be more specific ?

Of course: what I meant is that the issues you might have encountered
require fixing, and not documentation.  I can't tell how much of what
you describe is currently supported by the current ifenslave and vlan
packages, but it's where things should be implemented or fixed.

I know that I'm currently using things such as:

auto bond0
iface bond0 inet manual
   up insmod bonding -o bond0 mode=0 miimon=100
   up ifconfig bond0 hw ether 00:04:23:B1:C7:24
   up ifconfig $IFACE up
   down ifconfig $IFACE down
   down rmmod bond0

or even:

auto bond1
iface bond1 inet manual
   pre-up ifconfig eth1 up
   pre-up ifconfig eth3 up
   up insmod bonding -o $IFACE mode=0 miimon=100
   up ifconfig $IFACE hw ether 00:04:23:B1:C7:25
   up ifconfig $IFACE up
   up ifenslave -X $IFACE eth1 eth3
   up ifconfig $IFACE
   up ifconfig eth1
   up ifconfig eth3
   up echo success
   down ifconfig $IFACE
   down ifenslave -d $IFACE eth1 eth3
   down ifconfig $IFACE down
   down rmmod $IFACE
   post-down ifconfig eth3 down
   post-down ifconfig eth1 down

on a server running sarge, but it seems that the current ifenslave-2.6
packages already offer to switch to:

iface bond0 inet static
       slaves eth0 eth1

since this change:

ifenslave-2.6 (1.1.0-5) unstable; urgency=low

 * Add if-pre-up and if-post-down scripts to enslave and detach interfaces.
   Closes: #306993.

-- Guus Sliepen <guus@debian.org>  Fri, 30 Sep 2005 14:33:46 +0200
Well, then my code is useless, it does the same thing ! :-)
I'll check the new ifenslave package to see if there's anything to add to it, but given what you say, it does pretty much the same thing I just implemented :-) In any case, I'll send my eventual patches to the maintainer.

- provide a way to simplify the process by setting defaults enabling fast testing/setting up. Of course I realise that these defaults cannot be universal, that is the reason why I suggested the possibility to override them in network/options. You tell that this file is deprecated, so I should find some other way to do it, but the spirit remains the same : centralise some defaults, render the interface definition less verbose, lighter.

I suggest using /etc/default/<your-package>.
OK, I'll do that.

- allow one to configure easily the bonding module from inside the interface definition

That's good, and is not covered by the current ifenslave-2.6 scripts.
I suggest you file a request against ifenslave-2.6 with your proposalto
address this.
Will do.

I feel that those 3 goals are fixes to problems I have encountered myself.
As for my future goals (to be implemented, but I'm working on it and have something soon) : - Add logic to check VLANs are up only AFTER the main bond interface is up and attached to at least one slave (in order to ensure no mac address issues will occur) - Add logic to check that slave interfaces do not have IPs configured nor routes (in order to prevent conflicts) befor the bond is started - Add logic to check there is no IP attached to an underlying bond interface before bringing up VLANs.

That's useful indeed.  That would be another round of fixes for the
bonding if-up scripts.
Mmmhh ... are there already any bonding if-up scripts ? If so, I don't know in which package ...

- Add logic to configure arp monitoring using the default gw / resolvconf hooks / whaterver makes sense (suggestions welcome)

I don't see what you mean, but it certainly makes sense to add such a
support in the ifupdown package so that all interface types can benefit
from the change.
When not using miimon, you can configure bonding to send arp requests to specified IPs in order to check link connectivity. One of the problem I have with that mode is that whenever you change the IP of those reference machines or those individual hosts go down, your link is declared as down. If the ifup scripts for bonding could automatically select reasonable IPs to send ARPs to (after checking ARP requests are honored), it could allow sysadmins to transparently use arp monitoring without worrying too much about whether the bonding configuration is up-to-date. Good candidates for such reference hosts could be the declared default gateway for the interface and DNS servers (specified in network/interfaces through resolvconf package hooks). This is very specific to bonding, so I can't see how it can be usefull to other inteface type.

- Add logic to detect which modules are used by the slave interfaces and eventually unload themfirst and reload them after the bonding module (so that SNMP if queries will report the bond as the "owner" of its IP address). I realise that this impacts heavily the logic dictated by configuring modules in modprobe.conf / modprobe.d, maybe there is something that could be done with an external tool handling this (and storing configuration via debconf ?) ? What would be the "debian way" ?

I don't understand the above, but I'm sure you'll find your way in
Debian.  :)
A common problem with bonding + SNMP is that when you get the interface list of an SNMP-enabled host, the SNMP agent only returns the first interface to which a specific IP/mac address is attached (and this is normal behavior, not a bug). Say you have eth1 and eth2 slaves of bond0. If eth1 is brought up before bond0, its physicall interface definition will appear first in the SNMP interface table, thus it will be used instead of bond0 as "primary" interface for the corresponding IP address. This is wrong because it works only when eth1 is primary in a primary-backup scheme, but reports wrong stats in any other case (stats like traffic, dropped packets, etc.).

The only way to make sure the SNMP agent will report the correct interfaces indexes is to load the bonding module first, then load each slave driver (of course this works only with modules, not with monolithically compiled-in drivers). So what I was suggesting is :
- given a bond0 interface definition :
. get the modules driving the slaves interfaces
. unload those modules if already loaded
. load the bond0 instance of bonding
. load the slave modules
. attach the slaves
. finish configuration ...

Of course, this requires much work. Plus, you must be able to unload/load modules on a per-interface basis. This is possible with modprobe -o, but not the standard case. Have you noticed that is you use an alias in modprobe.conf, like alias bon0 bonding, then the resulting instance of the module is still called "bonding"when doing an lsmod ? Instead, when you use modprobe -o bond0, you have separate instances which you can load/unload separately. I assume this could be possible with most eth drivers, but would require some tweaking in the drivers code and the way modules are defined in modprobe.conf, what I called "impacts heavily the logic dictated by configuring modules in modprobe.conf".

Is it clearer now ?

All that to say that my goal is to provide code to solve issues, not only document them. However, at some point, I hacked down the README as much as a reminder of items to investigate than documentation for people willing to use the embryo of features already coded.

Ok, so it's more of a TODO.  :)
Yes :-)
But in that case, my TODO could be some other people's HOWTO ;-P

However, I'm a lost, because in order to acheive all those "automatic handling of all VLANs+bonding issues" it seems that a lot more changes would be required to the network modules handling + ifupdown architecture than just a few hooks.

I don't see why, the current approach of if-up scripts seems ok to me.
Well, for one, there is a major flaw when reconfiguring interfaces : in order for the if state to be properly handled, you are supposed to ifdown the interface before any change to the config. Otherwise, you loose the ability to properly ifdown it at a later time.

Also, I have not seen any mechanism provided so that the code handling one interface can be aware of the other ones (apart from parsing network/interfaces by hand, but this is redundant and should not be done in a hook script !). This prevent things like detecting if VLANs are defined (but not up yet) for the current bond interface, in which case setting an IP address to the interface should issue a warning/error/workaround .

See the docs of the ifupdown-scripts-zg2 package for other examples of limitations.

While I am at it, my scripts currently require bash (shopts extglod and such) to work. Is that something acceptable in network scripts ?

Sure, but you need to use #!/bin/bash in the shebang.
Good, I was not sure that bash was in base and allowed for such scripts by policy. #!/bin/sh is bash anyway on a default installation :-).

That file is deprecated in recent netbase packages.
Where can I find documentation about the reason why, the new policy, and the direction behind it ?

I don't know, I suppose you can mail the maintainer.  Probably, the
configured options had nothing to do in there and /etc/sysctl.conf
could be used instead.
I'll do some research. I don't like it because it expands scattering information around the place about the same semantic unit (a network interface and the way it behaves).

I agree. But what about a more complete bonding+vlan handling (see above) ? Would that still belong to individual packages providing the underlying feature (ifenslave) ? Or should scattering it should be avoided in order to maintain a coherent whole, more easily maintainable ?

Well, I don't see anything specific about doing vlan over bonding.
It's normal vlan usage, normal bonding usage.
Well, in most cases it is. But there are a few exceptions. Let me elaborate (most of this comes from Bonding.txt.gz in linux-doc-2.6) :

- order is critical. As the bond interface sets its own mac address from the first slave to be attached, this can impact VLAN. If you fail to ensure that at least one slave has been attached before adding a VLAN, then the VLAN interface will pick up a 00:00:00:00:00:00 address !

- when sending "internal" traffic (ARP mostly), the bond interface must be aware of the VLANs on top of it. This takes some discovery routine to learn the VLANs as they are added. This is documented and works. But looking at the code, I am not convinced that it would handle properly interface-local v.s. host-global VLAN type. I should try, but this should be tested and included in foolproof ifup/down scripts. Although many things in this area have been improved recently (look for shmulik.hen@intel.com in kernel 2.6.4 changelog), I am pretty convinced that doing VLAN over bonding requires special precaution and awareness, because it relies on a large amount of specific code and workarounds, which could not possibly cover ALL the cases, given it is not written as a general support but merely with per-issue fixes.

- If you detach all slaves from a bond which has VLAN on top of it, and start adding new slaves, the VLAN will be non-functionnal because of the very same hw address probleme I mentionned earlier.

- Changing manually the VLAN interface MAC address is not an option to handle this, because it would put the bond interface into promiscuous mode, with the effect of propagating it to the active(s) slave (s), which is probably not a desirable side-effect.

I hope this clears up why I think specific precautions should be taken when playing with VLANs + bonding. Of course, 99% of the time it will work fine with you being unaware of those issues. But the day you'll encounter one of those issues in a production environment ... well ... good luck if youŕe not aware of all the specificities OR have good ifup/ifdown mechanism handling all those subtilities for you, mainly guaranteeing modules and interfaces/VLAN starting order.

Well, I have always felt that the debian way was not to change the semantics of the upstream software. And automagic interface naming IS part of the vconfig semantics, see the set_name_type options. However, in this particular case, I share part of the same feeling : I would prefer not have the name of the interface carry any semantic at all. This can be misleading. The reason I mimicked this behavior was mostly to satisfy my crave for consistency, added to the fact that the vlan package being already into debian, I took for granted that this is the right way to do it :-)

One way to keep the autogeneration of interface names is to use
something in the lines of:

iface eth0 inet manual
    vlans 16 32 48
    vlan_name_type vlan_plus_vid_no_pad

iface eth0.16 inet static
    address x.x.x.x

The current way of detecting that eth0.16 means setup eth0 for VLANs
and add it to VLAN 16 is ugly IMO.
I agree this is ugly, but I disagree with your suggestion, witch is even more error-prone IMHO. The problem in your example is redundancy of information, without automatic cross-checking. I do prefer the vlan_raw_device stanza style, where all VLAN definition is grouped into the VLAN interface (a la RedHat) :

iface eth0 inet manual

iface eth0.16 inet static
vlan_raw_device eth0
vlan_id 16
# this could be host / interface
bind-type PER_KERNEL
address x.x.x.x

One simple reason I prefer the latest is because a VLAN may have multiple specific options besides it VLAN number, like header reordering and vlan-qos tagging. A problem, here, however, is that ignoring the rigid naming structure enforced by vconfig can lead to problems if the interface name if not checked to be proper and compatible with one of the vconfig recognized options. I guess this is the major reason why the vlan package maintainer chose to detect VLAN parameters from the name. Also, even if not in the debian manpage (as of sarge) yet, the often forgotten bind-type can be crucial if you need to use the same VLAN IDs on different interfaces, corresponding to totally different VLANs. All those options would need to be indexed in the raw interface stanza, which is heavy.

I think as you do, you made some perfectly valid remarks that I can
only ack.
By removing the lines before your answer, I could trick any new reader that you totally agree with all I ever said :-) But that would make me a politician .... More seriously, the fact that you agree with what I meant by robustness and simplicity makes me really happy :-) Because this is the core point of the whole discussion ... and it is the direction I'm desperatly trying to help with.

Can anyone share pointers with me as of what is the "debian way" ? Or at least what are the conflicting views on those matters ?

When I said the "Debian way" on the topic of networking I was mostly
referring to the ifupdown hooks.
Well, of course, this was intended to be sarcastic at all. I was merely asking what is the "debian way" of handling netwoking config files, or maybe some background on the principles behind the current structure, its goals, etc. It is often easy to break something you do not understand, and it takes a looog time re-discovering what lead to the initial design. I'm just trying to understand how things are, and WHY they are this way, so I can avoid misjudging what I might dislike at first sight. I guess I'll have to read the relevant packages changelogs, but I was hoping for some ML threads pointers, something like that.

In general, I agree with what you said.  Since all you offered to
comment on were ideas and documentation, I commented on that, and I
And I thank you again for that.

must say I'm now a bit confused on the actual things you're trying to
fix or change.  My feeling is that you want some broad changes to
happen, and Debian allows mainly small fixes on a per-package basis.
Well, I might get frustrated if everything is not as I dreamt of, but that won't prevent me from helping doing small steps. I am perfectly aware that huge changes often lead to broken things. And I don't want that to happen in Debian :-) My speech might be a little bit confusing, it's only that I easily get overexcited with those subjects ...

  I suggest you take a look at the ifenslave-2.6 package in unstable,
and see how its current if-up hooks can be enhanced, and also file a
request to include your vlan handling scripts against the vlan package.
I'll do that. The most reasonable thing to do is indeeed trying to contribute to the vlan and ifenslave packages, I guess.

  Thanks for improving Debian,
Thanks for keeping this ML alive :-) This is crucial for newcomers like me.

P.S.: Ca mérite une bière, quand tu veut :-)

Reply to: