Re: ifupdown-bonding package
Loïc Minier wrote:
On Tue, Oct 18, 2005, Jerome Martin wrote:
Well, I am a bit surprised by the way you see it. Could you be more
specific ?
Of course: what I meant is that the issues you might have encountered
require fixing, and not documentation. I can't tell how much of what
you describe is currently supported by the current ifenslave and vlan
packages, but it's where things should be implemented or fixed.
I know that I'm currently using things such as:
auto bond0
iface bond0 inet manual
up insmod bonding -o bond0 mode=0 miimon=100
up ifconfig bond0 hw ether 00:04:23:B1:C7:24
up ifconfig $IFACE up
down ifconfig $IFACE down
down rmmod bond0
or even:
auto bond1
iface bond1 inet manual
pre-up ifconfig eth1 up
pre-up ifconfig eth3 up
up insmod bonding -o $IFACE mode=0 miimon=100
up ifconfig $IFACE hw ether 00:04:23:B1:C7:25
up ifconfig $IFACE 127.0.0.2 up
up ifenslave -X $IFACE eth1 eth3
up ifconfig $IFACE 0.0.0.0
up ifconfig eth1 0.0.0.0
up ifconfig eth3 0.0.0.0
up echo success
down ifconfig $IFACE 127.0.0.2
down ifenslave -d $IFACE eth1 eth3
down ifconfig $IFACE down
down rmmod $IFACE
post-down ifconfig eth3 down
post-down ifconfig eth1 down
on a server running sarge, but it seems that the current ifenslave-2.6
packages already offer to switch to:
iface bond0 inet static
address 192.168.48.157
netmask 255.255.252.0
broadcast 192.168.48.255
gateway 192.168.48.1
slaves eth0 eth1
since this change:
ifenslave-2.6 (1.1.0-5) unstable; urgency=low
* Add if-pre-up and if-post-down scripts to enslave and detach interfaces.
Closes: #306993.
-- Guus Sliepen <guus@debian.org> Fri, 30 Sep 2005 14:33:46 +0200
Well, then my code is useless, it does the same thing ! :-)
I'll check the new ifenslave package to see if there's anything to add
to it, but given what you say, it does pretty much the same thing I just
implemented :-) In any case, I'll send my eventual patches to the
maintainer.
- provide a way to simplify the process by setting defaults enabling
fast testing/setting up. Of course I realise that these defaults cannot
be universal, that is the reason why I suggested the possibility to
override them in network/options. You tell that this file is deprecated,
so I should find some other way to do it, but the spirit remains the
same : centralise some defaults, render the interface definition less
verbose, lighter.
I suggest using /etc/default/<your-package>.
OK, I'll do that.
- allow one to configure easily the bonding module from inside the
interface definition
That's good, and is not covered by the current ifenslave-2.6 scripts.
I suggest you file a request against ifenslave-2.6 with your proposalto
address this.
Will do.
I feel that those 3 goals are fixes to problems I have encountered myself.
As for my future goals (to be implemented, but I'm working on it and
have something soon) :
- Add logic to check VLANs are up only AFTER the main bond interface is
up and attached to at least one slave (in order to ensure no mac address
issues will occur)
- Add logic to check that slave interfaces do not have IPs configured
nor routes (in order to prevent conflicts) befor the bond is started
- Add logic to check there is no IP attached to an underlying bond
interface before bringing up VLANs.
That's useful indeed. That would be another round of fixes for the
bonding if-up scripts.
Mmmhh ... are there already any bonding if-up scripts ? If so, I don't
know in which package ...
- Add logic to configure arp monitoring using the default gw /
resolvconf hooks / whaterver makes sense (suggestions welcome)
I don't see what you mean, but it certainly makes sense to add such a
support in the ifupdown package so that all interface types can benefit
from the change.
When not using miimon, you can configure bonding to send arp requests to
specified IPs in order to check link connectivity. One of the problem I
have with that mode is that whenever you change the IP of those
reference machines or those individual hosts go down, your link is
declared as down. If the ifup scripts for bonding could automatically
select reasonable IPs to send ARPs to (after checking ARP requests are
honored), it could allow sysadmins to transparently use arp monitoring
without worrying too much about whether the bonding configuration is
up-to-date. Good candidates for such reference hosts could be the
declared default gateway for the interface and DNS servers (specified in
network/interfaces through resolvconf package hooks). This is very
specific to bonding, so I can't see how it can be usefull to other
inteface type.
- Add logic to detect which modules are used by the slave interfaces and
eventually unload themfirst and reload them after the bonding module (so
that SNMP if queries will report the bond as the "owner" of its IP
address). I realise that this impacts heavily the logic dictated by
configuring modules in modprobe.conf / modprobe.d, maybe there is
something that could be done with an external tool handling this (and
storing configuration via debconf ?) ? What would be the "debian way" ?
I don't understand the above, but I'm sure you'll find your way in
Debian. :)
A common problem with bonding + SNMP is that when you get the interface
list of an SNMP-enabled host, the SNMP agent only returns the first
interface to which a specific IP/mac address is attached (and this is
normal behavior, not a bug). Say you have eth1 and eth2 slaves of bond0.
If eth1 is brought up before bond0, its physicall interface definition
will appear first in the SNMP interface table, thus it will be used
instead of bond0 as "primary" interface for the corresponding IP
address. This is wrong because it works only when eth1 is primary in a
primary-backup scheme, but reports wrong stats in any other case (stats
like traffic, dropped packets, etc.).
The only way to make sure the SNMP agent will report the correct
interfaces indexes is to load the bonding module first, then load each
slave driver (of course this works only with modules, not with
monolithically compiled-in drivers). So what I was suggesting is :
- given a bond0 interface definition :
. get the modules driving the slaves interfaces
. unload those modules if already loaded
. load the bond0 instance of bonding
. load the slave modules
. attach the slaves
. finish configuration ...
Of course, this requires much work. Plus, you must be able to
unload/load modules on a per-interface basis. This is possible with
modprobe -o, but not the standard case. Have you noticed that is you use
an alias in modprobe.conf, like alias bon0 bonding, then the resulting
instance of the module is still called "bonding"when doing an lsmod ?
Instead, when you use modprobe -o bond0, you have separate instances
which you can load/unload separately. I assume this could be possible
with most eth drivers, but would require some tweaking in the drivers
code and the way modules are defined in modprobe.conf, what I called
"impacts heavily the logic dictated by configuring modules in
modprobe.conf".
Is it clearer now ?
All that to say that my goal is to provide code to solve issues, not
only document them. However, at some point, I hacked down the README as
much as a reminder of items to investigate than documentation for people
willing to use the embryo of features already coded.
Ok, so it's more of a TODO. :)
Yes :-)
But in that case, my TODO could be some other people's HOWTO ;-P
However, I'm a lost, because in order to acheive all those "automatic
handling of all VLANs+bonding issues" it seems that a lot more changes
would be required to the network modules handling + ifupdown
architecture than just a few hooks.
I don't see why, the current approach of if-up scripts seems ok to me.
Well, for one, there is a major flaw when reconfiguring interfaces : in
order for the if state to be properly handled, you are supposed to
ifdown the interface before any change to the config. Otherwise, you
loose the ability to properly ifdown it at a later time.
Also, I have not seen any mechanism provided so that the code handling
one interface can be aware of the other ones (apart from parsing
network/interfaces by hand, but this is redundant and should not be done
in a hook script !). This prevent things like detecting if VLANs are
defined (but not up yet) for the current bond interface, in which case
setting an IP address to the interface should issue a
warning/error/workaround .
See the docs of the ifupdown-scripts-zg2 package for other examples of
limitations.
While I am at it, my scripts currently require bash (shopts extglod and
such) to work. Is that something acceptable in network scripts ?
Sure, but you need to use #!/bin/bash in the shebang.
Good, I was not sure that bash was in base and allowed for such scripts
by policy. #!/bin/sh is bash anyway on a default installation :-).
That file is deprecated in recent netbase packages.
Where can I find documentation about the reason why, the new policy, and
the direction behind it ?
I don't know, I suppose you can mail the maintainer. Probably, the
configured options had nothing to do in there and /etc/sysctl.conf
could be used instead.
I'll do some research. I don't like it because it expands scattering
information around the place about the same semantic unit (a network
interface and the way it behaves).
I agree. But what about a more complete bonding+vlan handling (see
above) ? Would that still belong to individual packages providing the
underlying feature (ifenslave) ? Or should scattering it should be
avoided in order to maintain a coherent whole, more easily maintainable ?
Well, I don't see anything specific about doing vlan over bonding.
It's normal vlan usage, normal bonding usage.
Well, in most cases it is. But there are a few exceptions. Let me
elaborate (most of this comes from Bonding.txt.gz in linux-doc-2.6) :
- order is critical. As the bond interface sets its own mac address from
the first slave to be attached, this can impact VLAN. If you fail to
ensure that at least one slave has been attached before adding a VLAN,
then the VLAN interface will pick up a 00:00:00:00:00:00 address !
- when sending "internal" traffic (ARP mostly), the bond interface must
be aware of the VLANs on top of it. This takes some discovery routine to
learn the VLANs as they are added. This is documented and works. But
looking at the code, I am not convinced that it would handle properly
interface-local v.s. host-global VLAN type. I should try, but this
should be tested and included in foolproof ifup/down scripts. Although
many things in this area have been improved recently (look for
shmulik.hen@intel.com in kernel 2.6.4 changelog), I am pretty convinced
that doing VLAN over bonding requires special precaution and awareness,
because it relies on a large amount of specific code and workarounds,
which could not possibly cover ALL the cases, given it is not written as
a general support but merely with per-issue fixes.
- If you detach all slaves from a bond which has VLAN on top of it, and
start adding new slaves, the VLAN will be non-functionnal because of the
very same hw address probleme I mentionned earlier.
- Changing manually the VLAN interface MAC address is not an option to
handle this, because it would put the bond interface into promiscuous
mode, with the effect of propagating it to the active(s) slave (s),
which is probably not a desirable side-effect.
I hope this clears up why I think specific precautions should be taken
when playing with VLANs + bonding.
Of course, 99% of the time it will work fine with you being unaware of
those issues. But the day you'll encounter one of those issues in a
production environment ... well ... good luck if youŕe not aware of all
the specificities OR have good ifup/ifdown mechanism handling all those
subtilities for you, mainly guaranteeing modules and interfaces/VLAN
starting order.
Well, I have always felt that the debian way was not to change the
semantics of the upstream software. And automagic interface naming IS
part of the vconfig semantics, see the set_name_type options. However,
in this particular case, I share part of the same feeling : I would
prefer not have the name of the interface carry any semantic at all.
This can be misleading. The reason I mimicked this behavior was mostly
to satisfy my crave for consistency, added to the fact that the vlan
package being already into debian, I took for granted that this is the
right way to do it :-)
One way to keep the autogeneration of interface names is to use
something in the lines of:
iface eth0 inet manual
vlans 16 32 48
vlan_name_type vlan_plus_vid_no_pad
iface eth0.16 inet static
address x.x.x.x
...
The current way of detecting that eth0.16 means setup eth0 for VLANs
and add it to VLAN 16 is ugly IMO.
I agree this is ugly, but I disagree with your suggestion, witch is even
more error-prone IMHO. The problem in your example is redundancy of
information, without automatic cross-checking. I do prefer the
vlan_raw_device stanza style, where all VLAN definition is grouped into
the VLAN interface (a la RedHat) :
iface eth0 inet manual
iface eth0.16 inet static
vlan_raw_device eth0
vlan_id 16
# this could be host / interface
bind-type PER_KERNEL
address x.x.x.x
One simple reason I prefer the latest is because a VLAN may have
multiple specific options besides it VLAN number, like header reordering
and vlan-qos tagging. A problem, here, however, is that ignoring the
rigid naming structure enforced by vconfig can lead to problems if the
interface name if not checked to be proper and compatible with one of
the vconfig recognized options. I guess this is the major reason why the
vlan package maintainer chose to detect VLAN parameters from the name.
Also, even if not in the debian manpage (as of sarge) yet, the often
forgotten bind-type can be crucial if you need to use the same VLAN IDs
on different interfaces, corresponding to totally different VLANs. All
those options would need to be indexed in the raw interface stanza,
which is heavy.
I think as you do, you made some perfectly valid remarks that I can
only ack.
By removing the lines before your answer, I could trick any new reader
that you totally agree with all I ever said :-) But that would make me a
politician .... More seriously, the fact that you agree with what I
meant by robustness and simplicity makes me really happy :-) Because
this is the core point of the whole discussion ... and it is the
direction I'm desperatly trying to help with.
Can anyone share pointers with me as of what is the "debian way" ? Or at
least what are the conflicting views on those matters ?
When I said the "Debian way" on the topic of networking I was mostly
referring to the ifupdown hooks.
Well, of course, this was intended to be sarcastic at all. I was merely
asking what is the "debian way" of handling netwoking config files, or
maybe some background on the principles behind the current structure,
its goals, etc. It is often easy to break something you do not
understand, and it takes a looog time re-discovering what lead to the
initial design. I'm just trying to understand how things are, and WHY
they are this way, so I can avoid misjudging what I might dislike at
first sight. I guess I'll have to read the relevant packages changelogs,
but I was hoping for some ML threads pointers, something like that.
In general, I agree with what you said. Since all you offered to
comment on were ideas and documentation, I commented on that, and I
And I thank you again for that.
must say I'm now a bit confused on the actual things you're trying to
fix or change. My feeling is that you want some broad changes to
happen, and Debian allows mainly small fixes on a per-package basis.
Well, I might get frustrated if everything is not as I dreamt of, but
that won't prevent me from helping doing small steps. I am perfectly
aware that huge changes often lead to broken things. And I don't want
that to happen in Debian :-) My speech might be a little bit confusing,
it's only that I easily get overexcited with those subjects ...
I suggest you take a look at the ifenslave-2.6 package in unstable,
and see how its current if-up hooks can be enhanced, and also file a
request to include your vlan handling scripts against the vlan package.
I'll do that. The most reasonable thing to do is indeeed trying to
contribute to the vlan and ifenslave packages, I guess.
Thanks for improving Debian,
Thanks for keeping this ML alive :-) This is crucial for newcomers like me.
P.S.: Ca mérite une bière, quand tu veut :-)
Reply to: