Re: 2 nics, 1 network, puzzle?

[Jason Healy <jhealy@logn.net>]

At 1017265404s since epoch (03/27/02 17:43:24 -0500 UTC), Shawn Yarbrough wrote:
> Or is an ARP reply actually an IP packet?  Maybe it is, but I don't think
> so, I'm assuming it's an ethernet packet.

This e-mail is turning out to be *very* long.  Here is the short
answer:

- ARP is an IP packet, not an ethernet frame.
- Your problem (as I've alluded in previous e-mails) is that you have
  two interfaces to the same network.  More to follow.
- The only way to use both cards at once is to hook them up to
  different networks (by different, I mean different media and
  different subnets).

Shawn, I see that in the time it's taken me to write this, you've
written another e-mail.  This e-mail lays the groundwork for
understanding what I'll write in my next e-mail.  <g>

===============
The LONG answer [Jason dusts off his networking textbook]:
(if you understand Ethernet frames and IP packets, skip to the section
below titled 'The Snorkel Analogy')

Ethernet frames look something like this (roughly...):

X | Dest | Source | Data Length | Data ... [padding] | Error Check

The "dest" and "source" are 48-bit MAC addresses, read off as "HWaddr"
in ifconfig: '00:12:34:56:78:9a' would be an example.

Pure ethernet only uses these addresses; the data in the data field
can be whatever you want: IP, Appletalk, Netware, etc.  It serves as
the Data Link Layer in the OSI networking model.  It has _no idea_
about IP data, including IP addresses and hostnames.

Now, let's move on to IP packets (again, edited for content):

Yada yada yada | Source IP | Dest IP | options | padding | data

The Source and Dest addresses are the good 'ol dotted quads
(12.34.56.78) that we know and love.  Note that IP packets have no
idea how they get from one place to another; they just say where
they're from and where they're going.

So, Ethernet has no idea about IP, and IP has no idea about
Ethernet... what to do???

That's the kernel's job.  The kernel generates IP packets, and then is
responsible for sending them out over the correct network interface.
For example, when the kernel wants to send an IP packet to a computer
on the same network, it looks up the Ethernet address of the local
computer in its ARP cache, and then constructs a frame with the
correct destination MAC address.

Now, how does the computer know the MAC addresses of 'local'
computers?  By using ARP.  ARP is an IP packet, not an Ethernet
frame.  The purpose of ARP is to link IP addresses and MAC addresses
together.  Here is what an ARP query and reply would look like:

arp who-has mysteryhost tell thequestioner (ca:fe:ba:be:fa:ce)
arp reply mysteryhost is-at de:ad:be:ef:f0:0d

Note that the ARP asks the whole network for a specific host, and the
reply lets everyone know the MAC address bound to the hostname.

Once a computer gets the reply to an ARP query, it caches it for
later.  Note that ARPs are only for local networks; you don't ARP for
address that are not on your immediate network segment.

How does the kernel decide if a computer is local (on its ethernet
segment) by looking at its IP address?  By using 'route'.  The kernel
applies the netmask for each possible route to the packet's
destination, and if it matches that network, then it sends it out in
an ethernet frame.  So, a packet with destination 192.168.1.XXX will
match a network interface with address 192.168.1.YYY and netmask
255.255.255.0, and so will be sent out on that interface.

If you have two interfaces with the same network address
(192.168.1.ZZZ/255.255.255.0), the kernel will simply pick one of the
two when it sends out packets destined for that network.  The kernel
has no notion of "fairness" when it comes to forwarding to the same
network address; it just picks the first one it finds (this is mainly
for performance). 

Okay, with that review out of the way, back to our regularly scheduled
discussion:

===================
The Snorkel Analogy

> In your analogy, inhaling is where my problem lies.  In theory I should be
> able to hook up the left snorkle to a room containing cold air and the
> right snorkle to another room containing warm air, and that should
> guarantee that my left and right nostrils get cold and warm air,
> respectively.

YOU ARE EXACTLY CORRECT!!! You should be able to do this, and you can!
The problem is that with NICs, each one must be hooked up to a
different network, a la:

eth0 192.168.1.1/255.255.255.0
eth1 192.168.2.1/255.255.255.0

What you have is two NICs on the *same* network:

eth0 192.168.1.130/255.255.255.0
eth1 192.168.1.131/255.255.255.0

In the snorkel analogy, this is like having both your snorkels go
into the same tank of air; it doesn't matter which snorkel you use, as
you're going to get the same air either way.  Your lungs will get warm
air, whether you use the left snorkel, the right snorkel, or both.

If you want different kinds of air (IP addresses), you need to have
different tanks (networks).  That is the problem you're having.
You're trying to make a distinction where none exists (at least
according to the IP protocol).

> It's not happening though, because the kernel, involving what I believe
> are mangled ARP replies, is giving wrong instructions to whoever hooks up
> the snorkles, and both hot and cold air are entering the same nostril,
> while the other nostril is going unused for inhaling.

Again, this would be valid, except that you don't have different kinds
of air.  The kernel doesn't differentiate, because there's nothing to
differentiate between.  Yes, you have two snorkels, but the
distinction is purely academic in that it's the same air.  This is why
the kernel seems to be maliciously ignoring you; it doesn't see any
difference between the "choices" you've given it, so it just ignores
one of them.

> All I care about is that when ethernet address E1 is bound to IP address
> I1, everybody else on the network knows it.  The kernel is telling
> everybody else on the network that E0 = I1, which is wrong.

Only because you say it's wrong.  I'm not trying to be mean, I'm just
presenting the kernel's point of view.  The kernel has bound the IP
address to the MAC address on the first network interface it found.
That MAC will receive and process the desired IP packets, so this is
valid.  Sure, it's not the one YOU would have picked, but delivery of
IP packets will work as you want (in that the machine sends and
receives packets at both IP addresses), so the kernel thinks that's
good enough.

> [ But given that the ARP reply must (I assume) contain the address E1, it
> seems like it would be really easy for the kernel to figure out that the
> ARP packet should be transmitted over E1. ]

This is the cause of your problem.  You have two interfaces on the
same ethernet segment.  When the kernel responds to an ARP, this is an
IP packet.  The kernel uses 'route' to determine the interface to send
the packet out on.  Because both eth0 and eth1 are on the same network
on your machine, the kernel will always pick the first one it comes
across.  The ARP will therefore go out on that interface, regardless
of the IP address its answering for.

As a semi-final aside, it *is* possible to get the kernel to answer
requests on specific interfaces; you would just have to crank the
netmask up to 255.255.255.255, which would essentially turn both your
NICs into PPP connections.  When the kernel needed to route an IP
packet, the highly specific netmask would force it to pick the exact
NIC you wanted.

Unfortunately, since the rest of the network is assuming a netmask of
255.255.255.0, it's highly doubtful that this setup would actually
work.  So much for that idea.  But that's basically what you'd have to do.

> Why do I want this?  At this point it's mostly academic, I know I can get
> my system working using only one network card + IP aliasing.  However, I
> think this ARP problem is a glaring problem that should be easy to fix and
> will also have benefits for people doing load balancing.

Again, on a shared media network (such as ethernet), you're not
actually balancing the load at all.  Only one node on an ethernet
segment may broadcast at any one time, so there could never be a time
where both cards were active.  In fact, you're probably just slowing
the machine down with two cards in it, since you'll be receiving every
ethernet frame twice, and filling up your routing table with redundant
IP routing info.

Remeber, ethernet and IP are both very old protocols at this point;
back when they were designed nobody thought that you would run
multiple IP networks on a single physical network segment.  Thus, the
assumptions that IP makes about the underlying physical layer may seem
a little weird.

The general rule is: one IP subnet per physical network segment.  And,
by extension, only one interface per machine is necessary (allowed?)
per physical network segment. 

Whew...  if you stuck with all that, then I think you deserve
1/2 credit towards a networking course, or something!  =)

Jason

--
Jason Healy    |    jhealy@logn.net    |   http://www.logn.net/


-- 
To UNSUBSCRIBE, email to debian-user-request@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org