Re: Traffic control with traffic shaper
--->
Limiting Outgoing and Incoming Bandwidth in Linux (for 2.2+ Kernels)
Brian Wilson - wilson@sentrisystems.com - 12/5/2000
Overview
--------
The point of this document is to walk you through setting up bandwidth
limiting for outgoing and incoming network packets under Linux 2.2.x+
kernel series. The limiting of the outgoing bandwidth is accomplished
using the built-in QoS Class Based Queueing (CBQ) functions in the 2.2.x
kernel. The limiting of the incoming bandwidth is accomplished using a
stand-alone kernel module called rshaper. NOTE: rshaper also requires
slight modififications to your network driver source code.
Before You Begin
----------------
You first need to download a few utilities.
- iproute
ftp://ftp.funet.fi/pub/mirrors/ftp.inr.ac.ru/ip-routing/iproute2-current.tar.gz
(and if you wish to limit incoming traffic as well, you'll need rshaper)
- rshaper
ftp://ftp.prosa.it/pub/software/rshaper-1.07.tar.gz
Installation
------------
You'll first need to configure your kernel for QoS and CBQ. This is
accomplished under the "Networking Options" menu. Here you need to enable
Routing Messages and the Kernel/User netlink socket. Next, under "QoS
and/or Fair Queueing" menu, you need to enable all of the options. Below
is the Networking Options portion of my kernel config. You should probably
mimic this setup (unless you want to build them as modules). If you do
choose to build them as modules, then you'll have to load all the specifc
QoS modules (sch_cbq sch_tbf sch_sfq sch_prio cls_u32) after your machine
boots (or allow them to be autoloaded). Since I don't like dealing with
a slew of modules, I usually try to compile everything in staticly.
<*> Packet socket
[*] Kernel/User netlink socket
[*] Routing messages
<*> Netlink device emulation
[*] Network firewalls
[*] Socket Filtering
<*> Unix domain sockets
[*] TCP/IP networking
[*] IP: multicasting
[*] IP: advanced router
[*] IP: policy routing
[*] IP: equal cost multipath
[*] IP: use TOS value as routing key
[ ] IP: verbose route monitoring
[ ] IP: large routing tables
[*] IP: fast network address translation
[ ] IP: kernel level autoconfiguration
[*] IP: firewalling
[ ] IP: firewall packet netlink device
[*] IP: use FWMARK value as routing key
[*] IP: masquerading
--- Protocol-specific masquerading support will be built as modules.
[*] IP: ICMP masquerading
--- Protocol-specific masquerading support will be built as modules.
[*] IP: masquerading special modules support
<*> IP: ipautofw masq support (EXPERIMENTAL)
<*> IP: ipportfw masq support (EXPERIMENTAL)
<*> IP: ip fwmark masq-forwarding support (EXPERIMENTAL)
[ ] IP: optimize as router not host
< > IP: tunneling
[*] IP: aliasing support
[ ] IP: ARP daemon support (EXPERIMENTAL)
[*] IP: TCP syncookie support (not enabled per default)
--- (it is safe to leave these untouched)
<*> IP: Reverse ARP
[*] IP: Allow large windows (not recommended if <16Mb of memory)
< > The IPv6 protocol (EXPERIMENTAL)
---
< > The IPX protocol
< > Appletalk DDP
< > CCITT X.25 Packet Layer (EXPERIMENTAL)
< > LAPB Data Link Driver (EXPERIMENTAL)
[ ] Bridging (EXPERIMENTAL)
[ ] 802.2 LLC (EXPERIMENTAL)
< > Acorn Econet/AUN protocols (EXPERIMENTAL)
< > WAN router
[*] CPU is too slow to handle full bandwidth
QoS and/or fair queueing --->
[*] QoS and/or fair queueing
<*> CBQ packet scheduler
<*> CSZ packet scheduler
<*> The simplest PRIO pseudoscheduler
<*> RED queue
<*> SFQ queue
<*> TEQL queue
<*> TBF queue
[*] QoS support
[*] Rate estimator
[*] Packet classifier API
<*> Routing table based classifier
<*> Firewall based classifier
<*> U32 classifier
<*> Special RSVP classifier
< > Special RSVP classifier for IPv6
[*] Ingres traffic policing
While you are in the kernel config, if you have decided to use rshaper and
limit incoming traffic, then go ahead and modularize your network device
driver (if its not already a module).
<M> Sun LANCE support
Doing so will be helpful in modifying the driver and setting it up for use
with rshaper. If you aren't going to be using rshaper, then you may skip the
next step.
Installing rshaper
------------------
Unpackage the source, "make" and then read the README! (from the README)
(1) in global space add the following line:
int (*net_shaper_rx_hook)(struct sk_buff *skb) = NULL;
(2) Only for Linux-2.2 (not for Linux-2.0), add this too:
EXPORT_SYMBOL(net_shaper_rx_hook)
this line must appear after the definition of the hook.
(3) in the receive function of the driver change
netif_rx(skb)
to
if (net_shaper_rx_hook)
(*net_shaper_rx_hook)(skb);
else
netif_rx(skb);
This should be it unless you have more than one network device or if you are
not compiling your network driver as a module. If this is the case, please
see the README included with rshaper.
Enabling Changes
----------------
Compile your kernel and modules, install them and reboot if necessary.
Now, we'll setup our outgoing bandwidth limitations (note, replace eth0 with
your network device).
Setup the queue, with your max bandwidth, mine is 10Mbit
# tc qdisc add dev eth0 root handle 10: cbq bandwidth 10Mbit avpkt 1000
Add your class. Again, bandwidth is 10Mbit. weight should be bandwidth / 10.
# tc class add dev eth0 parent 10:0 classid 10:1 cbq bandwidth 10Mbit \
rate 10Mbit allot 1514 weight 1Mbit prio 8 maxburst 20 avpkt 1000
Add your subclass where you'll limit the bandwidth. I'm limiting at a rate
of 1024Kbit's, weight should be less than the limit, preferrably by a factor
of 10 as above, but I didn't do that :)
# tc class add dev eth0 parent 10:1 classid 10:100 cbq bandwidth 10Mbit \
rate 1024kbit allot 1514 weight 768Kbit prio 5 maxburst 20 avpkt 1000 \
bounded
We've added the queues, now you must tell the kernel they exist and
how to manage them. We'll be using stochastic fairness queuing (sfq)
# tc qdisc add dev eth0 parent 10:100 sfq quantum 1514b perturb 15
Now we have to tell the kernel to send packets to the queue. Notice
the ip/mask. Change this depending on your target network to filter.
# tc filter add dev eth0 parent 10:0 protocol ip prio 100 u32 match ip \
dst 192.168.1.0/24 flowid 10:100
If you get "RTNETLINK answers: Invalid argument" errors, chances are you
didn't compile in all the necessary items into the kernel. Check my
config above and make sure necessary things are there. For instance, if
the last "tc" command above delivers this error, it means that QoS
u32 queueing is not compiled into the kernel or the module isn't loaded.
If you do not see any errors, then chances are you have correctly limited
the outgoing bandwidth of the above device. Issue the "tc -d qdisc" command.
It should look similar to this:
# tc -d qdisc
qdisc sfq 8001: dev eth0 quantum 1514b limit 128p flows 128/1024 perturb 15sec
qdisc cbq 10: dev eth0 rate 10Mbit cell 8b (bounded,isolated) prio no-transmit/8 weight 10Mbit allot 1514b
level 2 ewma 5 avpkt 1000b maxidle 23us
You can verify this by connecting to your machine from a remote machine and
comparing your download speeds. This is best done on an internal LAN where
you're speeds are pretty much guarentted maxes.
If you have chosen to also limit incoming bandwidth using the rshaper utility,
then you'll need to load the rshaper.o module at this time.
# insmod /usr/lib/rshaper.o
Then you need to setup your incoming bandwidth limitation using the rshaperctl
utility. For more information on this, see the rshaper README. In this case
my local network is 192.168.1.0/24.
# rshaperctl 192.168.1.0/24 128000
This shapes all incoming traffic on the local network to 128kb/s. Running
rshapectl with no options will show you the status of shaped hosts.
# rshaperctl
host mask bytes-per-sec max-queue-len (s)
192.168.1.0 255.255.255.0 128000 4
To test this setup, connect from your local machine to a remote machine and
compare your download speeds from this machine. Again, this is best done
on an internal LAN where you're speeds are pretty much guarenteed.
Scripts
-------
Below is the script I use to limit my outgoing and incoming bandwidth on my
machine:
----------------------------------------------------------------------------
#!/bin/sh
FILTERMASK="192.168.1.0/24"
# bandwidth in Kbits
IN_BANDWIDTH="1024"
# bandwidth in bytes
OUT_BANDWIDTH="128000"
DEV="eth0"
###################################################################
PATH=/usr/sbin:/sbin
echo -n "Shaping incoming traffic to ${IN_BANDWIDTH}Kbit/s "
tc qdisc add dev $DEV root handle 10: cbq bandwidth 10Mbit avpkt 1000
echo -n .
tc class add dev $DEV parent 10:0 classid 10:1 cbq bandwidth 10Mbit rate 10Mbit allot 1514 weight 1Mbit prio 8 maxburst 20 avpkt 1000
echo -n .
tc class add dev $DEV parent 10:1 classid 10:100 cbq bandwidth 10Mbit rate ${IN_BANDWIDTH}kbit allot 1514 weight 85kbit prio 5 maxburst 20 avpkt 1000 bounded
echo -n .
tc qdisc add dev $DEV parent 10:100 sfq quantum 1514b perturb 15
echo -n .
tc filter add dev $DEV parent 10:0 protocol ip prio 100 u32 match ip dst $FILTERMASK flowid 10:100
echo -n .
echo
echo -n "Shaping outgoing traffic to ${OUT_BANDWIDTH}bytes/s "
modprobe rshaper
rshaperctl $FILTERMASK $OUT_BANDWIDTH
echo -n .
echo
-------------------------------------------------------------------------
Questions/Comments/Credits
--------------------------
The information contained herein was gathered from a weekends worth of
searches on google. Hopefully you'll find it useful and usable. I
claim to be no expert on this topic by any means, so I don't know if
I'll be much help if you run into problems, but I'll do my best to help.
So please direct comments and questions to me, Brian Wilson
(wilson@sentrisystems.com). Thanks to z3d and sh00p for pointing me in
the right direction.
--
Prisacom
A. Ramos mailto:aramos@prisacom.com
Dpto. Admin. Sistemas
--
Reply to: