Re: Package conflicts, breaks, and problematic upgrades... with irqbalance

[Zach Marano <zmarano@google.com>]

On Mon, Feb 26, 2018 at 11:06 PM Bastian Blank <waldi@debian.org> wrote:

On Tue, Feb 27, 2018 at 01:46:15AM +0000, Zach Marano wrote:
> The problem is that irqbalance breaks on GCE and KVM (and likely other
> virtualization platforms). So, in our debian package for
> google-compute-engine [1], where we have a script that balances the
> interrupts for scsi and network devices across CPU's correctly, we conflict
> with irqbalance because it literally does the opposite.

Could you explain this a bit more?  This would mean "irqbalance" itself
is broken.

irqbalance is supposed to spread interrupts across all CPU's but in GCE (and from what I recall KVM as well) it ends up assigning all interrupts to CPU0. It may be considered broken except that it was originally meant for bare metal hardware and therefore- maybe its not actually broken in a virtual environment? Consequently, virtio_net Tx/Rx queues are assigned to CPU0 as well and likewise for virtio_scsi when using multiqueue. This has serious consequences for network and disk IO. For example, its the difference between 150k iops and 700k iops for LocalSSD's with multiqueue SCSI. This discussion is also relevant. Hence, why we really want to prevent it from being installed. The number of cases that used to come in about getting terrible IO performance or why is my *insert network IO heavy application here* performance so terrible all came down to someone installing or inadvertently installing irqbalance.

With irqbalance on a 32 core VM:

for i in $(seq 0 300); do grep . /proc/irq/$i/smp_affinity /dev/null 2>/dev/null; done

/proc/irq/0/smp_affinity:ffffffff

/proc/irq/1/smp_affinity:ffffffff

/proc/irq/2/smp_affinity:ffffffff

/proc/irq/3/smp_affinity:ffffffff

/proc/irq/4/smp_affinity:ffffffff

/proc/irq/5/smp_affinity:ffffffff

/proc/irq/6/smp_affinity:ffffffff

/proc/irq/7/smp_affinity:ffffffff

/proc/irq/8/smp_affinity:ffffffff

/proc/irq/9/smp_affinity:ffffffff

/proc/irq/10/smp_affinity:ffffffff

/proc/irq/11/smp_affinity:ffffffff

/proc/irq/12/smp_affinity:ffffffff

/proc/irq/13/smp_affinity:ffffffff

/proc/irq/14/smp_affinity:ffffffff

/proc/irq/15/smp_affinity:ffffffff

/proc/irq/24/smp_affinity:ffffffff

/proc/irq/25/smp_affinity:ffffffff

/proc/irq/26/smp_affinity:ffffffff

/proc/irq/27/smp_affinity:ffffffff

/proc/irq/28/smp_affinity:ffffffff

/proc/irq/29/smp_affinity:ffffffff

/proc/irq/30/smp_affinity:ffffffff

/proc/irq/31/smp_affinity:ffffffff

/proc/irq/32/smp_affinity:ffffffff

/proc/irq/33/smp_affinity:ffffffff

/proc/irq/34/smp_affinity:ffffffff

/proc/irq/35/smp_affinity:ffffffff

/proc/irq/36/smp_affinity:ffffffff

/proc/irq/37/smp_affinity:ffffffff

/proc/irq/38/smp_affinity:ffffffff

/proc/irq/39/smp_affinity:ffffffff

/proc/irq/40/smp_affinity:ffffffff

/proc/irq/41/smp_affinity:ffffffff

/proc/irq/42/smp_affinity:ffffffff

/proc/irq/43/smp_affinity:ffffffff

/proc/irq/44/smp_affinity:ffffffff

/proc/irq/45/smp_affinity:ffffffff

/proc/irq/46/smp_affinity:ffffffff

/proc/irq/47/smp_affinity:ffffffff

/proc/irq/48/smp_affinity:ffffffff

/proc/irq/49/smp_affinity:ffffffff

/proc/irq/50/smp_affinity:ffffffff

/proc/irq/51/smp_affinity:ffffffff

/proc/irq/52/smp_affinity:ffffffff

/proc/irq/53/smp_affinity:ffffffff

/proc/irq/54/smp_affinity:ffffffff

/proc/irq/55/smp_affinity:ffffffff

/proc/irq/56/smp_affinity:ffffffff

/proc/irq/57/smp_affinity:ffffffff

/proc/irq/58/smp_affinity:ffffffff

/proc/irq/59/smp_affinity:ffffffff

/proc/irq/60/smp_affinity:ffffffff

/proc/irq/61/smp_affinity:ffffffff

/proc/irq/62/smp_affinity:ffffffff

/proc/irq/63/smp_affinity:ffffffff

/proc/irq/64/smp_affinity:ffffffff

/proc/irq/65/smp_affinity:ffffffff

/proc/irq/66/smp_affinity:ffffffff

/proc/irq/67/smp_affinity:ffffffff

/proc/irq/68/smp_affinity:ffffffff

/proc/irq/69/smp_affinity:ffffffff

/proc/irq/70/smp_affinity:ffffffff

/proc/irq/71/smp_affinity:ffffffff

/proc/irq/72/smp_affinity:ffffffff

/proc/irq/73/smp_affinity:ffffffff

/proc/irq/74/smp_affinity:ffffffff

/proc/irq/75/smp_affinity:ffffffff

/proc/irq/76/smp_affinity:ffffffff

/proc/irq/77/smp_affinity:ffffffff

/proc/irq/78/smp_affinity:ffffffff

/proc/irq/79/smp_affinity:ffffffff

/proc/irq/80/smp_affinity:ffffffff

/proc/irq/81/smp_affinity:ffffffff

/proc/irq/82/smp_affinity:ffffffff

/proc/irq/83/smp_affinity:ffffffff

/proc/irq/84/smp_affinity:ffffffff

/proc/irq/85/smp_affinity:ffffffff

/proc/irq/86/smp_affinity:ffffffff

/proc/irq/87/smp_affinity:ffffffff

/proc/irq/88/smp_affinity:ffffffff

/proc/irq/89/smp_affinity:ffffffff

/proc/irq/90/smp_affinity:ffffffff

/proc/irq/91/smp_affinity:ffffffff

/proc/irq/92/smp_affinity:ffffffff

Without irqbalance on a 32 core VM (and our own scripts):

for i in $(seq 0 300); do grep . /proc/irq/$i/smp_affinity /dev/null 2>/dev/null; done

/proc/irq/0/smp_affinity:ffffffff

/proc/irq/1/smp_affinity:ffffffff

/proc/irq/2/smp_affinity:ffffffff

/proc/irq/3/smp_affinity:ffffffff

/proc/irq/4/smp_affinity:ffffffff

/proc/irq/5/smp_affinity:ffffffff

/proc/irq/6/smp_affinity:ffffffff

/proc/irq/7/smp_affinity:ffffffff

/proc/irq/8/smp_affinity:ffffffff

/proc/irq/9/smp_affinity:ffffffff

/proc/irq/10/smp_affinity:ffffffff

/proc/irq/11/smp_affinity:ffffffff

/proc/irq/12/smp_affinity:ffffffff

/proc/irq/13/smp_affinity:ffffffff

/proc/irq/14/smp_affinity:ffffffff

/proc/irq/15/smp_affinity:ffffffff

/proc/irq/24/smp_affinity:ffffffff

/proc/irq/25/smp_affinity:00000001

/proc/irq/26/smp_affinity:00000001

/proc/irq/27/smp_affinity:00000002

/proc/irq/28/smp_affinity:00000002

/proc/irq/29/smp_affinity:00000004

/proc/irq/30/smp_affinity:00000004

/proc/irq/31/smp_affinity:00000008

/proc/irq/32/smp_affinity:00000008

/proc/irq/33/smp_affinity:00000010

/proc/irq/34/smp_affinity:00000010

/proc/irq/35/smp_affinity:00000020

/proc/irq/36/smp_affinity:00000020

/proc/irq/37/smp_affinity:00000040

/proc/irq/38/smp_affinity:00000040

/proc/irq/39/smp_affinity:00000080

/proc/irq/40/smp_affinity:00000080

/proc/irq/41/smp_affinity:00000100

/proc/irq/42/smp_affinity:00000100

/proc/irq/43/smp_affinity:00000200

/proc/irq/44/smp_affinity:00000200

/proc/irq/45/smp_affinity:00000400

/proc/irq/46/smp_affinity:00000400

/proc/irq/47/smp_affinity:00000800

/proc/irq/48/smp_affinity:00000800

/proc/irq/49/smp_affinity:00001000

/proc/irq/50/smp_affinity:00001000

/proc/irq/51/smp_affinity:00002000

/proc/irq/52/smp_affinity:00002000

/proc/irq/53/smp_affinity:00004000

/proc/irq/54/smp_affinity:00004000

/proc/irq/55/smp_affinity:00008000

/proc/irq/56/smp_affinity:00008000

/proc/irq/57/smp_affinity:00010000

/proc/irq/58/smp_affinity:00010000

/proc/irq/59/smp_affinity:00020000

/proc/irq/60/smp_affinity:00020000

/proc/irq/61/smp_affinity:00040000

/proc/irq/62/smp_affinity:00040000

/proc/irq/63/smp_affinity:00080000

/proc/irq/64/smp_affinity:00080000

/proc/irq/65/smp_affinity:00100000

/proc/irq/66/smp_affinity:00100000

/proc/irq/67/smp_affinity:00200000

/proc/irq/68/smp_affinity:00200000

/proc/irq/69/smp_affinity:00400000

/proc/irq/70/smp_affinity:00400000

/proc/irq/71/smp_affinity:00800000

/proc/irq/72/smp_affinity:00800000

/proc/irq/73/smp_affinity:01000000

/proc/irq/74/smp_affinity:01000000

/proc/irq/75/smp_affinity:02000000

/proc/irq/76/smp_affinity:02000000

/proc/irq/77/smp_affinity:04000000

/proc/irq/78/smp_affinity:04000000

/proc/irq/79/smp_affinity:08000000

/proc/irq/80/smp_affinity:08000000

/proc/irq/81/smp_affinity:10000000

/proc/irq/82/smp_affinity:10000000

/proc/irq/83/smp_affinity:20000000

/proc/irq/84/smp_affinity:20000000

/proc/irq/85/smp_affinity:40000000

/proc/irq/86/smp_affinity:40000000

/proc/irq/87/smp_affinity:80000000

/proc/irq/88/smp_affinity:80000000

/proc/irq/89/smp_affinity:ffffffff

/proc/irq/90/smp_affinity:ffffffff

/proc/irq/91/smp_affinity:ffffffff

/proc/irq/92/smp_affinity:ffffffff

 

>                                                         The Debian kernel
> package however recommends irqbalance [2]. Some users running `apt upgrade`
> (not apt-get) run into a problem where our package gets removed because the
> kernel will install irqbalance when it gets updated.

Yes, removing this package is a valid solution.  That's why you never
want to conflict against or break any core package (BTDT, people cried).

So, irqbalance is a recommended package not a required package. But because its a recommended package by the kernel package it seems to get priority to be installed even though it is not installed already. The kernel package has priority 'optional' as does our own package. It does seem that if I set our package priority to `required`, apt correctly handles the dependencies because I guess the conflicts and breaks statements in our control file now take priority. From what I gather from the Debian policy manual, 'required' means: "Packages which are necessary for the proper functioning of the system"... and on GCE this could be considered the case for our package given that it provides you access to your VM and does things like handle interrupt assignments. So, I don't believe we would be violating policy by setting the priority to required. On the other hand, the policy seems to indicate (via example anyway) that 'required' is really only for things like dpkg. No idea what the right course of action is- but since I know that using 'required' priority works I am heavily leaning towards that.

> So, what is the right way to fix this? And yes, I do believe the having
> irqbalance as a recommends for the kernel package is just wrong- however
> lets deal with that separately.

There is no solution, at least no stable one.

Bastian

--
Vulcans do not approve of violence.
                -- Spock, "Journey to Babel", stardate 3842.4