Bug#932769: [moreinfo] DoS via DHCP request

Hi fellas,

Apologies for the brevity in the initial bug report. I was using the reportbug tool directly from the console of the VM I was working on, small resolution. Allow me to elaborate...

We initially discovered this bug testing our storage product, we had a Debian 10 VM running in a typical ESXi 6.7 environment with iSCSI backed storage. The VM ran in a VMDK file on a VMFS datastore volume. While the VM was running in memory, we removed the storage initiators from ESXi purposefully to test something unrelated, to simulate a storage outage. After a couple of minutes the OS will go into R/O mode without its disk, and at that time dhclient will rapidly request IP's from our ISC DHCP server. dhclient will take the IP, consume it from the DHCP pool and then request another. After some period of time this depletes the DHCP pool, several hours to days depending on the scopes size. This could also be replicated by deleting the hard disk from a running VM in a virtual environment.

When I look at systemctl for the dhclient service, I can see that there's an error, "can't create /var/lib/dhcp/dhclient.intname.leases Read Only file system", and then the DHCPREQUEST > DHCPACK > DHCPDECLINE sequence starts every few seconds, and occasionally the service will show "RTNETLINK answers: File Exists."

I'm guessing from the error that dhclient has a problem with not being able to read / write to the client leases file, declines the IP and requests another, but secretly holds on to the IP.

The DHCP server logs will show a final DHCPDECLINE after the ACK, and mark the address as abandoned. The VM will still have the address leased however. After a period of time VMware's guest tools will show all the consumed IP's belonging to that MAC address and virtual interface. Network gear ARP shows the IP's belonging to the same MAC as well.

We've consistently reproduced this bug in our lab, and performed the test simultaneously with a Debian 9, Centos and Ubuntu 16 instance to make sure it wasn't some kind of NetworkManager thing, or a broader Linux issue.

I see that someone reported this similar bug back in 2018 as well, I think they may be the same thing.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=888209

Thanks, just let me know if you have any questions.

On Tue, Jul 23, 2019 at 4:23 PM Tomáš Pospíšek <tpo@sourcepole.ch> wrote:

Am 23.07.19 um 17:57 schrieb Ben Hutchings:
> On Tue, 2019-07-23 at 16:51 -0400, Tomas Pospisek wrote:
>> Package: general
>> Followup-For: Bug #932769
>>
>> Could you privide a recipe on how to reproduce this? There's a lot of
>> very special setup below, that someone wwould need large amounts of time
>> to reporoduce I feel.
>>
>> Is it possible to reduce the problem to something easily demonstratable?
>>
>> This seems to be an important issue to me.
>>
>> I think the problem here *might* be a kernel problem? Re-assign this to
>> kernel package?
> [...]
>
> So far as I know, the kernel only ever does DHCP if you net-boot
> without an initramfs.

My focus was more on this issue here - aparenty:

Mark Hutchison wrote:

>> This DoS's the server [due to DHCP changing IPs rapidly
>> - my interpretation] and the interface attempts to take and discard
>> IP's in a rapid fashion.

-> changing IPs of an interface of a *VM* can DoS the server. Which I
think is not expected, and not terribly funny. It takes a bit of not so
straightforward circumstances (as far as I can understand the bug
report), but then an attacker can DoS the server via DHCP. Which is uh,
I mean ah, um.

Information is a bit sparse here, though.

<OT>If I may shoot completely off topic for a second: Woah, many thanks
for your terrific kernel maintenance work Ben. Truly amazing :-o!!!
Thanks so may times a lot! Woah :-)!!!! Thank you! (this doesn't exclude
the rest of the kernel team - my thanks extend to you all - it's just
that I have the honor to say thanks to a participating party in this
email exchange 8v)!</OT>
*t