[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: google compute engine health check or DoS?



Yow, those graphs don't look happy, indeed. Maybe your guess was right after all. I hope CloudFlare deters your attackers.

Good to know you have rebooted since the 7.4 kernel went out. That's a good sign that the virtio-scsi memory leak didn't cause this problem.

Note that GCE doesn't make kernels available and haven't since we switched APIs from v1beta16 to v1. We just boot the normal MBR and run whatever's there, in this case Debian's standard GRUB 2 (not pv_grub), just like standard KVM. We therefore use whatever kernels Debian provides. The backports image is configured to prefer the kernels (and initramfs-tools and soon ssh) from wheezy-backports, but both image flavors still use the kernels that Debian provides. We did submit the virtio-scsi fix for 7.4, but via Debian's BTS and kernel team.

- Jimmy


On Fri, Mar 21, 2014 at 9:34 PM, Jose R R <Jose.r.r@metztli-it.com> wrote:
Oh, by the way, Jimmy. Every time apt-get dist-upgrade's the kernel I
subsequently reboot from the command line; hence I am running the
latest kernel that GCE team makes available

On Fri, Mar 21, 2014 at 9:30 PM, Jose R R <Jose.r.r@metztli-it.com> wrote:
> Thanks Jimmy, I will try the gcutil resetinstance; looking at may GCE console
>
> https://twitter.com/Metztli_IT/status/447226775483858944/photo/1/large
>
> I think I may switch DNS to CloudFlare for resilience on this
> instance. First time it happens (sigh)
>
> Thank you, Jeremy, as well.
>
>
> Best Professional Regards
>
> On Fri, Mar 21, 2014 at 9:23 PM, Jimmy Kaplowitz <jkaplowitz@google.com> wrote:
>> gcutil resetinstance is a reasonable idea. That is the equivalent of
>> pressing a hard reset button on the system. Alternatively, gcutil
>> deleteinstance and declining when asked about deleting the boot disk (or
>> using --nodelete_boot_pd) is the equivalent of sending an ACPI power down
>> signal (there is no equivalent signal for soft reboots) and freeing up the
>> CPU, unreserved IP addresses, RAM, etc.
>>
>> Neither of these deletes your disk - the disk is called persistent disk for
>> a reason. (The soon-to-vanish v1beta16 API offered scratch disks, but unless
>> you're using a gcutil from before early December, you would have had to
>> choose that API explicitly.) I've used both commands many times, including
>> on critical instances.
>>
>> If you do ever delete and re-add the instance, you can give gcutil
>> addinstance a flag like --disk=<old instance name>,boot instead of
>> specifying an image to boot off the existing disk.
>>
>> Also, if you apt-get upgraded without rebooting, the kernel is the same one
>> you first booted with. However unless something got killed via OOM, I can't
>> see how the old kernel's memory leak would be related.
>>
>> Yes, I meant #debian-cloud on IRC. (As Jeremy said, that's irc.debian.org,
>> also known as OFTC.) I'm Hydroxide there.
>>
>> Heading to sleep soon - it is indeed evening here and I'm on an unusually
>> early schedule this week.
>>
>> Good luck!
>>
>> - Jimmy
>>
>>
>> On Fri, Mar 21, 2014 at 9:01 PM, Jose R R <Jose.r.r@metztli-it.com> wrote:
>>>
>>> Hi Jimmy-
>>>
>>> Sorry I did not think anybody would reply as quickly on a Friday
>>> evening (Calironia).
>>>
>>> I had not experienced any issues since approximately 2 months ago that
>>> I started the instance with Salt/Libcloud as a proof of concept.
>>> Accordingly the image has been upgraded since then with apt-get
>>> update/dist-upgrade.
>>>
>>> gcutil getserialportoutput ouputs a long output and at the end I get
>>> copious output with tail related to the database, example:
>>>
>>> Mar 21 20:23:31 my-google-instance mysqld: 0 [0, 0, 0, 0] ,
>>> Mar 21 20:23:43 my-google-instance mysqld: ibuf aio reads:140321
>>> 20:23:43 [Warning] Aborted connection 58427 to db: 'my-sql-db' user:
>>> 'my-self' host: 'localhost' (Unknown error)
>>> Mar 21 20:25:49 my-google-instance mysqld: 140321 20:25:49 [Warning]
>>> Aborted connection 58378 to db: 'my-sql-db' user: 'my-self' host:
>>> 'localhost' (Unknown error)
>>>
>>> I was going to try gcutil resetinstance to reboot my instance but I am
>>> not sure if it will wipe out my served content.
>>>
>>>
>>> Best Professional regards.
>>>
>>> PS. #debian-cloud irc?
>>>
>>>
>>> On Fri, Mar 21, 2014 at 7:27 PM, Jimmy Kaplowitz <jkaplowitz@google.com>
>>> wrote:
>>> > Hi Jose,
>>> >
>>> > No, us-central1-b should be working.  Does gcutil getserialportoutput
>>> > (or
>>> > the equivalent web console view) show anything informative? Might be an
>>> > OOM,
>>> > a kernel panic, really any of the possibilities for a normal system.
>>> >
>>> > I won't be awake many more hours, but I can briefly discuss in
>>> > #debian-cloud
>>> > if you'd like.
>>> >
>>> > One thing to note is that a memory leak in the virtio-scsi driver got
>>> > fixed
>>> > in the Debian 7.4 kernel, and is also absent from the backports kernel.
>>> > Either of these is recommended over older Debian kernels.
>>> >
>>> > We released our 7.4 images earlier this week, though of course apt-get
>>> > upgrade would have gotten you that sooner, and the backports image
>>> > already
>>> > offered a newer kernel partly for this reason.
>>> >
>>> > - Jimmy
>>> >
>>> > On Mar 21, 2014 7:10 PM, "Jose R R" <Jose.r.r@metztli-it.com> wrote:
>>> >>
>>> >> Niltze [hi!]-
>>> >>
>>> >> I am running a small Debian instance on GCE on us-central1-b (from
>>> >> another email account than this one) and was wondering if  there it's
>>> >> undergoing maintenance; otherwise I think it is being being DoS DDoS
>>> >> since
>>> >>
>>> >> gcutil nor ssh are not able to connect; nor the web server displays
>>> >> anything (loading...)
>>> >>
>>> >> If anyone from GCE reads this I'd appreciate your input.
>>> >>
>>> >>
>>> >> Best Professional Regards.
>>> >>
>>> >> --
>>> >> Jose R R
>>> >> http://www.metztli-it.com
>>> >>
>>> >>
>>> >> ---------------------------------------------------------------------------------------------
>>> >> NEW Apache OpenOffice 4.0.1! Download for GNU/Linux, Mac OS, Windows.
>>> >>
>>> >>
>>> >> ---------------------------------------------------------------------------------------------
>>> >> Daylight Saving Time in USA & Canada ends: Sunday, November 02, 2014
>>> >>
>>> >>
>>> >> ---------------------------------------------------------------------------------------------
>>> >>
>>> >>
>>> >> --
>>> >> To UNSUBSCRIBE, email to debian-cloud-request@lists.debian.org
>>> >> with a subject of "unsubscribe". Trouble? Contact
>>> >> listmaster@lists.debian.org
>>> >> Archive:
>>> >>
>>> >> [🔎] CAM12Q5SLEqE9TstrVL9PYrirvDH1UJcDA2ragD1zYeUeUxbzGA@mail.gmail.com" target="_blank">https://lists.debian.org/[🔎] CAM12Q5SLEqE9TstrVL9PYrirvDH1UJcDA2ragD1zYeUeUxbzGA@mail.gmail.com
>>> >>
>>> >
>>
>>


Reply to: