Re: google compute engine health check or DoS?
I appreciate your input on everything, Jimmy. Have a good night.
On Fri, Mar 21, 2014 at 9:42 PM, Jimmy Kaplowitz <jkaplowitz@google.com> wrote:
> Yow, those graphs don't look happy, indeed. Maybe your guess was right after
> all. I hope CloudFlare deters your attackers.
>
> Good to know you have rebooted since the 7.4 kernel went out. That's a good
> sign that the virtio-scsi memory leak didn't cause this problem.
>
> Note that GCE doesn't make kernels available and haven't since we switched
> APIs from v1beta16 to v1. We just boot the normal MBR and run whatever's
> there, in this case Debian's standard GRUB 2 (not pv_grub), just like
> standard KVM. We therefore use whatever kernels Debian provides. The
> backports image is configured to prefer the kernels (and initramfs-tools and
> soon ssh) from wheezy-backports, but both image flavors still use the
> kernels that Debian provides. We did submit the virtio-scsi fix for 7.4, but
> via Debian's BTS and kernel team.
>
> - Jimmy
>
>
> On Fri, Mar 21, 2014 at 9:34 PM, Jose R R <Jose.r.r@metztli-it.com> wrote:
>>
>> Oh, by the way, Jimmy. Every time apt-get dist-upgrade's the kernel I
>> subsequently reboot from the command line; hence I am running the
>> latest kernel that GCE team makes available
>>
>> On Fri, Mar 21, 2014 at 9:30 PM, Jose R R <Jose.r.r@metztli-it.com> wrote:
>> > Thanks Jimmy, I will try the gcutil resetinstance; looking at may GCE
>> > console
>> >
>> > https://twitter.com/Metztli_IT/status/447226775483858944/photo/1/large
>> >
>> > I think I may switch DNS to CloudFlare for resilience on this
>> > instance. First time it happens (sigh)
>> >
>> > Thank you, Jeremy, as well.
>> >
>> >
>> > Best Professional Regards
>> >
>> > On Fri, Mar 21, 2014 at 9:23 PM, Jimmy Kaplowitz <jkaplowitz@google.com>
>> > wrote:
>> >> gcutil resetinstance is a reasonable idea. That is the equivalent of
>> >> pressing a hard reset button on the system. Alternatively, gcutil
>> >> deleteinstance and declining when asked about deleting the boot disk
>> >> (or
>> >> using --nodelete_boot_pd) is the equivalent of sending an ACPI power
>> >> down
>> >> signal (there is no equivalent signal for soft reboots) and freeing up
>> >> the
>> >> CPU, unreserved IP addresses, RAM, etc.
>> >>
>> >> Neither of these deletes your disk - the disk is called persistent disk
>> >> for
>> >> a reason. (The soon-to-vanish v1beta16 API offered scratch disks, but
>> >> unless
>> >> you're using a gcutil from before early December, you would have had to
>> >> choose that API explicitly.) I've used both commands many times,
>> >> including
>> >> on critical instances.
>> >>
>> >> If you do ever delete and re-add the instance, you can give gcutil
>> >> addinstance a flag like --disk=<old instance name>,boot instead of
>> >> specifying an image to boot off the existing disk.
>> >>
>> >> Also, if you apt-get upgraded without rebooting, the kernel is the same
>> >> one
>> >> you first booted with. However unless something got killed via OOM, I
>> >> can't
>> >> see how the old kernel's memory leak would be related.
>> >>
>> >> Yes, I meant #debian-cloud on IRC. (As Jeremy said, that's
>> >> irc.debian.org,
>> >> also known as OFTC.) I'm Hydroxide there.
>> >>
>> >> Heading to sleep soon - it is indeed evening here and I'm on an
>> >> unusually
>> >> early schedule this week.
>> >>
>> >> Good luck!
>> >>
>> >> - Jimmy
>> >>
>> >>
>> >> On Fri, Mar 21, 2014 at 9:01 PM, Jose R R <Jose.r.r@metztli-it.com>
>> >> wrote:
>> >>>
>> >>> Hi Jimmy-
>> >>>
>> >>> Sorry I did not think anybody would reply as quickly on a Friday
>> >>> evening (Calironia).
>> >>>
>> >>> I had not experienced any issues since approximately 2 months ago that
>> >>> I started the instance with Salt/Libcloud as a proof of concept.
>> >>> Accordingly the image has been upgraded since then with apt-get
>> >>> update/dist-upgrade.
>> >>>
>> >>> gcutil getserialportoutput ouputs a long output and at the end I get
>> >>> copious output with tail related to the database, example:
>> >>>
>> >>> Mar 21 20:23:31 my-google-instance mysqld: 0 [0, 0, 0, 0] ,
>> >>> Mar 21 20:23:43 my-google-instance mysqld: ibuf aio reads:140321
>> >>> 20:23:43 [Warning] Aborted connection 58427 to db: 'my-sql-db' user:
>> >>> 'my-self' host: 'localhost' (Unknown error)
>> >>> Mar 21 20:25:49 my-google-instance mysqld: 140321 20:25:49 [Warning]
>> >>> Aborted connection 58378 to db: 'my-sql-db' user: 'my-self' host:
>> >>> 'localhost' (Unknown error)
>> >>>
>> >>> I was going to try gcutil resetinstance to reboot my instance but I am
>> >>> not sure if it will wipe out my served content.
>> >>>
>> >>>
>> >>> Best Professional regards.
>> >>>
>> >>> PS. #debian-cloud irc?
>> >>>
>> >>>
>> >>> On Fri, Mar 21, 2014 at 7:27 PM, Jimmy Kaplowitz
>> >>> <jkaplowitz@google.com>
>> >>> wrote:
>> >>> > Hi Jose,
>> >>> >
>> >>> > No, us-central1-b should be working. Does gcutil
>> >>> > getserialportoutput
>> >>> > (or
>> >>> > the equivalent web console view) show anything informative? Might be
>> >>> > an
>> >>> > OOM,
>> >>> > a kernel panic, really any of the possibilities for a normal system.
>> >>> >
>> >>> > I won't be awake many more hours, but I can briefly discuss in
>> >>> > #debian-cloud
>> >>> > if you'd like.
>> >>> >
>> >>> > One thing to note is that a memory leak in the virtio-scsi driver
>> >>> > got
>> >>> > fixed
>> >>> > in the Debian 7.4 kernel, and is also absent from the backports
>> >>> > kernel.
>> >>> > Either of these is recommended over older Debian kernels.
>> >>> >
>> >>> > We released our 7.4 images earlier this week, though of course
>> >>> > apt-get
>> >>> > upgrade would have gotten you that sooner, and the backports image
>> >>> > already
>> >>> > offered a newer kernel partly for this reason.
>> >>> >
>> >>> > - Jimmy
>> >>> >
>> >>> > On Mar 21, 2014 7:10 PM, "Jose R R" <Jose.r.r@metztli-it.com> wrote:
>> >>> >>
>> >>> >> Niltze [hi!]-
>> >>> >>
>> >>> >> I am running a small Debian instance on GCE on us-central1-b (from
>> >>> >> another email account than this one) and was wondering if there
>> >>> >> it's
>> >>> >> undergoing maintenance; otherwise I think it is being being DoS
>> >>> >> DDoS
>> >>> >> since
>> >>> >>
>> >>> >> gcutil nor ssh are not able to connect; nor the web server displays
>> >>> >> anything (loading...)
>> >>> >>
>> >>> >> If anyone from GCE reads this I'd appreciate your input.
>> >>> >>
>> >>> >>
>> >>> >> Best Professional Regards.
>> >>> >>
>> >>> >> --
>> >>> >> Jose R R
>> >>> >> http://www.metztli-it.com
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> ---------------------------------------------------------------------------------------------
>> >>> >> NEW Apache OpenOffice 4.0.1! Download for GNU/Linux, Mac OS,
>> >>> >> Windows.
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> ---------------------------------------------------------------------------------------------
>> >>> >> Daylight Saving Time in USA & Canada ends: Sunday, November 02,
>> >>> >> 2014
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> ---------------------------------------------------------------------------------------------
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> To UNSUBSCRIBE, email to debian-cloud-request@lists.debian.org
>> >>> >> with a subject of "unsubscribe". Trouble? Contact
>> >>> >> listmaster@lists.debian.org
>> >>> >> Archive:
>> >>> >>
>> >>> >>
>> >>> >> [🔎] CAM12Q5SLEqE9TstrVL9PYrirvDH1UJcDA2ragD1zYeUeUxbzGA@mail.gmail.com">https://lists.debian.org/[🔎] CAM12Q5SLEqE9TstrVL9PYrirvDH1UJcDA2ragD1zYeUeUxbzGA@mail.gmail.com
>> >>> >>
>> >>> >
>> >>
>> >>
>
>
Best Professional Regards.
-
Reply to: