Re: google compute engine health check or DoS?
Good morning, Jimmy-
On Sat, Mar 22, 2014 at 11:16 AM, Jimmy Kaplowitz <jkaplowitz@google.com> wrote:
> Thanks for the log data. Can I suggest that you mention what happened on the
> gce-discussion list linked from the support page? The right people will
> probably follow up there. (I'm there too.)
>
Well, I just glossed over StackOverflow's:
http://stackoverflow.com/questions/tagged/google-compute-engine
But since I usually favor the deductive/inductive approach to solve
technical problems, I went back to actually probe the issue and act
--subsequently-- on the feedback from the system.
It is only on rare occasions when really something stumps me that I
post to a list and hang around waiting for an answer to my specific
issue.
> It doesn't actually answer whether or not there was a DoS as the ultimate
> cause, only that your instance had suffered the kernel messages as the
> immediate cause of the outage. Certainly it provides less reason to think a
> DoS occurred.
>
> In the meantime, try the kernel from wheezy-backports. It's known to perform
> much better (both in GCE and elsewhere), and also has some important bug
> fixes that are harder than the memory leak fix to integrate into wheezy's
> 3.2 kernel. These are all a big part of why we offer the backports image,
> though of course users who know Debian enough can add it in the regular
> image.
>
> While the backports kernel doesn't get Debian security team support, the
> kernel team does maintain and periodically update the backport (admittedly
> not as fast as the security team patches).
>
> If you want to switch to the backports kernel and receive updates to it in
> routine apt-get dist-upgrades, examine /etc/apt/*.d/*backports* on a
> backports image-based instance or look at
> https://github.com/google/build-debian-cloud/blob/master/tasks/gce/21-apt-backport-sources
> and
> https://github.com/google/build-debian-cloud/blob/master/tasks/gce/21-prefer-backports-kernel
> to see how we add it. (You can also grab the backports image tarball via
> gsutil from gs://debian_images if you prefer to examine it locally.)
Indeed, I may try the backports kernel, although at the moment (not
being triggered by the suspected event that took it down) the current
kernel seems to perform fine once again.
When I was evaluating HP Cloud's OpenStack (and even Joyent's
SmartOS-based Cloud) I used to build my kernels and thus keep control
of that aspect of the system. However, unless I'm mistaken, I believe
it is still not possible on GCE, right? Nor does GCE support hosting a
type 2 hypervisor like VirtualBox, right?
Once again, Jimmy, thanks.
Best Professional Regards.
>
> Good morning, Jimmy-
>
> Thanks for https://cloud.google.com/support/ reference. I see many
> unanswered questions, though ;-)
>
> On Fri, Mar 21, 2014 at 10:23 PM, Jimmy Kaplowitz <jkaplowitz@google.com>
> wrote:
>> You too!
>>
>> By the way, if you want a broader range of people providing support
>> (including external peers and GCE Googlers who may not be part of the
>> Debian
>> community), check out the self-service and community support options here,
>> including a Google Group aka mailing list and a Stack overflow tag (paid
>> support also available):
>>
>> https://cloud.google.com/support/
>>
>> Now that I've finished an unrelated email, actually going to sleep for
>> real.
>> :) /me waves good night! Feel free to say hello on IRC any time.
>>
>> - Jimmy
>
> After changing DNS to CloudFlare's, I did not notice any
> improvement(s). Thus at around 23:11pm I rebooted (with gcutil
> resetinstance) and subsequently I was able to log in via gcutil
> (indeed ssh functions, too).
>
> Analyzing syslog it seems that the GCE instance began having issues at
> around 18:54 pm beginning with the line:
>
> Mar 21 18:54:34 my-gce-instance kernel: [3502320.420053] INFO: task
> kswapd0:18 blocked for more than 120 seconds.
> [...]
>
> Accordingly, issue that began this thread appears to be kernel
> related; I have sent you an edited version of syslog (leaving the
> kernel related stuff intact) in case you want to analyze it.
>
> uname -a
>
> Linux my-gce-instance 3.2.0-4-amd64 #1 SMP Debian 3.2.54-2 x86_64 GNU/Linux
>
> Although my GCE Console graph shows sustained spikes, the range of
> values seems normal; thus, it was not a DoS/DDoS after all.
>
>
> Best Professional Regards.
>
> $ cat /etc/os*
>
> PRETTY_NAME="Debian GNU/Linux 7 (wheezy)"
> NAME="Debian GNU/Linux"
> VERSION_ID="7"
> VERSION="7 (wheezy)"
> ID=debian
> ANSI_COLOR="1;31"
> HOME_URL="http://www.debian.org/";
> SUPPORT_URL="http://www.debian.org/support/";
> BUG_REPORT_URL="http://bugs.debian.org/";
>
> --
> Jose R R
> http://www.metztli-it.com
> ---------------------------------------------------------------------------------------------
> NEW Apache OpenOffice 4.0.1! Download for GNU/Linux, Mac OS, Windows.
> ---------------------------------------------------------------------------------------------
> Daylight Saving Time in USA & Canada ends: Sunday, November 02, 2014
> ---------------------------------------------------------------------------------------------
Reply to: