Re: Oracle Cloud Infrastructure

[Thomas Goirand <zigo@debian.org>]

Happy new year Paul,

On 12/31/18 4:56 PM, Paul Graydon wrote:
> On 12/29/18 12:55, Thomas Goirand wrote:
> 
>>> I may be misunderstanding your comment, but so far as runtime vs
>>> baked-in, we've been attempting to make everything as secure as possible
>>> from the get-go, and assume that customers ignore any good practices or
>>> suggestions we document.  Given the security risk of the iSCSI endpoint,
>>> we don't want to be taking risks there.
>> It looks like I'm missing exactly what you're doing. Could you please
>> explain what you're doing with iSCSI exactly? Maybe some configuration
>> example, etc. I have to admit I don't know much about it, apart from
>> having configured Cinder and let the magic happen with tgtd... :)
> 
> The actual configuration changes for iSCSI are relatively minor. Just
> tweaks to resiliency, iscsi pings etc, based on experiences in our
> operating environment.  Back when we launched, we just ran as default,
> we've been spending a couple of years tweaking things to strike the
> right balance between resiliency and performance.

Could you please share these changes? For example, just the config file
in an instance, or even better, a diff with the default, would be super
nice and helpful to share! Maybe the package maintainer in Debian can
have your defaults become the package defaults, for example.

> The security concerns around the endpoint are just that because it's a
> cloud environment, root has to be unauthenticated, or we'd either have
> to engineer ways to inject in to images the (launch-time?) generated
> credentials, etc.  iptables gives you the option to restrict traffic
> based on UID, so we just lock it down to UID 0.

This doesn't look very much a standard thing in Debian.

> One of the other design goals for the platform was "secure by default",
> and we're trying to encourage customers to think about defence in depth,
> putting the firewall on by default at each server, with a particular
> configuration is ideal.

There, we have a problem. I don't mind myself having a firewall
activated by default, but that's not the Debian standard. Under
OpenStack, also, this is useless, because there's at least the security
groups, and probably also neutron-fwaas (ie: firewall_v2) which is also
providing a firewall line of defence from the host.

If you wish to have this kind of setup activated by default, then we
need to have a serious discussion within the cloud team to make sure
that all of our images have the same type of configuration.

> I promise you, we don't make decisions based on "how do we make life
> hardest for ourselves", and believe me, with FIPS etc to consider,
> producing our own agent is most definitely not the easiest option.

What is FIPS referring to?

> We've worked with the main cloud-init developers repeatedly over the
> last two years, with bug fixing and the like. OCI has been repeatedly
> funding and submitting work upstream to various projects as part of the
> cloud build-out, based on issues we've found at scale, features we need
> etc.  Those fixes and improvements have been going in to everything from
> the kernel down to iPXE.  It's our strong preference to do things that
> way vs build it in-house (I know, you're probably not used to
> associating such a thing with Oracle)

Great!

> Why didn't we use cloud-init? For pretty much the same reason the other
> cloud vendors haven't, and there was fair bit of discussion about it at
> the cloud-init summit back in August.

I'm not sure what "other cloud vendors" you're referring to, but AWS use
it, and so are the vast majority of OpenStack based services (according
to Jeremy, that's about 72 cloud providers).

> RedHat doesn't backport features.

Neither does Debian in Stable. We have stable-backports, but it is the
view of the cloud team that we prefer to avoid using backports in our
images if that's possible.

> The Redhat based distributions are running comparatively ancient
> versions of cloud-init, and we can't inject newer versions in the images
> used on our platforms, or we fall afoul of their branding rules and
> wouldn't be able to call it "CentOS", for example.

You will have the very same problem with Debian (except maybe that
cloud-init isn't that old? I'm not sure what you call old, so I can't
tell about this specific point...)

> What you _can_ do
> there is add in extra packages, to a limited degree, so it's perfectly
> fine to add an instance agent, it's not okay to add in an updated
> version of cloud-init.

Our view is that an official Debian image must contain *only* packages
which are available in Debian. - full stop -

So, if there's a way to have an image work with cloud-init, then there
is a chance that we can produce a Stretch official image at some point
for the Oracle cloud. Otherwise, your only hope is to have your agent
packaged in Debian, or simply not hope for having the word *official*
stamped (ie: it's going to be your own version of Debian).

> When they added cloud-init to the redhat
> repositories, they basically tied all the cloud-vendor's hands. CentOS,
> RedHat etc. make up a large slice of the Linux server market, so if we
> relied on cloud-init for this runtime configuration, we'd be stuck
> without important feature support for customers.

I don't think it is reasonable to take CentOS / Red Hat as an excuse to
not make the Debian image right (sorry to be very strait when stating
this, but that's really my feeling).

> The most frustrating
> part of that, of course, is that cloud-init only serves one purpose.

What do you mean? Cloud-init does *many* things, not just one.

> Customers that use it are all going to be customers that are operating
> in some kind of cloud environment.

Do you mean owning a private cloud deployment?

> There are also other considerations such as Windows

In my company, we do have some Windows images available, and they do run
with cloud-init to setup the initial admin password. It works reasonably
well, as much as I know.

> feature release co-ordination etc

What kind of feature release co-ordination is needed?

> runtime features

The more I think about it, the more I see the need of having a daemon
running in cloud-init, so that we could add runtime stuff, indeed.

> one thing Google's instance agent
> supports is injecting new SSH keys to instances, which cloud-init as a
> "run and done" doesn't support.

Right. I don't see this as a key feature though. It's trivial automation
to add more keys when the VM is up.

>> If your question is:
>> - can I do: "curl $foo | bash" at startup and hardwire it in rc.local
>>
>> then my answer would be: of course not! Please do things properly, have
>> your agent packaged and uploaded to Debian. We have a bit more than a
>> month to have this happen if you don't want to miss Buster.
> curl | bash :shudder:  I hope I never see us do that.

Maybe I didn't make myself clear. The "curl | bash" is in fact what I
would consider you doing, if you're installing an agent which isn't
packaged and available in the standard Debian repository. Because the
thing you'd be installing would do whatever you decide, and the users
would have no way to know when it's going to be updated and what it does
under the hood.

>>> I'd like to make sure we're singing from the same hymn sheet. What's the
>>> route forwards?  Do we do the work and provide patches for review, do we
>>> need to be contracting with the Debian group to get the work done, or is
>>> the route forwards something else entirely?
>> IMO, the route forward is:
>> 1/ Get your instance agent packaged, and give me (or someone else) the
>> link to the source package for review. Probably best would even be if
>> the packaging was done in Salsa as a new repository here:
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__salsa.debian.org_cloud-2Dteam_&d=DwIFaQ&c=RoP1YumCXCgaWHvlZYR8PZh8Bv7qIrMUB65eapI_JnE&r=ZscEEr9nw3LSVvXoPtSeEPOB99CD0eOeziaYFoP_AX8&m=fKlH8BO1tnqF-BJl-j4mhx-XlF_dxhnsAcaM6bcN2NY&s=_MGmkWEn1vFQNQ4hd4i6CtE01S5OzcCNa-SpyGabxoI&e=
>>
>>
>> (or in another group, but there is fine...)
>>
>> 2/ Get patches done to debian-cloud-images so it produces an Oracle
>> image.
>>
>> 3/ Get Steve McIntire (aka: Sledge) to also publish your image in our
>> web site.
>>
>> 4/ Ultimately, get in touch with SPI once all of the above is done, so
>> that you can give a login / pass to Oracle, so that when we produce a
>> new Debian image (maybe because of a point release, or a security
>> update) then it gets uploaded automatically in your could. We can deal
>> with that last part later.
>>
>> I hope this helps,
>> Cheers,
>>
>> Thomas Goirand (zigo)
> 
> That's awesome, thanks.  We'll get working on it.  I'll speak to our
> instance agent devs about 1), I've already dropped them links and
> started a conversation with them on slack.

You can also tell him/her to reach me on IRC. I'm zigo in both OFTC and
FreeNode.

> The others will probably
> come back to me in one form or the other.  I'll get started and see what
> I can produce.

Exciting ! :)

Thomas Goirand (zigo)