Re: Finding new home for our builds and other security sensitive stuff
On Sun, Feb 27, 2022 at 04:14:03PM +0100, Thomas Goirand wrote:
> On 2/27/22 14:09, Bastian Blank wrote:
> > Sadly the problems regarding Salsa did just gain a new level. For those
> > who don't follow debian-private or the monthly meetings of the Cloud
> > team, this is the short version:
> >
> > - The instance was not updated for any of the last nine upstream
> > releases, it is now seven months out of upstream security support.
> > - It is now affected by a critical (aka pre-auth) vulnerability, which
> > leads to expossure of secrets stored in the instance.
Bastian - thanks for staying on top of this issue and taking action. I hope
the current unfortunate situation can improve somehow.
> > I don't see or hear anything that would make me think there will be any
> > meaningful change in maintenance procedures in the future.
> >
> > Our image management stuff uses capabilities of Salsa and also uses it
> > to store the secrets required to do privileged operations on Cloud
> > platforms. Those stored secrets are non-expiring and allow privileged
> > access to our releases on those platforms.
> >
> > After thinking about, I propose two projects:
> > - Move secrets to Vault.
> > - Move the critical projects to a properly maintainer GitLab instance.
> >
> > Using Hashicorp Vault as secrets store allows us tighter controls, like
> > - providing the jobs with temporary access credentials,
> > - restricting from where credentials can be read and
> > - get an audit log when, who, where credentials have been requested.
>
> We use Hashicorp Vault in my company, and we are very happy of it. It works
> well, it's safe, and has many good options. So I support the idea.
+1 - we should talk more about how this would look. I have some thoughts.
We could keep it simple: one VM in an autounseal supported cloud, probably
using a storage backend from the platform.
Or we could get fancy: HA, platform independent storage, and write some
software to automate unsealing. This is hard, though, since it requires secure
secret distribution.
> > Using another GitLab instance is a bit more problematic. Due to the
> > ressources we use, most of the instances out there are kind of out of
> > the question. Which remains is hosting one ourselves. That's not
> > ideal, by far.
gitlab.com could work - they could handle our artifacts, and we could bring our
own CI runners. This might not be popular for a variety of reasons (and I'm
not pushing for it). But I think it's important to note since:
a) it's technically feasible, and
b) it's probably the least effort (both migration & ongoing ops)
And since Debian is listed as a partner in their open source partner program,
it's possible that someone has already worked through exexcuting the legal
agreements we'd need. I'll reach out to their contact and inquire.
> If we need hosting space, we (at Infomaniak) have really enough resources to
> provide it in our public cloud. It work super well (zero issue since its
> launch in September, if we don't consider abusers registering all available
> public IPs), and has very new hardware. Worst case, I can grant you such a
> public cloud account for our operations.
Thanks! Bastian, do you remember how much artifact storage we use? IIRC, it's
surprisingly large. salsa is still down at the moment, so I'm unable to check.
> But, this is problematic not only for the cloud team. Let's hope this gets
> fixed "soon", no? Maybe we should set a deadline for ourselves?
100% agreed. I don't think we need to set a deadline yet, but I think we
should continue this conversation so we can build opinions about our options.
Ross
Reply to: