On 2/28/22 06:41, Ross Vandegrift wrote:
Thanks! Bastian, do you remember how much artifact storage we use? IIRC, it's surprisingly large. salsa is still down at the moment, so I'm unable to check.
I'm not sure what you mean by "large". These days, we setup servers with 12x18TB each, 6 servers at a time (because of 3 availability regions, and 2 zones per region). As data are replicated 3 times in swift, that's 432 TB of data per batch of 6 servers.
As we also offer 2 other storage services (swiss backup and kdrive), which we bill cheaper (by the month instead of by the hour), our public cloud Swift is not used much.
All this to say: we have plenty of space: hundreds of TBs if needed... The only thing is: it's best to store everything in Swift, on mechanical hard drives object store, rather than on expensive (Ceph backed) NVMe block storage. Though I believe Gitlab can handle that.
If we don't use Infomaniak public cloud for this, maybe we can use VMs to run our jobs (for gitlab runners?) if needed.
But, this is problematic not only for the cloud team. Let's hope this gets fixed "soon", no? Maybe we should set a deadline for ourselves?100% agreed. I don't think we need to set a deadline yet, but I think we should continue this conversation so we can build opinions about our options.
The ETA on IRC says 24 hours from now. That's IMO acceptable... Even though the situation could have been handled a way better (like with explanation of what was going on, which is the bare minimum in any standard...).
Waldi, if it's you who triggered the current events, thanks for it, indeed, it was kind of dangerous to keep things as they were.
Let's hope this all solve soon. Cheers, Thomas Goirand (zigo)