Re: disk space on ullmann.d.o / UDD cron disabled
On 19/08/21 at 16:29 +0100, Adam D. Barratt wrote:
> On Thu, 2021-08-19 at 11:37 +0200, Mattia Rizzolo wrote:
> > On Sat, Aug 14, 2021 at 11:20:47AM +0100, Adam D. Barratt wrote:
> > > ullmann.debian.org, the host for udd.d.o, filled its PostgreSQL
> > > partition overnight. DSA added another 5GB of space, but within a
> > > few hours around 3GB of that had already been used.
> >
> > Uh.
>
> That's similar to the reaction I had. :-)
>
> > > Looking at the PostgreSQL logs, it appears that auto-cleanup of
> > > deleted
> > > records in some of the tables hasn't been working correctly for a
> > > while, which may be adding to the space usage.
> >
> > I see (also through the #debian-admin logs) that the space got under
> > control after a few runs of the autovacuum.
>
> It's certainly better, but still a fair way above where things were a
> week ago.
>
> Judging from
> https://munin.debian.org/debian.org/ullmann.debian.org/df.html , the
> partition was around 50% used for the past month or so (~25-30GB).
> Starting on Thursday night, there was a gradual but sustained increase
> in usage until it filled in the early hours of Saturday morning and the
> additional 5GB was added.
>
> We're now at ~65%, _after_ the rounds of cleanup (both manual and
> automatic).
My guess would be that it's a combination of:
1/ the release of bullseye (and the addition of bookworm), which
explains the bump mid-august (there's a similar bump for the release of
Ubuntu 21.04)
2/ the usage pattern of postgresql by UDD, with large transactions that
mass-delete/insert: maybe autovacuum are not sufficient in that case,
and some fragmentation remains.
I ran a VACUUM FULL ANALYZE to clean up all stale data, and got from
~41 GB used down to 13 GB. (for example, bugs_usertags went from 10 GB to
25 MB).
I think that the lesson here is that from time to time, a VACUUM FULL
ANALYZE is needed...
Regarding disk usage of /srv (the question was raised on IRC):
The breakdown is:
15G /srv
9.8G /srv/udd.debian.org
3,4G /srv/udd.debian.org/udd
3.2G /srv/udd.debian.org/udd/web/dumps/: DB dumps to export
to udd-mirror. already compressed.
2,9G /srv/udd.debian.org/testing-status: I don't remember the role
of those files. I xz'ed those that weren't. Now down to
1.6G.
1,6G /srv/udd.debian.org/mirrors: local temporary mirrors, no problem here
other: home directories?
I suspect that there's someone with a large homedir, but I can't explore
this.
Someone made backups on Aug 11th in /srv/udd.debian.org/udd/web/dumps/,
they could be removed.
All in all, I think that we can re-cron everything and watch how things
go. Adam, Mattia, what do you think?
Lucas
Reply to: