GSoC 2025 Reflections: Building Salsa Status and contributing to Salsa CI
Greetings everyone!
I participated in the Google Summer of Code 2025 program with Debian
and want to share a recap of what I did this summer. The program
officially spanned June to August, and I have been actively
contributing to Salsa CI since April. Throughout the summer, I
dedicated 450+ hours to development, testing, and collaboration. My
main achievement was creating the Salsa Status page for visualizing
Salsa CI statistics, which is a result of 160+ commits and 9,500+
lines of code. I also contributed to Salsa CI in various areas,
enhancing pipeline speed (caching, auto-cancel) and fixing several
issues, and submitted 17 MRs. Overall, the summer was very productive,
and I’m proud of all that I accomplished.
## Salsa CI statistics (salsa-status.debian.net)
-------------------------------------------------------------
The status page was my most important goal and took 3 weeks of
planning and prototyping, and most of my time went toward it
throughout the summer. It provides both Salsa CI users and the Salsa
CI developer team with visibility into CI performance, helping
identify wasteful practices, broken configurations, and optimization
opportunities across the entire Debian package ecosystem.
In the last 15 days alone, we have recorded 8500+ pipelines across
3500+ projects that have consumed 250,000+ CI minutes. We also spotted
55+ projects that haven’t been updated in the last 2 or more years,
but still run daily/weekly scheduled pipelines that result in large
amounts of traffic and unnecessary wastage of resources. We will be
reaching out to them soon to resolve this.
**Features**: The status page aggregates and presents pipeline and job
statistics, success rates, performance trends, and detailed CI history
with insights across all projects that use Salsa CI. It tracks CI
duration trends and resource consumption patterns to optimize
performance and identify bottlenecks. The aggregated stats are
presented in an organized way and visualized using various charts.
Automated notifications for performance degradation are sent on the
Matrix room to stay informed of regression and dropped success rate of
jobs in real-time.
**Benefits to Salsa/Debian**: Among several benefits that can’t be
listed here, the status page helps in early detection of failures,
controlled feature rollouts, and post-merge regressions, all while
identifying resource wastage of CI resources (multiple long runs, high
failure rates). It helps with focused probing of issues using filters
and allows developers to compare their project trends over time. The
automated Matrix alerts help bring regressions to light in time.
**The backend**: Located at /api/v1, it provides an API and is
responsible for fetching, storing, and serving the data to the
frontend. The pipelines using Salsa CI make an API call with pipeline
identifiers when the `SALSA_CI_ENABLE_STATS` flag is on. The backend
registers the pipeline and inserts their stats and final status into
the database with their jobs, average time, result, project details,
maintainer, etc. I have also implemented cron jobs to aggregate and
insert hourly and daily stats for the pipelines and jobs in the
database. This helps serve them quickly and saves the database from
repeated strenuous calculations. Apart from this, we have cron jobs
for MRs to keep track of whether any merge or feature rollout on Salsa
CI pipeline resulted in regression. The backend sends Matrix alerts
about regressions to https://matrix.to/#/#salsa-stats:matrix.org
**The frontend**: Built using React, and the design is inspired from
Dribbble. It was a challenge to make the design look good yet keep the
implementation simple and easy to understand, but I feel I did a good
job. The status page contains mainly 4 pages:
Home Page - https://salsa-status.debian.net
- Header shows global stats for all-time pipelines, jobs, projects, and CI time.
- Interactive line chart with customizable durations (24h, 7d, 30d,
6m, 1y) visualizing total, passed, and failed pipelines with
switchable success/failure rate mode.
- MR avatar markers on chart lines (hover for MR details).
- Job Types table with descriptions of jobs and 30-day success rate sparklines.
Pipelines Page - https://salsa-status.debian.net/pipelines
- Trend line chart and real-time pipeline grid.
- Pipeline cards show ID, Gitlab-style status, timestamps, project
details, and Salsa link.
- Sortable by status/time/duration and filterable by status, duration
ranges, and date ranges.
Job Types Page - https://salsa-status.debian.net/job-types
- Treemap showing job type quantities and success rates by stages
(provisioning, build, test).
- Grid view with descriptions, sparklines, and 30-day stats (runs, CI
minutes, success rate, average duration).
- Clicking job types shows a detailed page with line chart and jobs grid.
Projects Page - https://salsa-status.debian.net/projects
- Grid view of all registered projects using Salsa CI.
- Cards show version (from pushed tags), last activity, maintainer
info (from d/control), and last 30 days stats.
- WIP: Project's CI config page.
The dashboard was initially built to be hosted on Gitlab pages and
backend on Render (render.com), but thanks to Otto, both frontend and
backend are now at salsa-status.debian.net. The name was kept generic
intentionally so that in the future, we may add more stats about Salsa
in general and not just be limited to Salsa CI.
The MRs on Salsa CI for enabling statistics:
!622: Enable optional pipeline statistics reporting to Salsa CI
Dashboard for public projects
https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/622
!638: Update dashboard statistics API URL to salsa-status.debian.net
https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/638
Perfecting it took time and is a result of 160+ commits and 9,500+
lines of code, all of which you can see at:
Frontend: https://salsa.debian.org/44yu5h/ci-dashboard
Backend: https://salsa.debian.org/44yu5h/salsa-ci-dashboard-backend
(These will be moved to the Salsa CI team namespace soon.)
Many of you may remember my earlier mail introducing the Salsa Status
dashboard. I’d be grateful for feedback on which statistics developers
find most useful (or still missing).
You can share feedback and find the most up-to-date information at
https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/413
## Improve the pipeline’s speed
------------------------------------------
I implemented buildah caching to accelerate image builds, which would
greatly improve contributors’ experience by reducing build times to
~65%. I also introduced auto-cancellation of redundant pipelines to
prevent unnecessary resource consumption in the internal pipelines.
These optimizations reduced execution time and improved efficiency.
MR 634: Implement buildah caching for faster image builds
Link to MR: https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/634
MR 641: Avoid running apt-get upgrades in the jobs
Link to MR: https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/641
MR 610: Replace the old timeout implementation with new
RUNNER_SCRIPT_TIMEOUT variable
Link to MR: https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/610
## Bugfixes
----------------
In addition to my main projects, I was also an active member on the
Salsa CI Team, where I fixed various issues, helped others with
debugging issues, and provided code reviews and help in testing.
Job Failure due to failure to pull docker images on parallel CI pipeline runs
Issue: https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/289
Fixed in MR: https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/615
Support piuparts and blhc arguments containing spaces or quotes
Issues:https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/127
and https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/325
Fixed in MR: https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/636
Use filter-output for blhc job and return artifacts for piuparts
Issue: https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/225
and https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/299
Fixed in MR: https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/612
**Other bugfixes/contributions**:
!593 (#427): Add zstd installation to missing-breaks job
!611 (#438): Fix autopkgtest images cleanup: use STABLE instead of RELEASE
!594 (#365): Clean Images job: Add missing arm64v8 images
!595 (#433): Clean Images job: Update Ubuntu release from oracular to plucky
!625 (#455): Revert "Workaround reprotest bug #1108550 by disabling
some variations"
!601: Docs: Add instructions for uploading public GPG key to keyserver
**Resolved old inconclusive issues**:
Issue: Autopkgtest pipeline doesn't save autopkgtest artifacts to the
salsa-ci artifacts
Link: https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/417
Issue: Reprotest fails chown on disorderfs
Link: https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/26
**Other side-work on pipeline**:
I worked on ways to skip parts of or the whole pipeline for faster
determination of outcomes. These are still some way from finding their
way into the main repo.
To skip extract-sources and the build jobs, for debugging tests when
the tests don’t necessarily depend on the build job (like autopkgtest,
piuparts, etc.); this can help greatly reduce iteration cycle time.
Link to it: https://salsa.debian.org/44yu5h/salsa-ci/-/commit/db76bce4806c4a41a02a5951a22bd7b1d13fcb52
To skip the pipeline on changes to some particular files.
Link to it: https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/437
#### Reflections
-----------------------
The initial few weeks were the most challenging for me. Getting
familiar with a large and established codebase, learning the
community's best practices, and connecting with new people all took
considerable time and effort. At that stage, even tasks that seem very
simple in hindsight took longer to complete, since I was still piecing
together how everything fit. With time and support from my mentor and
the Salsa CI Team, I was able to work more independently and gradually
pick up speed.
When I started the program, I had little experience with CI pipelines.
I learned a lot by working hands-on, experimenting, and learning from
mistakes. While not all the issues I worked on made it into my key
achievements, I learned something from each one. While building the
stats dashboard I learned how to design a database that can grow with
more data, deal with some of the quirks of Gitlab’s API, and tune the
backend so it stays efficient while keeping results up to date. On the
frontend, I got much more comfortable with React and data
visualization. I also realized how critical it is to keep performance
in mind early, since even small inefficiencies multiply quickly at
scale.
Working remotely, I wasn’t able to attend DebConf in person, but I’ve
been actively involved online with the Salsa CI team and the local
India community!
#### Acknowledgments
--------------------------------
I want to take a moment to thank my mentor, Otto Kekäläinen, for all
the guidance and patience (especially when I managed to break things
:). I’d also like to thank Santiago Ruano Rincón, Ahmed Siam, and
Aquila Macedo for their support and collaboration during the summer.
Santiago and Ahmed offered valuable reviews and feedback within Salsa
CI, while Aquila, a fellow GSoC contributor, was both a great
collaborator and a generous reviewer of my MRs.
Looking back on this summer, I’m proud of what I have accomplished.
The Salsa Status website now provides valuable insights for the Debian
community. I’m grateful for the support and collaboration throughout
the summer from Otto and the Salsa CI team, and I look forward to
seeing these contributions continue to benefit Salsa CI and make life
a little easier for Debian developers.
I truly appreciated the connections I built within the community.
Collaborating on shared challenges, exchanging ideas, and thoughtfully
debating different approaches not only helped us move things forward
but also helped foster meaningful bonds along the way.
#### What’s next?
------------------------
The summer has ended, but the adventure hasn’t. Hopefully you will see
me again around somewhere bashing my head on some random bug :p I
intend to stick with Debian and continue my journey to become a Debian
Maintainer.
Thank you so much!
Aayush Raj
My Salsa profile: https://salsa.debian.org/44yu5h
MRs in Salsa CI (12 of 17 merged):
https://salsa.debian.org/salsa-ci-team/pipeline/merge_requests?state=all&author_username=44yu5h
Reply to: