[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#961142: marked as done (snapshot.debian.org: python3 migration)

Your message dated Sun, 18 Sep 2022 11:03:20 +0200
with message-id <Yybe2MKtjab2F+/m@jcristau-z4>
and subject line Re: Bug#961142: snapshot.debian.org: python3 migration
has caused the Debian Bug report #961142,
regarding snapshot.debian.org: python3 migration
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@bugs.debian.org
immediately.)


-- 
961142: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=961142
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems
--- Begin Message --- 
Package: snapshot.debian.org
Severity: normal

Hi,

As discussed previously with pabs and weasel, I'm interested in taking
on the python 3 migration for snapshot.

With bullseye, the goal is to drop python 2 packages. As such the
framework used by snapshot, pylons, will be removed as the project is
mostly dead and not python3 compatible.

I'm willing to port snapshot to python 3, replacing the pylons framework
in the process.

After doing a bit of research on snapshot and how it works, I believe
that switching over to the flask framework [1] would be a fitting
option. flask is a lightweight python web framework, well maintained and
packaged in Debian.

That would allow us to keep the model part of snapshot, still using
psycopg2 as the driver to communicate with the postgreSQL backend.
Pylons specifics parts will have to be replaced with flask logic and
templates will have to be converted to jinja2.

Technically, I propose we use salsa to work our way on that migration:

- I port a small set of code to python3/flask on my fork
- I add the required tests to cover that set
- I submit a merge request to the snapshot project
- You review and merge the small set to a migration branch
- We go back to step 1 until migration is over

We can use this bug report to tracker progression or to exchange on
technical topics.

Once we are ready to migrate the production, you can merge the migration
branch to master.

To facilitate the migration, I do intend to cover the entire rewrite
with testing with stable+bpo and unstable as targets.

For easier communication, we could also create a #debian-snapshot on
OTFC. Else, I'm available on #-qa or #-admin.

Looking forward to your comments,

Best,
-- 
Baptiste BEAUPLAT - lyknode

Attachment: signature.asc
Description: OpenPGP digital signature


--- End Message ---
--- Begin Message --- 
This is now merged and deployed.  Thanks!

Cheers,
Julien

On Fri, Jul 16, 2021 at 12:31:25PM +0200, Baptiste BEAUPLAT wrote:
> 
> Hi,
> 
> After a couple of month, the migration from python 2 and pylons to
> python3 and flask is finally done. I've created a merge request
> available at:
> 
> https://salsa.debian.org/snapshot-team/snapshot/-/merge_requests/1
> 
> The following describe what changed in snapshot:
> 
> Snapshot script
> ===============
> 
> Ruby 'dbi' driver has been replaced by 'pg' ('dbi' was removed from
> unstable in 2016). Main differences between both driver are:
> 
> - 'pg' require use of positional arguments
> - renamed function/attributes (do/exec, ntuples and cmd_tuples)
> 
> I've tested the script both manually and using the frontend test suite.
> 
> Snapshot configuration
> ======================
> 
> Since ruby 'pg' module understand connection string, I've removed
> the duplicated yaml block information.
> 
> DB python upgrade scripts
> =========================
> 
> In addition to being converted to python 3, I've updated the scripts so
> they can be run either as a standalone script or imported from another
> python application. This allow running upgrade scripts from the frontend
> test suite, when provisionning the test database.
> 
> Other python scripts (frontend excluded)
> ========================================
> 
> I've updated all other python scripts that appeared releavent to me,
> including scripts in the following directories
> 
> - fsck/
> - master/
> - mirror/
> - fuse/
> 
> In fsck, I've also updated the C module to python 3.
> 
> Without doing extensive and exhaustive testing, I did run the script to
> validate they primary use cases.
> 
> Dependencies
> ============
> 
> I've updated the dependency list in README.md to match bullseye package
> versions (For fuse, ruby, scripts, frontend and fsck).
> 
> Frontend
> ========
> 
> Framework and layout
> --------------------
> 
> I've used the Flask framework to implement the frontend. That's a
> minimalist framework, more simple and without any defined pattern, as
> opposed to bigger framework such as Django.
> 
> It's a good fit in snapshot case because the backend is not in the
> traditionnal form of a "Model", using a proper ORM, but instead is
> already optimized, specific and implemented using PostgreSQL (psycopg2).
> 
> Flask provides all necessary tools to build around snapshot existing
> model implementation, adding the frontend logic and the templates pages.
> 
> The following directories exposes the layout used for the frontend:
> 
> - `docs/`, deployment documentation and file header
> - `public/`, root web folder
> - `snapshot/__init__.py` `wgsi.py`, entry point
> - `snapshot/views/`, frontend logic, contains all routes
> - `snapshot/controllers/`, intermediary code, between views and models
> - `snapshot/models/`, snapshot psycopg2 model
> - `snapshot/settings/`, application configuration
> - `snapshot/templates/`, jinja2 templates
> - `snapshot/static/`, static content
> - `snapshot/lib/`, generic code
> - `tests/conftest.py`, test entry point
> - `tests/controllers/`, test controllers
> - `tests/data/`, test static data (package template, keyring, dataset)
> - `tests/unit/`, unit and api tests
> - `tests/functional/`, user/tools oriented tests (apt, debsnap)
> 
> Additionally, the frontend root folder contains licensing,
> documentation, packaging and testing files.
> 
> Settings
> --------
> 
> Snapshot frontend settings are loaded from
> `snapshot/settings/snapshot.py`. This file does not exist in the
> repository and is intended to be a link to whatever environment
> configuration file you choose (`develop.py` or `prod.py`).
> 
> All environment configuration files load `common.py` where all defaults
> are defined, and overrides them when necessary.
> 
> Behavior changes
> ----------------
> 
> My goal for this migration is to provide a seamless transition for
> snapshot users. Meaning I've re-implemented as close as I can the
> behavior of the old frontend.
> 
> Generated html/json, minus the space indentation, should be identical to the
> previous version.
> 
> Exception from a couple of minors bugs fixed, I'm not aware of any
> behavior changes from the previous version.
> 
> Testing and QA
> --------------
> 
> I've added a fairly big test suite to ensure that the rewrite does not
> comes with any regression. The test suite is written using pytest and
> provide almost a full coverage of the code (grep `pragma` to see
> excluded code and why).
> 
> The test suite is working as described:
> 
> - Load testing settings from `snapshot/settings/test.py`
> - Create a temporary PostgreSQL database (ASCII encoded)
> - Import snapshot schema (using db/ files)
> - Build testing packages and create pooled archived using data from
>   `tests/data/dataset.py` and `tests/data/templates` (aptly is used to
>   create the archives)
> - Import and index the archives using the snapshot script
> 
> Once all those step are done, a snapshot of the database is made and
> restored before each test.
> 
> Unit and functional tests are run. Unit tests covers all routes while
> functional tests aims to cover user use cases (using snapshot with apt
> or debsnap).
> 
> Additionally, a flake8 checker is run and a branch coverage report is
> generated in web/app/htmlcov/.
> 
> The test suite is either run by:
> 
> - invoking the `tox` command. It will take care of settings up an
>   isolated environment to test with (postgresql is still a requirement)
> - Pushing commits to the repository, that will trigger the CI
> 
> Deployment
> ==========
> 
> Although, it requires some adjustments, I haven't touched the
> deployments files (speaking of `etc/apache.conf` and `web/deploy`).
> 
> From what I can gather, the current frontend setup is split between
> several directories:
> 
> - /srv/snapshot.debian.org/web/public/static/, for static files
> - /srv/snapshot.debian.org/web/public/, for web root
> - /srv/snapshot.debian.org/bin/, for the WSGI script
> - /srv/snapshot.debian.org/web-app/, for the actual frontend app
> 
> It looks like public/ and the WSGI script are actually links to the
> web-app subtree.
> 
> All of that is split from the local copy of the repository (living in
> `/srv/snapshot.debian.org/code/`).
> 
> Do you want to use the same setup for the new version? I usually expose
> directly the git repository, aliasing /robots.txt, /static and / (to the
> wsgi script) and it works. However, that technique might not fit
> security requirements for snapshot.d.o.
> 
> In any case, I've removed the egg-info directory from the repository
> (being generated files), so they would have to be generated by the
> deploy script using a `setup.py sdist`.
> 
> What's next
> ===========
> 
> Let me know how to proceed from here. I suppose you will want to review
> the code; I'm looking forward to your feedback on that.
> 
> Then, do you want to host that version, parallel to the previous one for
> a couple of weeks to allow testing by other people, before switching?
> 
> Let me know how can I be of any help.
> 
> I do intend to address bugs listed for snapshot.debian.org in the BTS
> and build on this new version of snapshot.
> 
> -- 
> Baptiste Beauplat - lyknode

--- End Message ---
Reply to: