Re: how to get list of debianmed packages
On Tue, Mar 15, 2016 at 09:09:15AM +0100, Ole Streicher wrote:
> >
> > I'm not sure whether we really need things like description translations
> > in this machine readable form since it would possibly increase the size
> > of the json file by one order of magnitude (at least I *hope* that we
> > have so many translations ;-)).
>
> I would recommend to include them. One use of such a file is to generate
> some other interface for the web or so.
>
> BTW, in the moment there is (inofficially, for debugging purposes)
> already a JSON generated, however for the whole debian-med blend
> together: http://blends.debian.org/med/tasks/tasks.json
Good. :-)
> This file is currently 13 MB (2,5 MB gzipped); containing all available
> languages. Since it is just a thumb dump of the internal structure, it
> even contains everything twice. This is not really the size where "an
> order of magnitude" is an issue.
ACK.
> Once we finish the two current tasks of the blends sentinel pages
> (completely switching to UDD, move to Python-3), I think it would be
> worth to clean this up; this however needs some change of the internal
> structure of the webtools.
Yes. When I rewrote bugs.py (which was formerly a mix from reading
tasks files + UDD as well but is UDD exclusively) I had the json export
in mind as well. This resulted in using Python dictionaries exclusively
which makes the code (even ;-)) less readable and thus I did not put
some preference for tasks.py on this internal structure for the moment.
> @Andreas: could you have a look for the differences between the UDD and
> non-UDD parts and tell me the current problems of the UDD? I'd volunteer
> to fix them; after that we could switch to Python-3, and then I am open
> (as time allows) to look into a better json.
I tried your converted tasks_udd.py and after fixing a simply syntax
error I was running into:
$ tasks_udd.py debian-med
Traceback (most recent call last):
File "./tasks_udd.py", line 28, in <module>
tasks.GetAllDependencies()
File "/home/andreas/debian-maintain/alioth/blends_git/website/webtools/blendstasktools_udd.py", line 895, in GetAllDependencies
if td.GetTaskDependencies(source):
File "/home/andreas/debian-maintain/alioth/blends_git/website/webtools/blendstasktools_udd.py", line 1279, in GetTaskDependencies
self.GetDepInfo(curs, dependencies, 1)
File "/home/andreas/debian-maintain/alioth/blends_git/website/webtools/blendstasktools_udd.py", line 1231, in GetDepInfo
dep.remark['long'] = MarkupString(long.encode('utf-8'), dep.pkg, 'RemarkLong')
File "/home/andreas/debian-maintain/alioth/blends_git/website/webtools/blendsmarkdown.py", line 114, in MarkupString
% (elem, pkg, lang, 'debug-string', str(err)))
TypeError: not all arguments converted during string formatting
This does not happen when rendering debian-astro.
If you have both outputs ("classic" and "udd") you can compare these
using test_output_udd.py.
The main issue I was working on is that the prospective packages which
are not yet in VCS (== exclusively mentioned inside the tasks files) are
not yet rendered. The data are gathered into UDD properly and are
residing inside the table blends_unknown_packages but not rendered yet.
Moreover it needs to be checked whether the remarks are properly
rendered. The blends_query_packages() function should contain this
information (it was added right before upstream_bugs and
upstream_repository) but I remember vaguely some issue with this.
I would need to check again.
> > Its generated from UDD twice a day. Currently some data for prospective
> > packages come from the original tasks files but I'm busy changing this
> > right now to get everything out of UDD. Considering this I admit that
> > I'm a bit hesitating to create this static json interface if you have a
> > quite flexible SQL interface in UDD which has all the relevant
> > information.
>
> The UDD ist quite difficult to use. Look into blendstasktools.py to see
> what an amount to code is needed to get it into a usable form. In
> contrary, the web pages could be generated from a JSON file with just
> applying the template.
I agree to some extend but I'd volunteer to inject handy SQL functions
into UDD to reduce complexity for certain tasks.
> >> In a more general way, it would be nice to query Debian archive from
> >> remote systems (get files in a packages get package info, search
> >> packages with X or Y etc...). We have information from many web
> >> sites/pages but it could be usefull to get all those info available
> >> via an easy web API. It would help users willing to integrate some
> >> Debian stuff into their tools.
>
> do you volunteer here? ;-)
:-)
> > There are examples for web APIs like the Maintainer dashboard[1] and it
> > would be probably not to hard to craft something Blends related follwing
> > this example.
>
> In principle, many of our problems are not specific to blends at all. I
> have the feeling that we are just a bit more concerned about our users :-)
+1
Alternatively we should run around and ask Debian developers: Did you
created your Blend today? At least this was the suggestion I've received
in my talk at DebConf13[1] :-) (and no Asheesh did not followed his own
advise yet - I've asked him at this years DebConf ;-))
Kind regards
Andreas.
[1] http://saimei.acc.umu.se/pub/debian-meetings/2013/debconf13/high/987_How_to_attract_new_developers_for_your_team.ogv
--
http://fam-tille.de
Reply to: