Re: Downloading all information in JSON format
On Sat, Jan 30, 2016 at 2:15 AM, Sébastien Delafond <sdelafond@gmail.com> wrote:
> On 2016-01-07, Grant Murphy <grantcmurphy@gmail.com> wrote:
>> - The file has not had whitespace removed (will reduce file size by
>> about 5mb)
>
> I've just pushed a change that does two things:
>
> * uses the most compact JSON representation; measured against the
> "stable-only" DB, it cuts down the download size around 30%, from
> 3.6Mo to 2.5Mo.
>
> * removes the hash sorting altogether; this brings the execution
> time from 3.7s to 2.5s ("stable-only" once again), which should
> hopefully help with the load on security-tracker.
>
> Those are ballpark figures, of course, and we'll need to see how it
> behaves against the full DB.
Cool thanks. I'm currently getting a 404 from this URL -
https://security-tracker.debian.org/tracker/data/json is this related?
>
>> - The file is not compressed (gzip will reduce __total__ file size
>> to ~2mb)
>
> Not sure why that stream isn't gzipped, will look into that.
>
That would be awesome. I haven't had a chance yet to dig into the code
but try to look into it also.
>> - It is generated per request
>> - The 'If-Modified-Since' header is not honored so the complete file
>> will need to be re-downloaded every time to check if it has
>> changed.
>
> Good points. I'm not too sure about the technical implications related
> to the integration of those two features, but your contribution would
> in any case be most welcome :)
>
> Cheers,
>
> --Seb
>
Reply to: