Re: Downloading all information in JSON format

[Sébastien Delafond <sdelafond@gmail.com>]

On 2016-01-07, Grant Murphy <grantcmurphy@gmail.com> wrote:
> - The file has not had whitespace removed (will reduce file size by
>   about 5mb)

I've just pushed a change that does two things:

  * uses the most compact JSON representation; measured against the
    "stable-only" DB, it cuts down the download size around 30%, from
    3.6Mo to 2.5Mo.

  * removes the hash sorting altogether; this brings the execution
    time from 3.7s to 2.5s ("stable-only" once again), which should
    hopefully help with the load on security-tracker.

Those are ballpark figures, of course, and we'll need to see how it
behaves against the full DB.

> - The file is not compressed (gzip will reduce __total__ file size
>   to ~2mb)

Not sure why that stream isn't gzipped, will look into that.

> - It is generated per request
> - The 'If-Modified-Since' header is not honored so the complete file
>   will need to be re-downloaded every time to check if it has
>   changed.

Good points. I'm not too sure about the technical implications related
to the integration of those two features, but your contribution would
in any case be most welcome :)

Cheers,

--Seb