Re: Archive changes

["Thomas Schmitt" <scdbackup@gmx.net>]

Hi,

Joerg Jaspert wrote:
> > > As of now, InRelease/Release files, Packages and Sources no longer
> > > provide MD5Sum and SHA1sums, only SHA256.

Steve McIntyre wrote:
> > We can't fix this easily in a short
> > timescale - current released jigdo clients (both in Debian and
> > externally) use md5 internally to reference files in the archive.

Joerg Jaspert wrote:
> Im not sure we *want* to support that, at least for sure not for more than
> stretch, but we could.

> > We need more time than that to fix things up.

> How much?

Currently Jigdo is produced by xorriso, libisofs, and libjte.
libjte upstream is Steve, with George Danchev and me as co-authors.
libisofs and xorriso upstream is me.

I only know of consumer jigit. Are there others (externals) ?

------------------------------------------------------------------
xorriso:

- Calls API function libjte_set_md5_path().
  Either there needs to be a new call for the SHA256 list file path,
  or we keep this function as half misnomer and let the MD5 file reader
  of libjte recognize SHA256 automatically.

libisofs:

- Seems not to refer to the checksum type.
  (Let's hope my assessment is true.)

libjte:

- Offers API call  libjte_set_md5_path().
  As mentioned above, one could let it recognize MD5 and SHA256 list
  content so that the callers do not have to care for the change.
  Only the runner of xorriso (debian-cd, i assume) would have to
  submit a suitable SHA256 list file instead of the old MD5 list.

- Internal function parse_md5_list() and its subordinates read
  the given list file and create a list of md5_list_entry_t.
    typedef struct _md5_list_entry
    {
        struct _md5_list_entry *next;
        unsigned char       MD5[16];
        uint64_t size;
        char               *filename;
    } md5_list_entry_t;
  This struct and the reader would have to be adapted to the new
  checksum list format.

- The equipment for SHA256 computation seems already present.
  At least there is a file sha256.c.

- Internal call list_file_in_jigdo() makes the decision, whether
  a data file in the future output matches a listed file item by
  basename and by MD5.
  This function needs to learn to process SHA256.

- Some message texts talk of "MD5" hardcodedly. There we'd need a
  variable text snippet.

- Internal function write_template_desc_entries() in one of its
  cases writes a jigdo_file_entry_t to the .template file.
    typedef struct _jigdo_file_entry
    {
        unsigned char type;
        unsigned char fileLen[6];
        unsigned char fileRsync[8];
        unsigned char fileMD5[16];
    } jigdo_file_entry_t;
  This will cause a layout change in the .template file.
  (Jigdo consumer jigit has Steve as upstrem.)

- Internal function add_file_entry() records a file_entry_t for
  each matching file:
    typedef struct _file_entry
    {
        unsigned char       md5[16];
        off_t               file_length;
        uint64_t            rsyncsum;
        char               *filename;
    } file_entry_t;
  Internal function write_jigdo_file() writes the MD5 of the .template
  file and the MD5s of the file_entry_t (as union member of entry_t).
  This will cause a layout change in the .jigdo file.

- The most probably unused "Traditional Data File API" deals with MD5
  sums computed from data files:
    int libjte_decide_file_jigdo(struct libjte_env *jte_handle,
                             char *filename, off_t size, char **mirror_name,
                             unsigned char md5[16]);
    int libjte_write_match_record(struct libjte_env *jte_handle,
                            char *filename, char *mirror_name, int sector_size,
                            off_t size, unsigned char md5[16]);
  One should consider to give up this API in favor of the
  "Simplified Data File API" as used by libisofs.

There are many more occurences of [Mm][Dd]5 in jte.c and libjte.c.
They seem to be consequences of the gestures described above.
All of them have to be checked and adapted, of course.

------------------------------------------------------------------
Summary:

Program jigit and library libjte would need a thorough overhaul.
The higher levels inside xorriso could stay as they are.

The impact on external Jidgo tools has to be judged by their
maintainers.

The effort in libjte would be at least one week of dense work and
testing. A substantial part of this is the plight to analyze the
old code which none of us touched in the last 5 years.


Have a nice day :)

Thomas