Appstream data not in UTF-8?

[Raphael Hertzog <hertzog@debian.org>]

Hello Matthias,

tracker.debian.org has been failing to import the appstream metadata for a
while (since November 26th) with this exception:

Task UpdateAppStreamStatsTask failed with the following traceback.

Traceback (most recent call last):
  File "/srv/tracker.debian.org/distro-tracker/distro_tracker/core/tasks/base.py", line 378, in run_task
    task.execute()
  File "/srv/tracker.debian.org/distro-tracker/distro_tracker/core/tasks/base.py", line 256, in execute
    call_methods_with_prefix(self, 'execute_')
  File "/srv/tracker.debian.org/distro-tracker/distro_tracker/core/utils/misc.py", line 44, in
call_methods_with_prefix
    method(*args, **kwargs)
  File "/srv/tracker.debian.org/distro-tracker/distro_tracker/vendor/debian/tracker_tasks.py", line 889,
in execute_main
    self._load_appstream_hint_stats(component, arch, all_stats)
  File "/srv/tracker.debian.org/distro-tracker/distro_tracker/vendor/debian/tracker_tasks.py", line 728,
in _load_appstream_hint_stats
    hints_json = get_resource_text(url, force_update=self.force_update)
  File "/srv/tracker.debian.org/distro-tracker/distro_tracker/core/utils/http.py", line 356, in
get_resource_text
    return content.decode(encoding)

Exception Type: UnicodeDecodeError
Exception Value: 'utf-8' codec can't decode byte 0xcc in position 990416: invalid continuation byte
Request data not supplied


Can you look into this and fix the data? (the precise position of the
error in the stream varied over the days, here you have the position as of
today, December 7th 09:36 UTC)

This refers to those two lines:
        url = 'https://appstream.debian.org/hints/sid/{}/Hints-{}.json.gz' \
              .format(section, arch)
        hints_json = get_resource_text(url, force_update=self.force_update)

arch is "amd64", an section is likely "main" but it's not impossible that
it's in one of the other sections.

Thank you in advance for your help!
-- 
  ⢀⣴⠾⠻⢶⣦⠀   Raphaël Hertzog <hertzog@debian.org>
  ⣾⠁⢠⠒⠀⣿⡁
  ⢿⡄⠘⠷⠚⠋    The Debian Handbook: https://debian-handbook.info/get/
  ⠈⠳⣄⠀⠀⠀⠀   Debian Long Term Support: https://deb.li/LTS