Open libvorbis CVEs

[Guido Günther <agx@sigxcpu.org>]

Dear xiph maintainers,

As part of fixing the open CVEs of vorbis in LTS I looked at:

* CVE-2017-14633
  https://gitlab.xiph.org/xiph/vorbis/issues/2329
  As far as I understadn things the maximum number of channels is
  hardcoded in vorbis:

    https://github.com/xiph/vorbis/blob/master/lib/backends.h#L132 

  so guard for this:
  
    https://github.com/xiph/vorbis/pull/34

  Theora doesn't check the result of that function so account for that
  as well:

    https://github.com/xiph/theora/pull/6

  Does this make sense?

* CVE-2017-14632 

  That's basically the second fix of the above PR
  https://github.com/xiph/vorbis/pull/34/commits/d5dbde2686b3e68c858d1a7033852b45d74bbd52

* CVE-2017-11333
  https://gitlab.xiph.org/xiph/vorbis/issues/2332
  
  The invocation there also hits the above chmuxlist limit. With the fix
  for CVE-2017-14633 applied sox needs to handle errors returned by the
  function as well:
  
    https://sourceforge.net/p/sox/mailman/message/36130427/

  (which Debian's sox maintainer picked up very quickly and forwarded
  it, thanks Måns).

So the above issues could be fixed by uploading vorbis, theora and
sox. However we have many more programs invoking the above function but
not handling errors returned by it (among them firefox and
gstreamer). It's unlikely that we break things for decoders reading
vorbis since the limit can only happen on encoding not decoding but I
might be overlooking something.

* CVE-2017-14160
  https://gitlab.xiph.org/xiph/vorbis/issues/2330

  I've not been able to find the root cause for this yet. I've posted
  a patch that makes the code more robust by also checking the loop
  variable but I'm yet lost about the real reason. Any ideas would be
  welcome.

Cheers,
 -- Guido