Bug#1107564: unblock: diffoscope/297
Package: release.debian.org
User: release.debian.org@packages.debian.org
Usertags: unblock
X-Debbugs-CC: reproducible-builds@lists.alioth.debian.org
Dear Release Team,
Please consider unblocking diffoscope 297. This is principally to fix
the use of potentially dangerous functionality in zipdetails
(addressed in diffoscope versions 295 and 206).
Regrettably, a new upload (297) was then made that introduced lzma
comparator support, and NuGet package support was included in 296. We
did not have our mind on the freeze, alas...
However, I foresee no regressions here, as new functionality like this
is fairly well compartmentalised in diffoscope, thus I think it is
safe to include — and not require first rolling back these changes or
similar.
The full debdiff is attached, but the relevant changelog entries are
included inline here:
diffoscope (297) unstable; urgency=medium
[ Will Hollywood ]
* Add a LZMA comparator and tests.
-- Chris Lamb <lamby@debian.org> Fri, 30 May 2025 09:34:34 -0700
diffoscope (296) unstable; urgency=medium
[ Chris Lamb ]
* Don't rely on zipdetails' --walk functionality to be available; only add
that argument after testing for a new enough versions.
(Closes: reproducible-builds/diffoscope#408)
* Disable and then re-enable failing on stable-bpo.
* Update copyright years.
[ Omair Majid ]
* Add NuGet package support.
-- Chris Lamb <lamby@debian.org> Fri, 16 May 2025 08:41:37 -0700
diffoscope (295) unstable; urgency=medium
[ Chris Lamb ]
* Use --walk over the potentially dangerous --scan argument of zipdetails(1).
(Closes: reproducible-builds/diffoscope#406)
-- Chris Lamb <lamby@debian.org> Fri, 09 May 2025 09:13:39 -0700
Best wishes,
--
,''`.
: :' : Chris Lamb
`. `'` lamby@debian.org / chris-lamb.co.uk
`-diff --git debian/changelog debian/changelog
index 47a350bf..95507593 100644
--- debian/changelog
+++ debian/changelog
@@ -1,3 +1,32 @@
+diffoscope (297) unstable; urgency=medium
+
+ [ Will Hollywood ]
+ * Add a LZMA comparator and tests.
+
+ -- Chris Lamb <lamby@debian.org> Fri, 30 May 2025 09:34:34 -0700
+
+diffoscope (296) unstable; urgency=medium
+
+ [ Chris Lamb ]
+ * Don't rely on zipdetails' --walk functionality to be available; only add
+ that argument after testing for a new enough versions.
+ (Closes: reproducible-builds/diffoscope#408)
+ * Disable and then re-enable failing on stable-bpo.
+ * Update copyright years.
+
+ [ Omair Majid ]
+ * Add NuGet package support.
+
+ -- Chris Lamb <lamby@debian.org> Fri, 16 May 2025 08:41:37 -0700
+
+diffoscope (295) unstable; urgency=medium
+
+ [ Chris Lamb ]
+ * Use --walk over the potentially dangerous --scan argument of zipdetails(1).
+ (Closes: reproducible-builds/diffoscope#406)
+
+ -- Chris Lamb <lamby@debian.org> Fri, 09 May 2025 09:13:39 -0700
+
diffoscope (294) unstable; urgency=medium
[ Chris Lamb ]
diff --git diffoscope/__init__.py diffoscope/__init__.py
index 93b0f258..f7a303d4 100644
--- diffoscope/__init__.py
+++ diffoscope/__init__.py
@@ -17,4 +17,4 @@
# You should have received a copy of the GNU General Public License
# along with diffoscope. If not, see <https://www.gnu.org/licenses/>.
-VERSION = "294"
+VERSION = "297"
diff --git diffoscope/comparators/__init__.py diffoscope/comparators/__init__.py
index 6621079d..1c51d080 100644
--- diffoscope/comparators/__init__.py
+++ diffoscope/comparators/__init__.py
@@ -82,6 +82,7 @@ class ComparatorManager:
("java.ClassFile",),
("lz4.Lz4File",),
("lzip.LzipFile",),
+ ("lzma.LzmaFile",),
("mono.MonoExeFile",),
("pdf.PdfFile",),
("png.PngFile",),
@@ -98,6 +99,7 @@ class ComparatorManager:
("ocaml.OcamlInterfaceFile",),
("docx.DocxFile",),
("zip.MozillaZipFile",),
+ ("zip.NuGetPackageFile",),
("zip.JmodJavaModule",),
("zip.ZipFile",),
("image.JPEGImageFile",),
diff --git diffoscope/comparators/lzma.py diffoscope/comparators/lzma.py
new file mode 100644
index 00000000..8836f98a
--- /dev/null
+++ diffoscope/comparators/lzma.py
@@ -0,0 +1,65 @@
+#
+# diffoscope: in-depth comparison of files, archives, and directories
+#
+# Copyright © 2014-2015 Jérémy Bobbio <lunar@debian.org>
+# Copyright © 2015-2020, 2024-2025 Chris Lamb <lamby@debian.org>
+#
+# diffoscope is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# diffoscope is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with diffoscope. If not, see <https://www.gnu.org/licenses/>.
+
+import re
+import logging
+import subprocess
+
+from diffoscope.tools import tool_required
+
+
+from .utils.file import File
+from .utils.archive import Archive
+
+logger = logging.getLogger(__name__)
+
+
+class LzmaContainer(Archive):
+ def open_archive(self):
+ return self
+
+ def close_archive(self):
+ pass
+
+ def get_member_names(self):
+ return [self.get_compressed_content_name(".lzma")]
+
+ @tool_required("xz")
+ def extract(self, member_name, dest_dir):
+ dest_path = self.get_path_name(dest_dir)
+ logger.debug("lzma extracting to %s", dest_path)
+ with open(dest_path, "wb") as fp:
+ subprocess.check_call(
+ [
+ "xz",
+ "--decompress",
+ "--format=lzma",
+ "--stdout",
+ self.source.path,
+ ],
+ stdout=fp,
+ stderr=None,
+ )
+ return dest_path
+
+
+class LzmaFile(File):
+ DESCRIPTION = "LZMA compressed files"
+ CONTAINER_CLASSES = [LzmaContainer]
+ FILE_TYPE_RE = re.compile(r"^LZMA compressed data\b")
diff --git diffoscope/comparators/zip.py diffoscope/comparators/zip.py
index 5801e37d..80dc85ae 100644
--- diffoscope/comparators/zip.py
+++ diffoscope/comparators/zip.py
@@ -32,7 +32,11 @@ from diffoscope.tempfiles import get_named_temporary_file
from .utils.file import File
from .directory import Directory
from .utils.archive import Archive, ArchiveMember
-from .utils.command import Command
+from .utils.command import Command, our_check_output
+
+
+def zipdetails_version():
+ return our_check_output(["zipdetails", "--version"]).decode("UTF-8")
class Zipinfo(Command):
@@ -157,7 +161,19 @@ def zipinfo_differences(file, other):
class Zipdetails(Command):
@tool_required("zipdetails")
def cmdline(self):
- return ["zipdetails", "--redact", "--scan", "--utc", self.path]
+ # See <https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/406>
+ # for discussion of zipdetails command line arguments.
+ #
+ # Older zipdetails does not support --walk; added in Debian
+ # 5.40.0~rc1-1, but "zipdetails --version" shipped in, say, perl
+ # 5.36.0-7+deb12u1 returns "2.104".
+ #
+ # See <https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/408>
+ #
+ if float(zipdetails_version()) < 4.0:
+ return ["zipdetails", "--redact", "--utc", self.path]
+
+ return ["zipdetails", "--redact", "--walk", "--utc", self.path]
class ZipDirectory(Directory, ArchiveMember):
@@ -340,6 +356,13 @@ class MozillaZipFile(ZipFile):
return file.file_header[4:8] == b"PK\x01\x02"
+class NuGetPackageFile(ZipFile):
+ DESCRIPTION = "NuGet packages"
+ CONTAINER_CLASSES = [ZipContainer]
+ FILE_TYPE_HEADER_PREFIX = b"PK\x03\x04"
+ FILE_EXTENSION_SUFFIX = {".nupkg"}
+
+
class JmodJavaModule(ZipFile):
DESCRIPTION = "Java .jmod modules"
FILE_TYPE_RE = re.compile(r"^(Zip archive data|Java jmod module)")
diff --git diffoscope/external_tools.py diffoscope/external_tools.py
index 3d698153..5abcae99 100644
--- diffoscope/external_tools.py
+++ diffoscope/external_tools.py
@@ -132,6 +132,7 @@ EXTERNAL_TOOLS = {
},
"lz4": {"debian": "lz4", "FreeBSD": "lz4", "guix": "lz4"},
"lzip": {"debian": "lzip", "guix": "lzip"},
+ "lzma": {"debian": "xz-utils", "arch": "xz", "guix": "xz"},
"msgunfmt": {
"debian": "gettext",
"arch": "gettext",
diff --git tests/comparators/test_lzma.py tests/comparators/test_lzma.py
new file mode 100644
index 00000000..696da3f0
--- /dev/null
+++ tests/comparators/test_lzma.py
@@ -0,0 +1,75 @@
+#
+# diffoscope: in-depth comparison of files, archives, and directories
+#
+# Copyright © 2015 Jérémy Bobbio <lunar@debian.org>
+# Copyright © 2016-2017, 2020, 2024-2025 Chris Lamb <lamby@debian.org>
+#
+# diffoscope is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# diffoscope is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with diffoscope. If not, see <https://www.gnu.org/licenses/>.
+
+import shutil
+import pytest
+
+from diffoscope.comparators.lzma import LzmaFile
+from diffoscope.comparators.binary import FilesystemFile
+from diffoscope.comparators.utils.specialize import specialize
+
+from ..utils.data import load_fixture, assert_diff
+from ..utils.tools import skip_unless_tools_exist
+from ..utils.nonexisting import assert_non_existing
+
+lzma1 = load_fixture("test1.lzma")
+lzma2 = load_fixture("test2.lzma")
+
+
+def test_identification(lzma1):
+ assert isinstance(lzma1, LzmaFile)
+
+
+def test_no_differences(lzma1):
+ difference = lzma1.compare(lzma1)
+ assert difference is None
+
+
+@pytest.fixture
+def differences(lzma1, lzma2):
+ return lzma1.compare(lzma2).details
+
+
+@skip_unless_tools_exist("xz")
+def test_content_source(differences):
+ assert differences[0].source1 == "test1"
+ assert differences[0].source2 == "test2"
+
+
+@skip_unless_tools_exist("xz")
+def test_content_source_without_extension(tmpdir, lzma1, lzma2):
+ path1 = str(tmpdir.join("test1"))
+ path2 = str(tmpdir.join("test2"))
+ shutil.copy(lzma1.path, path1)
+ shutil.copy(lzma2.path, path2)
+ lzma1 = specialize(FilesystemFile(path1))
+ lzma2 = specialize(FilesystemFile(path2))
+ difference = lzma1.compare(lzma2).details
+ assert difference[0].source1 == "test1-content"
+ assert difference[0].source2 == "test2-content"
+
+
+@skip_unless_tools_exist("xz")
+def test_content_diff(differences):
+ assert_diff(differences[0], "text_ascii_expected_diff")
+
+
+@skip_unless_tools_exist("xz")
+def test_compare_non_existing(monkeypatch, lzma1):
+ assert_non_existing(monkeypatch, lzma1)
diff --git tests/comparators/test_zip.py tests/comparators/test_zip.py
index 303b6f80..a5538c7b 100644
--- tests/comparators/test_zip.py
+++ tests/comparators/test_zip.py
@@ -2,7 +2,7 @@
# diffoscope: in-depth comparison of files, archives, and directories
#
# Copyright © 2015 Jérémy Bobbio <lunar@debian.org>
-# Copyright © 2015-2020, 2022, 2024 Chris Lamb <lamby@debian.org>
+# Copyright © 2015-2020, 2022, 2024-2025 Chris Lamb <lamby@debian.org>
#
# diffoscope is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@@ -20,7 +20,13 @@
import pytest
import subprocess
-from diffoscope.comparators.zip import ZipFile, MozillaZipFile, JmodJavaModule
+from diffoscope.comparators.zip import (
+ ZipFile,
+ MozillaZipFile,
+ NuGetPackageFile,
+ JmodJavaModule,
+ zipdetails_version,
+)
from ..utils.data import load_fixture, assert_diff
from ..utils.tools import skip_unless_tools_exist, skip_unless_tool_is_at_least
@@ -34,16 +40,14 @@ encrypted_zip1 = load_fixture("encrypted1.zip")
encrypted_zip2 = load_fixture("encrypted2.zip")
mozzip1 = load_fixture("test1.mozzip")
mozzip2 = load_fixture("test2.mozzip")
+nupkg1 = load_fixture("test1.nupkg")
+nupkg2 = load_fixture("test2.nupkg")
jmod1 = load_fixture("test1.jmod")
jmod2 = load_fixture("test2.jmod")
test_comment1 = load_fixture("test_comment1.zip")
test_comment2 = load_fixture("test_comment2.zip")
-def zipdetails_version():
- return subprocess.check_output(["zipdetails", "--version"]).decode("UTF-8")
-
-
def io_compress_zip_version():
try:
return subprocess.check_output(
@@ -135,6 +139,42 @@ def test_mozzip_compare_non_existing(monkeypatch, mozzip1):
assert_non_existing(monkeypatch, mozzip1)
+def test_nupkg_identification(nupkg1):
+ assert isinstance(nupkg1, NuGetPackageFile)
+
+
+def test_nupkg_no_differences(nupkg1):
+ difference = nupkg1.compare(nupkg1)
+ assert difference is None
+
+
+@pytest.fixture
+def nupkg_differences(nupkg1, nupkg2):
+ return nupkg1.compare(nupkg2).details
+
+
+@skip_unless_tools_exist("zipinfo")
+def test_nupkg_metadata(nupkg_differences, nupkg1, nupkg2):
+ assert_diff(nupkg_differences[0], "nupkg_expected_diff")
+
+
+@skip_unless_tools_exist("zipinfo")
+def test_nupkg_compressed_files(nupkg_differences):
+ assert (
+ nupkg_differences[-1].source1
+ == "package/services/metadata/core-properties/b44ebb537bbf4983b9527f9e3820fda6.psmdcp"
+ )
+ assert (
+ nupkg_differences[-1].source2
+ == "package/services/metadata/core-properties/08f1f9d8789a4668a128f78560bd0107.psmdcp"
+ )
+
+
+@skip_unless_tools_exist("zipinfo")
+def test_nupkg_compare_non_existing(monkeypatch, nupkg1):
+ assert_non_existing(monkeypatch, nupkg1)
+
+
def test_jmod_identification(jmod1):
assert isinstance(jmod1, JmodJavaModule)
diff --git tests/data/nupkg_expected_diff tests/data/nupkg_expected_diff
new file mode 100644
index 00000000..c8b39ff9
--- /dev/null
+++ tests/data/nupkg_expected_diff
@@ -0,0 +1,11 @@
+@@ -1,7 +1,7 @@
+-Zip file size: 3163 bytes, number of entries: 5
++Zip file size: 3162 bytes, number of entries: 5
+ -rw-r--r-- 2.0 unx 502 b- defN 25-Apr-14 18:14 _rels/.rels
+ -rw-r--r-- 2.0 unx 407 b- defN 25-Apr-14 18:14 ClassLibrary.nuspec
+ -rw-r--r-- 2.0 unx 3584 b- defN 25-Apr-14 22:14 lib/net8.0/ClassLibrary.dll
+ -rw-r--r-- 2.0 unx 459 b- defN 25-Apr-14 18:14 [Content_Types].xml
+--rw-r--r-- 2.0 unx 625 b- defN 25-Apr-14 18:14 package/services/metadata/core-properties/b44ebb537bbf4983b9527f9e3820fda6.psmdcp
+-5 files, 5577 bytes uncompressed, 2447 bytes compressed: 56.1%
++-rw-r--r-- 2.0 unx 625 b- defN 25-Apr-14 18:14 package/services/metadata/core-properties/08f1f9d8789a4668a128f78560bd0107.psmdcp
++5 files, 5577 bytes uncompressed, 2446 bytes compressed: 56.1%
diff --git tests/data/test1.lzma tests/data/test1.lzma
new file mode 100644
index 00000000..d7186e5d
Binary files /dev/null and tests/data/test1.lzma differ
diff --git tests/data/test1.nupkg tests/data/test1.nupkg
new file mode 100644
index 00000000..c1b85928
Binary files /dev/null and tests/data/test1.nupkg differ
diff --git tests/data/test2.lzma tests/data/test2.lzma
new file mode 100644
index 00000000..28d7d0d6
Binary files /dev/null and tests/data/test2.lzma differ
diff --git tests/data/test2.nupkg tests/data/test2.nupkg
new file mode 100644
index 00000000..c439e5b8
Binary files /dev/null and tests/data/test2.nupkg differ
Reply to: