[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#991804: unblock: debian-cd/3.1.25



Package: release.debian.org
Severity: normal
User: release.debian.org@packages.debian.org
Usertags: unblock

Hi!

Please unblock package debian-cd

Here's the traditional last-minute request for a debian-cd unblock,
such that the version in the archive for our release is up to date
with the software that we use to create it!

Two key changes to see here:

* Add brltty and espeakup to all images from netinst up, for blind
  users. Closes: #678065

* Add script to generate firmware metadata from appdata metadata,
  and use it when building a media tree. See: #989863

Both are tested and known working already, as we use the version from
git to build our daily and weekly images.

Debdiff attached.

unblock debian-cd/3.1.25

-- System Information:
Debian Release: 10.10
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable-debug'), (500, 'stable'), (500, 'oldstable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 5.10.0-0.bpo.5-amd64 (SMP w/4 CPU cores)
Kernel taint flags: TAINT_CPU_OUT_OF_SPEC
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE=en_GB:en (charmap=UTF-8)
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
diff -Nru debian-cd-3.1.34/debian/changelog debian-cd-3.1.35/debian/changelog
--- debian-cd-3.1.34/debian/changelog	2021-04-20 00:03:35.000000000 +0100
+++ debian-cd-3.1.35/debian/changelog	2021-07-26 01:02:03.000000000 +0100
@@ -1,3 +1,15 @@
+debian-cd (3.1.35) unstable; urgency=medium
+
+  [ Steve McIntyre ]
+  * Add brltty and espeakup to all images from netinst up, for blind
+    users. Closes: #678065
+
+  [ Cyril Brulebois ]
+  * Add script to generate firmware metadata from appdata metadata,
+    and use it when building a media tree. See: #989863
+
+ -- Steve McIntyre <93sam@debian.org>  Mon, 26 Jul 2021 01:02:03 +0100
+
 debian-cd (3.1.34) unstable; urgency=medium
 
   [ Wolfgang Schweer ]
@@ -10,7 +22,7 @@
   * debian/control: Drop Recommends: on netpbm and syslinux-utils. These are no
     longer needed now that custom splash images are used unmodified.
 
-  [ Petter Reinholdtse ]
+  [ Petter Reinholdtsen ]
   * Include eatmydata deb for d-i to use offline. Closes: #986772
 
   [ Steve McIntyre ]
diff -Nru debian-cd-3.1.34/debian/control debian-cd-3.1.35/debian/control
--- debian-cd-3.1.34/debian/control	2021-03-22 14:20:37.000000000 +0000
+++ debian-cd-3.1.35/debian/control	2021-07-25 20:39:31.000000000 +0100
@@ -15,7 +15,7 @@
 
 Package: debian-cd
 Architecture: all
-Depends: ${misc:Depends}, curl, perl, dpkg-dev, cpp, libdigest-md5-perl, libdigest-sha-perl, tofrodos, apt, make, xorriso | genisoimage, lynx, grep-dctrl, bc, libcompress-zlib-perl, bzip2, libdpkg-perl, wget
+Depends: ${misc:Depends}, curl, perl, dpkg-dev, cpp, libdigest-md5-perl, libdigest-sha-perl, tofrodos, apt, make, xorriso | genisoimage, lynx, grep-dctrl, bc, libcompress-zlib-perl, bzip2, libdpkg-perl, wget, libfile-slurp-perl, libyaml-libyaml-perl
 Recommends: hfsutils, isolinux, syslinux-common, mtools, dosfstools
 Description: Tools for building (Official) Debian CD set
  Debian-cd is the official tool for building Debian CD set since the potato
diff -Nru debian-cd-3.1.34/tasks/bullseye/forcd1 debian-cd-3.1.35/tasks/bullseye/forcd1
--- debian-cd-3.1.34/tasks/bullseye/forcd1	2021-04-14 09:45:37.000000000 +0100
+++ debian-cd-3.1.35/tasks/bullseye/forcd1	2021-05-16 15:44:54.000000000 +0100
@@ -6,11 +6,7 @@
 openssh-client
 /* could be used by debconf in certain configurations */
 libterm-readline-gnu-perl
-/* Accessibility stuff that is installed by base-installer
- * in certain situations, but too large for the netinst. */
-brltty
-espeakup
-alsa-utils
+
 /* See rationale in #630805 */
 apt-offline
 
diff -Nru debian-cd-3.1.34/tasks/kali-dev/forcd1 debian-cd-3.1.35/tasks/kali-dev/forcd1
--- debian-cd-3.1.34/tasks/kali-dev/forcd1	2021-04-14 09:45:37.000000000 +0100
+++ debian-cd-3.1.35/tasks/kali-dev/forcd1	2021-05-16 15:44:54.000000000 +0100
@@ -6,11 +6,7 @@
 openssh-client
 /* could be used by debconf in certain configurations */
 libterm-readline-gnu-perl
-/* Accessibility stuff that is installed by base-installer
- * in certain situations, but too large for the netinst. */
-brltty
-espeakup
-alsa-utils
+
 /* See rationale in #630805 */
 apt-offline
 
diff -Nru debian-cd-3.1.34/tasks/kali-last-snapshot/forcd1 debian-cd-3.1.35/tasks/kali-last-snapshot/forcd1
--- debian-cd-3.1.34/tasks/kali-last-snapshot/forcd1	2021-04-14 09:45:37.000000000 +0100
+++ debian-cd-3.1.35/tasks/kali-last-snapshot/forcd1	2021-05-16 15:44:54.000000000 +0100
@@ -6,11 +6,7 @@
 openssh-client
 /* could be used by debconf in certain configurations */
 libterm-readline-gnu-perl
-/* Accessibility stuff that is installed by base-installer
- * in certain situations, but too large for the netinst. */
-brltty
-espeakup
-alsa-utils
+
 /* See rationale in #630805 */
 apt-offline
 
diff -Nru debian-cd-3.1.34/tasks/kali-rolling/forcd1 debian-cd-3.1.35/tasks/kali-rolling/forcd1
--- debian-cd-3.1.34/tasks/kali-rolling/forcd1	2021-04-14 09:45:37.000000000 +0100
+++ debian-cd-3.1.35/tasks/kali-rolling/forcd1	2021-05-16 15:44:54.000000000 +0100
@@ -6,11 +6,7 @@
 openssh-client
 /* could be used by debconf in certain configurations */
 libterm-readline-gnu-perl
-/* Accessibility stuff that is installed by base-installer
- * in certain situations, but too large for the netinst. */
-brltty
-espeakup
-alsa-utils
+
 /* See rationale in #630805 */
 apt-offline
 
diff -Nru debian-cd-3.1.34/tasks/sid/forcd1 debian-cd-3.1.35/tasks/sid/forcd1
--- debian-cd-3.1.34/tasks/sid/forcd1	2021-04-14 09:45:37.000000000 +0100
+++ debian-cd-3.1.35/tasks/sid/forcd1	2021-05-16 15:44:54.000000000 +0100
@@ -6,11 +6,7 @@
 openssh-client
 /* could be used by debconf in certain configurations */
 libterm-readline-gnu-perl
-/* Accessibility stuff that is installed by base-installer
- * in certain situations, but too large for the netinst. */
-brltty
-espeakup
-alsa-utils
+
 /* See rationale in #630805 */
 apt-offline
 
diff -Nru debian-cd-3.1.34/tools/generate_di+k_list debian-cd-3.1.35/tools/generate_di+k_list
--- debian-cd-3.1.34/tools/generate_di+k_list	2019-07-07 16:52:46.000000000 +0100
+++ debian-cd-3.1.35/tools/generate_di+k_list	2021-05-16 15:44:50.000000000 +0100
@@ -93,6 +93,12 @@
 /* Needed for some languages with bi-directional support */
 libfribidi0
 
+/* Accessibility stuff that is installed by base-installer
+ * in certain situations. */
+alsa-utils
+brltty
+espeakup
+
 /* Needed for rootless installs. */
 sudo
 
diff -Nru debian-cd-3.1.34/tools/generate_firmware_patterns debian-cd-3.1.35/tools/generate_firmware_patterns
--- debian-cd-3.1.34/tools/generate_firmware_patterns	1970-01-01 01:00:00.000000000 +0100
+++ debian-cd-3.1.35/tools/generate_firmware_patterns	2021-07-26 01:01:55.000000000 +0100
@@ -0,0 +1,174 @@
+#!/usr/bin/perl
+# © 2021 Cyril Brulebois <kibi@debian.org>
+#
+# Generate ready-to-use .patterns files so that one can use grep to
+# perform hardware to firmware-package lookups in the installer
+# (see MODALIAS= lines in `udevadm info --export-db`), see the
+# hw-detect component.
+#
+# Parameters: dists/<suite>/*/dep11/Components-<arch>.yml.(gz|xz)
+#
+# We're limiting ourselves to packages announcing Type: firmware, and
+# they need to include such information in their metadata, e.g.
+# src:firmware-nonfree; the --test option can be used until such
+# packages appear in the archive.
+#
+# See: https://salsa.debian.org/kernel-team/firmware-nonfree/-/merge_requests/19
+
+use strict;
+use warnings;
+
+use File::Path qw(make_path);
+use File::Slurp;
+use Getopt::Long;
+use YAML::XS;
+
+
+# XXX: Drop support for --test once we have the required metadata in
+# the archive.
+my $output_dir = '.';
+my $test;
+my $verbose;
+my $pkgname = "ALL";
+
+GetOptions( "output-dir=s" => \$output_dir,
+            "test"         => \$test,
+            "verbose"      => \$verbose,
+            "package=s"    => \$pkgname)
+    or die "Error in command line arguments";
+
+
+sub format_alias {
+    return map { $a = $_; $a =~ s/[*]/.*/g; "^$a\$\n" } @_;
+}
+
+
+sub process_components {
+    my $input = shift;
+    my $content;
+    print STDERR "processing $input\n"
+        if $verbose;
+
+    if ($input =~ /\.gz$/) {
+        $content = `zcat $input`;
+    }
+    elsif ($input =~ /\.xz$/) {
+        $content = `xzcat $input`;
+    }
+    else {
+        die "only gz and xz suffixes are supported";
+    }
+
+    my @packages;
+    foreach my $array (Load $content) {
+        # XXX: Drop the condition once we have the required metadata
+        # in the archive.
+        if (! $test) {
+            next if not defined $array->{Type};
+            next if $array->{Type} ne 'firmware';
+        }
+        next if not defined $array->{Provides};
+        next if not defined $array->{Provides}->{modaliases};
+
+        print STDERR "found modaliases entries for firmware package ", $array->{Package}, "\n"
+            if $verbose;
+
+	if ($pkgname eq "ALL" or $pkgname eq $array->{Package}) {
+
+	    my $patterns_file = $output_dir . "/" . $array->{Package} . ".patterns";
+	    printf STDERR "writing %d entries to %s\n",
+		(scalar @{ $array->{Provides}->{modaliases} }),
+		$patterns_file;
+
+	    # For each alias, anchor the pattern on the left (^) and on
+	    # the right ($), and replace each '*' with '.*':
+	    write_file($patterns_file,
+		       format_alias( @{ $array->{Provides}->{modaliases} } ));
+	    push @packages, $array->{Package};
+	}
+    }
+    return @packages;
+}
+
+
+# Prepare output directory:
+if (! -d $output_dir) {
+    print STDERR "creating output directory $output_dir\n"
+        if $verbose;
+    make_path($output_dir, { verbose => $verbose });
+}
+
+write_file("$output_dir/README.txt",
+           "These files help Debian Installer detect helpful firmware packages (via hw-detect).\n");
+
+# Generate .patterns file to match firmware packages found in
+# Components-* files:
+my @all_packages;
+foreach my $component (@ARGV) {
+    my @packages = process_components($component);
+    push @all_packages, @packages;
+}
+@all_packages = sort @all_packages;
+print STDERR "firmware packages found across all components: @all_packages\n"
+    if $verbose;
+
+# Workaround for firmware-sof-signed, which doesn't advertise firmware
+# files it might need through the MODULE_FIRMWARE() macro (meaning no
+# chance to generate DEP-11 info from there): alias info manually
+# extracted on 2021-07-25 (linux-image-5.10.0-8-amd64, 5.10.46-2).
+#
+# XXX: To be kept in sync!
+my $SOF = 'firmware-sof-signed';
+if (! grep { $_ eq $SOF } @all_packages) {
+
+    if ($pkgname eq 'ALL' or $pkgname eq 'firmware-sof-signed') {
+
+	# Extract on amd64, from the installed package, with the following
+	# command. Note that descending under intel/ would lead to no
+	# aliases, so stick to the top-level directory for sof:
+	#
+	#     for x in $(dpkg -L linux-image-5.10.0-8-amd64 | grep kernel/sound/soc/sof/.*\.ko$); do /usr/sbin/modinfo $x | awk '/^alias:/ { print $2 }'; done | sort
+	#
+	# XXX: If that's not enough, there are other modules matching the
+	#      kernel/sound/soc/intel/boards/snd-soc-sof*.ko pattern.
+
+	my @sof_aliases = qw(
+	    pci:v00008086d000002C8sv*sd*bc*sc*i*
+	    pci:v00008086d000006C8sv*sd*bc*sc*i*
+	    pci:v00008086d0000119Asv*sd*bc*sc*i*
+	    pci:v00008086d00001A98sv*sd*bc*sc*i*
+	    pci:v00008086d00003198sv*sd*bc*sc*i*
+	    pci:v00008086d000034C8sv*sd*bc*sc*i*
+	    pci:v00008086d000038C8sv*sd*bc*sc*i*
+	    pci:v00008086d00003DC8sv*sd*bc*sc*i*
+	    pci:v00008086d000043C8sv*sd*bc*sc*i*
+	    pci:v00008086d00004B55sv*sd*bc*sc*i*
+	    pci:v00008086d00004B58sv*sd*bc*sc*i*
+	    pci:v00008086d00004DC8sv*sd*bc*sc*i*
+	    pci:v00008086d00005A98sv*sd*bc*sc*i*
+	    pci:v00008086d00009DC8sv*sd*bc*sc*i*
+	    pci:v00008086d0000A0C8sv*sd*bc*sc*i*
+	    pci:v00008086d0000A348sv*sd*bc*sc*i*
+	    pci:v00008086d0000A3F0sv*sd*bc*sc*i*
+	    platform:jsl_rt5682_max98360a
+	    platform:jsl_rt5682_rt1015
+	    platform:sof-audio
+	    platform:sof_rt5682
+	    platform:sof_sdw
+	    platform:tgl_max98357a_rt5682
+	    platform:tgl_max98373_rt5682
+	);
+
+	print STDERR "deploying manual workaround for $SOF\n"
+	    if $verbose;
+
+	my $sof_file = $output_dir . "/" . $SOF . ".patterns";
+	printf STDERR "writing %d entries to %s\n",
+	    (scalar @sof_aliases),
+	    $sof_file;
+
+	write_file($sof_file,
+		   format_alias(@sof_aliases));
+	push @all_packages, $SOF;
+    }
+}
diff -Nru debian-cd-3.1.34/tools/make_disc_trees.pl debian-cd-3.1.35/tools/make_disc_trees.pl
--- debian-cd-3.1.34/tools/make_disc_trees.pl	2021-02-03 16:33:35.000000000 +0000
+++ debian-cd-3.1.35/tools/make_disc_trees.pl	2021-07-25 20:39:31.000000000 +0100
@@ -172,6 +172,7 @@
 
 print "Starting to lay out packages into images:\n";
 
+# Read in the list of packages that we're expecting to include
 if (-e "$bdir/firmware-packages") {
     open(FWLIST, "$bdir/firmware-packages") or die "Unable to read firmware-packages file!\n";
     while (defined (my $pkg = <FWLIST>)) {
@@ -909,7 +910,7 @@
 	print "  Finishing off md5sum.txt\n";
 	# Just md5 the bits we won't have seen already
 	open(MD5LIST, ">>md5sum.txt") or die "Failed to open md5sum.txt file: $!\n";
-	find (\&md5_files_for_md5sum, ("./.disk", "./dists"));
+	find (\&md5_files_for_md5sum, ("./.disk", "./dists", "./firmware/dep11"));
 	close(MD5LIST);
 
 	# And sort; it should make things faster for people checking
@@ -1063,8 +1064,8 @@
 
     m/^Package: (\S+)/m and $p = $1;
     m/^Section: (\S+)/m and $section = $1;
-
     m/^Filename: (\S+)/mi and $file = $1;
+
     $idir = Packages_dir($dir, $file, $section, $in_backports) . "/i18n";
 
     if (! -d $idir) {
@@ -1164,6 +1165,62 @@
     return $blocks_added;
 }
 
+# Add sym-links and pattern files for firmware packages
+sub add_firmware_stuff {
+    my $dir = shift;
+    my $arch = shift;
+    my $in_backports = shift;
+    local $_ = shift;
+    my ($p, $file, $section, $dist, $dep11_dir);
+    my $blocks_added = 0;
+    my @args = ("$basedir/tools/generate_firmware_patterns",
+		"--output-dir", "$dir/firmware/dep11");
+
+    m/^Package: (\S+)/m and $p = $1;
+    m/^Section: (\S+)/m and $section = $1;
+    m/^Filename: (\S+)/mi and $file = $1;
+
+    if ($file =~ /\/main\//) {
+        $dist = "main";
+    } elsif ($file =~ /\/contrib\//) {
+        $dist = "contrib";
+    } elsif ($file =~ /\/non-free\//) {
+        $dist = "non-free";
+    } else {
+        $dist = "local";
+    }
+
+    $dep11_dir = "$mirror/dists/$codename/$dist/dep11";
+    if ($in_backports) {
+	$dep11_dir = "$mirror/dists/$codename-backports/$dist/dep11";
+    }
+
+    msg_ap(0, "Symlink fw package $p into /firmware\n");
+    symlink("../$file", "$dir/firmware/" . basename($file));
+    msg_ap(0, "Symlink ../$file $dir/firmware/.\n");
+    if (! -d "$dir/firmware") {
+	mkdir "$dir/firmware" or die "mkdir $dir/firmware failed $!\n";
+	mkdir "$dir/firmware/dep11" or die "mkdir $dir/firmware/dep11 failed $!\n";
+	$blocks_added += 2;
+    }
+
+    # Cope with maybe having the patterns file already
+    # (e.g. multi-arch), in which case we'll replace it here
+    if (-f "$dir/firmware/dep11/$p.patterns") {
+	$blocks_added -= get_file_blocks("$dir/firmware/dep11/$p.patterns");
+    }
+
+    msg_ap(0, "(Maybe) generate fw pattern file $dir/firmware/dep11/$p.patterns\n");
+    push(@args, "--package", "$p");
+    push(@args, "$dep11_dir/Components-$arch.yml.gz");
+    system(@args) == 0 or die "generate_firmware_patterns failed: $?";
+    if (-f "$dir/firmware/dep11/$p.patterns") {
+	$blocks_added += get_file_blocks("$dir/firmware/dep11/$p.patterns");
+    }
+
+    return $blocks_added;
+}
+
 # Roll back the results of add_Packages_entry()
 sub remove_Packages_entry {
     my $dir = shift;
@@ -1363,6 +1420,29 @@
     return $blocks_removed;
 }
 
+sub remove_firmware_stuff {
+    my $dir = shift;
+    my $arch = shift;
+    my $in_backports = shift;
+    my ($p, $file);
+    local $_ = shift;
+    my $blocks_removed = 0;
+
+    m/^Package: (\S+)/mi and $p = $1;
+    m/^Filename: (\S+)/mi and $file = $1;
+
+    msg_ap(0, "Remove symlink for fw package $p in /firmware\n");
+	unlink("$dir/firmware/" . basename($file));
+
+    if (-f "$dir/firmware/dep11/$p.patterns") {
+	$blocks_removed += get_file_blocks("$dir/firmware/dep11/$p.patterns");
+	msg_ap(0, "Remove $dir/firmware/dep11/$p.patterns\n");
+	unlink("$dir/firmware/dep11/$p.patterns");
+    }
+
+    return $blocks_removed;
+}
+
 sub get_file_blocks {
     my $realfile = shift;
     my $st;
@@ -1431,6 +1511,10 @@
             if (!($arch eq "source")) {
                 $total_blocks -= remove_trans_desc_entry($dir, $arch, $in_backports, $package_info);
             }
+	    
+	    if ($firmware_package{$pkgname}) {
+		$total_blocks -= remove_firmware_stuff($dir, $arch, $in_backports, $package_info);
+	    }
         
             foreach my $file (@files) {
                 my $missing = 0;
@@ -1479,12 +1563,7 @@
                     $total_blocks += good_link ($realfile, "$dir/$file");
                     msg_ap(0, "  Linked $dir/$file\n");
                     if ($firmware_package{$pkgname}) {
-                        msg_ap(0, "Symlink fw package $pkgname into /firmware\n");
-                        if (! -d "$dir/firmware") {
-                            mkdir "$dir/firmware" or die "symlink failed $!\n";
-                        }
-                        symlink("../$file", "$dir/firmware/" . basename($file));
-                        msg_ap(0, "Symlink ../$file $dir/firmware/.\n");
+			$total_blocks += add_firmware_stuff($dir, $arch, $in_backports, $package_info);
                     }
                 } else {
                     msg_ap(0, "  $dir/$file already linked in\n");

Reply to: