[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#1037198: marked as done (locales: please parallelise locale-gen)

Your message dated Sat, 17 Jun 2023 09:20:45 +0000
with message-id <E1qAS6v-006I9t-Ol@fasolo.debian.org>
and subject line Bug#1037198: fixed in glibc 2.37-2
has caused the Debian Bug report #1037198,
regarding locales: please parallelise locale-gen
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@bugs.debian.org
immediately.)


-- 
1037198: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1037198
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems
--- Begin Message --- 
Package: locales
Version: 2.36-9
Severity: wishlist
Tags: patch

Dear Maintainer,

Posting as a bug per comment from Andrej; originally posted 2022-05-06 as
  https://salsa.debian.org/glibc-team/glibc/-/merge_requests/7

Patch based on current Salsa HEAD attached, incl. analysis.

Best,
наб

-- System Information:
Debian Release: 12.0
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: x32 (x86_64)
Foreign Architectures: amd64, i386

Kernel: Linux 6.1.0-2-amd64 (SMP w/2 CPU threads; PREEMPT)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages locales depends on:
ii  debconf [debconf-2.0]  1.5.82
ii  libc-bin               2.36-9
ii  libc-l10n              2.36-9

locales recommends no packages.

locales suggests no packages.

-- debconf information:
* locales/locales_to_be_generated: en_GB.UTF-8 UTF-8
* locales/default_environment_locale: en_GB.UTF-8

From b6af0ad83f5517fd1987f9c7ac0493565bc0976d Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=D0=BD=D0=B0=D0=B1?= <nabijaczleweli@nabijaczleweli.xyz>
Date: Fri, 6 May 2022 01:22:10 +0200
Subject: [PATCH] Parallelise locale-gen if possible
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Mutt-PGP: OS

Assuming a very generous 200M free/localedef (because I saw a max RSS
of 147M w/time(1)), this will attempt to keep all jobs saturated,
and usually succeed. There's little starvation, since the vast majority
of time is spent in gzip(1) ‒ 1:14 user vs 27:55 sys

At 2.2ish seconds per locale, even on a low-end system of today with
4 CPUs (and 800 free MB), we can generate up to 4 locales at once
for 6.6s' speed-up. Assuming no super-pathological cases, this globally
scales in roughly ceil(locales/ncpus)*2.2s chunks, which is a massive
win

The only user-visible change is that, with nproc>1, the output is
  en_GB.UTF-8...
  <cursor here>
instead of
  en_GB.UTF-8... <cursor here, will print "done\n" when it's done>

MemFree: in /proc/meminfo is available on all supported Debian kernels,
and, indeed, exactly what procps free(1) uses
---
 debian/local/usr_sbin/locale-gen | 30 ++++++++++++++++++++++++++++--
 1 file changed, 28 insertions(+), 2 deletions(-)

diff --git a/debian/local/usr_sbin/locale-gen b/debian/local/usr_sbin/locale-gen
index 7fa3d772..f1632f4e 100755
--- a/debian/local/usr_sbin/locale-gen
+++ b/debian/local/usr_sbin/locale-gen
@@ -23,6 +23,18 @@ is_entry_ok() {
 	fi
 }
 
+nproc="$(nproc 2>/dev/null)" || nproc=1
+if [ "$nproc" -gt 1 ]; then
+	mem_free=0
+	while read -r k v _; do
+		[ "$k" = "MemFree:" ] && mem_free="$v" && break || :
+	done < /proc/meminfo || :
+	mem_free=$(( mem_free / 1024 / 200 ))
+	[ "$mem_free" -lt 1 ] && mem_free=1 || :
+	[ "$mem_free" -lt "$nproc" ] && nproc="$mem_free" || :
+	jobs=0; pids=
+fi 2>/dev/null
+
 echo "Generating locales (this might take a while)..."
 while read -r locale charset; do
 	if [ -z "$locale" ] || [ "${locale#\#}" != "$locale" ]; then continue; fi
@@ -35,6 +47,7 @@ while read -r locale charset; do
 	locale_at="${locale#*@}"
 	[ "$locale_at" = "$locale" ] && locale_at= || locale_at="@$locale_at"
 	printf "  %s.%s%s..." "$locale_base" "$charset" "$locale_at"
+	[ "$nproc" -gt 1 ] && echo || :
 
 	if [ -e "$USER_LOCALES/$locale" ]; then
 		input="$USER_LOCALES/$locale"
@@ -46,7 +59,20 @@ while read -r locale charset; do
 			input="$USER_LOCALES/$input"
 		fi
 	fi
-	localedef -i "$input" -c -f "$charset" -A /usr/share/locale/locale.alias "$locale" || :
-	echo " done"
+	localedef -i "$input" -c -f "$charset" -A /usr/share/locale/locale.alias "$locale" &
+	if [ "$nproc" -gt 1 ]; then
+		pids="$pids$! "
+		jobs=$(( jobs + 1 ))
+
+		if [ "$jobs" -ge "$nproc" ]; then
+			wait "${pids%% *}" || :
+			jobs=$(( jobs - 1 ))
+			pids="${pids#* }"
+		fi
+	else
+		wait
+		echo " done"
+	fi
 done < "$LOCALEGEN"
+wait
 echo "Generation complete."
-- 
2.30.2

Attachment: signature.asc
Description: PGP signature


--- End Message ---
--- Begin Message --- 
Source: glibc
Source-Version: 2.37-2
Done: Aurelien Jarno <aurel32@debian.org>

We believe that the bug you reported is fixed in the latest version of
glibc, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 1037198@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Aurelien Jarno <aurel32@debian.org> (supplier of updated glibc package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.8
Date: Sat, 17 Jun 2023 11:10:20 +0200
Source: glibc
Architecture: source
Version: 2.37-2
Distribution: experimental
Urgency: medium
Maintainer: GNU Libc Maintainers <debian-glibc@lists.debian.org>
Changed-By: Aurelien Jarno <aurel32@debian.org>
Closes: 1037198
Changes:
 glibc (2.37-2) experimental; urgency=medium
 .
   [ Samuel Thibault ]
   * debian/patches/hurd-i386/git-task_info_count.diff: Fix passing size to
     task_info.
 .
   [ Aurelien Jarno ]
   * debian/patches/git-updates.diff: update from upstream stable branch:
     - Fix FTBFS on hppa.
   * debian/sysdeps/alpha.mk, debian/control.in/main: build-depends on g++-12
     (>= 12.3.0-2) on alpha instead of forcing -mcpu=ev56.
 .
   [ Andrej Shadura ]
   * debian/po/sk.po: Fix typos in the Slovak translation.
 .
   [ Helmut Grohne ]
   * debian/rules.d/build.mk: disable -Werror when dpkg-builflags contains
     -Wno-error.
 .
   [ наб ]
   * debian/local/usr_sbin/locale-gen: parallelise locales generation if
     possible.  Closes: #1037198.
Checksums-Sha1:
 6e445e22051bf0f00e405d28bf0e2580766d1a00 9685 glibc_2.37-2.dsc
 68d7fdaf8b88521a7bb62855573269aa3423656b 786416 glibc_2.37-2.debian.tar.xz
 75749b933d4f11817d57b10ddb8078726b7e3bf0 9587 glibc_2.37-2_source.buildinfo
Checksums-Sha256:
 7734737fb59c29630a21b0a6e603f7fc73612d17122c77f94fc5c8e8b4a919f9 9685 glibc_2.37-2.dsc
 0c9a943ffff1c246f032d5058e28c7fb8871abe569094c08726e38c9e0ebf4aa 786416 glibc_2.37-2.debian.tar.xz
 ce1b6fbc1d3582e00bb38a46b9fd57b01105279a6cccad76370c62e9e5043f93 9587 glibc_2.37-2_source.buildinfo
Files:
 4149eafdec39ba67839484c2d921206b 9685 libs required glibc_2.37-2.dsc
 db3087bebb96abee19f80b8c27825f74 786416 libs required glibc_2.37-2.debian.tar.xz
 9d05ad59b80f6e2696ca8d32b0bc9eb8 9587 libs required glibc_2.37-2_source.buildinfo

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEUryGlb40+QrX1Ay4E4jA+JnoM2sFAmSNeQYACgkQE4jA+Jno
M2s4+hAAiPx2iASWGPrM9HftZCr4wVqygVhjEr8sdKXNpm8IFJB/Iu76Ic1wn4sX
K1Ovzl9mJeYhQJarIuqFBmUvPLcPDsGlr7Rw9aTnhFe8p7XS01gEUZlsID7dH5ex
gNc0DdLbUOzytpu75NDxFmALB3wjSWXzOnkVwZggryYulIO3kanhd9+75LiKHm74
/CHw0+NXY4cx8JJP/Ixwn2fxHyzcjlGgaUYu3fetvkq7fADnJSrkEpYhkn2PNB3A
Wikp53wV0FISiivkkCLX7rDjcSojvY3Wd5dX27z0RdbX+YIDkZu7yuX55zSP40Js
5L+twb+I/sM7WhmVLWsWEIF5LMSVbx0IvSYbQcCkP17+x+zECYVX/UaY0cFmbkxS
Rw1+TOFB3i+jJTxg2xgmtyXG77bB+6Ibmpg8DYS1TwFM8JoBfsID1TTkHyfmtAR+
U2GFog672z1bRZQEWrhBGCF7pONo2FDh4FXTi2SL8tLJjZ+wC8UOgrKtHFOPgTPO
EUm/UXZmbDaTJFI1AVf4gfNG2De4l5TZqPtMMNtv6heL5byUYr2JcvQr1j9l/DIq
i53KpGvOULHLiS9r0LkYejUxxk+y0Csi4LI3XYW1q97Okj1sYCpncejB1s2rU40M
jyW4uo5y4+/2DpApMCdmxn45J9up/3MmPFe+4HTClZIuHrRsEnU=
=LpsO
-----END PGP SIGNATURE-----

--- End Message ---
Reply to: