Bug#838188: marked as done (ocaml: temporary preprocessed file paths make the ocaml compiler produce unreproducible output)
Your message dated Thu, 22 Dec 2016 00:19:27 +0000
with message-id <E1cJr6V-0006a4-PJ@fasolo.debian.org>
and subject line Bug#838188: fixed in ocaml 4.02.3-9
has caused the Debian Bug report #838188,
regarding ocaml: temporary preprocessed file paths make the ocaml compiler produce unreproducible output
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@bugs.debian.org
immediately.)
--
838188: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=838188
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems
--- Begin Message ---
- To: Debian Bug Tracking System <submit@bugs.debian.org>
- Subject: ocaml: temporary preprocessed file paths make the ocaml compiler produce unreproducible output
- From: Johannes Schauer <josch@debian.org>
- Date: Sun, 18 Sep 2016 10:51:54 +0200
- Message-id: <147418871411.19537.8509715408333794006.reportbug@localhost>
Source: ocaml
Version: 4.02.3-7.1
Severity: wishlist
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertags: toolchain randomness
Hi,
currently, ocaml embeds the file paths of temporary files that a
preprocessor created into the debug output. This makes several source
packages in Debian unreproducible. To see the effect, look for example
at this diffoscope output of src:botch:
│ ├── data.tar.xz
│ │ ├── data.tar
│ │ │ ├── ./usr/lib/debug/.build-id/03/28382a2670552f3318cc61bdebc13bbeef8f2f.debug
│ │ │ │ ├── readelf --wide --symbols {}
│ │ │ │ │ @@ -56,15 +56,15 @@
│ │ │ │ │ 52: 0000000000830838 0 NOTYPE LOCAL DEFAULT 25 caml_startup__9
│ │ │ │ │ 53: 0000000000830868 0 NOTYPE LOCAL DEFAULT 25 caml_startup__10
│ │ │ │ │ 54: 0000000000830898 0 NOTYPE LOCAL DEFAULT 25 caml_startup__11
│ │ │ │ │ 55: 00000000008308c8 0 NOTYPE LOCAL DEFAULT 25 caml_startup__12
│ │ │ │ │ 56: 0000000000000000 0 FILE LOCAL DEFAULT ABS std_exit.ml
│ │ │ │ │ 57: 00000000005c4430 0 NOTYPE LOCAL DEFAULT 15 caml_negf_mask
│ │ │ │ │ 58: 00000000005c4440 0 NOTYPE LOCAL DEFAULT 15 caml_absf_mask
│ │ │ │ │ - 59: 0000000000000000 0 FILE LOCAL DEFAULT ABS /tmp/ocamlpp29daa7
│ │ │ │ │ + 59: 0000000000000000 0 FILE LOCAL DEFAULT ABS /tmp/ocamlpp4dfb7e
│ │ │ │ │ 60: 00000000005c4450 0 NOTYPE LOCAL DEFAULT 15 caml_negf_mask
│ │ │ │ │ 61: 00000000005c4460 0 NOTYPE LOCAL DEFAULT 15 caml_absf_mask
│ │ │ │ │ 62: 0000000000836558 0 NOTYPE LOCAL DEFAULT 25 camlAnnotate$2dstrong__30
│ │ │ │ │ 63: 0000000000836570 0 NOTYPE LOCAL DEFAULT 25 camlAnnotate$2dstrong__31
│ │ │ │ │ 64: 0000000000836588 0 NOTYPE LOCAL DEFAULT 25 camlAnnotate$2dstrong__32
│ │ │ │ │ 65: 00000000008365c0 0 NOTYPE LOCAL DEFAULT 25 camlAnnotate$2dstrong__2
│ │ │ │ │ 66: 0000000000836668 0 NOTYPE LOCAL DEFAULT 25 camlAnnotate$2dstrong__5
│ │ │ │ │ @@ -87,15 +87,15 @@
│ │ │ │ │ 83: 00000000008367e0 0 NOTYPE LOCAL DEFAULT 25 camlAnnotate$2dstrong__23
│ │ │ │ │ 84: 00000000008367f8 0 NOTYPE LOCAL DEFAULT 25 camlAnnotate$2dstrong__24
│ │ │ │ │ 85: 0000000000836810 0 NOTYPE LOCAL DEFAULT 25 camlAnnotate$2dstrong__25
│ │ │ │ │ 86: 0000000000836820 0 NOTYPE LOCAL DEFAULT 25 camlAnnotate$2dstrong__26
│ │ │ │ │ 87: 0000000000836868 0 NOTYPE LOCAL DEFAULT 25 camlAnnotate$2dstrong__27
│ │ │ │ │ 88: 0000000000836880 0 NOTYPE LOCAL DEFAULT 25 camlAnnotate$2dstrong__28
│ │ │ │ │ 89: 00000000008368c8 0 NOTYPE LOCAL DEFAULT 25 camlAnnotate$2dstrong__29
│ │ │ │ │ - 90: 0000000000000000 0 FILE LOCAL DEFAULT ABS /tmp/ocamlpp21639f
│ │ │ │ │ + 90: 0000000000000000 0 FILE LOCAL DEFAULT ABS /tmp/ocamlppfd0623
│ │ │ │ │ 91: 00000000005c4470 0 NOTYPE LOCAL DEFAULT 15 caml_negf_mask
│ │ │ │ │ 92: 00000000005c4480 0 NOTYPE LOCAL DEFAULT 15 caml_absf_mask
│ │ │ │ │ 93: 0000000000836fc0 0 NOTYPE LOCAL DEFAULT 25 camlSrcGraphExtras__43
│ │ │ │ │ 94: 0000000000836fd8 0 NOTYPE LOCAL DEFAULT 25 camlSrcGraphExtras__44
│ │ │ │ │ 95: 0000000000836ff0 0 NOTYPE LOCAL DEFAULT 25 camlSrcGraphExtras__45
│ │ │ │ │ 96: 0000000000837008 0 NOTYPE LOCAL DEFAULT 25 camlSrcGraphExtras__46
│ │ │ │ │ 97: 0000000000837028 0 NOTYPE LOCAL DEFAULT 25 camlSrcGraphExtras__9
I see two ways to fix this problem.
- instead of choosing a random temporary file name for the preprocessor
output, choose a stable file name
- do not include the path to the temporary file created by the
preprocessor in the debug information
I like the latter option because knowing this path is useless anyway
because the file is only temporary. Unfortunately, I was unable to
figure out a good way to implement this solution.
So instead, I implemented a solution that calculates the path of the
temporary files from the MD5 sum of the preprocessor name and the input
file path. The idea is, that running the same preprocessor on the same
file path should produce the same output and thus choosing the same
filename should not pose any problem. I chose to calculate a hash
instead of using the bare string values because the file paths contain
characters like the slash which must not appear in file names and also
because it allows a stable temporary filename length no matter the
length of the input path.
Here is the patch:
--- a/driver/pparse.ml
+++ b/driver/pparse.ml
@@ -19,9 +19,17 @@ type error =
exception Error of error
(* Optionally preprocess a source file *)
+external open_desc: string -> open_flag list -> int -> int = "caml_sys_open"
+external close_desc: int -> unit = "caml_sys_close"
let call_external_preprocessor sourcefile pp =
- let tmpfile = Filename.temp_file "ocamlpp" "" in
+ (* do not use Filename.temp_file as the resulting temporary file name will be
+ * recorded in the debug output of the resulting binary and thus make the
+ * output random and unreproducible *)
+ let temp_dir = Filename.get_temp_dir_name () in
+ let hash = Digest.to_hex (Digest.string (sourcefile^pp)) in
+ let tmpfile = Filename.concat temp_dir ("ocamlpp"^hash) in
+ close_desc(open_desc tmpfile [Open_wronly; Open_creat; Open_excl] 0o600);
let comm = Printf.sprintf "%s %s > %s"
pp (Filename.quote sourcefile) tmpfile
in
Applying this patch and rebuilding src:ocaml leads to src:botch becoming
reproducible.
I do not know whether the patch is suitable for inclusion into the
upstream project but I trust that you forward the issue accordingly.
Thanks!
cheers, josch
--- End Message ---
--- Begin Message ---
Source: ocaml
Source-Version: 4.02.3-9
We believe that the bug you reported is fixed in the latest version of
ocaml, which is due to be installed in the Debian FTP archive.
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to 838188@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Mehdi Dogguy <mehdi@debian.org> (supplier of updated ocaml package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Format: 1.8
Date: Thu, 22 Dec 2016 00:48:59 +0100
Source: ocaml
Binary: ocaml-nox ocaml ocaml-base-nox ocaml-base ocaml-native-compilers ocaml-source ocaml-interp ocaml-compiler-libs ocaml-mode
Architecture: source amd64 all
Version: 4.02.3-9
Distribution: unstable
Urgency: medium
Maintainer: Debian OCaml Maintainers <debian-ocaml-maint@lists.debian.org>
Changed-By: Mehdi Dogguy <mehdi@debian.org>
Description:
ocaml - ML language implementation with a class-based object system
ocaml-base - Runtime system for OCaml bytecode executables
ocaml-base-nox - Runtime system for OCaml bytecode executables (no X)
ocaml-compiler-libs - OCaml interpreter and standard libraries
ocaml-interp - OCaml interactive interpreter and standard libraries
ocaml-mode - major mode for editing Objective Caml in Emacs
ocaml-native-compilers - Native code compilers of the OCaml suite (the .opt ones)
ocaml-nox - ML implementation with a class-based object system (no X)
ocaml-source - Sources for Objective Caml
Closes: 792502 824139 838188
Changes:
ocaml (4.02.3-9) unstable; urgency=medium
.
* obey hardening LDFLAGS (Closes: #792502). Thanks to Török Edwin
for the patch!
- add patch 0013-Obey-ldflags.patch
* Compute a stable name for preprocessed files (Closes: #838188).
Thanks to Johannes Schauer for the patch!
* CVE-2015-8869 (Closes: #824139).
Checksums-Sha1:
afe010466cb39d0ba030e819b3c6977359ace2dc 2728 ocaml_4.02.3-9.dsc
16777daa6b2e4fed651d83132cfc87bff807dcb4 47924 ocaml_4.02.3-9.debian.tar.xz
67c6ece0e3dd4d7c92c47264e9e52ed8b44ba63b 7974 ocaml-base-dbgsym_4.02.3-9_amd64.deb
5790aa9220be9f693838ef243b4ca109a5fe523e 62876 ocaml-base-nox-dbgsym_4.02.3-9_amd64.deb
8a4cb47bc2ce284a46658731f8d5264f3401cb64 542860 ocaml-base-nox_4.02.3-9_amd64.deb
5f600c089e5f3e6d697bed4ac349dc7d01e554de 132524 ocaml-base_4.02.3-9_amd64.deb
1bb79133101981ab5c018f92775db1ea0a14cfb8 9483532 ocaml-compiler-libs_4.02.3-9_amd64.deb
77ab7e21d9efe6b65d8b2072a28dfdda6bc62b69 360590 ocaml-interp_4.02.3-9_amd64.deb
591314e5dd7dd7c9f3cd069766682b3bfb2a5018 137026 ocaml-mode_4.02.3-9_all.deb
9d6712c355b675fdb9e0b79e67c8a27d4f84e594 757654 ocaml-native-compilers-dbgsym_4.02.3-9_amd64.deb
00fcb0c97fa5a7a9795230372ff680b0ef4b9db2 3100004 ocaml-native-compilers_4.02.3-9_amd64.deb
327ccf933298e15f99e95ba01e41d2cb2dc7732f 1921812 ocaml-nox-dbgsym_4.02.3-9_amd64.deb
c0699980399c62f0f46b4941f7b0813d607561e8 5653690 ocaml-nox_4.02.3-9_amd64.deb
8fb9117ffdbfdb5dcc839102c0b700ee69b304e3 2163062 ocaml-source_4.02.3-9_all.deb
c1ed070d8ad0fd5c19f6c6f53532a9a6620fdfe8 8518 ocaml_4.02.3-9_amd64.buildinfo
8437a4604864b99967677b952243a39cb943bb73 121276 ocaml_4.02.3-9_amd64.deb
Checksums-Sha256:
040dc49756f6f7800720512db3f1f2608b2ef91b565b1aaab898fcb67ed10adf 2728 ocaml_4.02.3-9.dsc
56af57ead3af66295b50d1df39883f6ef0e16fa75c3c88efe0792f4eeb121dd0 47924 ocaml_4.02.3-9.debian.tar.xz
f29529592dbe46e2a945ab391188429e342ab0aabf679fb6ee45f4cb61273e07 7974 ocaml-base-dbgsym_4.02.3-9_amd64.deb
99262b94100ee0ec172a67ddf701474fdb98bf7b28e08ff1a7227463df09e63c 62876 ocaml-base-nox-dbgsym_4.02.3-9_amd64.deb
2ae76fea331b67e54870e32d915721a56e3fa292aacee1d2ead9d20ecc82751a 542860 ocaml-base-nox_4.02.3-9_amd64.deb
1ac2d4a114678938e4e2ea833c9c66973d017473b8a303237d940f89bb238082 132524 ocaml-base_4.02.3-9_amd64.deb
79fa8fc44cd399e6c6f8fa06a17fe6897ff032bd2c77f485a63f3c32539af089 9483532 ocaml-compiler-libs_4.02.3-9_amd64.deb
d215d1388e729c6733cb90cc0ab6db3f3bbb6d2d289604717917d260068f9e0d 360590 ocaml-interp_4.02.3-9_amd64.deb
10b0bfb47225a78cb751bf87f74c79f40b686e9f538f520227d838512a873a7c 137026 ocaml-mode_4.02.3-9_all.deb
0eec5aeebc0d700244f731e5a8dc5cb496c6d44a6fdaf684b4666b2f15a3dbee 757654 ocaml-native-compilers-dbgsym_4.02.3-9_amd64.deb
2d85e705cacf1b33fbec744eec347ad6f80dc0ab6ccbf7a6b764c035e6d299b1 3100004 ocaml-native-compilers_4.02.3-9_amd64.deb
9068b1d1dbb6eda256f81305832c2daf7044256c9a43e7ad7a876278574afdca 1921812 ocaml-nox-dbgsym_4.02.3-9_amd64.deb
af14b21f77a9c3f1f6e45c1f8ab6d4e14ee64c4d028c5d0982c724bb697209bf 5653690 ocaml-nox_4.02.3-9_amd64.deb
ea7c0bb4caf8a5ba4271908f8cc54a1512dcea581288a5c3a7633d3ae59142d3 2163062 ocaml-source_4.02.3-9_all.deb
5823ecd6682af0c103f00dd5c1677727aca288de8853002e7d14cfd246c21e2c 8518 ocaml_4.02.3-9_amd64.buildinfo
ee7ad69c0633a651f50a774839af8cc33ed0ddd170c330c895f5201dddfbe63d 121276 ocaml_4.02.3-9_amd64.deb
Files:
159069086e74add3937769ba89238025 2728 ocaml optional ocaml_4.02.3-9.dsc
62266f110ef3821dc266c7e11c214ad5 47924 ocaml optional ocaml_4.02.3-9.debian.tar.xz
e3e546eb89c5eed6df0e0c091fed4acf 7974 debug extra ocaml-base-dbgsym_4.02.3-9_amd64.deb
f77774cf1f0b1b5fe31816176f5556fa 62876 debug extra ocaml-base-nox-dbgsym_4.02.3-9_amd64.deb
8bdfa607728fe9a690afc504d329bf40 542860 ocaml optional ocaml-base-nox_4.02.3-9_amd64.deb
19290aedf51875ac114d3bcec1b41060 132524 ocaml optional ocaml-base_4.02.3-9_amd64.deb
e8ac451d1d18501a1fc708db94910be1 9483532 ocaml optional ocaml-compiler-libs_4.02.3-9_amd64.deb
20037b4a7f69d675edeefea9cc14bb70 360590 ocaml optional ocaml-interp_4.02.3-9_amd64.deb
a7461616990f71fa59cd325da5676fff 137026 ocaml optional ocaml-mode_4.02.3-9_all.deb
2fb2edb6782da8d0cb177c733489a2d6 757654 debug extra ocaml-native-compilers-dbgsym_4.02.3-9_amd64.deb
aa0c2e6b2f7982794e87946b2be8e767 3100004 ocaml optional ocaml-native-compilers_4.02.3-9_amd64.deb
0fabd645560138bf7e36bdd2e80018cd 1921812 debug extra ocaml-nox-dbgsym_4.02.3-9_amd64.deb
29d44c08fbb3f3e02776216d93717535 5653690 ocaml optional ocaml-nox_4.02.3-9_amd64.deb
1a1cb1e1d8110911c25dc3c6112d3a94 2163062 ocaml optional ocaml-source_4.02.3-9_all.deb
81240e87650c93789063e5466ac9e140 8518 ocaml optional ocaml_4.02.3-9_amd64.buildinfo
9ec04abab1afb6fc3ce432b616d254df 121276 ocaml optional ocaml_4.02.3-9_amd64.deb
-----BEGIN PGP SIGNATURE-----
iQJFBAEBCAAvFiEEvYOKK6r540CL2WRoM74aCowu2P8FAlhbF3kRHG1laGRpQGRl
Ymlhbi5vcmcACgkQM74aCowu2P8j3A//RPSxIqjvFPCY6kjy+CkSLsYZmFhjeHe/
+BYGC2CCWQmwDJ2mdhOniGORgZLRmJ6PUh/e6lrAiuZWJW8e+VUYzB0SEFpbxSty
6pQZMiODCfyCd90oYganLX1AFDphhf7ej8m5qxQdUMAre2HkwLgop1/eVC8XVrd9
Aterdm5YC6ACqhT6aW3pmS/etUiRcf7Obs19xJv6VKL1y0l7C9ca83vvYDnedh/H
a+nvBjv70jXWhBBX2uoTo6gf1HHqBO107just7o6EqL+dC4eqboToZamAriL+ofB
G1kle1IqXMBw2GBydRE30iJGWM9VKnEtJ9VpPsya6IywB/oJAg/yYB6mQd/ALAZU
TXjFRb22nNOq5N4MhCDRB4g00570+w4npHby7voh9oEybus2QIxzgo2uETmDC4kJ
HxmXGD1VUi0Kv5idFLJ5fDjDggLfGUrT/ERc9nF2+XMMPdz5y9KbJ0zDzuKQAWQI
EkaJKfiAlZigX9McwzsjyuRQWSfx/CEDJpUsxvY9hSeYTkOYVRfFqCqHl87WS56H
2VDDbLa+e/f6xrU0uU0ervDFgWjTKA1un9fFKAMeFgksKOVKUSxBAas7PGv2iUvP
VKcVV9ZDVIciK12wTOUEkPukeAE7/rrATznpOFXmFW+ZPdRfH5HB+Vlj3/rcyCez
hMSu9OZcP6g=
=YoIn
-----END PGP SIGNATURE-----
--- End Message ---
Reply to: