Your message dated Sun, 16 Apr 2023 21:56:10 +0200 with message-id <8cd3f9c3-f85f-d94f-82a0-9b383dfe69b6@debian.org> and subject line Re: Bug#1034376: unblock: opm-common/2022.10+ds-7 has caused the Debian Bug report #1034376, regarding unblock: opm-common/2022.10+ds-7 to be marked as done. This means that you claim that the problem has been dealt with. If this is not the case it is now your responsibility to reopen the Bug report if necessary, and/or fix the problem forthwith. (NB: If you are a system administrator and have no idea what this message is talking about, this may indicate a serious mail system misconfiguration somewhere. Please contact owner@bugs.debian.org immediately.) -- 1034376: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1034376 Debian Bug Tracking System Contact owner@bugs.debian.org with problems
--- Begin Message ---
- To: Debian Bug Tracking System <submit@bugs.debian.org>
- Subject: unblock: opm-common/2022.10+ds-7
- From: Markus Blatt <markus@dr-blatt.de>
- Date: Thu, 13 Apr 2023 22:33:54 +0200
- Message-id: <[🔎] 168141803459.411411.13542875514808057491.reportbug@smaug.dr-blatt.de>
Package: release.debian.org Severity: normal User: release.debian.org@packages.debian.org Usertags: unblock Please unblock package opm-common It contains an important fix of the printed version number. It is only blocked because britney thinks that autopkgtests have not run, but they have [ Reason ] It contains an important fix of the printed version number and reduces resource usage during the build. [ Impact ] No impact for users if there no binary rebuilds. In the case of rebuilds resource usage on buildd will be much higher than with new version. [ Tests ] All autopkgtest have run successfully for official architectures with binary packages [ Risks ] None that I can think of [ Checklist ] [X] all changes are documented in the d/changelog [X] I reviewed all changes and I approve them [X] attach debdiff against the package in testing [ Other info ] None unblock opm-common/2022.10+ds-7diff --git a/debian/changelog b/debian/changelog index 0af409a45..a011537d5 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,23 @@ +opm-common (2022.10+ds-7) unstable; urgency=medium + + * d/control: Limit the architectures to known working ones. + + -- Markus Blatt <markus@dr-blatt.de> Thu, 23 Mar 2023 08:34:59 +0100 + +opm-common (2022.10+ds-6) unstable; urgency=medium + + * Reduce compile time and resources with g++-12 + * d: Fixed version string used in binaries. + + -- Markus Blatt <markus@dr-blatt.de> Wed, 08 Mar 2023 23:05:15 +0100 + +opm-common (2022.10+ds-5) experimental; urgency=medium + + * Backported fixes for python tests from upstream (to fix mipsel*?) + * d/rules: Try to limit parallel builds to ensure 3GB RAM per process + + -- Markus Blatt <markus@dr-blatt.de> Sun, 05 Feb 2023 13:59:47 +0100 + opm-common (2022.10+ds-4) unstable; urgency=medium * Make sure copy_python is run before CopyHeaders Closes: #1029440 diff --git a/debian/control b/debian/control index 86112f71a..55debe51b 100644 --- a/debian/control +++ b/debian/control @@ -2,9 +2,9 @@ Source: opm-common Priority: optional Maintainer: Debian Science Maintainers <debian-science-maintainers@lists.alioth.debian.org> Uploaders: Arne Morten Kvarving <arne.morten.kvarving@sintef.no>, Markus Blatt <markus@dr-blatt.de> -Build-Depends: cmake (>=3.10), mpi-default-bin, mpi-default-dev, +Build-Depends: cmake (>=3.10), mpi-default-bin, mpi-default-dev, bc, procps, debhelper-compat (= 12), libcjson-dev, libfmt-dev, quilt, dh-python, - pkg-config, git, libtool, doxygen, graphviz, + pkg-config, lsb-release, libtool, doxygen, graphviz, texlive-latex-extra, texlive-latex-recommended, ghostscript, libboost-system-dev, libboost-test-dev, zlib1g-dev, python3-dev, libpython3-dev, python3-numpy, python3-distutils, @@ -19,7 +19,7 @@ Rules-Requires-Root: no Package: libopm-common Pre-Depends: ${misc:Pre-Depends} -Architecture: any +Architecture: amd64 arm64 armel ia64 m68k mips64el mipsel ppc64el riscv64 Multi-Arch: same Depends: ${shlibs:Depends}, ${misc:Depends}, ${python3:Depends} Provides: ${opm:shared-library} @@ -36,7 +36,7 @@ Description: Tools for Eclipse reservoir simulation files -- library Package: libopm-common-bin Pre-Depends: ${misc:Pre-Depends} -Architecture: any +Architecture: amd64 arm64 armel ia64 m68k mips64el mipsel ppc64el riscv64 Multi-Arch: foreign Depends: ${shlibs:Depends}, ${misc:Depends} Replaces: libopm-common1-bin @@ -52,7 +52,7 @@ Description: Tools for Eclipse reservoir simulation files -- utility programs Package: libopm-common-dev Section: libdevel -Architecture: any +Architecture: amd64 arm64 armel ia64 m68k mips64el mipsel ppc64el riscv64 Multi-Arch: same Suggests: libopm-common-doc Replaces: libopm-common1-dev @@ -85,7 +85,7 @@ Description: Tools for Eclipse reservoir simulation files -- documentation Package: python3-opm-common Section: python Pre-Depends: ${misc:Pre-Depends} -Architecture: any +Architecture: amd64 arm64 armel ia64 m68k mips64el mipsel ppc64el riscv64 Depends: ${shlibs:Depends}, ${misc:Depends}, ${python3:Depends}, libopm-common, python3-numpy, python3-decorator Description: Tools for Eclipse reservoir simulation files -- Python wrappers The Open Porous Media (OPM) software suite provides libraries and diff --git a/debian/opm-debian.mk b/debian/opm-debian.mk index 3b74fbe07..ea37112a8 100644 --- a/debian/opm-debian.mk +++ b/debian/opm-debian.mk @@ -2,10 +2,10 @@ include /usr/share/dpkg/architecture.mk include /usr/share/dpkg/pkg-info.mk include /usr/share/dune/dune-debian.env -LSB_RELEASE = lsb_release -r | sed "s/.*:\s*\([^\s]*\).*/\1/" +LSB_RELEASE = lsb_release -d | sed "s/.*:\s\+\(.*\)/\1/" # Needed for reproducable builds as we use __FILE__ export DEB_BUILD_MAINT_OPTIONS += reproducible=+fixfilepath -OPM_DEBIAN_CMAKE_FLAGS += -DBUILD_SHARED_LIBS=1 -DCMAKE_INSTALL_DOCDIR=share/doc/lib$(DEB_SOURCE) -DPYTHON_INSTALL_PREFIX=lib/python3/dist-packages -DOPM_INSTALL_COMPILED_PYTHON=OFF -DUSE_RUNPATH=OFF -DWITH_NATIVE=OFF -DUSE_MPI=ON -DUSE_BASH_COMPLETIONS_DIR=ON -DOPM_BINARY_PACKAGE_VERSION="$(DEB_VENDOR) $(LSB_RELEASE): $(DEB_VERSION)" +OPM_DEBIAN_CMAKE_FLAGS += -DBUILD_SHARED_LIBS=1 -DCMAKE_INSTALL_DOCDIR=share/doc/lib$(DEB_SOURCE) -DPYTHON_INSTALL_PREFIX=lib/python3/dist-packages -DOPM_INSTALL_COMPILED_PYTHON=OFF -DUSE_RUNPATH=OFF -DWITH_NATIVE=OFF -DUSE_MPI=ON -DUSE_BASH_COMPLETIONS_DIR=ON -DOPM_BINARY_PACKAGE_VERSION="$(LSB_RELEASE): $(DEB_VERSION)" OPM_DEBIAN_SHLIB = $(subst ~,.,lib$(DEB_SOURCE)-$(DEB_VERSION_UPSTREAM)) diff --git a/debian/patches/0011-python-Use-assertEqual-instead-of-deprecated-assertE.patch b/debian/patches/0011-python-Use-assertEqual-instead-of-deprecated-assertE.patch new file mode 100644 index 000000000..72eb9524d --- /dev/null +++ b/debian/patches/0011-python-Use-assertEqual-instead-of-deprecated-assertE.patch @@ -0,0 +1,27 @@ +From: Markus Blatt <markus@dr-blatt.de> +Date: Fri, 13 Jan 2023 16:59:26 +0100 +Subject: [python] Use assertEqual instead of deprecated assertEquals- + +Prevents warnings like +``` +1: test_faults (tests.test_state.TestState2.test_faults) ... /<<PKGBUILDDIR>>/obj-arm-linux-gnueabihf/python/tests/test_state.py:124: DeprecationWarning: Please use assertEqual instead. +1: self.assertEquals([], self.state.faultNames()) +``` +On Debian unstable. +--- + python/tests/test_state.py | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/python/tests/test_state.py b/python/tests/test_state.py +index 9051df7..f756e11 100644 +--- a/python/tests/test_state.py ++++ b/python/tests/test_state.py +@@ -121,7 +121,7 @@ SATNUM + self.state.tables().evaluate(tab, 0, 'NO', 1) + + def test_faults(self): +- self.assertEquals([], self.state.faultNames()) ++ self.assertEqual([], self.state.faultNames()) + parser = Parser() + faultdeck = parser.parse_string(self.FAULTS_DECK) + faultstate = EclipseState(faultdeck) diff --git a/debian/patches/0012-python-Prevent-ResourceWarning-unclosed-file-_io.Tex.patch b/debian/patches/0012-python-Prevent-ResourceWarning-unclosed-file-_io.Tex.patch new file mode 100644 index 000000000..45218786f --- /dev/null +++ b/debian/patches/0012-python-Prevent-ResourceWarning-unclosed-file-_io.Tex.patch @@ -0,0 +1,22 @@ +From: Markus Blatt <markus@dr-blatt.de> +Date: Fri, 13 Jan 2023 17:06:43 +0100 +Subject: [python] Prevent "ResourceWarning: unclosed file <_io.TextIOWrapper + +--- + python/tests/test_time_vector.py | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +diff --git a/python/tests/test_time_vector.py b/python/tests/test_time_vector.py +index 4f27413..6c81782 100644 +--- a/python/tests/test_time_vector.py ++++ b/python/tests/test_time_vector.py +@@ -83,7 +83,8 @@ class TestTimeVector(unittest.TestCase): + datetime.datetime(1998, 8, 1)]) + + def test_str(self): +- tv = TimeVector(datetime.date(1997, 11, 6), base_string = open(test_path("data/schedule/part1.sch")).read()) ++ with open(test_path("data/schedule/part1.sch")) as f: ++ tv = TimeVector(datetime.date(1997, 11, 6), base_string = f.read()) + tv.load(test_path("data/schedule/part3.sch")) + tv.load(test_path("data/schedule/part2.sch")) + diff --git a/debian/patches/0013-python-Run-unit-tests-directly-and-not-via-setup.py-.patch b/debian/patches/0013-python-Run-unit-tests-directly-and-not-via-setup.py-.patch new file mode 100644 index 000000000..c99ed6222 --- /dev/null +++ b/debian/patches/0013-python-Run-unit-tests-directly-and-not-via-setup.py-.patch @@ -0,0 +1,26 @@ +From: Markus Blatt <markus@dr-blatt.de> +Date: Fri, 13 Jan 2023 18:12:31 +0100 +Subject: [python] Run unit tests directly and not via setup.py (deprecated). + +Thus we get rid of the warning: + +WARNING: Testing via this command is deprecated and will be removed in +a future version. Users looking for a generic test entry point +independent of test runner are encouraged to use tox. +--- + CMakeLists.txt | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/CMakeLists.txt b/CMakeLists.txt +index 9039dc2..d057ece 100644 +--- a/CMakeLists.txt ++++ b/CMakeLists.txt +@@ -477,7 +477,7 @@ if (OPM_ENABLE_PYTHON) + # testing. + add_test(NAME python_tests + WORKING_DIRECTORY ${PROJECT_BINARY_DIR}/python +- COMMAND ${CMAKE_COMMAND} -E env LD_LIBRARY_PATH=${PROJECT_BINARY_DIR}/lib ${Python3_EXECUTABLE} setup.py build_ext --dry-run --build-lib ${PROJECT_BINARY_DIR}/python test ++ COMMAND ${CMAKE_COMMAND} -E env LD_LIBRARY_PATH=${PROJECT_BINARY_DIR}/lib ${Python3_EXECUTABLE} -m unittest discover + ) + + set_target_properties(opmcommon PROPERTIES POSITION_INDEPENDENT_CODE ON) diff --git a/debian/patches/0014-Refactored-ParserInit.cpp-to-reduce-ram-needed-for-c.patch b/debian/patches/0014-Refactored-ParserInit.cpp-to-reduce-ram-needed-for-c.patch new file mode 100644 index 000000000..a5a6b1bde --- /dev/null +++ b/debian/patches/0014-Refactored-ParserInit.cpp-to-reduce-ram-needed-for-c.patch @@ -0,0 +1,225 @@ +From: Markus Blatt <markus@dr-blatt.de> +Date: Fri, 20 Jan 2023 19:49:37 +0100 +Subject: Refactored ParserInit.cpp to reduce ram needed for compilation + +--- + CMakeLists.txt | 1 + + CopyHeaders.cmake | 3 + + GenerateKeywords.cmake | 8 +- + opm/input/eclipse/Generator/KeywordGenerator.hpp | 2 +- + .../input/eclipse/Generator/KeywordGenerator.cpp | 93 ++++++++++++++++++---- + .../eclipse/Parser/createDefaultKeywordList.cpp | 2 +- + 6 files changed, 90 insertions(+), 19 deletions(-) + +diff --git a/CMakeLists.txt b/CMakeLists.txt +index d057ece..e5ae79e 100644 +--- a/CMakeLists.txt ++++ b/CMakeLists.txt +@@ -136,6 +136,7 @@ macro (sources_hook) + list(INSERT opm-common_SOURCES 0 ${PROJECT_BINARY_DIR}/ParserInit.cpp) + foreach (name A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) + list(INSERT opm-common_SOURCES 0 ${PROJECT_BINARY_DIR}/ParserKeywords/${name}.cpp) ++ list(INSERT opm-common_SOURCES 0 ${PROJECT_BINARY_DIR}/ParserKeywords/ParserInit${name}.cpp) + list(INSERT opm-common_HEADERS 0 ${PROJECT_BINARY_DIR}/include/opm/input/eclipse/Parser/ParserKeywords/${name}.hpp) + endforeach() + if (OPM_ENABLE_EMBEDDED_PYTHON) +diff --git a/CopyHeaders.cmake b/CopyHeaders.cmake +index 044fbb3..4c243e3 100644 +--- a/CopyHeaders.cmake ++++ b/CopyHeaders.cmake +@@ -31,4 +31,7 @@ foreach (name A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) + execute_process(COMMAND ${CMAKE_COMMAND} -E copy_if_different + ${BASE_DIR}/tmp_gen/ParserKeywords/${name}.cpp + ${BASE_DIR}/ParserKeywords/${name}.cpp) ++ execute_process(COMMAND ${CMAKE_COMMAND} -E copy_if_different ++ ${BASE_DIR}/tmp_gen/ParserKeywords/ParserInit${name}.cpp ++ ${BASE_DIR}/ParserKeywords/ParserInit${name}.cpp) + endforeach() +diff --git a/GenerateKeywords.cmake b/GenerateKeywords.cmake +index 16a94c4..9320f92 100644 +--- a/GenerateKeywords.cmake ++++ b/GenerateKeywords.cmake +@@ -54,9 +54,13 @@ set( genkw_argv keyword_list.argv + + foreach (name A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) + list(APPEND _tmp_output ${PROJECT_BINARY_DIR}/tmp_gen/ParserKeywords/${name}.cpp +- ${PROJECT_BINARY_DIR}/tmp_gen/include/opm/input/eclipse/Parser/ParserKeywords/${name}.hpp) ++ ${PROJECT_BINARY_DIR}/tmp_gen/include/opm/input/eclipse/Parser/ParserKeywords/${name}.hpp ++ ${PROJECT_BINARY_DIR}/tmp_gen/ParserKeywords/ParserInit${name}.cpp ++ ${PROJECT_BINARY_DIR}/tmp_gen/include/opm/input/eclipse/Parser/ParserKeywords/ParserInit${name}.hpp) + list(APPEND _target_output ${PROJECT_BINARY_DIR}/ParserKeywords/${name}.cpp +- ${PROJECT_BINARY_DIR}/include/opm/input/eclipse/Parser/ParserKeywords/${name}.hpp) ++ ${PROJECT_BINARY_DIR}/include/opm/input/eclipse/Parser/ParserKeywords/${name}.hpp ++ ${PROJECT_BINARY_DIR}/ParserKeywords/ParserInit${name}.cpp ++ ${PROJECT_BINARY_DIR}/include/opm/input/eclipse/Parser/ParserKeywords/ParserInit${name}.hpp) + endforeach() + + foreach(name TestKeywords.cpp ParserInit.cpp) +diff --git a/opm/input/eclipse/Generator/KeywordGenerator.hpp b/opm/input/eclipse/Generator/KeywordGenerator.hpp +index e571193..7c24f18 100644 +--- a/opm/input/eclipse/Generator/KeywordGenerator.hpp ++++ b/opm/input/eclipse/Generator/KeywordGenerator.hpp +@@ -39,7 +39,7 @@ namespace Opm { + static void updateFile(const std::stringstream& newContent, const std::string& filename); + + void updateBuiltInHeader(const KeywordLoader& loader, const std::string& headerBuildPath, const std::string& headerPath) const; +- void updateInitSource(const KeywordLoader& loader, const std::string& sourceFile ) const; ++ void updateInitSource(const KeywordLoader& loader, const std::string& sourceFile, const std::string& sourcePath) const; + void updateKeywordSource(const KeywordLoader& loader, const std::string& sourceFile ) const; + void updatePybindSource(const KeywordLoader& loader , const std::string& sourceFile ) const; + void updateHeader(const KeywordLoader& loader, const std::string& headerBuildPath, const std::string& headerPath) const; +diff --git a/src/opm/input/eclipse/Generator/KeywordGenerator.cpp b/src/opm/input/eclipse/Generator/KeywordGenerator.cpp +index f2179cf..418f311 100644 +--- a/src/opm/input/eclipse/Generator/KeywordGenerator.cpp ++++ b/src/opm/input/eclipse/Generator/KeywordGenerator.cpp +@@ -65,18 +65,34 @@ namespace Opm { + std::filesystem::create_directories( file.parent_path()); + } + +- void KeywordGenerator::updateFile(const std::stringstream& newContent , const std::string& filename) { +- ensurePath(filename); ++ void ++ updateFile(const std::string& newContent, const std::string& filename) ++ { ++ KeywordGenerator::ensurePath(filename); + std::ofstream outputStream(filename); +- outputStream << newContent.str(); ++ outputStream << newContent; ++ } ++ void ++ KeywordGenerator::updateFile(const std::stringstream& newContent, const std::string& filename) ++ { ++ Opm::updateFile(newContent.str(), filename); + } + +- static void write_file( const std::stringstream& stream, const std::string& file, bool verbose, std::string desc = "source" ) { +- KeywordGenerator::updateFile( stream, file ); +- if( verbose ) ++ void ++ write_file(const std::string& content, const std::string& file, bool verbose, std::string desc = "source") ++ { ++ Opm::updateFile(content, file); ++ if (verbose) + fmt::print("Updated {} file written to {}\n", desc, file); + } + ++ ++ static void ++ write_file(const std::stringstream& stream, const std::string& file, bool verbose, std::string desc = "source") ++ { ++ write_file(stream.str(), file, verbose, desc); ++ } ++ + void KeywordGenerator::updateBuiltInHeader(const KeywordLoader& loader, const std::string& headerBuildPath, const std::string& headerPath) const { + std::stringstream newSource; + newSource << R"(#ifndef PARSER_KEYWORDS_BUILTIN_HPP +@@ -141,7 +157,11 @@ private: + write_file( newSource, final_path, m_verbose, "header" ); + } + +- void KeywordGenerator::updateInitSource(const KeywordLoader& loader , const std::string& sourceFile ) const { ++ void KeywordGenerator::updateInitSource(const KeywordLoader& loader , const std::string& sourceFile, ++ const std::string& sourcePath ) const { ++ std::filesystem::path parserInitSource(sourceFile); ++ std::string stem = parserInitSource.stem(); ++ std::filesystem::path parentPath = parserInitSource.parent_path(); + std::stringstream newSource; + newSource << R"( + #include <opm/input/eclipse/Parser/Parser.hpp> +@@ -150,7 +170,52 @@ private: + + for(const auto& kw_pair : loader) { + const auto& first_char = kw_pair.first; +- newSource << fmt::format("#include <opm/input/eclipse/Parser/ParserKeywords/{}.hpp>\n", first_char); ++ const std::string header = fmt::format(R"( ++#ifndef OPM_PARSER_INIT_{0}_HH ++#define OPM_PARSER_INIT_{0}_HH ++ ++namespace Opm {{ ++class Parser; ++namespace ParserKeywords {{ ++void addDefaultKeywords{0}(Parser& p); ++}} ++}} ++#endif ++)", ++ first_char); ++ auto charHeaderFile = parserInitSource; ++ charHeaderFile.replace_filename( ++ fmt::format("include/opm/input/eclipse/Parser/ParserKeywords/ParserInit{}.hpp", first_char)); ++ write_file(header, charHeaderFile, m_verbose, fmt::format("init header for {}", first_char)); ++ std::stringstream sourceStr; ++ sourceStr << fmt::format(R"( ++#include <opm/input/eclipse/Parser/Parser.hpp> ++#include <opm/input/eclipse/Parser/ParserKeywords/Builtin.hpp> ++#include<opm/input/eclipse/Parser/ParserKeywords/ParserInit{0}.hpp> ++#include <opm/input/eclipse/Parser/ParserKeywords/{0}.hpp> ++ ++namespace Opm {{ ++namespace ParserKeywords {{ ++void addDefaultKeywords{0}(Parser& p){{ ++ Builtin keywords; ++)", ++ first_char); ++ for (const auto& kw_pair : loader) { ++ const auto& keywords = kw_pair.second; ++ for (const auto& kw : keywords) ++ sourceStr << fmt::format(" p.addParserKeyword( keywords.{} );", kw.className()) << std::endl; ++ } ++ sourceStr << R"( ++ ++} ++} ++} ++)"; ++ auto charSourceFile = std::filesystem::path(sourcePath) / fmt::format("ParserInit{}.cpp", first_char); ++ write_file(sourceStr, charSourceFile, m_verbose, fmt::format("init source for {}", first_char)); ++ ++ newSource << fmt::format("#include <opm/input/eclipse/Parser/ParserKeywords/ParserInit{}.hpp>\n", ++ first_char); + } + + newSource << R"( +@@ -158,24 +223,22 @@ namespace Opm { + namespace ParserKeywords { + void addDefaultKeywords(Parser& p); + void addDefaultKeywords(Parser& p) { +- Builtin keywords; + )"; + +- for(const auto& kw_pair : loader) { +- const auto& keywords = kw_pair.second; +- for (const auto& kw: keywords) +- newSource << " p.addParserKeyword( keywords." << kw.className() << " );" << std::endl; ++ for(const auto& [first_char, keywords] : loader) { ++ newSource << fmt::format(" addDefaultKeywords{}(p);", first_char) << std::endl; + } + + newSource << R"( + } + } + void Parser::addDefaultKeywords() { +- ParserKeywords::addDefaultKeywords(*this); ++ ParserKeywords::addDefaultKeywords(*this); + } + } + )"; +- write_file( newSource, sourceFile, m_verbose, "init" ); ++ ++ write_file(newSource, sourceFile, m_verbose, "init"); + } + + +diff --git a/src/opm/input/eclipse/Parser/createDefaultKeywordList.cpp b/src/opm/input/eclipse/Parser/createDefaultKeywordList.cpp +index 9c65405..88c8cf1 100644 +--- a/src/opm/input/eclipse/Parser/createDefaultKeywordList.cpp ++++ b/src/opm/input/eclipse/Parser/createDefaultKeywordList.cpp +@@ -58,7 +58,7 @@ int main(int argc, char ** argv) { + Opm::KeywordGenerator generator( true ); + + generator.updateKeywordSource(loader , source_file_path ); +- generator.updateInitSource(loader , init_file_name ); ++ generator.updateInitSource(loader , init_file_name , source_file_path ); + generator.updateHeader(loader, header_file_base_path, header_file_path ); + generator.updateBuiltInHeader(loader, header_file_base_path, header_file_path ); + generator.updateTest( loader , test_file_name ); diff --git a/debian/patches/0015-Split-Builtin.hpp-into-multiple-compile-units-for-g-.patch b/debian/patches/0015-Split-Builtin.hpp-into-multiple-compile-units-for-g-.patch new file mode 100644 index 000000000..5c394e5b2 --- /dev/null +++ b/debian/patches/0015-Split-Builtin.hpp-into-multiple-compile-units-for-g-.patch @@ -0,0 +1,236 @@ +From: Markus Blatt <markus@dr-blatt.de> +Date: Wed, 8 Mar 2023 19:47:36 +0100 +Subject: Split Builtin.hpp into multiple compile units for g++-12. + +g++-12 needs quite some memory and time to compile it. +We now use several Builtin*.cpp for the imlementations for different +starting letters of eclipse keywords to use less resources for +compiling. +--- + CMakeLists.txt | 1 + + CopyHeaders.cmake | 3 + + GenerateKeywords.cmake | 2 + + opm/input/eclipse/Generator/KeywordGenerator.hpp | 3 +- + .../input/eclipse/Generator/KeywordGenerator.cpp | 88 ++++++++++++++-------- + .../eclipse/Parser/createDefaultKeywordList.cpp | 3 +- + 6 files changed, 65 insertions(+), 35 deletions(-) + +diff --git a/CMakeLists.txt b/CMakeLists.txt +index e5ae79e..6a00d68 100644 +--- a/CMakeLists.txt ++++ b/CMakeLists.txt +@@ -137,6 +137,7 @@ macro (sources_hook) + foreach (name A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) + list(INSERT opm-common_SOURCES 0 ${PROJECT_BINARY_DIR}/ParserKeywords/${name}.cpp) + list(INSERT opm-common_SOURCES 0 ${PROJECT_BINARY_DIR}/ParserKeywords/ParserInit${name}.cpp) ++ list(INSERT opm-common_SOURCES 0 ${PROJECT_BINARY_DIR}/ParserKeywords/Builtin${name}.cpp) + list(INSERT opm-common_HEADERS 0 ${PROJECT_BINARY_DIR}/include/opm/input/eclipse/Parser/ParserKeywords/${name}.hpp) + endforeach() + if (OPM_ENABLE_EMBEDDED_PYTHON) +diff --git a/CopyHeaders.cmake b/CopyHeaders.cmake +index 4c243e3..faf9f4e 100644 +--- a/CopyHeaders.cmake ++++ b/CopyHeaders.cmake +@@ -34,4 +34,7 @@ foreach (name A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) + execute_process(COMMAND ${CMAKE_COMMAND} -E copy_if_different + ${BASE_DIR}/tmp_gen/ParserKeywords/ParserInit${name}.cpp + ${BASE_DIR}/ParserKeywords/ParserInit${name}.cpp) ++ execute_process(COMMAND ${CMAKE_COMMAND} -E copy_if_different ++ ${BASE_DIR}/tmp_gen/ParserKeywords/Builtin${name}.cpp ++ ${BASE_DIR}/ParserKeywords/Builtin${name}.cpp) + endforeach() +diff --git a/GenerateKeywords.cmake b/GenerateKeywords.cmake +index 9320f92..ac21320 100644 +--- a/GenerateKeywords.cmake ++++ b/GenerateKeywords.cmake +@@ -56,10 +56,12 @@ foreach (name A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) + list(APPEND _tmp_output ${PROJECT_BINARY_DIR}/tmp_gen/ParserKeywords/${name}.cpp + ${PROJECT_BINARY_DIR}/tmp_gen/include/opm/input/eclipse/Parser/ParserKeywords/${name}.hpp + ${PROJECT_BINARY_DIR}/tmp_gen/ParserKeywords/ParserInit${name}.cpp ++ ${PROJECT_BINARY_DIR}/tmp_gen/ParserKeywords/Builtin${name}.cpp + ${PROJECT_BINARY_DIR}/tmp_gen/include/opm/input/eclipse/Parser/ParserKeywords/ParserInit${name}.hpp) + list(APPEND _target_output ${PROJECT_BINARY_DIR}/ParserKeywords/${name}.cpp + ${PROJECT_BINARY_DIR}/include/opm/input/eclipse/Parser/ParserKeywords/${name}.hpp + ${PROJECT_BINARY_DIR}/ParserKeywords/ParserInit${name}.cpp ++ ${PROJECT_BINARY_DIR}/ParserKeywords/Builtin${name}.cpp + ${PROJECT_BINARY_DIR}/include/opm/input/eclipse/Parser/ParserKeywords/ParserInit${name}.hpp) + endforeach() + +diff --git a/opm/input/eclipse/Generator/KeywordGenerator.hpp b/opm/input/eclipse/Generator/KeywordGenerator.hpp +index 7c24f18..aa7af85 100644 +--- a/opm/input/eclipse/Generator/KeywordGenerator.hpp ++++ b/opm/input/eclipse/Generator/KeywordGenerator.hpp +@@ -38,7 +38,8 @@ namespace Opm { + static std::string headerHeader( const std::string& ); + static void updateFile(const std::stringstream& newContent, const std::string& filename); + +- void updateBuiltInHeader(const KeywordLoader& loader, const std::string& headerBuildPath, const std::string& headerPath) const; ++ void updateBuiltInHeader(const KeywordLoader& loader, const std::string& headerBuildPath, ++ const std::string& headerPath, const std::string& sourcePath) const; + void updateInitSource(const KeywordLoader& loader, const std::string& sourceFile, const std::string& sourcePath) const; + void updateKeywordSource(const KeywordLoader& loader, const std::string& sourceFile ) const; + void updatePybindSource(const KeywordLoader& loader , const std::string& sourceFile ) const; +diff --git a/src/opm/input/eclipse/Generator/KeywordGenerator.cpp b/src/opm/input/eclipse/Generator/KeywordGenerator.cpp +index 418f311..b1a2dd7 100644 +--- a/src/opm/input/eclipse/Generator/KeywordGenerator.cpp ++++ b/src/opm/input/eclipse/Generator/KeywordGenerator.cpp +@@ -93,60 +93,79 @@ namespace Opm { + write_file(stream.str(), file, verbose, desc); + } + +- void KeywordGenerator::updateBuiltInHeader(const KeywordLoader& loader, const std::string& headerBuildPath, const std::string& headerPath) const { +- std::stringstream newSource; +- newSource << R"(#ifndef PARSER_KEYWORDS_BUILTIN_HPP ++ void KeywordGenerator::updateBuiltInHeader(const KeywordLoader& loader, const std::string& headerBuildPath, const std::string& headerPath, ++ const std::string& sourcePath ) const { ++ std::stringstream newHeader; ++ std::map<char, std::stringstream> newSources; ++ ++ newHeader << R"(#ifndef PARSER_KEYWORDS_BUILTIN_HPP + #define PARSER_KEYWORDS_BUILTIN_HPP + #include <unordered_map> + #include <fmt/format.h> ++#include <opm/input/eclipse/Parser/ParserKeyword.hpp> + )"; + + for(const auto& kw_pair : loader) { + const auto& first_char = kw_pair.first; +- newSource << fmt::format("#include <opm/input/eclipse/Parser/ParserKeywords/{}.hpp>\n", first_char); ++ newSources[first_char] << fmt::format("#include <opm/input/eclipse/Parser/ParserKeywords/{}.hpp>\n", first_char) ++ << fmt::format("#include <{}/Builtin.hpp>\n", headerPath) ++ << "namespace Opm { namespace ParserKeywords {\n"; + } + +- newSource << R"( ++ newHeader << R"( + namespace Opm { + namespace ParserKeywords { + struct Builtin { + )"; + for(const auto& kw_pair : loader) { + const auto& keywords = kw_pair.second; ++ auto& source = newSources[kw_pair.first]; + for (const auto& kw: keywords) +- newSource << fmt::format(" const ::Opm::ParserKeywords::{0} {0};\n", kw.className()); +- } +- +- for(const auto& kw_pair : loader) { +- const auto& keywords = kw_pair.second; +- for (const auto& kw: keywords) +- newSource << fmt::format(" const ::Opm::ParserKeyword& get_{0}() {{ return this->{0}; }};\n",kw.className()); ++ { ++ newHeader << fmt::format(" const ::Opm::ParserKeyword get_{0}();\n",kw.className()); ++ source << fmt::format("const ::Opm::ParserKeyword Builtin::get_{0}() {{ return {0}(); }};\n",kw.className()); ++ } + } + +- newSource << R"( +- const ::Opm::ParserKeyword& operator[](const std::string& keyword) const { +- if (this->keywords.empty()) { ++ newHeader << R"( ++ const ::Opm::ParserKeyword& operator[](const std::string& keyword) const { ++ if (this->keywords.empty()) { + )"; ++ std::stringstream declareEmplace; + + for(const auto& kw_pair : loader) { + const auto& keywords = kw_pair.second; ++ auto& source = newSources[kw_pair.first]; ++ newHeader << fmt::format(" emplace{}(this->keywords);\n", kw_pair.first); ++ source << fmt::format(R"( ++void Builtin::emplace{}([[maybe_unused]] std::unordered_map<std::string, ::Opm::ParserKeyword>& keywords) const {{ ++)", ++ kw_pair.first); ++ declareEmplace << fmt::format(R"( ++ void emplace{}(std::unordered_map<std::string, ::Opm::ParserKeyword>& keywords) const; ++)", ++ kw_pair.first); ++ + for (const auto& kw: keywords) +- newSource << fmt::format(" this->keywords.emplace(\"{0}\", this->{0});\n", kw.className()); ++ source << fmt::format(" keywords.emplace(\"{0}\", {0}());\n", kw.className()); ++ source <<"}\n"; ++ source <<"} }\n"; + } + +- newSource << R"( } +- const auto kw_iter = this->keywords.find(keyword); +- if (kw_iter == this->keywords.end()) +- throw std::invalid_argument(fmt::format("No builtin keyword: {}", keyword)); +- return kw_iter->second; +-} ++ newHeader << R"( } ++ const auto kw_iter = this->keywords.find(keyword); ++ if (kw_iter == this->keywords.end()) ++ throw std::invalid_argument(fmt::format("No builtin keyword: {}", keyword)); ++ return kw_iter->second; ++ } + +- const ::Opm::ParserKeyword& getKeyword(const std::string& keyword) const { return this->operator[](keyword); } ++ const ::Opm::ParserKeyword& getKeyword(const std::string& keyword) const { return this->operator[](keyword); } + )"; + +- newSource << R"( +-private: +- mutable std::unordered_map<std::string, ::Opm::ParserKeyword> keywords; ++ newHeader << "private:\n"; ++ newHeader << declareEmplace.str(); ++ newHeader << R"( ++ mutable std::unordered_map<std::string, ::Opm::ParserKeyword> keywords; + }; + } + } +@@ -154,7 +173,13 @@ private: + )"; + + const auto final_path = headerBuildPath + headerPath+ "/Builtin.hpp"; +- write_file( newSource, final_path, m_verbose, "header" ); ++ write_file( newHeader, final_path, m_verbose, "header" ); ++ for(auto&& [first_char, source]: newSources) ++ { ++ auto sourceFile = std::filesystem::path(sourcePath) / fmt::format("Builtin{}.cpp", ++ first_char); ++ write_file(source, sourceFile, m_verbose, fmt::format("builtin source for {}", first_char)); ++ } + } + + void KeywordGenerator::updateInitSource(const KeywordLoader& loader , const std::string& sourceFile, +@@ -190,21 +215,18 @@ void addDefaultKeywords{0}(Parser& p); + std::stringstream sourceStr; + sourceStr << fmt::format(R"( + #include <opm/input/eclipse/Parser/Parser.hpp> +-#include <opm/input/eclipse/Parser/ParserKeywords/Builtin.hpp> + #include<opm/input/eclipse/Parser/ParserKeywords/ParserInit{0}.hpp> + #include <opm/input/eclipse/Parser/ParserKeywords/{0}.hpp> + + namespace Opm {{ + namespace ParserKeywords {{ +-void addDefaultKeywords{0}(Parser& p){{ +- Builtin keywords; ++void addDefaultKeywords{0}([[maybe_unused]] Parser& p){{ ++ //Builtin keywords; + )", + first_char); +- for (const auto& kw_pair : loader) { + const auto& keywords = kw_pair.second; + for (const auto& kw : keywords) +- sourceStr << fmt::format(" p.addParserKeyword( keywords.{} );", kw.className()) << std::endl; +- } ++ sourceStr << fmt::format(" p.addParserKeyword( {}() );", kw.className()) << std::endl; + sourceStr << R"( + + } +diff --git a/src/opm/input/eclipse/Parser/createDefaultKeywordList.cpp b/src/opm/input/eclipse/Parser/createDefaultKeywordList.cpp +index 88c8cf1..ec8d73d 100644 +--- a/src/opm/input/eclipse/Parser/createDefaultKeywordList.cpp ++++ b/src/opm/input/eclipse/Parser/createDefaultKeywordList.cpp +@@ -60,7 +60,8 @@ int main(int argc, char ** argv) { + generator.updateKeywordSource(loader , source_file_path ); + generator.updateInitSource(loader , init_file_name , source_file_path ); + generator.updateHeader(loader, header_file_base_path, header_file_path ); +- generator.updateBuiltInHeader(loader, header_file_base_path, header_file_path ); ++ generator.updateBuiltInHeader(loader, header_file_base_path, header_file_path, ++ source_file_path ); + generator.updateTest( loader , test_file_name ); + if (argc >= 8) + generator.updatePybindSource(loader , argv[7]); diff --git a/debian/patches/0016-Added-missing-include-to-ParserTests.cpp.patch b/debian/patches/0016-Added-missing-include-to-ParserTests.cpp.patch new file mode 100644 index 000000000..f663a4aef --- /dev/null +++ b/debian/patches/0016-Added-missing-include-to-ParserTests.cpp.patch @@ -0,0 +1,29 @@ +From: Markus Blatt <markus@dr-blatt.de> +Date: Wed, 8 Mar 2023 19:48:38 +0100 +Subject: Added missing include to ParserTests.cpp + +--- + tests/parser/ParserTests.cpp | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +diff --git a/tests/parser/ParserTests.cpp b/tests/parser/ParserTests.cpp +index 4ab0ecf..85b3086 100644 +--- a/tests/parser/ParserTests.cpp ++++ b/tests/parser/ParserTests.cpp +@@ -35,6 +35,7 @@ + #include <opm/input/eclipse/Parser/ParserKeyword.hpp> + #include <opm/input/eclipse/Parser/ParserKeywords/A.hpp> + #include <opm/input/eclipse/Parser/ParserKeywords/S.hpp> ++#include <opm/input/eclipse/Parser/ParserKeywords/T.hpp> + #include <opm/input/eclipse/Parser/ParserKeywords/Builtin.hpp> + #include <opm/input/eclipse/Parser/ParserRecord.hpp> + +@@ -2355,7 +2356,7 @@ GUIDERAT + )"; + + BOOST_CHECK_EQUAL( parser.size(), 0 ); +- parser.addParserKeyword( builtin.GUIDERAT ); ++ parser.addParserKeyword( builtin.get_GUIDERAT() ); + BOOST_CHECK_EQUAL( parser.size(), 1 ); + auto deck = parser.parseString(deck_string); + BOOST_CHECK( deck.hasKeyword("GUIDERAT") ); diff --git a/debian/patches/series b/debian/patches/series index 735303415..0a9f71111 100644 --- a/debian/patches/series +++ b/debian/patches/series @@ -8,3 +8,9 @@ 0008-cmake-Limit-dimensions-for-Alberta.patch 0009-cmake-Also-search-for-Alberta-with-pkgconfig-to-be-D.patch 0011-Make-sure-copy_python-is-run-before-CopyHeaders-Clos.patch +0011-python-Use-assertEqual-instead-of-deprecated-assertE.patch +0012-python-Prevent-ResourceWarning-unclosed-file-_io.Tex.patch +0013-python-Run-unit-tests-directly-and-not-via-setup.py-.patch +0014-Refactored-ParserInit.cpp-to-reduce-ram-needed-for-c.patch +0015-Split-Builtin.hpp-into-multiple-compile-units-for-g-.patch +0016-Added-missing-include-to-ParserTests.cpp.patch diff --git a/debian/rules b/debian/rules index 71031ce82..6b310e5e9 100755 --- a/debian/rules +++ b/debian/rules @@ -7,10 +7,14 @@ include /usr/share/dpkg/architecture.mk OPM_DEBIAN_CMAKE_FLAGS = -DOPM_ENABLE_PYTHON=1 -DOPM_INSTALL_PYTHON=1 -DPYTHON_EXECUTABLE=/usr/bin/python3 -DOPM_INSTALL_COMPILED_PYTHON=OFF -DOPM_ENABLE_EMBEDDED_PYTHON=1 - +need_gb_ram_per_process=3 +free_ram=$(shell free -g | sed -n 2p| sed "s/ \+/ /g"| cut -d " " -f 2) +max_procs=$(shell echo "$(free_ram)/$(need_gb_ram_per_process)" | bc) +parallel_procs=$(shell if test "$(max_procs)" -lt "1"; then echo 1; else echo "$(max_procs)"; fi) %: - dh $@ --with python3 + echo "ram in gb: $(free_ram), needed ram per thread: $(need_gb_ram_per_process), threads: $(parallel_procs)" + dh $@ --with python3 --max-parallel=$(parallel_procs) include debian/opm-debian.mk include debian/opm-lib-debian.mk # for makeshlibs
--- End Message ---
--- Begin Message ---
- To: Markus Blatt <markus@dr-blatt.de>, 1034376-done@bugs.debian.org
- Subject: Re: Bug#1034376: unblock: opm-common/2022.10+ds-7
- From: Paul Gevers <elbrus@debian.org>
- Date: Sun, 16 Apr 2023 21:56:10 +0200
- Message-id: <8cd3f9c3-f85f-d94f-82a0-9b383dfe69b6@debian.org>
- In-reply-to: <[🔎] 168141803459.411411.13542875514808057491.reportbug@smaug.dr-blatt.de>
- References: <[🔎] 168141803459.411411.13542875514808057491.reportbug@smaug.dr-blatt.de>
Hi On 13-04-2023 22:33, Markus Blatt wrote:All autopkgtest have run successfully for official architectures with binary packagesTechnically not true because arch:all binaries are available everywhere.unblock opm-common/2022.10+ds-7Done. PaulAttachment: OpenPGP_signature
Description: OpenPGP digital signature
--- End Message ---