--- Begin Message ---
Package: release.debian.org
Severity: normal
User: release.debian.org@packages.debian.org
Usertags: unblock
Dear Release Team,
Please unblock package translate-toolkit.
We have been tracking the beta releases and release candidates of this
package over the past 6ish months so that we could deliver a version in
stretch that had Python 3 support. Stretch currently includes rc3 of
translate-toolkit and the diff to the final released version is minimal;
some additional tests, a couple of small bug fixes and documentation updates.
The package has a pretty good test-suite which is happily passing on
ci.debian.net.
A source debdiff attached and summarised below if that helps:
debian/changelog | 6
docs/conf.py | 2
docs/developers/releasing.rst | 4
docs/releases/2.0.0.rst | 256 ++++++++++++++++++++++++
docs/releases/index.rst | 1
tests/cli/data/test_po2txt/one.po | 13 +
tests/cli/data/test_po2txt/out.txt | 7
tests/cli/data/test_po2txt_threshold/one.po | 13 +
tests/cli/data/test_pofilter_manpage/stdout.txt | 2
tests/cli/test_po2txt.sh | 6
tests/cli/test_po2txt_threshold.sh | 7
translate/__version__.py | 6
translate/convert/convert.py | 15 -
translate/storage/mozilla_lang.py | 14 -
translate/storage/test_mozilla_lang.py | 39 +++
15 files changed, 374 insertions(+), 17 deletions(-)
unblock translate-toolkit/2.0.0-1
Thanks for considering,
Stuart on behalf of the Debian l10n developers team
diff -Nru translate-toolkit-2.0.0~rc3/debian/changelog translate-toolkit-2.0.0/debian/changelog
--- translate-toolkit-2.0.0~rc3/debian/changelog 2017-01-25 13:31:22.000000000 +1100
+++ translate-toolkit-2.0.0/debian/changelog 2017-02-26 23:39:30.000000000 +1100
@@ -1,3 +1,9 @@
+translate-toolkit (2.0.0-1) unstable; urgency=medium
+
+ * New upstream release.
+
+ -- Stuart Prescott <stuart@debian.org> Sun, 26 Feb 2017 23:39:30 +1100
+
translate-toolkit (2.0.0~rc3-1) unstable; urgency=medium
* New upstream release candidate.
diff -Nru translate-toolkit-2.0.0~rc3/docs/conf.py translate-toolkit-2.0.0/docs/conf.py
--- translate-toolkit-2.0.0~rc3/docs/conf.py 2017-01-24 05:25:31.000000000 +1100
+++ translate-toolkit-2.0.0/docs/conf.py 2017-01-28 06:51:52.000000000 +1100
@@ -67,7 +67,7 @@
# The short X.Y version.
version = '2.0.0'
# The full version, including alpha/beta/rc tags.
-release = '2.0.0rc3'
+release = '2.0.0'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
diff -Nru translate-toolkit-2.0.0~rc3/docs/developers/releasing.rst translate-toolkit-2.0.0/docs/developers/releasing.rst
--- translate-toolkit-2.0.0~rc3/docs/developers/releasing.rst 2017-01-24 05:25:31.000000000 +1100
+++ translate-toolkit-2.0.0/docs/developers/releasing.rst 2017-01-28 06:51:52.000000000 +1100
@@ -136,7 +136,7 @@
.. code-block:: console
$ mkvirtualenv build-ttk-release
- (build-ttk-release)$ pip install --upgrade pip
+ (build-ttk-release)$ pip install --upgrade setuptools pip
(build-ttk-release)$ pip install -r requirements/dev.txt
(build-ttk-release)$ make build
(build-ttk-release)$ deactivate
@@ -159,7 +159,7 @@
.. code-block:: console
$ mkvirtualenv test-ttk-release
- (test-ttk-release)$ pip install --upgrade pip
+ (test-ttk-release)$ pip install --upgrade setuptools pip
(test-ttk-release)$ pip install dist/translate-toolkit-$version.tar.bz2
diff -Nru translate-toolkit-2.0.0~rc3/docs/releases/2.0.0.rst translate-toolkit-2.0.0/docs/releases/2.0.0.rst
--- translate-toolkit-2.0.0~rc3/docs/releases/2.0.0.rst 1970-01-01 10:00:00.000000000 +1000
+++ translate-toolkit-2.0.0/docs/releases/2.0.0.rst 2017-01-28 06:51:52.000000000 +1100
@@ -0,0 +1,256 @@
+Translate Toolkit 2.0.0
+***********************
+
+*Released on 27 January 2017*
+
+This release contains many improvements and bug fixes. While it contains many
+general improvements, it also specifically contains needed changes and
+optimizations for the upcoming `Pootle <http://pootle.translatehouse.org/>`_
+2.8.0 and `Virtaal <http://virtaal.translatehouse.org>`_ releases.
+
+
+2.0.0 vs 2.0.0rc3
+=================
+
+- :doc:`po2txt </commands/txt2po>` works correctly again when ``--threshold``
+ option is passed (:issue:`3499`)
+- Minor fixes for Mozilla .lang format
+- Minor Python 3 fix
+
+
+Major changes
+=============
+
+- Python 3 compatibility thanks to Claude Paroz
+- Dropped support for Python 2.6
+- Support for new l20n format
+- Translate Toolkit can now easily be installed on Windows
+- Changes in storage API to expose a more standardized API
+
+
+Detailed changes
+================
+
+Python 3 support
+----------------
+
+- Translate Toolkit went through a massive code cleanup looking forward Python
+ 3 compatibility. There might be quirks that need to be fixed, so please test
+ and `report any issue <https://github.com/translate/translate/issues/new>`_
+ you might find.
+- Python 3.3-3.5 is now supported.
+
+
+Requirements
+------------
+
+- ``lxml`` requirement was raised to 3.5.0 in order to simplify code.
+- Updated and pinned requirements
+- Removed misleading extra requirements files
+
+
+Formats and Converters
+----------------------
+
+- PO
+
+ - msgid comments (KDE style) are only parsed from msgid now.
+ - Fixed parsing of PO files with first entry in unicode
+ - Fixed parsing of locations with percent char
+
+- XLIFF
+
+ - Unaccepted ASCII control characters are now escaped in XLIFF
+
+- DTD
+
+ - Newlines are now skipped when parsing (:issue:`3390`).
+ - Invalid ampersands are not scrubbed anymore.
+ - label+accesskey is now only extracted if it is not followed by space.
+
+- Properties
+
+ - Keys can contain delimiters if they are properly wrapped (:issue:`3275`).
+ - Fix control characters escaping for utf-8 encoding.
+ - :ref:`po2prop <po2prop>` removes fully untranslated units if
+ required
+ - :ref:`po2prop <po2prop>` skips first entry in PO file (:issue:`3463`)
+
+- Mozilla .lang
+
+ - ``{ok}`` marker is now more cleanly removed
+ - Always output last unit followed by trailing newlines
+ - Added support for headers and tag comments
+ - ``MAX_LENGTH`` is now parsed into comment
+ - File line endings are now remembered, defaulting to Unix LF
+
+- Mozilla's l20n
+
+ - Added this new format storage class
+ - Added variants and traits support
+ - Added new converters :doc:`l20n2po </commands/l20n2po>` and
+ :ref:`po2l20n <po2l20n>`
+
+- Android
+
+ - Unknown locales no longer produce failures.
+ - Simplify newlines handling as the format now handles \n and newline equally
+ (:issue:`3262`)
+ - Moved all namespaces to <resources> element.
+ - Simplified newlines handling
+
+- ODF
+
+ - :doc:`odf2xliff </commands/odf2xliff>` now extracts all the text
+ (:issue:`3239`).
+
+- ts
+
+ - XML declaration is written with double quotes.
+ - Self-closing for 'location' elements are not output anymore.
+
+- JSON
+
+ - Output now includes a trailing newline.
+ - Unit ordering is maintained (:issue:`3394`).
+ - Added ``--removeuntranslated`` option to :doc:`po2json </commands/json2po>`
+
+- YAML
+
+ - YAML format support has been added.
+
+- txt
+
+ - :doc:`po2txt </commands/txt2po>` works correctly again when ``--threshold``
+ option is passed (:issue:`3499`)
+
+- ical
+
+ - Enabled this format for Python 3 too.
+
+- TermBase eXchange (TBX)
+
+ - :doc:`tbx2po </commands/tbx2po>` converter added
+ - Added basic support for Parts of Speech and term definitions.
+
+- Fixed error when writing back to the same file (:issue:`3419`).
+
+
+Filters and Checks
+------------------
+
+- Added the ability to skip some checks for some languages in specific checkers
+- ``accelerators`` check reports an error if accelerator is present for several
+ Indic languages in ``MozillaChecker`` checker.
+- Added ``l20nChecker`` to do custom checking for Mozilla's new l20n format.
+- LibreOffice checker no longer checks for Python brace format (:issue:`3303`).
+- LibreOffice validxml check correctly matches self-closing tags.
+- Numbers check now handles non latin numbers. Support for non latin numbers
+ has been added for Arabic, Assamese, Bengali and Persian languages.
+- Fixed issue that prevented standard checks from being used in Pootle with
+ default settings.
+- Fixed missing attribute warning displayed when using ``GnomeChecker``,
+ ``LibreOfficeChecker`` and ``MozillaChecker`` checkers.
+- Added language specific ``RomanianChecker``.
+
+
+Tools
+-----
+
+- :doc:`posegment </commands/posegment>` now correctly segments Japanese
+ strings with half width punctuation sign (:issue:`3280`).
+- :doc:`pocount </commands/pocount>` now outputs csv header in one line. It
+ also outputs using color.
+- ``buildxpi`` was adjusted to current Mozilla needs
+
+
+Languages
+---------
+
+- Fixed plural form for Montenegro, Macedonian, Songhay, Tajik, Slovenian and
+ Turkish.
+- Added plural forms for Bengali (Bangladesh), Konkani, Kashmiri, Sanskrit,
+ Silesian and Yue (Cantonese).
+- Added valid accelerators for Polish.
+- Renamed Oriya to Odia.
+- Altered Manipuri name to include its most common name Meithei.
+- Added language settings for Brazilian Portuguese.
+- Added Danish valid accelerators characters (:issue:`3487`).
+- Added additional special characters for Scottish Gaelic.
+
+
+Setup
+-----
+
+- Fixed Inno Setup builds allowing to easily install Translate Toolkit on
+ Windows using the ``pip`` installer. Commands are compiled to .exe files.
+- Updated installation instructions for Windows
+
+
+API changes
+-----------
+
+- Dropped ``translate.misc.dictutils.ordereddict`` in favor of
+ ``collections.OrderedDict``.
+- Added encoding handling in base ``TranslationStore`` class exposing a single
+ API.
+- Encoding detection in ``TranslationStore`` has been improved.
+- Standardized UnitClass definition across ``TranslationStore`` subclasses.
+- ``translate.misc.multistring.multistring``:
+
+ - Fixed list coercion to text
+ - Fixed comparison regression with multistrings (:issue:`3404`).
+ - Re-added ``str`` method (:issue:`3428`).
+ - Fixed ``__hash__`` (:issue:`3434`).
+
+
+API deprecation
+---------------
+
+- Passing non-ASCII bytes to the ``multistring`` class has been deprecated, as
+ well as the ``encoding`` argument to it.
+ Applications should always construct ``multistring`` objects by passing
+ characters (``unicode`` in Python 2, ``str`` in Python 3), not bytes. Support
+ for passing non-ASCII bytes will be removed in the next version.
+- ``TxtFile.getoutput()`` and ``dtdfile.getoutput()`` have been deprecated.
+ Either call ``bytes(<file_instance>)`` or use the
+ ``file_instance.serialize()`` API if you need to get the serialized store
+ content of a ``TxtFile`` or ``dtdfile`` instance.
+
+
+General
+-------
+
+- Dropped support for Python 2.6 since it is no longer supported by the Python
+ Foundation. Sticking to it was making us difficult to maintain code while we
+ move to Python 3.
+- Misc docs cleanups.
+- Added more tests.
+- Increased Python code health.
+- Legacy, deprecated and unused code cleansing:
+
+ - Dropped code for no longer supported Python versions.
+ - Removed unused code from various places across codebase.
+ - The legacy ``translate.search.indexing.PyLuceneIndexer1`` was removed.
+ - The deprecated ``translate.storage.properties.find_delimiter()`` was
+ removed and replaced by the
+ ``translate.storage.properties.Dialect.find_delimiter()`` class method.
+ - Python scripts are now available via `console_scripts` entry point, thus
+ allowing to drop dummy files for exposing the scripts.
+
+
+...and loads of general code cleanups and of course many many bugfixes.
+
+
+Contributors
+============
+
+This release was made possible by the following people:
+
+Claude Paroz, Leandro Regueiro, Dwayne Bailey, Michal Čihař, Taras Semenenko,
+Ryan Northey, Stuart Prescott, Kai Pastor, Julen Ruiz Aizpuru, Friedel Wolff,
+Hiroshi Miura, Thorbjørn Lindeijer, Melvi Ts, Jobava, Jerome Leclanche, Jakub
+Wilk, Adhika Setya Pramudita, Zibi Braniecki, Zdenek Juran, Yann Diorcet, Nick
+Shaforostoff, Jaka Kranjc, Christian Lohmaier, beernarrd.
+
+And to all our bug finders and testers, a Very BIG Thank You.
diff -Nru translate-toolkit-2.0.0~rc3/docs/releases/index.rst translate-toolkit-2.0.0/docs/releases/index.rst
--- translate-toolkit-2.0.0~rc3/docs/releases/index.rst 2017-01-24 05:25:31.000000000 +1100
+++ translate-toolkit-2.0.0/docs/releases/index.rst 2017-01-28 06:51:52.000000000 +1100
@@ -8,6 +8,7 @@
.. toctree::
:maxdepth: 1
+ 2.0.0 <2.0.0>
2.0.0rc3 <2.0.0rc3>
2.0.0rc2 <2.0.0rc2>
2.0.0rc1 <2.0.0rc1>
diff -Nru translate-toolkit-2.0.0~rc3/tests/cli/data/test_po2txt/one.po translate-toolkit-2.0.0/tests/cli/data/test_po2txt/one.po
--- translate-toolkit-2.0.0~rc3/tests/cli/data/test_po2txt/one.po 1970-01-01 10:00:00.000000000 +1000
+++ translate-toolkit-2.0.0/tests/cli/data/test_po2txt/one.po 2017-01-28 06:51:52.000000000 +1100
@@ -0,0 +1,13 @@
+msgctxt "first"
+msgid "One"
+msgstr "Een"
+
+msgctxt "second"
+msgid "One"
+msgstr "Won"
+
+msgid "Two two"
+msgstr "Twee twee"
+
+msgid "Three three"
+msgstr ""
diff -Nru translate-toolkit-2.0.0~rc3/tests/cli/data/test_po2txt/out.txt translate-toolkit-2.0.0/tests/cli/data/test_po2txt/out.txt
--- translate-toolkit-2.0.0~rc3/tests/cli/data/test_po2txt/out.txt 1970-01-01 10:00:00.000000000 +1000
+++ translate-toolkit-2.0.0/tests/cli/data/test_po2txt/out.txt 2017-01-28 06:51:52.000000000 +1100
@@ -0,0 +1,7 @@
+Een
+
+Won
+
+Twee twee
+
+Three three
\ No newline at end of file
diff -Nru translate-toolkit-2.0.0~rc3/tests/cli/data/test_po2txt_threshold/one.po translate-toolkit-2.0.0/tests/cli/data/test_po2txt_threshold/one.po
--- translate-toolkit-2.0.0~rc3/tests/cli/data/test_po2txt_threshold/one.po 1970-01-01 10:00:00.000000000 +1000
+++ translate-toolkit-2.0.0/tests/cli/data/test_po2txt_threshold/one.po 2017-01-28 06:51:52.000000000 +1100
@@ -0,0 +1,13 @@
+msgctxt "first"
+msgid "One"
+msgstr "Een"
+
+msgctxt "second"
+msgid "One"
+msgstr "Won"
+
+msgid "Two two"
+msgstr "Twee twee"
+
+msgid "Three three"
+msgstr ""
diff -Nru translate-toolkit-2.0.0~rc3/tests/cli/data/test_pofilter_manpage/stdout.txt translate-toolkit-2.0.0/tests/cli/data/test_pofilter_manpage/stdout.txt
--- translate-toolkit-2.0.0~rc3/tests/cli/data/test_pofilter_manpage/stdout.txt 2017-01-24 05:25:31.000000000 +1100
+++ translate-toolkit-2.0.0/tests/cli/data/test_pofilter_manpage/stdout.txt 2017-01-28 06:51:52.000000000 +1100
@@ -1,5 +1,5 @@
.\" Autogenerated manpage
-.TH pofilter 1 "Translate Toolkit 2.0.0rc3" "" "Translate Toolkit 2.0.0rc3"
+.TH pofilter 1 "Translate Toolkit 2.0.0" "" "Translate Toolkit 2.0.0"
.SH NAME
pofilter \- Perform quality checks on Gettext PO, XLIFF and TMX localization files.
.SH SYNOPSIS
diff -Nru translate-toolkit-2.0.0~rc3/tests/cli/test_po2txt.sh translate-toolkit-2.0.0/tests/cli/test_po2txt.sh
--- translate-toolkit-2.0.0~rc3/tests/cli/test_po2txt.sh 1970-01-01 10:00:00.000000000 +1000
+++ translate-toolkit-2.0.0/tests/cli/test_po2txt.sh 2017-01-28 06:51:52.000000000 +1100
@@ -0,0 +1,6 @@
+#!/bin/bash
+
+source $(dirname $0)/test.inc.sh
+
+po2txt --progress=none $one $out
+check_results
diff -Nru translate-toolkit-2.0.0~rc3/tests/cli/test_po2txt_threshold.sh translate-toolkit-2.0.0/tests/cli/test_po2txt_threshold.sh
--- translate-toolkit-2.0.0~rc3/tests/cli/test_po2txt_threshold.sh 1970-01-01 10:00:00.000000000 +1000
+++ translate-toolkit-2.0.0/tests/cli/test_po2txt_threshold.sh 2017-01-28 06:51:52.000000000 +1100
@@ -0,0 +1,7 @@
+#!/bin/bash
+
+source $(dirname $0)/test.inc.sh
+
+# We expect no output as translations are not complete
+po2txt --progress=none --threshold=100 $one $out
+check_results
diff -Nru translate-toolkit-2.0.0~rc3/translate/convert/convert.py translate-toolkit-2.0.0/translate/convert/convert.py
--- translate-toolkit-2.0.0~rc3/translate/convert/convert.py 2017-01-24 05:25:31.000000000 +1100
+++ translate-toolkit-2.0.0/translate/convert/convert.py 2017-01-28 06:51:52.000000000 +1100
@@ -481,13 +481,16 @@
from translate.storage import statsdb
- units = filter(lambda unit: unit.istranslatable(), store.units)
- translated = filter(lambda unit: unit.istranslated(), units)
- wordcounts = dict(map(lambda unit: (unit, statsdb.wordsinunit(unit)), units))
- sourcewords = lambda elementlist: sum(map(lambda unit: wordcounts[unit][0], elementlist))
- tranlated_count = sourcewords(translated)
+ units = list(filter(lambda unit: unit.istranslatable(), store.units))
+ translated = list(filter(lambda unit: unit.istranslated(), units))
+ wordcounts = dict(map(lambda unit: (unit.getid(), statsdb.wordsinunit(unit)), units))
+
+ def sourcewords(elementlist):
+ return sum(map(lambda unit: wordcounts[unit.getid()][0], elementlist))
+
+ translated_count = sourcewords(translated)
total_count = sourcewords(units)
- percent = tranlated_count * 100 / total_count
+ percent = translated_count * 100 / total_count
return percent >= threshold
diff -Nru translate-toolkit-2.0.0~rc3/translate/storage/mozilla_lang.py translate-toolkit-2.0.0/translate/storage/mozilla_lang.py
--- translate-toolkit-2.0.0~rc3/translate/storage/mozilla_lang.py 2017-01-24 05:25:31.000000000 +1100
+++ translate-toolkit-2.0.0/translate/storage/mozilla_lang.py 2017-01-28 06:51:52.000000000 +1100
@@ -56,7 +56,8 @@
if self.source == self.target:
target = self.target + " {ok}"
if (self.rawtarget is not None
- and self.target == strip_ok(self.rawtarget)):
+ and self.target == strip_ok(self.rawtarget)
+ and len(self.target) != len(strip_ok(self.rawtarget))):
target = self.rawtarget
if self.getnotes():
notes = (self.eol).join(
@@ -107,7 +108,11 @@
self.is_active = True
continue
- if line.startswith("## ") and not line.startswith('## TAG'):
+ header_meta_data = (
+ line.startswith("## ")
+ and not line.startswith('## TAG')
+ and not line.startswith('## MAX_LENGTH'))
+ if header_meta_data:
self._headers.append(line)
continue
@@ -130,9 +135,10 @@
is_comment = (
line.startswith('#')
and (not line.startswith("##")
- or line.startswith('## TAG')))
+ or line.startswith('## TAG')
+ or line.startswith('## MAX_LENGTH')))
if is_comment:
- # Read comments, *including* meta tags (i.e. '## TAG')
+ # Read comments, *including* meta tags (e.g. '## TAG')
comment += line[1:].strip() + "\n"
if line.startswith(';'):
diff -Nru translate-toolkit-2.0.0~rc3/translate/storage/test_mozilla_lang.py translate-toolkit-2.0.0/translate/storage/test_mozilla_lang.py
--- translate-toolkit-2.0.0~rc3/translate/storage/test_mozilla_lang.py 2017-01-24 05:25:31.000000000 +1100
+++ translate-toolkit-2.0.0/translate/storage/test_mozilla_lang.py 2017-01-28 06:51:52.000000000 +1100
@@ -64,6 +64,31 @@
"# TAG: goodbye"
in unit.getnotes(origin="developer").split("\n"))
+ def test_copy_target(self):
+ """Validate that self.rawtarget does not break a valid translation.
+
+ self.rawtarget is used to preserve strange format anomalies related to
+ {ok}. But when units got translated it sometimes caused issues, in
+ that the unit was correctly translated but when serialised it used
+ self.rawtarget and thus output an untranslated unit. This checks that
+ translation actually results in rawformat being ignored.
+ """
+ unit = self.UnitClass("Device")
+ unit.target == ""
+ unit.rawtarget = "Device"
+ assert not unit.istranslated()
+ assert str(unit) == ";Device\nDevice"
+
+ other = self.UnitClass("Device")
+ other.target = "Device"
+ assert other.istranslated()
+ assert str(other) == ";Device\nDevice {ok}"
+
+ unit.target = other.target
+ assert unit.istranslated()
+ assert unit.target == "Device"
+ assert str(unit) == ";Device\nDevice {ok}"
+
class TestMozLangFile(test_base.TestTranslationStore):
StoreClass = mozilla_lang.LangStore
@@ -260,3 +285,17 @@
assert (
"# TAG: another_important_tag"
in store.units[0].getnotes(origin="developer").split("\n"))
+
+ def test_maxlength(self):
+ """Ensure we can handle MAX_LENGTH meta data"""
+ lang = ("## MAX_LENGTH: 80\n"
+ "# Comment\n"
+ ";Source\n"
+ "Target\n"
+ "\n\n")
+ store = self.StoreClass.parsestring(lang)
+ assert not store.getlangheaders()
+ assert bytes(store).decode('utf-8') == lang
+ assert (
+ "# MAX_LENGTH: 80"
+ in store.units[0].getnotes(origin="developer").split("\n"))
diff -Nru translate-toolkit-2.0.0~rc3/translate/__version__.py translate-toolkit-2.0.0/translate/__version__.py
--- translate-toolkit-2.0.0~rc3/translate/__version__.py 2017-01-24 05:25:31.000000000 +1100
+++ translate-toolkit-2.0.0/translate/__version__.py 2017-01-28 06:51:52.000000000 +1100
@@ -1,6 +1,6 @@
# -*- coding: utf-8 -*-
#
-# Copyright 2008-2009 Zuza Software Foundation
+# Copyright 2008-2017 Zuza Software Foundation
#
# This file is part of the Translate Toolkit.
#
@@ -19,14 +19,14 @@
"""This file contains the version of the Translate Toolkit."""
-build = 20010
+build = 20011
"""The build number is used by external users of the Translate Toolkit to
trigger refreshes. Thus increase the build number whenever changes are made to
code touching stats or quality checks. An increased build number will force a
toolkit user, like Pootle, to regenerate it's stored stats and check
results."""
-sver = "2.0.0rc3"
+sver = "2.0.0"
"""Human readable version number. Used for version number display."""
ver = (2, 0, 0)
--- End Message ---