Package: release.debian.org
Severity: important
Tags: stretch
User: release.debian.org@packages.debian.org
Usertags: pu
git-annex in stretch is vulnerable to CVE-2018-10857 and
CVE-2018-10859. This update is a minimal fix for those CVEs prepared by
its upstream, Joey Hess:
git-annex (6.20170101-1+deb9u2) stretch; urgency=high
[ Joey Hess ]
* CVE-2018-10857:
- Added annex.security.allowed-url-schemes setting, which defaults
to only allowing http, https, and ftp URLs. Note especially that file:/
is no longer enabled by default.
- Removed annex.web-download-command, since its interface does not allow
supporting annex.security.allowed-url-schemes across redirects.
If you used this setting, you may want to instead use annex.web-options
to pass options to curl.
- git-annex will refuse to download content from the web, to prevent
accidental exposure of data on private webservers on localhost and the
LAN. This can be overridden with the
annex.security.allowed-http-addresses setting.
(The S3, glacier, and webdav special remotes are still allowed to
download from the web.)
* CVE-2018-10857 and CVE-2018-10859:
- Refuse to download content, that cannot be verified with a hash,
from encrypted special remotes (for CVE-2018-10859),
and from all external special remotes (for CVE-2018-10857).
In particular, URL and WORM keys stored on such remotes won't
be downloaded. If this affects your files, you can run
`git-annex migrate` on the affected files, to convert them
to use a hash.
- Added annex.security.allow-unverified-downloads, which can override
the above.
-- Sean Whitton <spwhitton@spwhitton.name> Fri, 22 Jun 2018 16:42:37 +0100
The security team have decided that because a point release is close and
only certain usecases of git-annex are impacted, they won't issue a DSA.
I have not yet uploaded to stretch-proposed-updates for two reasons:
(i) although the fix is minimal and much smaller than the fix for the
CVEs included in git-annex in unstable, the source debdiff
(attached) is still quite large, so I thought I should wait for an
explicit ACK from the release team
(ii) there is already a +deb9u1 version of git-annex in
stretch-security, but not stretch, responding to a different CVE.
I have based my work on the +deb9u1 upload, and I assume that
uploading my +deb9u2 to stretch-proposed-updates will cause it to
take precedence over the import of the +deb9u1 upload. But in case
things are more fragile than that, I'm holding back.
Note that my debdiff is from +deb9u1 to +deb9u2.
I ran the test suite and manually smoke tested my proposed update
against stretch as the latter stood about a week ago, and plan to run
the test suite again right before uploading to stretch-proposed-updates.
Thank you for your attention!
-- System Information:
Debian Release: 9.4
APT prefers stable
APT policy: (900, 'stable'), (500, 'stable-updates')
Architecture: i386 (i686)
Kernel: Linux 4.9.0-6-686-pae (SMP w/2 CPU cores)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
--
Sean Whitton
diff -Nru git-annex-6.20170101/debian/changelog git-annex-6.20170101/debian/changelog
--- git-annex-6.20170101/debian/changelog 2017-10-26 15:28:29.000000000 +0100
+++ git-annex-6.20170101/debian/changelog 2018-06-22 16:42:37.000000000 +0100
@@ -1,3 +1,33 @@
+git-annex (6.20170101-1+deb9u2) stretch; urgency=high
+
+ [ Joey Hess ]
+ * CVE-2018-10857:
+ - Added annex.security.allowed-url-schemes setting, which defaults
+ to only allowing http, https, and ftp URLs. Note especially that file:/
+ is no longer enabled by default.
+ - Removed annex.web-download-command, since its interface does not allow
+ supporting annex.security.allowed-url-schemes across redirects.
+ If you used this setting, you may want to instead use annex.web-options
+ to pass options to curl.
+ - git-annex will refuse to download content from the web, to prevent
+ accidental exposure of data on private webservers on localhost and the
+ LAN. This can be overridden with the
+ annex.security.allowed-http-addresses setting.
+ (The S3, glacier, and webdav special remotes are still allowed to
+ download from the web.)
+ * CVE-2018-10857 and CVE-2018-10859:
+ - Refuse to download content, that cannot be verified with a hash,
+ from encrypted special remotes (for CVE-2018-10859),
+ and from all external special remotes (for CVE-2018-10857).
+ In particular, URL and WORM keys stored on such remotes won't
+ be downloaded. If this affects your files, you can run
+ `git-annex migrate` on the affected files, to convert them
+ to use a hash.
+ - Added annex.security.allow-unverified-downloads, which can override
+ the above.
+
+ -- Sean Whitton <spwhitton@spwhitton.name> Fri, 22 Jun 2018 16:42:37 +0100
+
git-annex (6.20170101-1+deb9u1) stretch-security; urgency=high
* Non-maintainer upload by the Security Team.
diff -Nru git-annex-6.20170101/debian/patches/add-retrievalsecuritypolicy.patch git-annex-6.20170101/debian/patches/add-retrievalsecuritypolicy.patch
--- git-annex-6.20170101/debian/patches/add-retrievalsecuritypolicy.patch 1970-01-01 01:00:00.000000000 +0100
+++ git-annex-6.20170101/debian/patches/add-retrievalsecuritypolicy.patch 2018-06-22 16:42:37.000000000 +0100
@@ -0,0 +1,293 @@
+From: Joey Hess <joeyh@joeyh.name>
+Date: Thu, 21 Jun 2018 11:35:27 -0400
+X-Dgit-Generated: 6.20170101-1+deb9u2 fe511617f5bde916f1bde799b5a51ddc9404eea1
+Subject: add retrievalSecurityPolicy
+
+This will be used to protect against CVE-2018-10859, where an encrypted
+special remote is fed the wrong encrypted data, and so tricked into
+decrypting something that the user encrypted with their gpg key and did
+not store in git-annex.
+
+It also protects against CVE-2018-10857, where a remote follows a http
+redirect to a file:// url or to a local private web server. While that's
+already been prevented in git-annex's own use of http, external special
+remotes, hooks, etc use other http implementations and could still be
+vulnerable.
+
+The policy is not yet enforced, this commit only adds the appropriate
+metadata to remotes.
+
+This commit was sponsored by Boyd Stephen Smith Jr. on Patreon.
+
+(cherry picked from commit 4315bb9e421f2c643e517d8982c6c35b1909c78b)
+
+---
+
+--- git-annex-6.20170101.orig/Remote/BitTorrent.hs
++++ git-annex-6.20170101/Remote/BitTorrent.hs
+@@ -57,6 +57,8 @@ gen r _ c gc =
+ , storeKey = uploadKey
+ , retrieveKeyFile = downloadKey
+ , retrieveKeyFileCheap = downloadKeyCheap
++ -- Bittorrent does its own hash checks.
++ , retrievalSecurityPolicy = RetrievalAllKeysSecure
+ , removeKey = dropKey
+ , lockContent = Nothing
+ , checkPresent = checkKey
+--- git-annex-6.20170101.orig/Remote/Bup.hs
++++ git-annex-6.20170101/Remote/Bup.hs
+@@ -56,6 +56,9 @@ gen r u c gc = do
+ , storeKey = storeKeyDummy
+ , retrieveKeyFile = retreiveKeyFileDummy
+ , retrieveKeyFileCheap = retrieveCheap buprepo
++ -- Bup uses git, which cryptographically verifies content
++ -- (with SHA1, but sufficiently for this).
++ , retrievalSecurityPolicy = RetrievalAllKeysSecure
+ , removeKey = removeKeyDummy
+ , lockContent = Nothing
+ , checkPresent = checkPresentDummy
+--- git-annex-6.20170101.orig/Remote/Ddar.hs
++++ git-annex-6.20170101/Remote/Ddar.hs
+@@ -56,6 +56,8 @@ gen r u c gc = do
+ , storeKey = storeKeyDummy
+ , retrieveKeyFile = retreiveKeyFileDummy
+ , retrieveKeyFileCheap = retrieveCheap
++ -- Unsure about this, safe default until Robie answers.
++ , retrievalSecurityPolicy = RetrievalVerifiableKeysSecure
+ , removeKey = removeKeyDummy
+ , lockContent = Nothing
+ , checkPresent = checkPresentDummy
+--- git-annex-6.20170101.orig/Remote/Directory.hs
++++ git-annex-6.20170101/Remote/Directory.hs
+@@ -54,6 +54,7 @@ gen r u c gc = do
+ , storeKey = storeKeyDummy
+ , retrieveKeyFile = retreiveKeyFileDummy
+ , retrieveKeyFileCheap = retrieveCheap dir chunkconfig
++ , retrievalSecurityPolicy = RetrievalAllKeysSecure
+ , removeKey = removeKeyDummy
+ , lockContent = Nothing
+ , checkPresent = checkPresentDummy
+--- git-annex-6.20170101.orig/Remote/External.hs
++++ git-annex-6.20170101/Remote/External.hs
+@@ -81,6 +81,11 @@ gen r u c gc
+ , storeKey = storeKeyDummy
+ , retrieveKeyFile = retreiveKeyFileDummy
+ , retrieveKeyFileCheap = \_ _ _ -> return False
++ -- External special remotes use many http libraries
++ -- and have no protection against redirects to
++ -- local private web servers, or in some cases
++ -- to file:// urls.
++ , retrievalSecurityPolicy = RetrievalVerifiableKeysSecure
+ , removeKey = removeKeyDummy
+ , lockContent = Nothing
+ , checkPresent = checkPresentDummy
+--- git-annex-6.20170101.orig/Remote/GCrypt.hs
++++ git-annex-6.20170101/Remote/GCrypt.hs
+@@ -111,6 +111,7 @@ gen' r u c gc = do
+ , storeKey = storeKeyDummy
+ , retrieveKeyFile = retreiveKeyFileDummy
+ , retrieveKeyFileCheap = \_ _ _ -> return False
++ , retrievalSecurityPolicy = RetrievalAllKeysSecure
+ , removeKey = removeKeyDummy
+ , lockContent = Nothing
+ , checkPresent = checkPresentDummy
+--- git-annex-6.20170101.orig/Remote/Git.hs
++++ git-annex-6.20170101/Remote/Git.hs
+@@ -147,6 +147,7 @@ gen r u c gc
+ , storeKey = copyToRemote new
+ , retrieveKeyFile = copyFromRemote new
+ , retrieveKeyFileCheap = copyFromRemoteCheap new
++ , retrievalSecurityPolicy = RetrievalAllKeysSecure
+ , removeKey = dropKey new
+ , lockContent = Just (lockKey new)
+ , checkPresent = inAnnex new
+--- git-annex-6.20170101.orig/Remote/Glacier.hs
++++ git-annex-6.20170101/Remote/Glacier.hs
+@@ -53,6 +53,9 @@ gen r u c gc = new <$> remoteCost gc ver
+ , storeKey = storeKeyDummy
+ , retrieveKeyFile = retreiveKeyFileDummy
+ , retrieveKeyFileCheap = retrieveCheap this
++ -- glacier-cli does not follow redirects and does
++ -- not support file://, so this is secure.
++ , retrievalSecurityPolicy = RetrievalAllKeysSecure
+ , removeKey = removeKeyDummy
+ , lockContent = Nothing
+ , checkPresent = checkPresentDummy
+--- git-annex-6.20170101.orig/Remote/Helper/Special.hs
++++ git-annex-6.20170101/Remote/Helper/Special.hs
+@@ -161,6 +161,14 @@ specialRemote' cfg c preparestorer prepa
+ (retrieveKeyFileCheap baser k f d)
+ -- retrieval of encrypted keys is never cheap
+ (\_ -> return False)
++ -- When encryption is used, the remote could provide
++ -- some other content encrypted by the user, and trick
++ -- git-annex into decrypting it, leaking the decryption
++ -- into the git-annex repository. Verifiable keys
++ -- are the main protection against this attack.
++ , retrievalSecurityPolicy = if isencrypted
++ then RetrievalVerifiableKeysSecure
++ else retrievalSecurityPolicy baser
+ , removeKey = \k -> cip >>= removeKeyGen k
+ , checkPresent = \k -> cip >>= checkPresentGen k
+ , cost = if isencrypted
+--- git-annex-6.20170101.orig/Remote/Hook.hs
++++ git-annex-6.20170101/Remote/Hook.hs
+@@ -47,6 +47,9 @@ gen r u c gc = do
+ , storeKey = storeKeyDummy
+ , retrieveKeyFile = retreiveKeyFileDummy
+ , retrieveKeyFileCheap = retrieveCheap hooktype
++ -- A hook could use http and be vulnerable to
++ -- redirect to file:// attacks, etc.
++ , retrievalSecurityPolicy = RetrievalVerifiableKeysSecure
+ , removeKey = removeKeyDummy
+ , lockContent = Nothing
+ , checkPresent = checkPresentDummy
+--- git-annex-6.20170101.orig/Remote/P2P.hs
++++ git-annex-6.20170101/Remote/P2P.hs
+@@ -53,6 +53,7 @@ chainGen addr r u c gc = do
+ , storeKey = store u addr connpool
+ , retrieveKeyFile = retrieve u addr connpool
+ , retrieveKeyFileCheap = \_ _ _ -> return False
++ , retrievalSecurityPolicy = RetrievalAllKeysSecure
+ , removeKey = remove u addr connpool
+ , lockContent = Just (lock u addr connpool)
+ , checkPresent = checkpresent u addr connpool
+--- git-annex-6.20170101.orig/Remote/Rsync.hs
++++ git-annex-6.20170101/Remote/Rsync.hs
+@@ -69,6 +69,7 @@ gen r u c gc = do
+ , storeKey = storeKeyDummy
+ , retrieveKeyFile = retreiveKeyFileDummy
+ , retrieveKeyFileCheap = retrieveCheap o
++ , retrievalSecurityPolicy = RetrievalAllKeysSecure
+ , removeKey = removeKeyDummy
+ , lockContent = Nothing
+ , checkPresent = checkPresentDummy
+--- git-annex-6.20170101.orig/Remote/S3.hs
++++ git-annex-6.20170101/Remote/S3.hs
+@@ -86,6 +86,9 @@ gen r u c gc = do
+ , storeKey = storeKeyDummy
+ , retrieveKeyFile = retreiveKeyFileDummy
+ , retrieveKeyFileCheap = retrieveCheap
++ -- HttpManagerRestricted is used here, so this is
++ -- secure.
++ , retrievalSecurityPolicy = RetrievalAllKeysSecure
+ , removeKey = removeKeyDummy
+ , lockContent = Nothing
+ , checkPresent = checkPresentDummy
+--- git-annex-6.20170101.orig/Remote/Tahoe.hs
++++ git-annex-6.20170101/Remote/Tahoe.hs
+@@ -71,6 +71,8 @@ gen r u c gc = do
+ , storeKey = store u hdl
+ , retrieveKeyFile = retrieve u hdl
+ , retrieveKeyFileCheap = \_ _ _ -> return False
++ -- Tahoe cryptographically verifies content.
++ , retrievalSecurityPolicy = RetrievalAllKeysSecure
+ , removeKey = remove
+ , lockContent = Nothing
+ , checkPresent = checkKey u hdl
+--- git-annex-6.20170101.orig/Remote/Web.hs
++++ git-annex-6.20170101/Remote/Web.hs
+@@ -46,6 +46,9 @@ gen r _ c gc =
+ , storeKey = uploadKey
+ , retrieveKeyFile = downloadKey
+ , retrieveKeyFileCheap = downloadKeyCheap
++ -- HttpManagerRestricted is used here, so this is
++ -- secure.
++ , retrievalSecurityPolicy = RetrievalAllKeysSecure
+ , removeKey = dropKey
+ , lockContent = Nothing
+ , checkPresent = checkKey
+--- git-annex-6.20170101.orig/Remote/WebDAV.hs
++++ git-annex-6.20170101/Remote/WebDAV.hs
+@@ -64,6 +64,9 @@ gen r u c gc = new <$> remoteCost gc exp
+ , storeKey = storeKeyDummy
+ , retrieveKeyFile = retreiveKeyFileDummy
+ , retrieveKeyFileCheap = retrieveCheap
++ -- HttpManagerRestricted is used here, so this is
++ -- secure.
++ , retrievalSecurityPolicy = RetrievalAllKeysSecure
+ , removeKey = removeKeyDummy
+ , lockContent = Nothing
+ , checkPresent = checkPresentDummy
+--- git-annex-6.20170101.orig/Types/Key.hs
++++ git-annex-6.20170101/Types/Key.hs
+@@ -1,6 +1,6 @@
+ {- git-annex Key data type
+ -
+- - Copyright 2011-2016 Joey Hess <id@joeyh.name>
++ - Copyright 2011-2018 Joey Hess <id@joeyh.name>
+ -
+ - Licensed under the GNU GPL version 3 or higher.
+ -}
+@@ -160,3 +160,13 @@ prop_isomorphic_key_decode f
+ normalfieldorder = fields `isPrefixOf` "smSC"
+ fields = map (f !!) $ filter (< length f) $ map succ $
+ elemIndices fieldSep f
++
++{- Is the Key variety backed by a hash, which allows verifying content?
++ - It does not have to be cryptographically secure against eg birthday
++ - attacks.
++ -}
++isVerifiable :: Key -> Bool
++isVerifiable k = case keyBackendName k of
++ "WORM" -> False
++ "URL" -> False
++ _ -> True
+--- git-annex-6.20170101.orig/Types/Remote.hs
++++ git-annex-6.20170101/Types/Remote.hs
+@@ -2,7 +2,7 @@
+ -
+ - Most things should not need this, using Types instead
+ -
+- - Copyright 2011-2014 Joey Hess <id@joeyh.name>
++ - Copyright 2011-2018 Joey Hess <id@joeyh.name>
+ -
+ - Licensed under the GNU GPL version 3 or higher.
+ -}
+@@ -17,6 +17,7 @@ module Types.Remote
+ , Availability(..)
+ , Verification(..)
+ , unVerified
++ , RetrievalSecurityPolicy(..)
+ )
+ where
+
+@@ -75,6 +76,8 @@ data RemoteA a = Remote {
+ -- Retrieves a key's contents to a tmp file, if it can be done cheaply.
+ -- It's ok to create a symlink or hardlink.
+ retrieveKeyFileCheap :: Key -> AssociatedFile -> FilePath -> a Bool,
++ -- Security policy for reteiving keys from this remote.
++ retrievalSecurityPolicy :: RetrievalSecurityPolicy,
+ -- Removes a key's contents (succeeds if the contents are not present)
+ removeKey :: Key -> a Bool,
+ -- Uses locking to prevent removal of a key's contents,
+@@ -145,3 +148,29 @@ unVerified :: Monad m => m Bool -> m (Bo
+ unVerified a = do
+ ok <- a
+ return (ok, UnVerified)
++
++-- Security policy indicating what keys can be safely retrieved from a
++-- remote.
++data RetrievalSecurityPolicy
++ = RetrievalVerifiableKeysSecure
++ -- ^ Transfer of keys whose content can be verified
++ -- with a hash check is secure; transfer of unverifiable keys is
++ -- not secure and should not be allowed.
++ --
++ -- This is used eg, when HTTP to a remote could be redirected to a
++ -- local private web server or even a file:// url, causing private
++ -- data from it that is not the intended content of a key to make
++ -- its way into the git-annex repository.
++ --
++ -- It's also used when content is stored encrypted on a remote,
++ -- which could replace it with a different encrypted file, and
++ -- trick git-annex into decrypting it and leaking the decryption
++ -- into the git-annex repository.
++ --
++ -- It's not (currently) used when the remote could alter the
++ -- content stored on it, because git-annex does not provide
++ -- strong guarantees about the content of keys that cannot be
++ -- verified with a hash check.
++ -- (But annex.securehashesonly does provide such guarantees.)
++ | RetrievalAllKeysSecure
++ -- ^ Any key can be securely retrieved.
diff -Nru git-annex-6.20170101/debian/patches/block-url-downloads-by-default.patch git-annex-6.20170101/debian/patches/block-url-downloads-by-default.patch
--- git-annex-6.20170101/debian/patches/block-url-downloads-by-default.patch 1970-01-01 01:00:00.000000000 +0100
+++ git-annex-6.20170101/debian/patches/block-url-downloads-by-default.patch 2018-06-22 16:42:37.000000000 +0100
@@ -0,0 +1,154 @@
+From: Joey Hess <id@joeyh.name>
+Date: Mon, 18 Jun 2018 17:40:50 -0400
+X-Dgit-Generated: 6.20170101-1+deb9u2 139539615478c1f618525715bef184d450b38558
+Subject: block url downloads by default
+
+git-annex will refuse to download content from the web, to prevent
+accidental exposure of data on private webservers on localhost and the
+LAN. This can be overridden with the
+annex.security.allowed-http-addresses setting.
+
+This is the simplest possible fix for the security hole. A better fix
+has been developed for newer versions of git-annex but would be a lot of
+work to backport, and perhaps too big a diff.
+
+There are several sets of git-annex users who will be impacted
+in different ways by this:
+
+* Users who have a git-annex repository but don't use the web special
+ remote. Unaffected.
+
+* Users who have a git-annex repository that is for private use only.
+ They will have to read enough docs to find the setting to allow
+ git annex addurl to work again.
+
+* Users who have a git-annex repositry that is shared with people they
+ don't fully trust.
+ They will not be able to use the web special remote with this version
+ of git-annex. They'll have to upgrade.
+
+The S3, glacier, and webdav special remotes are still allowed to
+download from the web. There are other potential attacks involving the
+web server they connect to redirecting to a local private web server,
+and tricking them from downloading content from it which then leaks back
+to the attacker. Those attacks are not addressed here, but they also
+seem fairly unlikely. Further analysis is needed; preliminary analysis
+of glacier-cli, for example, suggests it does not follow redirects and
+so is not vulnerable to such attacks.
+
+---
+
+--- git-annex-6.20170101.orig/Annex/Quvi.hs
++++ git-annex-6.20170101/Annex/Quvi.hs
+@@ -11,8 +11,8 @@ module Annex.Quvi where
+
+ import Annex.Common
+ import qualified Annex
++import Annex.Url
+ import Utility.Quvi
+-import Utility.Url
+
+ withQuviOptions :: forall a. Query a -> [QuviParams] -> URLString -> Annex a
+ withQuviOptions a ps url = do
+@@ -21,7 +21,14 @@ withQuviOptions a ps url = do
+ liftIO $ a v (concatMap (\mkp -> mkp v) ps ++ opts) url
+
+ quviSupported :: URLString -> Annex Bool
+-quviSupported u = liftIO . flip supported u =<< quviVersion
++quviSupported u = ifM httpAddressesUnlimited
++ ( liftIO . flip supported u =<< quviVersion
++ -- Don't allow any url schemes to be used when
++ -- there's a limit on the allowed addresses, because
++ -- there is no way to prevent quvi from
++ -- redirecting to any address.
++ , return False
++ )
+
+ quviVersion :: Annex QuviVersion
+ quviVersion = go =<< Annex.getState Annex.quviversion
+--- git-annex-6.20170101.orig/Annex/Url.hs
++++ git-annex-6.20170101/Annex/Url.hs
+@@ -11,6 +11,7 @@ module Annex.Url (
+ withUrlOptions,
+ getUrlOptions,
+ getUserAgent,
++ httpAddressesUnlimited,
+ ) where
+
+ import Annex.Common
+@@ -18,6 +19,8 @@ import qualified Annex
+ import Utility.Url as U
+ import qualified Build.SysConfig as SysConfig
+
++import qualified Data.Set as S
++
+ defaultUserAgent :: U.UserAgent
+ defaultUserAgent = "git-annex/" ++ SysConfig.packageversion
+
+@@ -30,7 +33,7 @@ getUrlOptions = mkUrlOptions
+ <$> getUserAgent
+ <*> headers
+ <*> options
+- <*> (annexAllowedUrlSchemes <$> Annex.getGitConfig)
++ <*> urlschemes
+ where
+ headers = do
+ v <- annexHttpHeadersCommand <$> Annex.getGitConfig
+@@ -38,6 +41,18 @@ getUrlOptions = mkUrlOptions
+ Just cmd -> lines <$> liftIO (readProcess "sh" ["-c", cmd])
+ Nothing -> annexHttpHeaders <$> Annex.getGitConfig
+ options = map Param . annexWebOptions <$> Annex.getGitConfig
++ urlschemes = ifM httpAddressesUnlimited
++ ( annexAllowedUrlSchemes <$> Annex.getGitConfig
++ -- Don't allow any url schemes to be used when
++ -- there's a limit on the allowed addresses, because
++ -- there is no way to prevent curl or wget from
++ -- redirecting to any address.
++ , pure S.empty
++ )
++
++httpAddressesUnlimited :: Annex Bool
++httpAddressesUnlimited =
++ ("all" == ) . annexAllowedHttpAddresses <$> Annex.getGitConfig
+
+ withUrlOptions :: (U.UrlOptions -> IO a) -> Annex a
+ withUrlOptions a = liftIO . a =<< getUrlOptions
+--- git-annex-6.20170101.orig/NEWS
++++ git-annex-6.20170101/NEWS
+@@ -4,6 +4,12 @@ git-annex (6.20170101-1+deb9u2) stretch-
+ URL schemes by default. You can enable other URL schemes, at your own risk,
+ using annex.security.allowed-url-schemes.
+
++ A related security fix prevents git-annex from downloading content from
++ the web. This can be overridden with the
++ annex.security.allowed-http-addresses setting.
++ (The S3, glacier, and webdav special remotes are still allowed to
++ download from the web.)
++
+ The annex.web-download-command configuration has been removed,
+ use annex.web-options instead.
+
+--- git-annex-6.20170101.orig/doc/git-annex.mdwn
++++ git-annex-6.20170101/doc/git-annex.mdwn
+@@ -1253,6 +1253,21 @@ Here are all the supported configuration
+ Some special remotes support their own domain-specific URL
+ schemes; those are not affected by this configuration setting.
+
++* `annex.security.allowed-http-addresses`
++
++ By default, this version of git-annex refuses to download the content of
++ annexed files from the web. Newer versions of git-annex allow downloading
++ from the web, but only when the web server is not on a private IP address.
++
++ To relax this security check and allow getting annexed files from
++ anywhere on the web, set this to "all".
++
++ Think very carefully before changing this; there are security
++ implications. Anyone who can get a commit into your git-annex repository
++ could `git annex addurl` an url on a private http server, possibly
++ causing it to be downloaded into your repository and transferred to
++ other remotes, exposing its content.
++
+ * `annex.secure-erase-command`
+
+ This can be set to a command that should be run whenever git-annex
diff -Nru git-annex-6.20170101/debian/patches/dont-assume-boto-will-remain-secure.patch git-annex-6.20170101/debian/patches/dont-assume-boto-will-remain-secure.patch
--- git-annex-6.20170101/debian/patches/dont-assume-boto-will-remain-secure.patch 1970-01-01 01:00:00.000000000 +0100
+++ git-annex-6.20170101/debian/patches/dont-assume-boto-will-remain-secure.patch 2018-06-22 16:42:37.000000000 +0100
@@ -0,0 +1,27 @@
+From: Joey Hess <joeyh@joeyh.name>
+Date: Thu, 21 Jun 2018 14:14:56 -0400
+X-Dgit-Generated: 6.20170101-1+deb9u2 acdcbe25bc02919a8c2241155e399f9ef9533c5f
+Subject: don't assume boto will remain secure
+
+On second thought, best to default to being secure even if boto changes
+http libraries to one that happens to follow redirects.
+
+(cherry picked from commit f1b29dbeb4277bf8a7febc795fe52e7b109c6d59)
+
+---
+
+--- git-annex-6.20170101.orig/Remote/Glacier.hs
++++ git-annex-6.20170101/Remote/Glacier.hs
+@@ -54,8 +54,10 @@ gen r u c gc = new <$> remoteCost gc ver
+ , retrieveKeyFile = retreiveKeyFileDummy
+ , retrieveKeyFileCheap = retrieveCheap this
+ -- glacier-cli does not follow redirects and does
+- -- not support file://, so this is secure.
+- , retrievalSecurityPolicy = RetrievalAllKeysSecure
++ -- not support file://, as far as we know, but
++ -- there's no guarantee that will continue to be
++ -- the case, so require verifiable keys.
++ , retrievalSecurityPolicy = RetrievalVerifiableKeysSecure
+ , removeKey = removeKeyDummy
+ , lockContent = Nothing
+ , checkPresent = checkPresentDummy
diff -Nru git-annex-6.20170101/debian/patches/enforce-retrievalsecuritypolicy.patch git-annex-6.20170101/debian/patches/enforce-retrievalsecuritypolicy.patch
--- git-annex-6.20170101/debian/patches/enforce-retrievalsecuritypolicy.patch 1970-01-01 01:00:00.000000000 +0100
+++ git-annex-6.20170101/debian/patches/enforce-retrievalsecuritypolicy.patch 2018-06-22 16:42:37.000000000 +0100
@@ -0,0 +1,438 @@
+From: Joey Hess <joeyh@joeyh.name>
+Date: Thu, 21 Jun 2018 13:34:11 -0400
+X-Dgit-Generated: 6.20170101-1+deb9u2 cd7e9410a9835b63fb06ef5c99b31d91f111c4c3
+Subject: enforce retrievalSecurityPolicy
+
+Leveraged the existing verification code by making it also check the
+retrievalSecurityPolicy.
+
+Also, prevented getViaTmp from running the download action at all when the
+retrievalSecurityPolicy is going to prevent verifying and so storing it.
+
+Added annex.security.allow-unverified-downloads. A per-remote version
+would be nice to have too, but would need more plumbing, so KISS.
+(Bill the Cat reference not too over the top I hope. The point is to
+make this something the user reads the documentation for before using.)
+
+A few calls to verifyKeyContent and getViaTmp, that don't
+involve downloads from remotes, have RetrievalAllKeysSecure hard-coded.
+It was also hard-coded for P2P.Annex and Command.RecvKey,
+to match the values of the corresponding remotes.
+
+A few things use retrieveKeyFile/retrieveKeyFileCheap without going
+through getViaTmp.
+* Command.Fsck when downloading content from a remote to verify it.
+ That content does not get into the annex, so this is ok.
+* Command.AddUrl when using a remote to download an url; this is new
+ content being added, so this is ok.
+
+This commit was sponsored by Fernando Jimenez on Patreon.
+
+(cherry picked from commit b657242f5d946efae4cc77e8aef95dd2a306cd6b)
+
+---
+
+--- git-annex-6.20170101.orig/Annex/Content.hs
++++ git-annex-6.20170101/Annex/Content.hs
+@@ -1,6 +1,6 @@
+ {- git-annex file content managing
+ -
+- - Copyright 2010-2015 Joey Hess <id@joeyh.name>
++ - Copyright 2010-2018 Joey Hess <id@joeyh.name>
+ -
+ - Licensed under the GNU GPL version 3 or higher.
+ -}
+@@ -15,6 +15,7 @@ module Annex.Content (
+ lockContentShared,
+ lockContentForRemoval,
+ ContentRemovalLock,
++ RetrievalSecurityPolicy(..),
+ getViaTmp,
+ getViaTmp',
+ checkDiskSpaceToGet,
+@@ -74,7 +75,7 @@ import qualified Annex.Content.Direct as
+ import Annex.ReplaceFile
+ import Annex.LockPool
+ import Messages.Progress
+-import Types.Remote (unVerified, Verification(..))
++import Types.Remote (unVerified, Verification(..), RetrievalSecurityPolicy(..))
+ import qualified Types.Remote
+ import qualified Types.Backend
+ import qualified Backend
+@@ -294,19 +295,19 @@ lockContentUsing locker key a = do
+ {- Runs an action, passing it the temp file to get,
+ - and if the action succeeds, verifies the file matches
+ - the key and moves the file into the annex as a key's content. -}
+-getViaTmp :: VerifyConfig -> Key -> (FilePath -> Annex (Bool, Verification)) -> Annex Bool
+-getViaTmp v key action = checkDiskSpaceToGet key False $
+- getViaTmp' v key action
++getViaTmp :: RetrievalSecurityPolicy -> VerifyConfig -> Key -> (FilePath -> Annex (Bool, Verification)) -> Annex Bool
++getViaTmp rsp v key action = checkDiskSpaceToGet key False $
++ getViaTmp' rsp v key action
+
+ {- Like getViaTmp, but does not check that there is enough disk space
+ - for the incoming key. For use when the key content is already on disk
+ - and not being copied into place. -}
+-getViaTmp' :: VerifyConfig -> Key -> (FilePath -> Annex (Bool, Verification)) -> Annex Bool
+-getViaTmp' v key action = do
++getViaTmp' :: RetrievalSecurityPolicy -> VerifyConfig -> Key -> (FilePath -> Annex (Bool, Verification)) -> Annex Bool
++getViaTmp' rsp v key action = checkallowed $ do
+ tmpfile <- prepTmp key
+ (ok, verification) <- action tmpfile
+ if ok
+- then ifM (verifyKeyContent v verification key tmpfile)
++ then ifM (verifyKeyContent rsp v verification key tmpfile)
+ ( do
+ moveAnnex key tmpfile
+ logStatus key InfoPresent
+@@ -319,22 +320,45 @@ getViaTmp' v key action = do
+ -- On transfer failure, the tmp file is left behind, in case
+ -- caller wants to resume its transfer
+ else return False
++ where
++ -- Avoid running the action to get the content when the
++ -- RetrievalSecurityPolicy would cause verification to always fail.
++ checkallowed a = case rsp of
++ RetrievalAllKeysSecure -> a
++ RetrievalVerifiableKeysSecure
++ | isVerifiable key -> a
++ | otherwise -> ifM (annexAllowUnverifiedDownloads <$> Annex.getGitConfig)
++ ( a
++ , warnUnverifiableInsecure key >> return False
++ )
+
+ {- Verifies that a file is the expected content of a key.
++ -
+ - Configuration can prevent verification, for either a
+- - particular remote or always.
++ - particular remote or always, unless the RetrievalSecurityPolicy
++ - requires verification.
+ -
+ - Most keys have a known size, and if so, the file size is checked.
+ -
+- - When the key's backend allows verifying the content (eg via checksum),
++ - When the key's backend allows verifying the content (via checksum),
+ - it is checked.
++ -
++ - If the RetrievalSecurityPolicy requires verification and the key's
++ - backend doesn't support it, the verification will fail.
+ -}
+-verifyKeyContent :: VerifyConfig -> Verification -> Key -> FilePath -> Annex Bool
+-verifyKeyContent _ Verified _ _ = return True
+-verifyKeyContent v UnVerified k f = ifM (shouldVerify v)
+- ( verifysize <&&> verifycontent
+- , return True
+- )
++verifyKeyContent :: RetrievalSecurityPolicy -> VerifyConfig -> Verification -> Key -> FilePath -> Annex Bool
++verifyKeyContent _ _ Verified _ _ = return True
++verifyKeyContent rsp v UnVerified k f = case rsp of
++ RetrievalVerifiableKeysSecure
++ | isVerifiable k -> verifysize <&&> verifycontent
++ | otherwise -> ifM (annexAllowUnverifiedDownloads <$> Annex.getGitConfig)
++ ( verifysize <&&> verifycontent
++ , warnUnverifiableInsecure k >> return False
++ )
++ _ -> ifM (shouldVerify v)
++ ( verifysize <&&> verifycontent
++ , return True
++ )
+ where
+ verifysize = case keySize k of
+ Nothing -> return True
+@@ -345,6 +369,16 @@ verifyKeyContent v UnVerified k f = ifM
+ Nothing -> return True
+ Just verifier -> verifier k f
+
++warnUnverifiableInsecure :: Key -> Annex ()
++warnUnverifiableInsecure k = warning $ unwords
++ [ "Getting " ++ kv ++ " keys with this remote is not secure;"
++ , "the content cannot be verified to be correct."
++ , "(Use annex.security.allow-unverified-downloads to bypass"
++ , "this safety check.)"
++ ]
++ where
++ kv = keyBackendName k
++
+ data VerifyConfig = AlwaysVerify | NoVerify | RemoteVerify Remote | DefaultVerify
+
+ shouldVerify :: VerifyConfig -> Annex Bool
+@@ -790,7 +824,7 @@ isUnmodified key f = go =<< geti
+ go (Just fc) = cheapcheck fc <||> expensivecheck fc
+ cheapcheck fc = anyM (compareInodeCaches fc)
+ =<< Database.Keys.getInodeCaches key
+- expensivecheck fc = ifM (verifyKeyContent AlwaysVerify UnVerified key f)
++ expensivecheck fc = ifM (verifyKeyContent RetrievalAllKeysSecure AlwaysVerify UnVerified key f)
+ -- The file could have been modified while it was
+ -- being verified. Detect that.
+ ( geti >>= maybe (return False) (compareInodeCaches fc)
+--- git-annex-6.20170101.orig/Command/Get.hs
++++ git-annex-6.20170101/Command/Get.hs
+@@ -108,7 +108,7 @@ getKey' key afile = dispatch
+ | Remote.hasKeyCheap r =
+ either (const False) id <$> Remote.hasKey r key
+ | otherwise = return True
+- docopy r witness = getViaTmp (RemoteVerify r) key $ \dest ->
++ docopy r witness = getViaTmp (Remote.retrievalSecurityPolicy r) (RemoteVerify r) key $ \dest ->
+ download (Remote.uuid r) key afile forwardRetry
+ (\p -> do
+ showAction $ "from " ++ Remote.name r
+--- git-annex-6.20170101.orig/Command/Move.hs
++++ git-annex-6.20170101/Command/Move.hs
+@@ -179,7 +179,7 @@ fromPerform src move key afile = ifM (in
+ go = notifyTransfer Download afile $
+ download (Remote.uuid src) key afile forwardRetry $ \p -> do
+ showAction $ "from " ++ Remote.name src
+- getViaTmp (RemoteVerify src) key $ \t ->
++ getViaTmp (Remote.retrievalSecurityPolicy src) (RemoteVerify src) key $ \t ->
+ Remote.retrieveKeyFile src key afile t p
+ dispatch _ False = stop -- failed
+ dispatch False True = next $ return True -- copy complete
+--- git-annex-6.20170101.orig/Command/ReKey.hs
++++ git-annex-6.20170101/Command/ReKey.hs
+@@ -83,7 +83,7 @@ linkKey file oldkey newkey = ifM (isJust
+ - This avoids hard linking to content linked to an
+ - unlocked file, which would leave the new key unlocked
+ - and vulnerable to corruption. -}
+- ( getViaTmp' DefaultVerify newkey $ \tmp -> unVerified $ do
++ ( getViaTmp' RetrievalAllKeysSecure DefaultVerify newkey $ \tmp -> unVerified $ do
+ oldobj <- calcRepo (gitAnnexLocation oldkey)
+ linkOrCopy' (return True) newkey oldobj tmp Nothing
+ , do
+--- git-annex-6.20170101.orig/Command/RecvKey.hs
++++ git-annex-6.20170101/Command/RecvKey.hs
+@@ -13,6 +13,7 @@ import Annex.Action
+ import Annex
+ import Utility.Rsync
+ import Types.Transfer
++import Types.Remote (RetrievalSecurityPolicy(..))
+ import Command.SendKey (fieldTransfer)
+ import qualified CmdLine.GitAnnexShell.Fields as Fields
+
+@@ -31,7 +32,9 @@ start key = fieldTransfer Download key $
+ fromunlocked <- (isJust <$> Fields.getField Fields.unlocked)
+ <||> (isJust <$> Fields.getField Fields.direct)
+ let verify = if fromunlocked then AlwaysVerify else DefaultVerify
+- ifM (getViaTmp verify key go)
++ -- This matches the retrievalSecurityPolicy of Remote.Git
++ let rsp = RetrievalAllKeysSecure
++ ifM (getViaTmp rsp verify key go)
+ ( do
+ -- forcibly quit after receiving one key,
+ -- and shutdown cleanly
+--- git-annex-6.20170101.orig/Command/Reinject.hs
++++ git-annex-6.20170101/Command/Reinject.hs
+@@ -44,7 +44,7 @@ startSrcDest (src:dest:[])
+ | otherwise = notAnnexed src $ do
+ showStart "reinject" dest
+ next $ ifAnnexed dest
+- (\key -> perform src key (verifyKeyContent DefaultVerify UnVerified key src))
++ (\key -> perform src key (verifyKeyContent RetrievalAllKeysSecure DefaultVerify UnVerified key src))
+ stop
+ startSrcDest _ = giveup "specify a src file and a dest file"
+
+--- git-annex-6.20170101.orig/Command/SetKey.hs
++++ git-annex-6.20170101/Command/SetKey.hs
+@@ -33,7 +33,7 @@ perform file key = do
+ -- the file might be on a different filesystem, so moveFile is used
+ -- rather than simply calling moveAnnex; disk space is also
+ -- checked this way.
+- ok <- getViaTmp DefaultVerify key $ \dest -> unVerified $
++ ok <- getViaTmp RetrievalAllKeysSecure DefaultVerify key $ \dest -> unVerified $
+ if dest /= file
+ then liftIO $ catchBoolIO $ do
+ moveFile file dest
+--- git-annex-6.20170101.orig/Command/TestRemote.hs
++++ git-annex-6.20170101/Command/TestRemote.hs
+@@ -154,7 +154,7 @@ test st r k =
+ Just b -> case Backend.verifyKeyContent b of
+ Nothing -> return True
+ Just verifier -> verifier k (key2file k)
+- get = getViaTmp (RemoteVerify r) k $ \dest ->
++ get = getViaTmp (Remote.retrievalSecurityPolicy r) (RemoteVerify r) k $ \dest ->
+ Remote.retrieveKeyFile r k Nothing dest nullMeterUpdate
+ store = Remote.storeKey r k Nothing nullMeterUpdate
+ remove = Remote.removeKey r k
+@@ -168,10 +168,10 @@ testUnavailable st r k =
+ , check (`notElem` [Right True, Right False]) "checkPresent" $
+ Remote.checkPresent r k
+ , check (== Right False) "retrieveKeyFile" $
+- getViaTmp (RemoteVerify r) k $ \dest ->
++ getViaTmp (Remote.retrievalSecurityPolicy r) (RemoteVerify r) k $ \dest ->
+ Remote.retrieveKeyFile r k Nothing dest nullMeterUpdate
+ , check (== Right False) "retrieveKeyFileCheap" $
+- getViaTmp (RemoteVerify r) k $ \dest -> unVerified $
++ getViaTmp (Remote.retrievalSecurityPolicy r) (RemoteVerify r) k $ \dest -> unVerified $
+ Remote.retrieveKeyFileCheap r k Nothing dest
+ ]
+ where
+--- git-annex-6.20170101.orig/Command/TransferKey.hs
++++ git-annex-6.20170101/Command/TransferKey.hs
+@@ -60,7 +60,7 @@ toPerform key file remote = go Upload fi
+ fromPerform :: Key -> AssociatedFile -> Remote -> CommandPerform
+ fromPerform key file remote = go Upload file $
+ download (uuid remote) key file forwardRetry $ \p ->
+- getViaTmp (RemoteVerify remote) key $
++ getViaTmp (retrievalSecurityPolicy remote) (RemoteVerify remote) key $
+ \t -> Remote.retrieveKeyFile remote key file t p
+
+ go :: Direction -> AssociatedFile -> (NotifyWitness -> Annex Bool) -> CommandPerform
+--- git-annex-6.20170101.orig/Command/TransferKeys.hs
++++ git-annex-6.20170101/Command/TransferKeys.hs
+@@ -42,7 +42,7 @@ start = do
+ return ok
+ | otherwise = notifyTransfer direction file $
+ download (Remote.uuid remote) key file forwardRetry $ \p ->
+- getViaTmp (RemoteVerify remote) key $ \t -> do
++ getViaTmp (Remote.retrievalSecurityPolicy remote) (RemoteVerify remote) key $ \t -> do
+ r <- Remote.retrieveKeyFile remote key file t p
+ -- Make sure we get the current
+ -- associated files data for the key,
+--- git-annex-6.20170101.orig/NEWS
++++ git-annex-6.20170101/NEWS
+@@ -1,5 +1,11 @@
+ git-annex (6.20170101-1+deb9u2) stretch-security; urgency=high
+
++ A security fix has changed git-annex to refuse to download content from
++ some special remotes when the content cannot be verified with a hash check.
++ In particular URL and WORM keys stored on such remotes won't be downloaded.
++ See the documentation of the annex.security.allow-unverified-downloads
++ configuration for how to deal with this if it affects your files.
++
+ A security fix has changed git-annex to only support http, https, and ftp
+ URL schemes by default. You can enable other URL schemes, at your own risk,
+ using annex.security.allowed-url-schemes.
+--- git-annex-6.20170101.orig/P2P/Annex.hs
++++ git-annex-6.20170101/P2P/Annex.hs
+@@ -23,6 +23,7 @@ import P2P.Protocol
+ import P2P.IO
+ import Logs.Location
+ import Types.NumCopies
++import Types.Remote (RetrievalSecurityPolicy(..))
+ import Utility.Metered
+ import Utility.Tor
+ import Annex.UUID
+@@ -77,9 +78,12 @@ runLocal runmode runner a = case a of
+ Right Nothing -> runner (next False)
+ Left e -> return (Left (show e))
+ StoreContent k af o l getb next -> do
++ -- This is the same as the retrievalSecurityPolicy of
++ -- Remote.P2P and Remote.Git.
++ let rsp = RetrievalAllKeysSecure
+ ok <- flip catchNonAsync (const $ return False) $
+ transfer download k af $ \p ->
+- getViaTmp AlwaysVerify k $ \tmp ->
++ getViaTmp rsp AlwaysVerify k $ \tmp ->
+ unVerified $ storefile tmp o l getb p
+ runner (next ok)
+ StoreContentTo dest o l getb next -> do
+--- git-annex-6.20170101.orig/Remote.hs
++++ git-annex-6.20170101/Remote.hs
+@@ -12,6 +12,7 @@ module Remote (
+ storeKey,
+ retrieveKeyFile,
+ retrieveKeyFileCheap,
++ retrievalSecurityPolicy,
+ removeKey,
+ hasKey,
+ hasKeyCheap,
+--- git-annex-6.20170101.orig/Remote/Git.hs
++++ git-annex-6.20170101/Remote/Git.hs
+@@ -571,10 +571,11 @@ copyToRemote' r key file meterupdate
+ ensureInitialized
+ copier <- mkCopier hardlink params
+ let verify = Annex.Content.RemoteVerify r
++ let rsp = RetrievalAllKeysSecure
+ runTransfer (Transfer Download u key) file forwardRetry $ \p ->
+ let p' = combineMeterUpdate meterupdate p
+ in Annex.Content.saveState True `after`
+- Annex.Content.getViaTmp verify key
++ Annex.Content.getViaTmp rsp verify key
+ (\dest -> copier object dest p' (liftIO checksuccessio))
+ )
+
+--- git-annex-6.20170101.orig/Types/GitConfig.hs
++++ git-annex-6.20170101/Types/GitConfig.hs
+@@ -74,6 +74,7 @@ data GitConfig = GitConfig
+ , annexAddUnlocked :: Bool
+ , annexAllowedUrlSchemes :: S.Set Scheme
+ , annexAllowedHttpAddresses :: String
++ , annexAllowUnverifiedDownloads :: Bool
+ , coreSymlinks :: Bool
+ , coreSharedRepository :: SharedRepository
+ , gcryptId :: Maybe String
+@@ -128,6 +129,8 @@ extractGitConfig r = GitConfig
+ getmaybe (annex "security.allowed-url-schemes")
+ , annexAllowedHttpAddresses = fromMaybe "" $
+ getmaybe (annex "security.allowed-http-addresses")
++ , annexAllowUnverifiedDownloads = (== Just "ACKTHPPT") $
++ getmaybe (annex "security.allow-unverified-downloads")
+ , coreSymlinks = getbool "core.symlinks" True
+ , coreSharedRepository = getSharedRepository r
+ , gcryptId = getmaybe "core.gcrypt-id"
+--- git-annex-6.20170101.orig/Types/Key.hs
++++ git-annex-6.20170101/Types/Key.hs
+@@ -15,6 +15,7 @@ module Types.Key (
+ chunkKeyOffset,
+ isChunkKey,
+ isKeyPrefix,
++ isVerifiable,
+
+ prop_isomorphic_key_encode,
+ prop_isomorphic_key_decode
+--- git-annex-6.20170101.orig/doc/git-annex.mdwn
++++ git-annex-6.20170101/doc/git-annex.mdwn
+@@ -1126,6 +1126,10 @@ Here are all the supported configuration
+ from remotes. If you trust a remote and don't want the overhead
+ of these checksums, you can set this to `false`.
+
++ Note that even when this is set to `false`, git-annex does verification
++ in some edge cases, where it's likely the case than an
++ object was downloaded incorrectly, or when needed for security.
++
+ * `remote.<name>.annexUrl`
+
+ Can be used to specify a different url than the regular `remote.<name>.url`
+@@ -1268,6 +1272,43 @@ Here are all the supported configuration
+ causing it to be downloaded into your repository and transferred to
+ other remotes, exposing its content.
+
++* `annex.security.allow-unverified-downloads`,
++
++ For security reasons, git-annex refuses to download content from
++ most special remotes when it cannot check a hash to verify
++ that the correct content was downloaded. This particularly impacts
++ downloading the content of URL or WORM keys, which lack hashes.
++
++ The best way to avoid problems due to this is to migrate files
++ away from such keys, before their content reaches a special remote.
++ See [[git-annex-migrate]](1).
++
++ When the content is only available from a special remote, you can
++ use this configuration to force git-annex to download it.
++ But you do so at your own risk, and it's very important you read and
++ understand the information below first!
++
++ Downloading unverified content from encrypted special remotes is
++ prevented, because the special remote could send some other encrypted
++ content than what you expect, causing git-annex to decrypt data that you
++ never checked into git-annex, and risking exposing the decrypted
++ data to any non-encrypted remotes you send content to.
++
++ Downloading unverified content from (non-encrypted)
++ external special remotes is prevented, because they could follow
++ http redirects to web servers on localhost or on a private network,
++ or in some cases to a file:/// url.
++
++ If you decide to bypass this security check, the best thing to do is
++ to only set it temporarily while running the command that gets the file.
++ The value to set the config to is "ACKTHPPT".
++ For example:
++
++ git -c annex.security.allow-unverified-downloads=ACKTHPPT annex get myfile
++
++ It would be a good idea to check that it downloaded the file you expected,
++ too.
++
+ * `annex.secure-erase-command`
+
+ This can be set to a command that should be run whenever git-annex
diff -Nru git-annex-6.20170101/debian/patches/limit-url-downloads-to-whitelisted-schem.patch git-annex-6.20170101/debian/patches/limit-url-downloads-to-whitelisted-schem.patch
--- git-annex-6.20170101/debian/patches/limit-url-downloads-to-whitelisted-schem.patch 1970-01-01 01:00:00.000000000 +0100
+++ git-annex-6.20170101/debian/patches/limit-url-downloads-to-whitelisted-schem.patch 2018-06-22 16:42:37.000000000 +0100
@@ -0,0 +1,331 @@
+From: Joey Hess <id@joeyh.name>
+Date: Mon, 18 Jun 2018 15:38:25 -0400
+X-Dgit-Generated: 6.20170101-1+deb9u2 3a4e7ae8749102cf318a41f32598eb40ec22e92e
+Subject: limit url downloads to whitelisted schemes
+
+backported from 28720c795ff57a55b48e56d15f9b6bcb977f48d9
+
+Security fix! Allowing any schemes, particularly file: and
+possibly others like scp: allowed file exfiltration by anyone who had
+write access to the git repository, since they could add an annexed file
+using such an url, or using an url that redirected to such an url,
+and wait for the victim to get it into their repository and send them a copy.
+
+* Added annex.security.allowed-url-schemes setting, which defaults
+ to only allowing http and https URLs. Note especially that file:/
+ is no longer enabled by default.
+
+* Removed annex.web-download-command, since its interface does not allow
+ supporting annex.security.allowed-url-schemes across redirects.
+ If you used this setting, you may want to instead use annex.web-options
+ to pass options to curl.
+
+With annex.web-download-command removed, nearly all url accesses in
+git-annex are made via Utility.Url via http-client or curl. http-client
+only supports http and https, so no problem there.
+(Disabling one and not the other is not implemented.)
+
+Used curl --proto to limit the allowed url schemes.
+wget only supports http https ftp, so does not need any limiting.
+
+Note that this will cause git annex fsck --from web to mark files using
+a disallowed url scheme as not being present in the web. That seems
+acceptable; fsck --from web also does that when a web server is not available.
+
+quvi was not changed; it only supports a hardcoded set of urls, which
+are http, not file urls.
+
+This does not address any external special remotes that might download
+an url themselves. Current thinking is all external special remotes will
+need to be audited for this problem, although many of them will use
+http libraries that only support http and not curl's menagarie.
+
+The related problem of accessing private localhost and LAN urls is not
+addressed by this commit.
+
+This commit was sponsored by Brett Eisenberg on Patreon.
+
+---
+
+--- git-annex-6.20170101.orig/Annex/Content.hs
++++ git-annex-6.20170101/Annex/Content.hs
+@@ -902,24 +902,15 @@ saveState nocommit = doSideAction $ do
+
+ {- Downloads content from any of a list of urls. -}
+ downloadUrl :: Key -> MeterUpdate -> [Url.URLString] -> FilePath -> Annex Bool
+-downloadUrl k p urls file = meteredFile file (Just p) k $
+- go =<< annexWebDownloadCommand <$> Annex.getGitConfig
++downloadUrl k p urls file = meteredFile file (Just p) k go
+ where
+- go Nothing = do
++ go = do
+ a <- ifM commandProgressDisabled
+ ( return Url.downloadQuiet
+ , return Url.download
+ )
+ Url.withUrlOptions $ \uo ->
+ anyM (\u -> a u file uo) urls
+- go (Just basecmd) = anyM (downloadcmd basecmd) urls
+- downloadcmd basecmd url =
+- progressCommand "sh" [Param "-c", Param $ gencmd url basecmd]
+- <&&> liftIO (doesFileExist file)
+- gencmd url = massReplace
+- [ ("%file", shellEscape file)
+- , ("%url", shellEscape url)
+- ]
+
+ {- Copies a key's content, when present, to a temp file.
+ - This is used to speed up some rsyncs. -}
+--- git-annex-6.20170101.orig/Annex/Url.hs
++++ git-annex-6.20170101/Annex/Url.hs
+@@ -30,6 +30,7 @@ getUrlOptions = mkUrlOptions
+ <$> getUserAgent
+ <*> headers
+ <*> options
++ <*> (annexAllowedUrlSchemes <$> Annex.getGitConfig)
+ where
+ headers = do
+ v <- annexHttpHeadersCommand <$> Annex.getGitConfig
+--- git-annex-6.20170101.orig/NEWS
++++ git-annex-6.20170101/NEWS
+@@ -1,3 +1,14 @@
++git-annex (6.20170101-1+deb9u2) stretch-security; urgency=high
++
++ A security fix has changed git-annex to only support http, https, and ftp
++ URL schemes by default. You can enable other URL schemes, at your own risk,
++ using annex.security.allowed-url-schemes.
++
++ The annex.web-download-command configuration has been removed,
++ use annex.web-options instead.
++
++ -- Joey Hess <id@joeyh.name> Fri, 15 Jun 2018 17:54:23 -0400
++
+ git-annex (6.20170101) unstable; urgency=medium
+
+ XMPP support has been removed from the assistant in this release.
+--- git-annex-6.20170101.orig/Types/GitConfig.hs
++++ git-annex-6.20170101/Types/GitConfig.hs
+@@ -28,6 +28,9 @@ import Types.RefSpec
+ import Utility.HumanTime
+ import Utility.Gpg (GpgCmd, mkGpgCmd)
+ import Utility.ThreadScheduler (Seconds(..))
++import Utility.Url (Scheme, mkScheme)
++
++import qualified Data.Set as S
+
+ {- Main git-annex settings. Each setting corresponds to a git-config key
+ - such as annex.foo -}
+@@ -51,7 +54,6 @@ data GitConfig = GitConfig
+ , annexWebOptions :: [String]
+ , annexQuviOptions :: [String]
+ , annexAriaTorrentOptions :: [String]
+- , annexWebDownloadCommand :: Maybe String
+ , annexCrippledFileSystem :: Bool
+ , annexLargeFiles :: Maybe String
+ , annexAddSmallFiles :: Bool
+@@ -70,6 +72,8 @@ data GitConfig = GitConfig
+ , annexPidLock :: Bool
+ , annexPidLockTimeout :: Seconds
+ , annexAddUnlocked :: Bool
++ , annexAllowedUrlSchemes :: S.Set Scheme
++ , annexAllowedHttpAddresses :: String
+ , coreSymlinks :: Bool
+ , coreSharedRepository :: SharedRepository
+ , gcryptId :: Maybe String
+@@ -98,7 +102,6 @@ extractGitConfig r = GitConfig
+ , annexWebOptions = getwords (annex "web-options")
+ , annexQuviOptions = getwords (annex "quvi-options")
+ , annexAriaTorrentOptions = getwords (annex "aria-torrent-options")
+- , annexWebDownloadCommand = getmaybe (annex "web-download-command")
+ , annexCrippledFileSystem = getbool (annex "crippledfilesystem") False
+ , annexLargeFiles = getmaybe (annex "largefiles")
+ , annexAddSmallFiles = getbool (annex "addsmallfiles") True
+@@ -120,6 +123,11 @@ extractGitConfig r = GitConfig
+ , annexPidLockTimeout = Seconds $ fromMaybe 300 $
+ getmayberead (annex "pidlocktimeout")
+ , annexAddUnlocked = getbool (annex "addunlocked") False
++ , annexAllowedUrlSchemes = S.fromList $ map mkScheme $
++ maybe ["http", "https", "ftp"] words $
++ getmaybe (annex "security.allowed-url-schemes")
++ , annexAllowedHttpAddresses = fromMaybe "" $
++ getmaybe (annex "security.allowed-http-addresses")
+ , coreSymlinks = getbool "core.symlinks" True
+ , coreSharedRepository = getSharedRepository r
+ , gcryptId = getmaybe "core.gcrypt-id"
+--- git-annex-6.20170101.orig/Utility/Url.hs
++++ git-annex-6.20170101/Utility/Url.hs
+@@ -15,6 +15,9 @@ module Utility.Url (
+ managerSettings,
+ URLString,
+ UserAgent,
++ Scheme,
++ mkScheme,
++ allowedScheme,
+ UrlOptions,
+ mkUrlOptions,
+ check,
+@@ -38,6 +41,7 @@ import Network.HTTP.Types
+ import qualified Data.CaseInsensitive as CI
+ import qualified Data.ByteString as B
+ import qualified Data.ByteString.UTF8 as B8
++import qualified Data.Set as S
+ import Control.Monad.Trans.Resource
+ import Network.HTTP.Conduit hiding (closeManager)
+
+@@ -63,6 +67,15 @@ type Headers = [String]
+
+ type UserAgent = String
+
++newtype Scheme = Scheme (CI.CI String)
++ deriving (Eq, Ord)
++
++mkScheme :: String -> Scheme
++mkScheme = Scheme . CI.mk
++
++fromScheme :: Scheme -> String
++fromScheme (Scheme s) = CI.original s
++
+ data UrlOptions = UrlOptions
+ { userAgent :: Maybe UserAgent
+ , reqHeaders :: Headers
+@@ -72,15 +85,17 @@ data UrlOptions = UrlOptions
+ #else
+ , applyRequest :: forall m. Request m -> Request m
+ #endif
++ , allowedSchemes :: S.Set Scheme
+ }
+
+ instance Default UrlOptions
+ where
+- def = UrlOptions Nothing [] [] id
++ def = UrlOptions Nothing [] [] id
++ (S.fromList $ map mkScheme ["http", "https", "ftp"])
+
+-mkUrlOptions :: Maybe UserAgent -> Headers -> [CommandParam] -> UrlOptions
+-mkUrlOptions defuseragent reqheaders reqparams =
+- UrlOptions useragent reqheaders reqparams applyrequest
++mkUrlOptions :: Maybe UserAgent -> Headers -> [CommandParam] -> S.Set Scheme -> UrlOptions
++mkUrlOptions defuseragent reqheaders reqparams allowedschemes =
++ UrlOptions useragent reqheaders reqparams applyrequest allowedschemes
+ where
+ applyrequest = \r -> r { requestHeaders = requestHeaders r ++ addedheaders }
+ addedheaders = uaheader ++ otherheaders
+@@ -104,6 +119,28 @@ addUserAgent uo ps = case userAgent uo o
+ -- --user-agent works for both wget and curl commands
+ Just ua -> ps ++ [Param "--user-agent", Param ua]
+
++checkPolicy :: UrlOptions -> URI -> a -> IO a -> IO a
++checkPolicy uo u onerr a
++ | allowedScheme uo u = a
++ | otherwise = do
++ hPutStrLn stderr $
++ "Configuration does not allow accessing " ++ show u
++ hFlush stderr
++ return onerr
++
++curlSchemeParams :: UrlOptions -> [CommandParam]
++curlSchemeParams uo =
++ [ Param "--proto"
++ , Param $ intercalate "," ("-all" : schemelist)
++ ]
++ where
++ schemelist = map fromScheme $ S.toList $ allowedSchemes uo
++
++allowedScheme :: UrlOptions -> URI -> Bool
++allowedScheme uo u = uscheme `S.member` allowedSchemes uo
++ where
++ uscheme = mkScheme $ takeWhile (/=':') (uriScheme u)
++
+ {- Checks that an url exists and could be successfully downloaded,
+ - also checking that its size, if available, matches a specified size. -}
+ checkBoth :: URLString -> Maybe Integer -> UrlOptions -> IO Bool
+@@ -137,7 +174,7 @@ assumeUrlExists = UrlInfo True Nothing N
+ - also returning its size and suggested filename if available. -}
+ getUrlInfo :: URLString -> UrlOptions -> IO UrlInfo
+ getUrlInfo url uo = case parseURIRelaxed url of
+- Just u -> case parseurlconduit (show u) of
++ Just u -> checkPolicy uo u dne' $ case parseurlconduit (show u) of
+ Just req -> catchJust
+ -- When http redirects to a protocol which
+ -- conduit does not support, it will throw
+@@ -161,7 +198,8 @@ getUrlInfo url uo = case parseURIRelaxed
+ | otherwise -> dne
+ Nothing -> dne
+ where
+- dne = return $ UrlInfo False Nothing Nothing
++ dne = return dne'
++ dne' = UrlInfo False Nothing Nothing
+ found sz f = return $ UrlInfo True sz f
+
+ curlparams = addUserAgent uo $
+@@ -169,7 +207,7 @@ getUrlInfo url uo = case parseURIRelaxed
+ , Param "--head"
+ , Param "-L", Param url
+ , Param "-w", Param "%{http_code}"
+- ] ++ concatMap (\h -> [Param "-H", Param h]) (reqHeaders uo) ++ (reqParams uo)
++ ] ++ concatMap (\h -> [Param "-H", Param h]) (reqHeaders uo) ++ (reqParams uo) ++ curlSchemeParams uo
+
+ extractlencurl s = case lastMaybe $ filter ("Content-Length:" `isPrefixOf`) (lines s) of
+ Just l -> case lastMaybe $ words l of
+@@ -263,9 +301,10 @@ downloadQuiet = download' True
+ download' :: Bool -> URLString -> FilePath -> UrlOptions -> IO Bool
+ download' quiet url file uo = do
+ case parseURIRelaxed url of
+- Just u
+- | uriScheme u == "file:" -> curl
+- | otherwise -> ifM (inPath "wget") (wget , curl)
++ Just u -> checkPolicy uo u False $
++ if uriScheme u == "file:"
++ then curl
++ else ifM (inPath "wget") (wget , curl)
+ _ -> return False
+ where
+ headerparams = map (\h -> Param $ "--header=" ++ h) (reqHeaders uo)
+@@ -276,6 +315,10 @@ download' quiet url file uo = do
+ -
+ - When the wget version is new enough, pass options for
+ - a less cluttered download display.
++ -
++ - wget only supports https, http, and ftp, not file, which are
++ - all always allowed, so its url schemes do not need to be
++ - limited.
+ -}
+ #ifndef __ANDROID__
+ wgetparams = concat
+@@ -296,7 +339,7 @@ download' quiet url file uo = do
+ -- curl does not create destination file
+ -- if the url happens to be empty, so pre-create.
+ writeFile file ""
+- go "curl" $ headerparams ++ quietopt "-s" ++
++ go "curl" $ headerparams ++ quietopt "-s" ++ curlSchemeParams uo ++
+ [Param "-f", Param "-L", Param "-C", Param "-", Param "-#", Param "-o"]
+
+ {- Run wget in a temp directory because it has been buggy
+--- git-annex-6.20170101.orig/doc/git-annex.mdwn
++++ git-annex-6.20170101/doc/git-annex.mdwn
+@@ -1238,13 +1238,20 @@ Here are all the supported configuration
+ If set, the command is run and each line of its output is used as a HTTP
+ header. This overrides annex.http-headers.
+
+-* `annex.web-download-command`
++* `annex.security.allowed-url-schemes`
+
+- Use to specify a command to run to download a file from the web.
+- (The default is to use wget or curl.)
++ List of URL schemes that git-annex is allowed to download content from.
++ The default is "http https ftp".
+
+- In the command line, %url is replaced with the url to download,
+- and %file is replaced with the file that it should be saved to.
++ Think very carefully before changing this; there are security
++ implications. For example, if it's changed to allow "file" URLs, then
++ anyone who can get a commit into your git-annex repository could
++ `git-annex addurl` a pointer to a private file located outside that
++ repository, possibly causing it to be copied into your repository
++ and transferred on to other remotes, exposing its content.
++
++ Some special remotes support their own domain-specific URL
++ schemes; those are not affected by this configuration setting.
+
+ * `annex.secure-erase-command`
+
diff -Nru git-annex-6.20170101/debian/patches/series git-annex-6.20170101/debian/patches/series
--- git-annex-6.20170101/debian/patches/series 2017-10-26 15:28:29.000000000 +0100
+++ git-annex-6.20170101/debian/patches/series 2018-06-22 16:42:37.000000000 +0100
@@ -1 +1,8 @@
CVE-2017-12976.patch
+limit-url-downloads-to-whitelisted-schem.patch
+block-url-downloads-by-default.patch
+update-test-suite-for-security-fix.patch
+add-retrievalsecuritypolicy.patch
+enforce-retrievalsecuritypolicy.patch
+dont-assume-boto-will-remain-secure.patch
+set-ddar-to-retrievalallkeyssecure.patch
diff -Nru git-annex-6.20170101/debian/patches/set-ddar-to-retrievalallkeyssecure.patch git-annex-6.20170101/debian/patches/set-ddar-to-retrievalallkeyssecure.patch
--- git-annex-6.20170101/debian/patches/set-ddar-to-retrievalallkeyssecure.patch 1970-01-01 01:00:00.000000000 +0100
+++ git-annex-6.20170101/debian/patches/set-ddar-to-retrievalallkeyssecure.patch 2018-06-22 16:42:37.000000000 +0100
@@ -0,0 +1,25 @@
+From: Joey Hess <joeyh@joeyh.name>
+Date: Thu, 21 Jun 2018 16:38:47 -0400
+X-Dgit-Generated: 6.20170101-1+deb9u2 fd8086fe4a9c4bd64e95fd9152b1800b1c0f6b89
+Subject: set ddar to RetrievalAllKeysSecure
+
+Based on information from Robie Basak.
+
+(cherry picked from commit 05ecee0db43919b3cfd8d2e2772022e16a804a04)
+
+---
+
+--- git-annex-6.20170101.orig/Remote/Ddar.hs
++++ git-annex-6.20170101/Remote/Ddar.hs
+@@ -56,8 +56,9 @@ gen r u c gc = do
+ , storeKey = storeKeyDummy
+ , retrieveKeyFile = retreiveKeyFileDummy
+ , retrieveKeyFileCheap = retrieveCheap
+- -- Unsure about this, safe default until Robie answers.
+- , retrievalSecurityPolicy = RetrievalVerifiableKeysSecure
++ -- ddar communicates over ssh, not subject to http redirect
++ -- type attacks
++ , retrievalSecurityPolicy = RetrievalAllKeysSecure
+ , removeKey = removeKeyDummy
+ , lockContent = Nothing
+ , checkPresent = checkPresentDummy
diff -Nru git-annex-6.20170101/debian/patches/update-test-suite-for-security-fix.patch git-annex-6.20170101/debian/patches/update-test-suite-for-security-fix.patch
--- git-annex-6.20170101/debian/patches/update-test-suite-for-security-fix.patch 1970-01-01 01:00:00.000000000 +0100
+++ git-annex-6.20170101/debian/patches/update-test-suite-for-security-fix.patch 2018-06-22 16:42:37.000000000 +0100
@@ -0,0 +1,31 @@
+From: Joey Hess <id@joeyh.name>
+Date: Mon, 18 Jun 2018 18:18:06 -0400
+X-Dgit-Generated: 6.20170101-1+deb9u2 9208ae8c4e1c9c6b1fd4cb004b4cda568ffd9406
+Subject: update test suite for security fix
+
+
+---
+
+--- git-annex-6.20170101.orig/Test.hs
++++ git-annex-6.20170101/Test.hs
+@@ -1688,12 +1688,18 @@ test_add_subdirs = intmpclonerepo $ do
+ test_addurl :: Assertion
+ test_addurl = intmpclonerepo $ do
+ -- file:// only; this test suite should not hit the network
++ let filecmd c ps = git_annex c
++ ( "-cannex.security.allowed-url-schemes=file"
++ : "-cannex.security.allowed-http-addresses=all"
++ : ps
++ )
+ f <- absPath "myurl"
+ let url = replace "\\" "/" ("file:///" ++ dropDrive f)
+ writeFile f "foo"
+- git_annex "addurl" [url] @? ("addurl failed on " ++ url)
++ not <$> git_annex "addurl" [url] @? "addurl failed to fail on file url"
++ filecmd "addurl" [url] @? ("addurl failed on " ++ url)
+ let dest = "addurlurldest"
+- git_annex "addurl" ["--file", dest, url] @? ("addurl failed on " ++ url ++ " with --file")
++ filecmd "addurl" ["--file", dest, url] @? ("addurl failed on " ++ url ++ " with --file")
+ doesFileExist dest @? (dest ++ " missing after addurl --file")
+
+ -- This is equivilant to running git-annex, but it's all run in-process
Attachment:
signature.asc
Description: PGP signature