Your message dated Sat, 14 Jul 2018 11:21:20 +0100 with message-id <1531563680.2095.30.camel@adam-barratt.org.uk> and subject line Closing bugs for updates included in 9.5 has caused the Debian Bug report #903037, regarding stretch-pu: package git-annex/6.20170101-1+b1 to be marked as done. This means that you claim that the problem has been dealt with. If this is not the case it is now your responsibility to reopen the Bug report if necessary, and/or fix the problem forthwith. (NB: If you are a system administrator and have no idea what this message is talking about, this may indicate a serious mail system misconfiguration somewhere. Please contact owner@bugs.debian.org immediately.) -- 903037: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=903037 Debian Bug Tracking System Contact owner@bugs.debian.org with problems
--- Begin Message ---
- To: Debian Bug Tracking System <submit@bugs.debian.org>
- Subject: stretch-pu: package git-annex/6.20170101-1+b1
- From: Sean Whitton <spwhitton@spwhitton.name>
- Date: Thu, 5 Jul 2018 13:59:23 +0100
- Message-id: <[🔎] 20180705125923.3d6xtpy5dzpauzlv@zephyr.silentflame.com>
Package: release.debian.org Severity: important Tags: stretch User: release.debian.org@packages.debian.org Usertags: pu git-annex in stretch is vulnerable to CVE-2018-10857 and CVE-2018-10859. This update is a minimal fix for those CVEs prepared by its upstream, Joey Hess: git-annex (6.20170101-1+deb9u2) stretch; urgency=high [ Joey Hess ] * CVE-2018-10857: - Added annex.security.allowed-url-schemes setting, which defaults to only allowing http, https, and ftp URLs. Note especially that file:/ is no longer enabled by default. - Removed annex.web-download-command, since its interface does not allow supporting annex.security.allowed-url-schemes across redirects. If you used this setting, you may want to instead use annex.web-options to pass options to curl. - git-annex will refuse to download content from the web, to prevent accidental exposure of data on private webservers on localhost and the LAN. This can be overridden with the annex.security.allowed-http-addresses setting. (The S3, glacier, and webdav special remotes are still allowed to download from the web.) * CVE-2018-10857 and CVE-2018-10859: - Refuse to download content, that cannot be verified with a hash, from encrypted special remotes (for CVE-2018-10859), and from all external special remotes (for CVE-2018-10857). In particular, URL and WORM keys stored on such remotes won't be downloaded. If this affects your files, you can run `git-annex migrate` on the affected files, to convert them to use a hash. - Added annex.security.allow-unverified-downloads, which can override the above. -- Sean Whitton <spwhitton@spwhitton.name> Fri, 22 Jun 2018 16:42:37 +0100 The security team have decided that because a point release is close and only certain usecases of git-annex are impacted, they won't issue a DSA. I have not yet uploaded to stretch-proposed-updates for two reasons: (i) although the fix is minimal and much smaller than the fix for the CVEs included in git-annex in unstable, the source debdiff (attached) is still quite large, so I thought I should wait for an explicit ACK from the release team (ii) there is already a +deb9u1 version of git-annex in stretch-security, but not stretch, responding to a different CVE. I have based my work on the +deb9u1 upload, and I assume that uploading my +deb9u2 to stretch-proposed-updates will cause it to take precedence over the import of the +deb9u1 upload. But in case things are more fragile than that, I'm holding back. Note that my debdiff is from +deb9u1 to +deb9u2. I ran the test suite and manually smoke tested my proposed update against stretch as the latter stood about a week ago, and plan to run the test suite again right before uploading to stretch-proposed-updates. Thank you for your attention! -- System Information: Debian Release: 9.4 APT prefers stable APT policy: (900, 'stable'), (500, 'stable-updates') Architecture: i386 (i686) Kernel: Linux 4.9.0-6-686-pae (SMP w/2 CPU cores) Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE=en_GB.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) -- Sean Whittondiff -Nru git-annex-6.20170101/debian/changelog git-annex-6.20170101/debian/changelog --- git-annex-6.20170101/debian/changelog 2017-10-26 15:28:29.000000000 +0100 +++ git-annex-6.20170101/debian/changelog 2018-06-22 16:42:37.000000000 +0100 @@ -1,3 +1,33 @@ +git-annex (6.20170101-1+deb9u2) stretch; urgency=high + + [ Joey Hess ] + * CVE-2018-10857: + - Added annex.security.allowed-url-schemes setting, which defaults + to only allowing http, https, and ftp URLs. Note especially that file:/ + is no longer enabled by default. + - Removed annex.web-download-command, since its interface does not allow + supporting annex.security.allowed-url-schemes across redirects. + If you used this setting, you may want to instead use annex.web-options + to pass options to curl. + - git-annex will refuse to download content from the web, to prevent + accidental exposure of data on private webservers on localhost and the + LAN. This can be overridden with the + annex.security.allowed-http-addresses setting. + (The S3, glacier, and webdav special remotes are still allowed to + download from the web.) + * CVE-2018-10857 and CVE-2018-10859: + - Refuse to download content, that cannot be verified with a hash, + from encrypted special remotes (for CVE-2018-10859), + and from all external special remotes (for CVE-2018-10857). + In particular, URL and WORM keys stored on such remotes won't + be downloaded. If this affects your files, you can run + `git-annex migrate` on the affected files, to convert them + to use a hash. + - Added annex.security.allow-unverified-downloads, which can override + the above. + + -- Sean Whitton <spwhitton@spwhitton.name> Fri, 22 Jun 2018 16:42:37 +0100 + git-annex (6.20170101-1+deb9u1) stretch-security; urgency=high * Non-maintainer upload by the Security Team. diff -Nru git-annex-6.20170101/debian/patches/add-retrievalsecuritypolicy.patch git-annex-6.20170101/debian/patches/add-retrievalsecuritypolicy.patch --- git-annex-6.20170101/debian/patches/add-retrievalsecuritypolicy.patch 1970-01-01 01:00:00.000000000 +0100 +++ git-annex-6.20170101/debian/patches/add-retrievalsecuritypolicy.patch 2018-06-22 16:42:37.000000000 +0100 @@ -0,0 +1,293 @@ +From: Joey Hess <joeyh@joeyh.name> +Date: Thu, 21 Jun 2018 11:35:27 -0400 +X-Dgit-Generated: 6.20170101-1+deb9u2 fe511617f5bde916f1bde799b5a51ddc9404eea1 +Subject: add retrievalSecurityPolicy + +This will be used to protect against CVE-2018-10859, where an encrypted +special remote is fed the wrong encrypted data, and so tricked into +decrypting something that the user encrypted with their gpg key and did +not store in git-annex. + +It also protects against CVE-2018-10857, where a remote follows a http +redirect to a file:// url or to a local private web server. While that's +already been prevented in git-annex's own use of http, external special +remotes, hooks, etc use other http implementations and could still be +vulnerable. + +The policy is not yet enforced, this commit only adds the appropriate +metadata to remotes. + +This commit was sponsored by Boyd Stephen Smith Jr. on Patreon. + +(cherry picked from commit 4315bb9e421f2c643e517d8982c6c35b1909c78b) + +--- + +--- git-annex-6.20170101.orig/Remote/BitTorrent.hs ++++ git-annex-6.20170101/Remote/BitTorrent.hs +@@ -57,6 +57,8 @@ gen r _ c gc = + , storeKey = uploadKey + , retrieveKeyFile = downloadKey + , retrieveKeyFileCheap = downloadKeyCheap ++ -- Bittorrent does its own hash checks. ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = dropKey + , lockContent = Nothing + , checkPresent = checkKey +--- git-annex-6.20170101.orig/Remote/Bup.hs ++++ git-annex-6.20170101/Remote/Bup.hs +@@ -56,6 +56,9 @@ gen r u c gc = do + , storeKey = storeKeyDummy + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = retrieveCheap buprepo ++ -- Bup uses git, which cryptographically verifies content ++ -- (with SHA1, but sufficiently for this). ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy +--- git-annex-6.20170101.orig/Remote/Ddar.hs ++++ git-annex-6.20170101/Remote/Ddar.hs +@@ -56,6 +56,8 @@ gen r u c gc = do + , storeKey = storeKeyDummy + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = retrieveCheap ++ -- Unsure about this, safe default until Robie answers. ++ , retrievalSecurityPolicy = RetrievalVerifiableKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy +--- git-annex-6.20170101.orig/Remote/Directory.hs ++++ git-annex-6.20170101/Remote/Directory.hs +@@ -54,6 +54,7 @@ gen r u c gc = do + , storeKey = storeKeyDummy + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = retrieveCheap dir chunkconfig ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy +--- git-annex-6.20170101.orig/Remote/External.hs ++++ git-annex-6.20170101/Remote/External.hs +@@ -81,6 +81,11 @@ gen r u c gc + , storeKey = storeKeyDummy + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = \_ _ _ -> return False ++ -- External special remotes use many http libraries ++ -- and have no protection against redirects to ++ -- local private web servers, or in some cases ++ -- to file:// urls. ++ , retrievalSecurityPolicy = RetrievalVerifiableKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy +--- git-annex-6.20170101.orig/Remote/GCrypt.hs ++++ git-annex-6.20170101/Remote/GCrypt.hs +@@ -111,6 +111,7 @@ gen' r u c gc = do + , storeKey = storeKeyDummy + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = \_ _ _ -> return False ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy +--- git-annex-6.20170101.orig/Remote/Git.hs ++++ git-annex-6.20170101/Remote/Git.hs +@@ -147,6 +147,7 @@ gen r u c gc + , storeKey = copyToRemote new + , retrieveKeyFile = copyFromRemote new + , retrieveKeyFileCheap = copyFromRemoteCheap new ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = dropKey new + , lockContent = Just (lockKey new) + , checkPresent = inAnnex new +--- git-annex-6.20170101.orig/Remote/Glacier.hs ++++ git-annex-6.20170101/Remote/Glacier.hs +@@ -53,6 +53,9 @@ gen r u c gc = new <$> remoteCost gc ver + , storeKey = storeKeyDummy + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = retrieveCheap this ++ -- glacier-cli does not follow redirects and does ++ -- not support file://, so this is secure. ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy +--- git-annex-6.20170101.orig/Remote/Helper/Special.hs ++++ git-annex-6.20170101/Remote/Helper/Special.hs +@@ -161,6 +161,14 @@ specialRemote' cfg c preparestorer prepa + (retrieveKeyFileCheap baser k f d) + -- retrieval of encrypted keys is never cheap + (\_ -> return False) ++ -- When encryption is used, the remote could provide ++ -- some other content encrypted by the user, and trick ++ -- git-annex into decrypting it, leaking the decryption ++ -- into the git-annex repository. Verifiable keys ++ -- are the main protection against this attack. ++ , retrievalSecurityPolicy = if isencrypted ++ then RetrievalVerifiableKeysSecure ++ else retrievalSecurityPolicy baser + , removeKey = \k -> cip >>= removeKeyGen k + , checkPresent = \k -> cip >>= checkPresentGen k + , cost = if isencrypted +--- git-annex-6.20170101.orig/Remote/Hook.hs ++++ git-annex-6.20170101/Remote/Hook.hs +@@ -47,6 +47,9 @@ gen r u c gc = do + , storeKey = storeKeyDummy + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = retrieveCheap hooktype ++ -- A hook could use http and be vulnerable to ++ -- redirect to file:// attacks, etc. ++ , retrievalSecurityPolicy = RetrievalVerifiableKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy +--- git-annex-6.20170101.orig/Remote/P2P.hs ++++ git-annex-6.20170101/Remote/P2P.hs +@@ -53,6 +53,7 @@ chainGen addr r u c gc = do + , storeKey = store u addr connpool + , retrieveKeyFile = retrieve u addr connpool + , retrieveKeyFileCheap = \_ _ _ -> return False ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = remove u addr connpool + , lockContent = Just (lock u addr connpool) + , checkPresent = checkpresent u addr connpool +--- git-annex-6.20170101.orig/Remote/Rsync.hs ++++ git-annex-6.20170101/Remote/Rsync.hs +@@ -69,6 +69,7 @@ gen r u c gc = do + , storeKey = storeKeyDummy + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = retrieveCheap o ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy +--- git-annex-6.20170101.orig/Remote/S3.hs ++++ git-annex-6.20170101/Remote/S3.hs +@@ -86,6 +86,9 @@ gen r u c gc = do + , storeKey = storeKeyDummy + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = retrieveCheap ++ -- HttpManagerRestricted is used here, so this is ++ -- secure. ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy +--- git-annex-6.20170101.orig/Remote/Tahoe.hs ++++ git-annex-6.20170101/Remote/Tahoe.hs +@@ -71,6 +71,8 @@ gen r u c gc = do + , storeKey = store u hdl + , retrieveKeyFile = retrieve u hdl + , retrieveKeyFileCheap = \_ _ _ -> return False ++ -- Tahoe cryptographically verifies content. ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = remove + , lockContent = Nothing + , checkPresent = checkKey u hdl +--- git-annex-6.20170101.orig/Remote/Web.hs ++++ git-annex-6.20170101/Remote/Web.hs +@@ -46,6 +46,9 @@ gen r _ c gc = + , storeKey = uploadKey + , retrieveKeyFile = downloadKey + , retrieveKeyFileCheap = downloadKeyCheap ++ -- HttpManagerRestricted is used here, so this is ++ -- secure. ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = dropKey + , lockContent = Nothing + , checkPresent = checkKey +--- git-annex-6.20170101.orig/Remote/WebDAV.hs ++++ git-annex-6.20170101/Remote/WebDAV.hs +@@ -64,6 +64,9 @@ gen r u c gc = new <$> remoteCost gc exp + , storeKey = storeKeyDummy + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = retrieveCheap ++ -- HttpManagerRestricted is used here, so this is ++ -- secure. ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy +--- git-annex-6.20170101.orig/Types/Key.hs ++++ git-annex-6.20170101/Types/Key.hs +@@ -1,6 +1,6 @@ + {- git-annex Key data type + - +- - Copyright 2011-2016 Joey Hess <id@joeyh.name> ++ - Copyright 2011-2018 Joey Hess <id@joeyh.name> + - + - Licensed under the GNU GPL version 3 or higher. + -} +@@ -160,3 +160,13 @@ prop_isomorphic_key_decode f + normalfieldorder = fields `isPrefixOf` "smSC" + fields = map (f !!) $ filter (< length f) $ map succ $ + elemIndices fieldSep f ++ ++{- Is the Key variety backed by a hash, which allows verifying content? ++ - It does not have to be cryptographically secure against eg birthday ++ - attacks. ++ -} ++isVerifiable :: Key -> Bool ++isVerifiable k = case keyBackendName k of ++ "WORM" -> False ++ "URL" -> False ++ _ -> True +--- git-annex-6.20170101.orig/Types/Remote.hs ++++ git-annex-6.20170101/Types/Remote.hs +@@ -2,7 +2,7 @@ + - + - Most things should not need this, using Types instead + - +- - Copyright 2011-2014 Joey Hess <id@joeyh.name> ++ - Copyright 2011-2018 Joey Hess <id@joeyh.name> + - + - Licensed under the GNU GPL version 3 or higher. + -} +@@ -17,6 +17,7 @@ module Types.Remote + , Availability(..) + , Verification(..) + , unVerified ++ , RetrievalSecurityPolicy(..) + ) + where + +@@ -75,6 +76,8 @@ data RemoteA a = Remote { + -- Retrieves a key's contents to a tmp file, if it can be done cheaply. + -- It's ok to create a symlink or hardlink. + retrieveKeyFileCheap :: Key -> AssociatedFile -> FilePath -> a Bool, ++ -- Security policy for reteiving keys from this remote. ++ retrievalSecurityPolicy :: RetrievalSecurityPolicy, + -- Removes a key's contents (succeeds if the contents are not present) + removeKey :: Key -> a Bool, + -- Uses locking to prevent removal of a key's contents, +@@ -145,3 +148,29 @@ unVerified :: Monad m => m Bool -> m (Bo + unVerified a = do + ok <- a + return (ok, UnVerified) ++ ++-- Security policy indicating what keys can be safely retrieved from a ++-- remote. ++data RetrievalSecurityPolicy ++ = RetrievalVerifiableKeysSecure ++ -- ^ Transfer of keys whose content can be verified ++ -- with a hash check is secure; transfer of unverifiable keys is ++ -- not secure and should not be allowed. ++ -- ++ -- This is used eg, when HTTP to a remote could be redirected to a ++ -- local private web server or even a file:// url, causing private ++ -- data from it that is not the intended content of a key to make ++ -- its way into the git-annex repository. ++ -- ++ -- It's also used when content is stored encrypted on a remote, ++ -- which could replace it with a different encrypted file, and ++ -- trick git-annex into decrypting it and leaking the decryption ++ -- into the git-annex repository. ++ -- ++ -- It's not (currently) used when the remote could alter the ++ -- content stored on it, because git-annex does not provide ++ -- strong guarantees about the content of keys that cannot be ++ -- verified with a hash check. ++ -- (But annex.securehashesonly does provide such guarantees.) ++ | RetrievalAllKeysSecure ++ -- ^ Any key can be securely retrieved. diff -Nru git-annex-6.20170101/debian/patches/block-url-downloads-by-default.patch git-annex-6.20170101/debian/patches/block-url-downloads-by-default.patch --- git-annex-6.20170101/debian/patches/block-url-downloads-by-default.patch 1970-01-01 01:00:00.000000000 +0100 +++ git-annex-6.20170101/debian/patches/block-url-downloads-by-default.patch 2018-06-22 16:42:37.000000000 +0100 @@ -0,0 +1,154 @@ +From: Joey Hess <id@joeyh.name> +Date: Mon, 18 Jun 2018 17:40:50 -0400 +X-Dgit-Generated: 6.20170101-1+deb9u2 139539615478c1f618525715bef184d450b38558 +Subject: block url downloads by default + +git-annex will refuse to download content from the web, to prevent +accidental exposure of data on private webservers on localhost and the +LAN. This can be overridden with the +annex.security.allowed-http-addresses setting. + +This is the simplest possible fix for the security hole. A better fix +has been developed for newer versions of git-annex but would be a lot of +work to backport, and perhaps too big a diff. + +There are several sets of git-annex users who will be impacted +in different ways by this: + +* Users who have a git-annex repository but don't use the web special + remote. Unaffected. + +* Users who have a git-annex repository that is for private use only. + They will have to read enough docs to find the setting to allow + git annex addurl to work again. + +* Users who have a git-annex repositry that is shared with people they + don't fully trust. + They will not be able to use the web special remote with this version + of git-annex. They'll have to upgrade. + +The S3, glacier, and webdav special remotes are still allowed to +download from the web. There are other potential attacks involving the +web server they connect to redirecting to a local private web server, +and tricking them from downloading content from it which then leaks back +to the attacker. Those attacks are not addressed here, but they also +seem fairly unlikely. Further analysis is needed; preliminary analysis +of glacier-cli, for example, suggests it does not follow redirects and +so is not vulnerable to such attacks. + +--- + +--- git-annex-6.20170101.orig/Annex/Quvi.hs ++++ git-annex-6.20170101/Annex/Quvi.hs +@@ -11,8 +11,8 @@ module Annex.Quvi where + + import Annex.Common + import qualified Annex ++import Annex.Url + import Utility.Quvi +-import Utility.Url + + withQuviOptions :: forall a. Query a -> [QuviParams] -> URLString -> Annex a + withQuviOptions a ps url = do +@@ -21,7 +21,14 @@ withQuviOptions a ps url = do + liftIO $ a v (concatMap (\mkp -> mkp v) ps ++ opts) url + + quviSupported :: URLString -> Annex Bool +-quviSupported u = liftIO . flip supported u =<< quviVersion ++quviSupported u = ifM httpAddressesUnlimited ++ ( liftIO . flip supported u =<< quviVersion ++ -- Don't allow any url schemes to be used when ++ -- there's a limit on the allowed addresses, because ++ -- there is no way to prevent quvi from ++ -- redirecting to any address. ++ , return False ++ ) + + quviVersion :: Annex QuviVersion + quviVersion = go =<< Annex.getState Annex.quviversion +--- git-annex-6.20170101.orig/Annex/Url.hs ++++ git-annex-6.20170101/Annex/Url.hs +@@ -11,6 +11,7 @@ module Annex.Url ( + withUrlOptions, + getUrlOptions, + getUserAgent, ++ httpAddressesUnlimited, + ) where + + import Annex.Common +@@ -18,6 +19,8 @@ import qualified Annex + import Utility.Url as U + import qualified Build.SysConfig as SysConfig + ++import qualified Data.Set as S ++ + defaultUserAgent :: U.UserAgent + defaultUserAgent = "git-annex/" ++ SysConfig.packageversion + +@@ -30,7 +33,7 @@ getUrlOptions = mkUrlOptions + <$> getUserAgent + <*> headers + <*> options +- <*> (annexAllowedUrlSchemes <$> Annex.getGitConfig) ++ <*> urlschemes + where + headers = do + v <- annexHttpHeadersCommand <$> Annex.getGitConfig +@@ -38,6 +41,18 @@ getUrlOptions = mkUrlOptions + Just cmd -> lines <$> liftIO (readProcess "sh" ["-c", cmd]) + Nothing -> annexHttpHeaders <$> Annex.getGitConfig + options = map Param . annexWebOptions <$> Annex.getGitConfig ++ urlschemes = ifM httpAddressesUnlimited ++ ( annexAllowedUrlSchemes <$> Annex.getGitConfig ++ -- Don't allow any url schemes to be used when ++ -- there's a limit on the allowed addresses, because ++ -- there is no way to prevent curl or wget from ++ -- redirecting to any address. ++ , pure S.empty ++ ) ++ ++httpAddressesUnlimited :: Annex Bool ++httpAddressesUnlimited = ++ ("all" == ) . annexAllowedHttpAddresses <$> Annex.getGitConfig + + withUrlOptions :: (U.UrlOptions -> IO a) -> Annex a + withUrlOptions a = liftIO . a =<< getUrlOptions +--- git-annex-6.20170101.orig/NEWS ++++ git-annex-6.20170101/NEWS +@@ -4,6 +4,12 @@ git-annex (6.20170101-1+deb9u2) stretch- + URL schemes by default. You can enable other URL schemes, at your own risk, + using annex.security.allowed-url-schemes. + ++ A related security fix prevents git-annex from downloading content from ++ the web. This can be overridden with the ++ annex.security.allowed-http-addresses setting. ++ (The S3, glacier, and webdav special remotes are still allowed to ++ download from the web.) ++ + The annex.web-download-command configuration has been removed, + use annex.web-options instead. + +--- git-annex-6.20170101.orig/doc/git-annex.mdwn ++++ git-annex-6.20170101/doc/git-annex.mdwn +@@ -1253,6 +1253,21 @@ Here are all the supported configuration + Some special remotes support their own domain-specific URL + schemes; those are not affected by this configuration setting. + ++* `annex.security.allowed-http-addresses` ++ ++ By default, this version of git-annex refuses to download the content of ++ annexed files from the web. Newer versions of git-annex allow downloading ++ from the web, but only when the web server is not on a private IP address. ++ ++ To relax this security check and allow getting annexed files from ++ anywhere on the web, set this to "all". ++ ++ Think very carefully before changing this; there are security ++ implications. Anyone who can get a commit into your git-annex repository ++ could `git annex addurl` an url on a private http server, possibly ++ causing it to be downloaded into your repository and transferred to ++ other remotes, exposing its content. ++ + * `annex.secure-erase-command` + + This can be set to a command that should be run whenever git-annex diff -Nru git-annex-6.20170101/debian/patches/dont-assume-boto-will-remain-secure.patch git-annex-6.20170101/debian/patches/dont-assume-boto-will-remain-secure.patch --- git-annex-6.20170101/debian/patches/dont-assume-boto-will-remain-secure.patch 1970-01-01 01:00:00.000000000 +0100 +++ git-annex-6.20170101/debian/patches/dont-assume-boto-will-remain-secure.patch 2018-06-22 16:42:37.000000000 +0100 @@ -0,0 +1,27 @@ +From: Joey Hess <joeyh@joeyh.name> +Date: Thu, 21 Jun 2018 14:14:56 -0400 +X-Dgit-Generated: 6.20170101-1+deb9u2 acdcbe25bc02919a8c2241155e399f9ef9533c5f +Subject: don't assume boto will remain secure + +On second thought, best to default to being secure even if boto changes +http libraries to one that happens to follow redirects. + +(cherry picked from commit f1b29dbeb4277bf8a7febc795fe52e7b109c6d59) + +--- + +--- git-annex-6.20170101.orig/Remote/Glacier.hs ++++ git-annex-6.20170101/Remote/Glacier.hs +@@ -54,8 +54,10 @@ gen r u c gc = new <$> remoteCost gc ver + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = retrieveCheap this + -- glacier-cli does not follow redirects and does +- -- not support file://, so this is secure. +- , retrievalSecurityPolicy = RetrievalAllKeysSecure ++ -- not support file://, as far as we know, but ++ -- there's no guarantee that will continue to be ++ -- the case, so require verifiable keys. ++ , retrievalSecurityPolicy = RetrievalVerifiableKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy diff -Nru git-annex-6.20170101/debian/patches/enforce-retrievalsecuritypolicy.patch git-annex-6.20170101/debian/patches/enforce-retrievalsecuritypolicy.patch --- git-annex-6.20170101/debian/patches/enforce-retrievalsecuritypolicy.patch 1970-01-01 01:00:00.000000000 +0100 +++ git-annex-6.20170101/debian/patches/enforce-retrievalsecuritypolicy.patch 2018-06-22 16:42:37.000000000 +0100 @@ -0,0 +1,438 @@ +From: Joey Hess <joeyh@joeyh.name> +Date: Thu, 21 Jun 2018 13:34:11 -0400 +X-Dgit-Generated: 6.20170101-1+deb9u2 cd7e9410a9835b63fb06ef5c99b31d91f111c4c3 +Subject: enforce retrievalSecurityPolicy + +Leveraged the existing verification code by making it also check the +retrievalSecurityPolicy. + +Also, prevented getViaTmp from running the download action at all when the +retrievalSecurityPolicy is going to prevent verifying and so storing it. + +Added annex.security.allow-unverified-downloads. A per-remote version +would be nice to have too, but would need more plumbing, so KISS. +(Bill the Cat reference not too over the top I hope. The point is to +make this something the user reads the documentation for before using.) + +A few calls to verifyKeyContent and getViaTmp, that don't +involve downloads from remotes, have RetrievalAllKeysSecure hard-coded. +It was also hard-coded for P2P.Annex and Command.RecvKey, +to match the values of the corresponding remotes. + +A few things use retrieveKeyFile/retrieveKeyFileCheap without going +through getViaTmp. +* Command.Fsck when downloading content from a remote to verify it. + That content does not get into the annex, so this is ok. +* Command.AddUrl when using a remote to download an url; this is new + content being added, so this is ok. + +This commit was sponsored by Fernando Jimenez on Patreon. + +(cherry picked from commit b657242f5d946efae4cc77e8aef95dd2a306cd6b) + +--- + +--- git-annex-6.20170101.orig/Annex/Content.hs ++++ git-annex-6.20170101/Annex/Content.hs +@@ -1,6 +1,6 @@ + {- git-annex file content managing + - +- - Copyright 2010-2015 Joey Hess <id@joeyh.name> ++ - Copyright 2010-2018 Joey Hess <id@joeyh.name> + - + - Licensed under the GNU GPL version 3 or higher. + -} +@@ -15,6 +15,7 @@ module Annex.Content ( + lockContentShared, + lockContentForRemoval, + ContentRemovalLock, ++ RetrievalSecurityPolicy(..), + getViaTmp, + getViaTmp', + checkDiskSpaceToGet, +@@ -74,7 +75,7 @@ import qualified Annex.Content.Direct as + import Annex.ReplaceFile + import Annex.LockPool + import Messages.Progress +-import Types.Remote (unVerified, Verification(..)) ++import Types.Remote (unVerified, Verification(..), RetrievalSecurityPolicy(..)) + import qualified Types.Remote + import qualified Types.Backend + import qualified Backend +@@ -294,19 +295,19 @@ lockContentUsing locker key a = do + {- Runs an action, passing it the temp file to get, + - and if the action succeeds, verifies the file matches + - the key and moves the file into the annex as a key's content. -} +-getViaTmp :: VerifyConfig -> Key -> (FilePath -> Annex (Bool, Verification)) -> Annex Bool +-getViaTmp v key action = checkDiskSpaceToGet key False $ +- getViaTmp' v key action ++getViaTmp :: RetrievalSecurityPolicy -> VerifyConfig -> Key -> (FilePath -> Annex (Bool, Verification)) -> Annex Bool ++getViaTmp rsp v key action = checkDiskSpaceToGet key False $ ++ getViaTmp' rsp v key action + + {- Like getViaTmp, but does not check that there is enough disk space + - for the incoming key. For use when the key content is already on disk + - and not being copied into place. -} +-getViaTmp' :: VerifyConfig -> Key -> (FilePath -> Annex (Bool, Verification)) -> Annex Bool +-getViaTmp' v key action = do ++getViaTmp' :: RetrievalSecurityPolicy -> VerifyConfig -> Key -> (FilePath -> Annex (Bool, Verification)) -> Annex Bool ++getViaTmp' rsp v key action = checkallowed $ do + tmpfile <- prepTmp key + (ok, verification) <- action tmpfile + if ok +- then ifM (verifyKeyContent v verification key tmpfile) ++ then ifM (verifyKeyContent rsp v verification key tmpfile) + ( do + moveAnnex key tmpfile + logStatus key InfoPresent +@@ -319,22 +320,45 @@ getViaTmp' v key action = do + -- On transfer failure, the tmp file is left behind, in case + -- caller wants to resume its transfer + else return False ++ where ++ -- Avoid running the action to get the content when the ++ -- RetrievalSecurityPolicy would cause verification to always fail. ++ checkallowed a = case rsp of ++ RetrievalAllKeysSecure -> a ++ RetrievalVerifiableKeysSecure ++ | isVerifiable key -> a ++ | otherwise -> ifM (annexAllowUnverifiedDownloads <$> Annex.getGitConfig) ++ ( a ++ , warnUnverifiableInsecure key >> return False ++ ) + + {- Verifies that a file is the expected content of a key. ++ - + - Configuration can prevent verification, for either a +- - particular remote or always. ++ - particular remote or always, unless the RetrievalSecurityPolicy ++ - requires verification. + - + - Most keys have a known size, and if so, the file size is checked. + - +- - When the key's backend allows verifying the content (eg via checksum), ++ - When the key's backend allows verifying the content (via checksum), + - it is checked. ++ - ++ - If the RetrievalSecurityPolicy requires verification and the key's ++ - backend doesn't support it, the verification will fail. + -} +-verifyKeyContent :: VerifyConfig -> Verification -> Key -> FilePath -> Annex Bool +-verifyKeyContent _ Verified _ _ = return True +-verifyKeyContent v UnVerified k f = ifM (shouldVerify v) +- ( verifysize <&&> verifycontent +- , return True +- ) ++verifyKeyContent :: RetrievalSecurityPolicy -> VerifyConfig -> Verification -> Key -> FilePath -> Annex Bool ++verifyKeyContent _ _ Verified _ _ = return True ++verifyKeyContent rsp v UnVerified k f = case rsp of ++ RetrievalVerifiableKeysSecure ++ | isVerifiable k -> verifysize <&&> verifycontent ++ | otherwise -> ifM (annexAllowUnverifiedDownloads <$> Annex.getGitConfig) ++ ( verifysize <&&> verifycontent ++ , warnUnverifiableInsecure k >> return False ++ ) ++ _ -> ifM (shouldVerify v) ++ ( verifysize <&&> verifycontent ++ , return True ++ ) + where + verifysize = case keySize k of + Nothing -> return True +@@ -345,6 +369,16 @@ verifyKeyContent v UnVerified k f = ifM + Nothing -> return True + Just verifier -> verifier k f + ++warnUnverifiableInsecure :: Key -> Annex () ++warnUnverifiableInsecure k = warning $ unwords ++ [ "Getting " ++ kv ++ " keys with this remote is not secure;" ++ , "the content cannot be verified to be correct." ++ , "(Use annex.security.allow-unverified-downloads to bypass" ++ , "this safety check.)" ++ ] ++ where ++ kv = keyBackendName k ++ + data VerifyConfig = AlwaysVerify | NoVerify | RemoteVerify Remote | DefaultVerify + + shouldVerify :: VerifyConfig -> Annex Bool +@@ -790,7 +824,7 @@ isUnmodified key f = go =<< geti + go (Just fc) = cheapcheck fc <||> expensivecheck fc + cheapcheck fc = anyM (compareInodeCaches fc) + =<< Database.Keys.getInodeCaches key +- expensivecheck fc = ifM (verifyKeyContent AlwaysVerify UnVerified key f) ++ expensivecheck fc = ifM (verifyKeyContent RetrievalAllKeysSecure AlwaysVerify UnVerified key f) + -- The file could have been modified while it was + -- being verified. Detect that. + ( geti >>= maybe (return False) (compareInodeCaches fc) +--- git-annex-6.20170101.orig/Command/Get.hs ++++ git-annex-6.20170101/Command/Get.hs +@@ -108,7 +108,7 @@ getKey' key afile = dispatch + | Remote.hasKeyCheap r = + either (const False) id <$> Remote.hasKey r key + | otherwise = return True +- docopy r witness = getViaTmp (RemoteVerify r) key $ \dest -> ++ docopy r witness = getViaTmp (Remote.retrievalSecurityPolicy r) (RemoteVerify r) key $ \dest -> + download (Remote.uuid r) key afile forwardRetry + (\p -> do + showAction $ "from " ++ Remote.name r +--- git-annex-6.20170101.orig/Command/Move.hs ++++ git-annex-6.20170101/Command/Move.hs +@@ -179,7 +179,7 @@ fromPerform src move key afile = ifM (in + go = notifyTransfer Download afile $ + download (Remote.uuid src) key afile forwardRetry $ \p -> do + showAction $ "from " ++ Remote.name src +- getViaTmp (RemoteVerify src) key $ \t -> ++ getViaTmp (Remote.retrievalSecurityPolicy src) (RemoteVerify src) key $ \t -> + Remote.retrieveKeyFile src key afile t p + dispatch _ False = stop -- failed + dispatch False True = next $ return True -- copy complete +--- git-annex-6.20170101.orig/Command/ReKey.hs ++++ git-annex-6.20170101/Command/ReKey.hs +@@ -83,7 +83,7 @@ linkKey file oldkey newkey = ifM (isJust + - This avoids hard linking to content linked to an + - unlocked file, which would leave the new key unlocked + - and vulnerable to corruption. -} +- ( getViaTmp' DefaultVerify newkey $ \tmp -> unVerified $ do ++ ( getViaTmp' RetrievalAllKeysSecure DefaultVerify newkey $ \tmp -> unVerified $ do + oldobj <- calcRepo (gitAnnexLocation oldkey) + linkOrCopy' (return True) newkey oldobj tmp Nothing + , do +--- git-annex-6.20170101.orig/Command/RecvKey.hs ++++ git-annex-6.20170101/Command/RecvKey.hs +@@ -13,6 +13,7 @@ import Annex.Action + import Annex + import Utility.Rsync + import Types.Transfer ++import Types.Remote (RetrievalSecurityPolicy(..)) + import Command.SendKey (fieldTransfer) + import qualified CmdLine.GitAnnexShell.Fields as Fields + +@@ -31,7 +32,9 @@ start key = fieldTransfer Download key $ + fromunlocked <- (isJust <$> Fields.getField Fields.unlocked) + <||> (isJust <$> Fields.getField Fields.direct) + let verify = if fromunlocked then AlwaysVerify else DefaultVerify +- ifM (getViaTmp verify key go) ++ -- This matches the retrievalSecurityPolicy of Remote.Git ++ let rsp = RetrievalAllKeysSecure ++ ifM (getViaTmp rsp verify key go) + ( do + -- forcibly quit after receiving one key, + -- and shutdown cleanly +--- git-annex-6.20170101.orig/Command/Reinject.hs ++++ git-annex-6.20170101/Command/Reinject.hs +@@ -44,7 +44,7 @@ startSrcDest (src:dest:[]) + | otherwise = notAnnexed src $ do + showStart "reinject" dest + next $ ifAnnexed dest +- (\key -> perform src key (verifyKeyContent DefaultVerify UnVerified key src)) ++ (\key -> perform src key (verifyKeyContent RetrievalAllKeysSecure DefaultVerify UnVerified key src)) + stop + startSrcDest _ = giveup "specify a src file and a dest file" + +--- git-annex-6.20170101.orig/Command/SetKey.hs ++++ git-annex-6.20170101/Command/SetKey.hs +@@ -33,7 +33,7 @@ perform file key = do + -- the file might be on a different filesystem, so moveFile is used + -- rather than simply calling moveAnnex; disk space is also + -- checked this way. +- ok <- getViaTmp DefaultVerify key $ \dest -> unVerified $ ++ ok <- getViaTmp RetrievalAllKeysSecure DefaultVerify key $ \dest -> unVerified $ + if dest /= file + then liftIO $ catchBoolIO $ do + moveFile file dest +--- git-annex-6.20170101.orig/Command/TestRemote.hs ++++ git-annex-6.20170101/Command/TestRemote.hs +@@ -154,7 +154,7 @@ test st r k = + Just b -> case Backend.verifyKeyContent b of + Nothing -> return True + Just verifier -> verifier k (key2file k) +- get = getViaTmp (RemoteVerify r) k $ \dest -> ++ get = getViaTmp (Remote.retrievalSecurityPolicy r) (RemoteVerify r) k $ \dest -> + Remote.retrieveKeyFile r k Nothing dest nullMeterUpdate + store = Remote.storeKey r k Nothing nullMeterUpdate + remove = Remote.removeKey r k +@@ -168,10 +168,10 @@ testUnavailable st r k = + , check (`notElem` [Right True, Right False]) "checkPresent" $ + Remote.checkPresent r k + , check (== Right False) "retrieveKeyFile" $ +- getViaTmp (RemoteVerify r) k $ \dest -> ++ getViaTmp (Remote.retrievalSecurityPolicy r) (RemoteVerify r) k $ \dest -> + Remote.retrieveKeyFile r k Nothing dest nullMeterUpdate + , check (== Right False) "retrieveKeyFileCheap" $ +- getViaTmp (RemoteVerify r) k $ \dest -> unVerified $ ++ getViaTmp (Remote.retrievalSecurityPolicy r) (RemoteVerify r) k $ \dest -> unVerified $ + Remote.retrieveKeyFileCheap r k Nothing dest + ] + where +--- git-annex-6.20170101.orig/Command/TransferKey.hs ++++ git-annex-6.20170101/Command/TransferKey.hs +@@ -60,7 +60,7 @@ toPerform key file remote = go Upload fi + fromPerform :: Key -> AssociatedFile -> Remote -> CommandPerform + fromPerform key file remote = go Upload file $ + download (uuid remote) key file forwardRetry $ \p -> +- getViaTmp (RemoteVerify remote) key $ ++ getViaTmp (retrievalSecurityPolicy remote) (RemoteVerify remote) key $ + \t -> Remote.retrieveKeyFile remote key file t p + + go :: Direction -> AssociatedFile -> (NotifyWitness -> Annex Bool) -> CommandPerform +--- git-annex-6.20170101.orig/Command/TransferKeys.hs ++++ git-annex-6.20170101/Command/TransferKeys.hs +@@ -42,7 +42,7 @@ start = do + return ok + | otherwise = notifyTransfer direction file $ + download (Remote.uuid remote) key file forwardRetry $ \p -> +- getViaTmp (RemoteVerify remote) key $ \t -> do ++ getViaTmp (Remote.retrievalSecurityPolicy remote) (RemoteVerify remote) key $ \t -> do + r <- Remote.retrieveKeyFile remote key file t p + -- Make sure we get the current + -- associated files data for the key, +--- git-annex-6.20170101.orig/NEWS ++++ git-annex-6.20170101/NEWS +@@ -1,5 +1,11 @@ + git-annex (6.20170101-1+deb9u2) stretch-security; urgency=high + ++ A security fix has changed git-annex to refuse to download content from ++ some special remotes when the content cannot be verified with a hash check. ++ In particular URL and WORM keys stored on such remotes won't be downloaded. ++ See the documentation of the annex.security.allow-unverified-downloads ++ configuration for how to deal with this if it affects your files. ++ + A security fix has changed git-annex to only support http, https, and ftp + URL schemes by default. You can enable other URL schemes, at your own risk, + using annex.security.allowed-url-schemes. +--- git-annex-6.20170101.orig/P2P/Annex.hs ++++ git-annex-6.20170101/P2P/Annex.hs +@@ -23,6 +23,7 @@ import P2P.Protocol + import P2P.IO + import Logs.Location + import Types.NumCopies ++import Types.Remote (RetrievalSecurityPolicy(..)) + import Utility.Metered + import Utility.Tor + import Annex.UUID +@@ -77,9 +78,12 @@ runLocal runmode runner a = case a of + Right Nothing -> runner (next False) + Left e -> return (Left (show e)) + StoreContent k af o l getb next -> do ++ -- This is the same as the retrievalSecurityPolicy of ++ -- Remote.P2P and Remote.Git. ++ let rsp = RetrievalAllKeysSecure + ok <- flip catchNonAsync (const $ return False) $ + transfer download k af $ \p -> +- getViaTmp AlwaysVerify k $ \tmp -> ++ getViaTmp rsp AlwaysVerify k $ \tmp -> + unVerified $ storefile tmp o l getb p + runner (next ok) + StoreContentTo dest o l getb next -> do +--- git-annex-6.20170101.orig/Remote.hs ++++ git-annex-6.20170101/Remote.hs +@@ -12,6 +12,7 @@ module Remote ( + storeKey, + retrieveKeyFile, + retrieveKeyFileCheap, ++ retrievalSecurityPolicy, + removeKey, + hasKey, + hasKeyCheap, +--- git-annex-6.20170101.orig/Remote/Git.hs ++++ git-annex-6.20170101/Remote/Git.hs +@@ -571,10 +571,11 @@ copyToRemote' r key file meterupdate + ensureInitialized + copier <- mkCopier hardlink params + let verify = Annex.Content.RemoteVerify r ++ let rsp = RetrievalAllKeysSecure + runTransfer (Transfer Download u key) file forwardRetry $ \p -> + let p' = combineMeterUpdate meterupdate p + in Annex.Content.saveState True `after` +- Annex.Content.getViaTmp verify key ++ Annex.Content.getViaTmp rsp verify key + (\dest -> copier object dest p' (liftIO checksuccessio)) + ) + +--- git-annex-6.20170101.orig/Types/GitConfig.hs ++++ git-annex-6.20170101/Types/GitConfig.hs +@@ -74,6 +74,7 @@ data GitConfig = GitConfig + , annexAddUnlocked :: Bool + , annexAllowedUrlSchemes :: S.Set Scheme + , annexAllowedHttpAddresses :: String ++ , annexAllowUnverifiedDownloads :: Bool + , coreSymlinks :: Bool + , coreSharedRepository :: SharedRepository + , gcryptId :: Maybe String +@@ -128,6 +129,8 @@ extractGitConfig r = GitConfig + getmaybe (annex "security.allowed-url-schemes") + , annexAllowedHttpAddresses = fromMaybe "" $ + getmaybe (annex "security.allowed-http-addresses") ++ , annexAllowUnverifiedDownloads = (== Just "ACKTHPPT") $ ++ getmaybe (annex "security.allow-unverified-downloads") + , coreSymlinks = getbool "core.symlinks" True + , coreSharedRepository = getSharedRepository r + , gcryptId = getmaybe "core.gcrypt-id" +--- git-annex-6.20170101.orig/Types/Key.hs ++++ git-annex-6.20170101/Types/Key.hs +@@ -15,6 +15,7 @@ module Types.Key ( + chunkKeyOffset, + isChunkKey, + isKeyPrefix, ++ isVerifiable, + + prop_isomorphic_key_encode, + prop_isomorphic_key_decode +--- git-annex-6.20170101.orig/doc/git-annex.mdwn ++++ git-annex-6.20170101/doc/git-annex.mdwn +@@ -1126,6 +1126,10 @@ Here are all the supported configuration + from remotes. If you trust a remote and don't want the overhead + of these checksums, you can set this to `false`. + ++ Note that even when this is set to `false`, git-annex does verification ++ in some edge cases, where it's likely the case than an ++ object was downloaded incorrectly, or when needed for security. ++ + * `remote.<name>.annexUrl` + + Can be used to specify a different url than the regular `remote.<name>.url` +@@ -1268,6 +1272,43 @@ Here are all the supported configuration + causing it to be downloaded into your repository and transferred to + other remotes, exposing its content. + ++* `annex.security.allow-unverified-downloads`, ++ ++ For security reasons, git-annex refuses to download content from ++ most special remotes when it cannot check a hash to verify ++ that the correct content was downloaded. This particularly impacts ++ downloading the content of URL or WORM keys, which lack hashes. ++ ++ The best way to avoid problems due to this is to migrate files ++ away from such keys, before their content reaches a special remote. ++ See [[git-annex-migrate]](1). ++ ++ When the content is only available from a special remote, you can ++ use this configuration to force git-annex to download it. ++ But you do so at your own risk, and it's very important you read and ++ understand the information below first! ++ ++ Downloading unverified content from encrypted special remotes is ++ prevented, because the special remote could send some other encrypted ++ content than what you expect, causing git-annex to decrypt data that you ++ never checked into git-annex, and risking exposing the decrypted ++ data to any non-encrypted remotes you send content to. ++ ++ Downloading unverified content from (non-encrypted) ++ external special remotes is prevented, because they could follow ++ http redirects to web servers on localhost or on a private network, ++ or in some cases to a file:/// url. ++ ++ If you decide to bypass this security check, the best thing to do is ++ to only set it temporarily while running the command that gets the file. ++ The value to set the config to is "ACKTHPPT". ++ For example: ++ ++ git -c annex.security.allow-unverified-downloads=ACKTHPPT annex get myfile ++ ++ It would be a good idea to check that it downloaded the file you expected, ++ too. ++ + * `annex.secure-erase-command` + + This can be set to a command that should be run whenever git-annex diff -Nru git-annex-6.20170101/debian/patches/limit-url-downloads-to-whitelisted-schem.patch git-annex-6.20170101/debian/patches/limit-url-downloads-to-whitelisted-schem.patch --- git-annex-6.20170101/debian/patches/limit-url-downloads-to-whitelisted-schem.patch 1970-01-01 01:00:00.000000000 +0100 +++ git-annex-6.20170101/debian/patches/limit-url-downloads-to-whitelisted-schem.patch 2018-06-22 16:42:37.000000000 +0100 @@ -0,0 +1,331 @@ +From: Joey Hess <id@joeyh.name> +Date: Mon, 18 Jun 2018 15:38:25 -0400 +X-Dgit-Generated: 6.20170101-1+deb9u2 3a4e7ae8749102cf318a41f32598eb40ec22e92e +Subject: limit url downloads to whitelisted schemes + +backported from 28720c795ff57a55b48e56d15f9b6bcb977f48d9 + +Security fix! Allowing any schemes, particularly file: and +possibly others like scp: allowed file exfiltration by anyone who had +write access to the git repository, since they could add an annexed file +using such an url, or using an url that redirected to such an url, +and wait for the victim to get it into their repository and send them a copy. + +* Added annex.security.allowed-url-schemes setting, which defaults + to only allowing http and https URLs. Note especially that file:/ + is no longer enabled by default. + +* Removed annex.web-download-command, since its interface does not allow + supporting annex.security.allowed-url-schemes across redirects. + If you used this setting, you may want to instead use annex.web-options + to pass options to curl. + +With annex.web-download-command removed, nearly all url accesses in +git-annex are made via Utility.Url via http-client or curl. http-client +only supports http and https, so no problem there. +(Disabling one and not the other is not implemented.) + +Used curl --proto to limit the allowed url schemes. +wget only supports http https ftp, so does not need any limiting. + +Note that this will cause git annex fsck --from web to mark files using +a disallowed url scheme as not being present in the web. That seems +acceptable; fsck --from web also does that when a web server is not available. + +quvi was not changed; it only supports a hardcoded set of urls, which +are http, not file urls. + +This does not address any external special remotes that might download +an url themselves. Current thinking is all external special remotes will +need to be audited for this problem, although many of them will use +http libraries that only support http and not curl's menagarie. + +The related problem of accessing private localhost and LAN urls is not +addressed by this commit. + +This commit was sponsored by Brett Eisenberg on Patreon. + +--- + +--- git-annex-6.20170101.orig/Annex/Content.hs ++++ git-annex-6.20170101/Annex/Content.hs +@@ -902,24 +902,15 @@ saveState nocommit = doSideAction $ do + + {- Downloads content from any of a list of urls. -} + downloadUrl :: Key -> MeterUpdate -> [Url.URLString] -> FilePath -> Annex Bool +-downloadUrl k p urls file = meteredFile file (Just p) k $ +- go =<< annexWebDownloadCommand <$> Annex.getGitConfig ++downloadUrl k p urls file = meteredFile file (Just p) k go + where +- go Nothing = do ++ go = do + a <- ifM commandProgressDisabled + ( return Url.downloadQuiet + , return Url.download + ) + Url.withUrlOptions $ \uo -> + anyM (\u -> a u file uo) urls +- go (Just basecmd) = anyM (downloadcmd basecmd) urls +- downloadcmd basecmd url = +- progressCommand "sh" [Param "-c", Param $ gencmd url basecmd] +- <&&> liftIO (doesFileExist file) +- gencmd url = massReplace +- [ ("%file", shellEscape file) +- , ("%url", shellEscape url) +- ] + + {- Copies a key's content, when present, to a temp file. + - This is used to speed up some rsyncs. -} +--- git-annex-6.20170101.orig/Annex/Url.hs ++++ git-annex-6.20170101/Annex/Url.hs +@@ -30,6 +30,7 @@ getUrlOptions = mkUrlOptions + <$> getUserAgent + <*> headers + <*> options ++ <*> (annexAllowedUrlSchemes <$> Annex.getGitConfig) + where + headers = do + v <- annexHttpHeadersCommand <$> Annex.getGitConfig +--- git-annex-6.20170101.orig/NEWS ++++ git-annex-6.20170101/NEWS +@@ -1,3 +1,14 @@ ++git-annex (6.20170101-1+deb9u2) stretch-security; urgency=high ++ ++ A security fix has changed git-annex to only support http, https, and ftp ++ URL schemes by default. You can enable other URL schemes, at your own risk, ++ using annex.security.allowed-url-schemes. ++ ++ The annex.web-download-command configuration has been removed, ++ use annex.web-options instead. ++ ++ -- Joey Hess <id@joeyh.name> Fri, 15 Jun 2018 17:54:23 -0400 ++ + git-annex (6.20170101) unstable; urgency=medium + + XMPP support has been removed from the assistant in this release. +--- git-annex-6.20170101.orig/Types/GitConfig.hs ++++ git-annex-6.20170101/Types/GitConfig.hs +@@ -28,6 +28,9 @@ import Types.RefSpec + import Utility.HumanTime + import Utility.Gpg (GpgCmd, mkGpgCmd) + import Utility.ThreadScheduler (Seconds(..)) ++import Utility.Url (Scheme, mkScheme) ++ ++import qualified Data.Set as S + + {- Main git-annex settings. Each setting corresponds to a git-config key + - such as annex.foo -} +@@ -51,7 +54,6 @@ data GitConfig = GitConfig + , annexWebOptions :: [String] + , annexQuviOptions :: [String] + , annexAriaTorrentOptions :: [String] +- , annexWebDownloadCommand :: Maybe String + , annexCrippledFileSystem :: Bool + , annexLargeFiles :: Maybe String + , annexAddSmallFiles :: Bool +@@ -70,6 +72,8 @@ data GitConfig = GitConfig + , annexPidLock :: Bool + , annexPidLockTimeout :: Seconds + , annexAddUnlocked :: Bool ++ , annexAllowedUrlSchemes :: S.Set Scheme ++ , annexAllowedHttpAddresses :: String + , coreSymlinks :: Bool + , coreSharedRepository :: SharedRepository + , gcryptId :: Maybe String +@@ -98,7 +102,6 @@ extractGitConfig r = GitConfig + , annexWebOptions = getwords (annex "web-options") + , annexQuviOptions = getwords (annex "quvi-options") + , annexAriaTorrentOptions = getwords (annex "aria-torrent-options") +- , annexWebDownloadCommand = getmaybe (annex "web-download-command") + , annexCrippledFileSystem = getbool (annex "crippledfilesystem") False + , annexLargeFiles = getmaybe (annex "largefiles") + , annexAddSmallFiles = getbool (annex "addsmallfiles") True +@@ -120,6 +123,11 @@ extractGitConfig r = GitConfig + , annexPidLockTimeout = Seconds $ fromMaybe 300 $ + getmayberead (annex "pidlocktimeout") + , annexAddUnlocked = getbool (annex "addunlocked") False ++ , annexAllowedUrlSchemes = S.fromList $ map mkScheme $ ++ maybe ["http", "https", "ftp"] words $ ++ getmaybe (annex "security.allowed-url-schemes") ++ , annexAllowedHttpAddresses = fromMaybe "" $ ++ getmaybe (annex "security.allowed-http-addresses") + , coreSymlinks = getbool "core.symlinks" True + , coreSharedRepository = getSharedRepository r + , gcryptId = getmaybe "core.gcrypt-id" +--- git-annex-6.20170101.orig/Utility/Url.hs ++++ git-annex-6.20170101/Utility/Url.hs +@@ -15,6 +15,9 @@ module Utility.Url ( + managerSettings, + URLString, + UserAgent, ++ Scheme, ++ mkScheme, ++ allowedScheme, + UrlOptions, + mkUrlOptions, + check, +@@ -38,6 +41,7 @@ import Network.HTTP.Types + import qualified Data.CaseInsensitive as CI + import qualified Data.ByteString as B + import qualified Data.ByteString.UTF8 as B8 ++import qualified Data.Set as S + import Control.Monad.Trans.Resource + import Network.HTTP.Conduit hiding (closeManager) + +@@ -63,6 +67,15 @@ type Headers = [String] + + type UserAgent = String + ++newtype Scheme = Scheme (CI.CI String) ++ deriving (Eq, Ord) ++ ++mkScheme :: String -> Scheme ++mkScheme = Scheme . CI.mk ++ ++fromScheme :: Scheme -> String ++fromScheme (Scheme s) = CI.original s ++ + data UrlOptions = UrlOptions + { userAgent :: Maybe UserAgent + , reqHeaders :: Headers +@@ -72,15 +85,17 @@ data UrlOptions = UrlOptions + #else + , applyRequest :: forall m. Request m -> Request m + #endif ++ , allowedSchemes :: S.Set Scheme + } + + instance Default UrlOptions + where +- def = UrlOptions Nothing [] [] id ++ def = UrlOptions Nothing [] [] id ++ (S.fromList $ map mkScheme ["http", "https", "ftp"]) + +-mkUrlOptions :: Maybe UserAgent -> Headers -> [CommandParam] -> UrlOptions +-mkUrlOptions defuseragent reqheaders reqparams = +- UrlOptions useragent reqheaders reqparams applyrequest ++mkUrlOptions :: Maybe UserAgent -> Headers -> [CommandParam] -> S.Set Scheme -> UrlOptions ++mkUrlOptions defuseragent reqheaders reqparams allowedschemes = ++ UrlOptions useragent reqheaders reqparams applyrequest allowedschemes + where + applyrequest = \r -> r { requestHeaders = requestHeaders r ++ addedheaders } + addedheaders = uaheader ++ otherheaders +@@ -104,6 +119,28 @@ addUserAgent uo ps = case userAgent uo o + -- --user-agent works for both wget and curl commands + Just ua -> ps ++ [Param "--user-agent", Param ua] + ++checkPolicy :: UrlOptions -> URI -> a -> IO a -> IO a ++checkPolicy uo u onerr a ++ | allowedScheme uo u = a ++ | otherwise = do ++ hPutStrLn stderr $ ++ "Configuration does not allow accessing " ++ show u ++ hFlush stderr ++ return onerr ++ ++curlSchemeParams :: UrlOptions -> [CommandParam] ++curlSchemeParams uo = ++ [ Param "--proto" ++ , Param $ intercalate "," ("-all" : schemelist) ++ ] ++ where ++ schemelist = map fromScheme $ S.toList $ allowedSchemes uo ++ ++allowedScheme :: UrlOptions -> URI -> Bool ++allowedScheme uo u = uscheme `S.member` allowedSchemes uo ++ where ++ uscheme = mkScheme $ takeWhile (/=':') (uriScheme u) ++ + {- Checks that an url exists and could be successfully downloaded, + - also checking that its size, if available, matches a specified size. -} + checkBoth :: URLString -> Maybe Integer -> UrlOptions -> IO Bool +@@ -137,7 +174,7 @@ assumeUrlExists = UrlInfo True Nothing N + - also returning its size and suggested filename if available. -} + getUrlInfo :: URLString -> UrlOptions -> IO UrlInfo + getUrlInfo url uo = case parseURIRelaxed url of +- Just u -> case parseurlconduit (show u) of ++ Just u -> checkPolicy uo u dne' $ case parseurlconduit (show u) of + Just req -> catchJust + -- When http redirects to a protocol which + -- conduit does not support, it will throw +@@ -161,7 +198,8 @@ getUrlInfo url uo = case parseURIRelaxed + | otherwise -> dne + Nothing -> dne + where +- dne = return $ UrlInfo False Nothing Nothing ++ dne = return dne' ++ dne' = UrlInfo False Nothing Nothing + found sz f = return $ UrlInfo True sz f + + curlparams = addUserAgent uo $ +@@ -169,7 +207,7 @@ getUrlInfo url uo = case parseURIRelaxed + , Param "--head" + , Param "-L", Param url + , Param "-w", Param "%{http_code}" +- ] ++ concatMap (\h -> [Param "-H", Param h]) (reqHeaders uo) ++ (reqParams uo) ++ ] ++ concatMap (\h -> [Param "-H", Param h]) (reqHeaders uo) ++ (reqParams uo) ++ curlSchemeParams uo + + extractlencurl s = case lastMaybe $ filter ("Content-Length:" `isPrefixOf`) (lines s) of + Just l -> case lastMaybe $ words l of +@@ -263,9 +301,10 @@ downloadQuiet = download' True + download' :: Bool -> URLString -> FilePath -> UrlOptions -> IO Bool + download' quiet url file uo = do + case parseURIRelaxed url of +- Just u +- | uriScheme u == "file:" -> curl +- | otherwise -> ifM (inPath "wget") (wget , curl) ++ Just u -> checkPolicy uo u False $ ++ if uriScheme u == "file:" ++ then curl ++ else ifM (inPath "wget") (wget , curl) + _ -> return False + where + headerparams = map (\h -> Param $ "--header=" ++ h) (reqHeaders uo) +@@ -276,6 +315,10 @@ download' quiet url file uo = do + - + - When the wget version is new enough, pass options for + - a less cluttered download display. ++ - ++ - wget only supports https, http, and ftp, not file, which are ++ - all always allowed, so its url schemes do not need to be ++ - limited. + -} + #ifndef __ANDROID__ + wgetparams = concat +@@ -296,7 +339,7 @@ download' quiet url file uo = do + -- curl does not create destination file + -- if the url happens to be empty, so pre-create. + writeFile file "" +- go "curl" $ headerparams ++ quietopt "-s" ++ ++ go "curl" $ headerparams ++ quietopt "-s" ++ curlSchemeParams uo ++ + [Param "-f", Param "-L", Param "-C", Param "-", Param "-#", Param "-o"] + + {- Run wget in a temp directory because it has been buggy +--- git-annex-6.20170101.orig/doc/git-annex.mdwn ++++ git-annex-6.20170101/doc/git-annex.mdwn +@@ -1238,13 +1238,20 @@ Here are all the supported configuration + If set, the command is run and each line of its output is used as a HTTP + header. This overrides annex.http-headers. + +-* `annex.web-download-command` ++* `annex.security.allowed-url-schemes` + +- Use to specify a command to run to download a file from the web. +- (The default is to use wget or curl.) ++ List of URL schemes that git-annex is allowed to download content from. ++ The default is "http https ftp". + +- In the command line, %url is replaced with the url to download, +- and %file is replaced with the file that it should be saved to. ++ Think very carefully before changing this; there are security ++ implications. For example, if it's changed to allow "file" URLs, then ++ anyone who can get a commit into your git-annex repository could ++ `git-annex addurl` a pointer to a private file located outside that ++ repository, possibly causing it to be copied into your repository ++ and transferred on to other remotes, exposing its content. ++ ++ Some special remotes support their own domain-specific URL ++ schemes; those are not affected by this configuration setting. + + * `annex.secure-erase-command` + diff -Nru git-annex-6.20170101/debian/patches/series git-annex-6.20170101/debian/patches/series --- git-annex-6.20170101/debian/patches/series 2017-10-26 15:28:29.000000000 +0100 +++ git-annex-6.20170101/debian/patches/series 2018-06-22 16:42:37.000000000 +0100 @@ -1 +1,8 @@ CVE-2017-12976.patch +limit-url-downloads-to-whitelisted-schem.patch +block-url-downloads-by-default.patch +update-test-suite-for-security-fix.patch +add-retrievalsecuritypolicy.patch +enforce-retrievalsecuritypolicy.patch +dont-assume-boto-will-remain-secure.patch +set-ddar-to-retrievalallkeyssecure.patch diff -Nru git-annex-6.20170101/debian/patches/set-ddar-to-retrievalallkeyssecure.patch git-annex-6.20170101/debian/patches/set-ddar-to-retrievalallkeyssecure.patch --- git-annex-6.20170101/debian/patches/set-ddar-to-retrievalallkeyssecure.patch 1970-01-01 01:00:00.000000000 +0100 +++ git-annex-6.20170101/debian/patches/set-ddar-to-retrievalallkeyssecure.patch 2018-06-22 16:42:37.000000000 +0100 @@ -0,0 +1,25 @@ +From: Joey Hess <joeyh@joeyh.name> +Date: Thu, 21 Jun 2018 16:38:47 -0400 +X-Dgit-Generated: 6.20170101-1+deb9u2 fd8086fe4a9c4bd64e95fd9152b1800b1c0f6b89 +Subject: set ddar to RetrievalAllKeysSecure + +Based on information from Robie Basak. + +(cherry picked from commit 05ecee0db43919b3cfd8d2e2772022e16a804a04) + +--- + +--- git-annex-6.20170101.orig/Remote/Ddar.hs ++++ git-annex-6.20170101/Remote/Ddar.hs +@@ -56,8 +56,9 @@ gen r u c gc = do + , storeKey = storeKeyDummy + , retrieveKeyFile = retreiveKeyFileDummy + , retrieveKeyFileCheap = retrieveCheap +- -- Unsure about this, safe default until Robie answers. +- , retrievalSecurityPolicy = RetrievalVerifiableKeysSecure ++ -- ddar communicates over ssh, not subject to http redirect ++ -- type attacks ++ , retrievalSecurityPolicy = RetrievalAllKeysSecure + , removeKey = removeKeyDummy + , lockContent = Nothing + , checkPresent = checkPresentDummy diff -Nru git-annex-6.20170101/debian/patches/update-test-suite-for-security-fix.patch git-annex-6.20170101/debian/patches/update-test-suite-for-security-fix.patch --- git-annex-6.20170101/debian/patches/update-test-suite-for-security-fix.patch 1970-01-01 01:00:00.000000000 +0100 +++ git-annex-6.20170101/debian/patches/update-test-suite-for-security-fix.patch 2018-06-22 16:42:37.000000000 +0100 @@ -0,0 +1,31 @@ +From: Joey Hess <id@joeyh.name> +Date: Mon, 18 Jun 2018 18:18:06 -0400 +X-Dgit-Generated: 6.20170101-1+deb9u2 9208ae8c4e1c9c6b1fd4cb004b4cda568ffd9406 +Subject: update test suite for security fix + + +--- + +--- git-annex-6.20170101.orig/Test.hs ++++ git-annex-6.20170101/Test.hs +@@ -1688,12 +1688,18 @@ test_add_subdirs = intmpclonerepo $ do + test_addurl :: Assertion + test_addurl = intmpclonerepo $ do + -- file:// only; this test suite should not hit the network ++ let filecmd c ps = git_annex c ++ ( "-cannex.security.allowed-url-schemes=file" ++ : "-cannex.security.allowed-http-addresses=all" ++ : ps ++ ) + f <- absPath "myurl" + let url = replace "\\" "/" ("file:///" ++ dropDrive f) + writeFile f "foo" +- git_annex "addurl" [url] @? ("addurl failed on " ++ url) ++ not <$> git_annex "addurl" [url] @? "addurl failed to fail on file url" ++ filecmd "addurl" [url] @? ("addurl failed on " ++ url) + let dest = "addurlurldest" +- git_annex "addurl" ["--file", dest, url] @? ("addurl failed on " ++ url ++ " with --file") ++ filecmd "addurl" ["--file", dest, url] @? ("addurl failed on " ++ url ++ " with --file") + doesFileExist dest @? (dest ++ " missing after addurl --file") + + -- This is equivilant to running git-annex, but it's all run in-processAttachment: signature.asc
Description: PGP signature
--- End Message ---
--- Begin Message ---
- To: 873330-done@bugs.debian.org, 880037-done@bugs.debian.org, 882197-done@bugs.debian.org, 885333-done@bugs.debian.org, 885617-done@bugs.debian.org, 887160-done@bugs.debian.org, 888766-done@bugs.debian.org, 888909-done@bugs.debian.org, 889940-done@bugs.debian.org, 890791-done@bugs.debian.org, 891415-done@bugs.debian.org, 891466-done@bugs.debian.org, 891470-done@bugs.debian.org, 891559-done@bugs.debian.org, 891563-done@bugs.debian.org, 891576-done@bugs.debian.org, 891580-done@bugs.debian.org, 891587-done@bugs.debian.org, 891655-done@bugs.debian.org, 891658-done@bugs.debian.org, 891668-done@bugs.debian.org, 891772-done@bugs.debian.org, 891784-done@bugs.debian.org, 891788-done@bugs.debian.org, 891791-done@bugs.debian.org, 891793-done@bugs.debian.org, 892278-done@bugs.debian.org, 892734-done@bugs.debian.org, 892836-done@bugs.debian.org, 892857-done@bugs.debian.org, 892940-done@bugs.debian.org, 893043-done@bugs.debian.org, 893278-done@bugs.debian.org, 893803-done@bugs.debian.org, 893866-done@bugs.debian.org, 893971-done@bugs.debian.org, 894713-done@bugs.debian.org, 894960-done@bugs.debian.org, 894971-done@bugs.debian.org, 895596-done@bugs.debian.org, 895766-done@bugs.debian.org, 895846-done@bugs.debian.org, 895888-done@bugs.debian.org, 895936-done@bugs.debian.org, 896170-done@bugs.debian.org, 896689-done@bugs.debian.org, 896918-done@bugs.debian.org, 896940-done@bugs.debian.org, 896945-done@bugs.debian.org, 897188-done@bugs.debian.org, 897368-done@bugs.debian.org, 897912-done@bugs.debian.org, 897923-done@bugs.debian.org, 899006-done@bugs.debian.org, 899014-done@bugs.debian.org, 900920-done@bugs.debian.org, 901033-done@bugs.debian.org, 901089-done@bugs.debian.org, 901192-done@bugs.debian.org, 901288-done@bugs.debian.org, 901331-done@bugs.debian.org, 901355-done@bugs.debian.org, 901426-done@bugs.debian.org, 901476-done@bugs.debian.org, 901639-done@bugs.debian.org, 901811-done@bugs.debian.org, 902042-done@bugs.debian.org, 902282-done@bugs.debian.org, 902283-done@bugs.debian.org, 902294-done@bugs.debian.org, 902319-done@bugs.debian.org, 902325-done@bugs.debian.org, 902326-done@bugs.debian.org, 902688-done@bugs.debian.org, 902758-done@bugs.debian.org, 902832-done@bugs.debian.org, 903005-done@bugs.debian.org, 903015-done@bugs.debian.org, 903037-done@bugs.debian.org, 903052-done@bugs.debian.org, 903088-done@bugs.debian.org, 903199-done@bugs.debian.org, 903203-done@bugs.debian.org, 903216-done@bugs.debian.org, 859731-done@bugs.debian.org, 899189-done@bugs.debian.org, 901744-done@bugs.debian.org
- Subject: Closing bugs for updates included in 9.5
- From: "Adam D. Barratt" <adam@adam-barratt.org.uk>
- Date: Sat, 14 Jul 2018 11:21:20 +0100
- Message-id: <1531563680.2095.30.camel@adam-barratt.org.uk>
Version: 9.5 Hi, The update referenced by each of these bugs was included in this morning's stretch point release. Regards, Adam
--- End Message ---