Re: Why is package X not in testing yet?
> From: Sven Bartscher <sven.bartscher@weltraumschlangen.de>
Hi again Sven,
>
> Greetings,
>
> If we have a package, which doesn't migrate to testing, we usually
> check the "Why does package X not in testing yet?" page or the PTS.
> Usually they do a great job in telling us why our package doesn't
> migrate.
>
> But sometimes you have packages, which have complicated dependencies,
> that don't make it easy to tell why our package doesn't migrate.
> Usually the PTS and "Why is package X not in testing yet?" fail at
> those packages and don't give any useful explanation. For example look
> at the page for haskell-hgettext[1].
>
> Those packages usually have to go into testing together with a few other
> packages. If it's getting worse your package needs to go into testing
> with a lot of other packages (usually if your package is part of a
> bigger transition).
>
> If your package is part of a transition you might have luck and the
> transition page can tell you what the problem is. But sometimes not
> even that pages help.
>
> For that reason I made a tool that takes a package name and tries to
> find out why your package doesn't migrate to testing.
>
> When run, it gathers all packages that block our given package X.
> Then it fetches all excuses for these packages and throws all of them
> away, except those that are identified as interesting. These types of
> excuses are identified as interesting:
>
> - out of date on <arch>
> - <pkg> has new bugs
> - Too young
>
> This takes a long time (for me 3 Minutes) depending on your internet
> connection.
>
> This is mostly useful for haskell packages, as they have very close
> dependencies. However, it might be interesting in any other transition.
>
> The source code is attached and can be found in the tools repository[2]
> of the Haskell Group.
> In order to compile it you need the following packages:
> - ghc
> - libghc-regex-pcre-dev
> - libpcre++-dev (This should be a dependency of libghc-regex-pcre-dev
> but it isn't due to a bug)
>
> Compile it with:
> ghc --make reasons.hs
>
> To run it you need the following packages:
> - devscripts
> - wget
> - ca-certificates
> - locales (you should use an UTF-8 encoding, otherwise its guaranteed
> you will have problems.)
> Also you must have enabled the source URIs in you sources.list.
>
> There are still some rough edges. The most notable ones are:
> - Most errors that can happen aren't catched. So an haskell exception
> will be thrown, which gives not very much information of the problem.
> - The excuses are fetched with grep-excuses. So the excuses file is
> downloaded over ad over again. There is already a bug with a patch
> filed against grep-excuses to fix this.
> - "out of date" excuses are all considered interesting, even though it
> would be better to only include those that aren't in state B-D
> unistallable.
>
> A bit more detail on the workflow of this tool is described in this[3]
> post.
>
I would just ask one (I hope little) feature, I'm not an haskell guy, so I find rather difficult to submit a patch :)
I would like to see also this
outputUrl = "people.canonical.com/~ubuntu-archive/proposed-migration/update_output.txt"
(maybe with a deletion of ~/.reasons/update_output.txt)
and replace "doesn't" with "doesn't " ;)
maybe something like renaming update_output.txt.{ubuntu,debian} and passing something like -d {debian,ubuntu} by command line will make this program universally used and easily extensible to other debian based distros.
thanks again for the nice and useful work you did!
cheers,
Gianfranco
>
> [1]:
> https://release.debian.org/migration/testing.pl?package=3Dhaskell-hgettext
> [2]: http://anonscm.debian.org/cgit/pkg-haskell/tools.git/
> [3]: https://lists.debian.org/debian-haskell/2014/08/msg00027.html
>
> --MP_/ptLm4kgDdV4o5zwF+5AjRiy
> Content-Type: text/x-haskell
> Content-Transfer-Encoding: quoted-printable
> Content-Disposition: attachment; filename=reasons.hs
>
> import Text.Regex.PCRE
> import System.Environment
> import System.Exit
> import System.Process
> import System.IO
> import Data.Maybe
> import Data.List
> import Data.Char
> import qualified Data.Set as S
> import Control.Exception
> import System.IO.Error
> import System.Directory
> import Debug.Trace
>
> data Excuses =3D Excuses String [String]
>
> isEmpty :: Excuses -> Bool
> isEmpty (Excuses _ []) =3D False
> isEmpty (Excuses _ _) =3D True
>
> excuses2String :: Excuses -> String
> excuses2String (Excuses pkg excuses) =3D unlines $ (pkg ++ ":"):(map
> (" =
> " ++) excuses)
>
> main =3D do
> package <- getArgs >>=3D parse
> output <- fmap lines acquireBritneyOut
> let bins =3D getBinBlockers output package
> result <- try (fmap nub $ mapM getSrcPackage bins) :: IO (Either ErrorCal=
> l [String])
> srcBlockers <- case result of
> Left e -> putStrLn packageNotFoundMsg >> exitFailure
> Right pkgs -> return pkgs
> excuses <- mapM getExcuse srcBlockers
> additionalExcuses <- getAdditionalExcuses srcBlockers excuses
> let filteredExcuses =3D filterExcuses isInteresting $ excuses ++ addition=
> alExcuses
> mapM_ putStrLn $ map excuses2String filteredExcuses
>
> acquireBritneyOut :: IO String
> acquireBritneyOut =3D do
> cachePath <- chooseCachePath
> case cachePath of=20
> Nothing -> readProcess "/usr/bin/wget" ["-q",
> "-O", "-", outputUrl] ""
> Just path -> do
> createDirectoryIfMissing False path
> setCurrentDirectory path
> readProcess "/usr/bin/wget" ["-q", "-N",
> outputUrl] ""
> readFile "update_output.txt"
>
> chooseCachePath :: IO (Maybe String)
> chooseCachePath =3D do
> result <- tryJust shouldCatch $ getAppUserDataDirectory "reasons"
> hasHome <- getHomeDirectory >>=3D doesDirectoryExist
> return $ case result of
> Right dir -> if hasHome
> then Just dir
> else Nothing
> Left _ -> Nothing
> where shouldCatch e =3D if isDoesNotExistError e
> then Just e
> else Nothing
>
> outputUrl :: String
> outputUrl =3D "release.debian.org/britney/update_output.txt"
>
> parse :: [String] -> IO String
> parse [package] =3D return package
> parse _ =3D printUsage >> exitFailure
>
> printUsage :: IO ()
> printUsage =3D do
> progName <- getProgName
> putStrLn $ "Usage: " ++ progName ++ " package-name"
>
> packageNotFoundMsg :: String
> packageNotFoundMsg
> =3D "The package you requested was not processed by the
> autohinter.\n\
> \grep-excuses <pkg> should list all reasons why this package
> doesn't\
> \migrate."
>
> filterExcuses :: (String -> Bool) -> [Excuses] -> [Excuses]
> filterExcuses f excuses =3D filter isEmpty $ map filterPkgExcuses excuses
> where filterPkgExcuses (Excuses pkg excuses) =3D Excuses pkg
> $ filter f excuses
>
> isInteresting :: String -> Bool
> isInteresting excuse =3D "out of date on" `isPrefixOf` excuse=20
> || "introduces new bugs" `isInfixOf`
> excuse
> || "Too young" `isPrefixOf` excuse
>
> isInterestingDependency :: [String] -> String -> Bool
> isInterestingDependency pkgs excuse =3D "(not considered)"
> `isSuffixOf` exc=
> use=20
> && (mangleDependency excuse)
> `notElem=
> ` pkgs
>
> mangleDependency :: String -> String
> mangleDependency excuse
> | null dependency =3D ""
> | otherwise =3D tail $ dropWhile (/=3D ' ') dependency
> where dependency =3D excuse =3D~ "(?<=3DDepends: ).*(?=3D
> \\(not consid=
> ered\\))"
>
> -- Takes a list of already fetched excuses and returns the excuses of missi=
> ng dependencies
> getAdditionalExcuses :: [String] -> [Excuses] -> IO [Excuses]
> getAdditionalExcuses _ [] =3D return []
> getAdditionalExcuses pkgs excuses =3D do
> let interestingDepends =3D filterExcuses
> (isInterestingDependency pkgs)
> excuses
> dependencies =3D nub $ map mangleDependency $ flattenExcuses interest=
> ingDepends
> excuses <- mapM getExcuse dependencies
> evenMoreExcuses <- getAdditionalExcuses (pkgs ++ dependencies) excuses
> return $ excuses ++ evenMoreExcuses
>
> flattenExcuses :: [Excuses] -> [String]
> flattenExcuses excuses =3D concat $ map unpackExcuses excuses
>
> unpackExcuses :: Excuses -> [String]
> unpackExcuses (Excuses _ excuses) =3D excuses
>
> maybeTail :: [a] -> Maybe [a]
> maybeTail [] =3D Nothing
> maybeTail (x:xs) =3D Just xs
>
> getExcuse :: String -> IO Excuses
> getExcuse pkg =3D do
> hPutStrLn stderr $ "retrievieng excuses for " ++ pkg
> excuses <- readProcess "/usr/bin/grep-excuses" [pkg] ""
> return $ Excuses pkg $ map (dropWhile isSpace)
> $ fromMaybe [] $ maybeTail $ lines excuses
>
> getSrcPackage :: String -> IO String
> getSrcPackage bin =3D do
> hPutStrLn stderr $ "querying source for " ++ bin
> packageDesc <- readProcess "/usr/bin/apt-cache"
> ["showsrc", bin] ""
> return $ parseDesc packageDesc
>
> parseDesc :: String -> String
> parseDesc desc =3D let ls =3D lines desc
> srcln =3D findSourceLine ls
> in removeFieldPrefix srcln
>
> findSourceLine :: [String] -> String
> findSourceLine (curLine:rest)
> | "Package: " `isPrefixOf` curLine =3D curLine
> | otherwise =3D findSourceLine rest
>
> getBinBlockers :: [String] -> String -> [String]
> getBinBlockers output package =3D let arches =3D getArches package output
> in nub $ map stripComma
> $ concat
> $ map words
> $ map removeFieldPrefix arches
> where stripComma str =3D if last str =
> =3D=3D ','
> then init str
> else str
>
> removeFieldPrefix :: String -> String
> removeFieldPrefix arch =3D drop 2 $ dropWhile (/=3D ':') arch
>
> getArches :: String -> [String] -> [String]
> getArches package output =3D get $ removeStats $ fromJust $ findAutohint pa=
> ckage output
> where get (line:rest)
> | line `matches` " *\\* .*:" =3D line : get rest
> | otherwise =3D []
>
> removeStats :: [String] -> [String]
> removeStats =3D drop 4
>
> findAutohint :: String -> [String] -> Maybe [String]
> findAutohint _ [] =3D Nothing
> findAutohint package (curLine:rest)
> | curLine `matches` ("Trying easy from autohinter.*" ++ package)
> =3D Just rest
> | otherwise =3D findAutohint package rest
>
> matches :: String -> String -> Bool
> str `matches` pattern =3D (not . null) (str =3D~ pattern :: String)
>
> --MP_/ptLm4kgDdV4o5zwF+5AjRiy--
>
> --Sig_/FxDr0M5M=m3X3vo7h1MYZq.
> Content-Type: application/pgp-signature; name=signature.asc
> Content-Disposition: attachment; filename=signature.asc
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
>
> iQIcBAEBCAAGBQJT8iWBAAoJEN9tdsRNaW9rVywP/2lYcEpkAvtgDDuUfNLuucL2
> 1brrS7QTo6Uqi6nV2b/FGwSQtH+cAlfQAwgbRqrhu9Wkb6aBRps1GGuyU3Cqw+XD
> fbcO3ChgnrLnVtcsg3DwCRamA0u3UavV0Ufz6YLzGfMB8o+sU+oTHkWU6iD+2zTQ
> sC7qK/pck2b6N2kioq2eKxk+tsMXI/GA2Z5zWeQztSNTgKIfPQY5YZojeSzIVDXB
> gMrrquvAsyc2OhDJtx4J/N1AmzQTc2saSGDbHB4lTfVJvYj8NnDJq7ovnOjBG5Kn
> XvINhlpnsVP61yPBpS5qlKT/UNqMa0TpgbkL8pAIzK84hqBwcmro1QWj9KM2OdSW
> EMwuGjFZemh8aCPMm+0cZh+CL7B1R6HZ+gJ+oY5RD3H59HOQ2hDnTn36wLeagBfL
> +1IqxVRdqnRqa0PMXNDr3eAd1ljIF66G4jUIWEwHUpr94eO59dXRAxNFWr0oZQ1W
> PuAlqDOE7QgdHjdStMU0V1DsyOLypTlOKEWUdu1Vxlf52AZl+NCs+ucbbzngfSM9
> VC1HmKA6Vei6BGeoXzjX4p6ISoWEC1HNnYCwb84ppyGmwxkY541jmcgJPdeGeJJy
> cAVpqeLMhqzDC7Jj3Gu3nBw2/p/5ScZpyVF8V9ak3sGEUVFeC07gUYRlRVoNrOAW
> CBkgtCewBRUw5ZC7ErE5
> =sZEq
> -----END PGP SIGNATURE-----
>
> --Sig_/FxDr0M5M=m3X3vo7h1MYZq.--
>
Reply to: