Re: [Pkg-ruby-extras-maintainers]
On 27/11/05 at 23:05 +0000, Esteban Manchado Velázquez wrote:
> > > > > About unit tests: it would be great to have a common architecture to
> > > > > deal with our unit tests. This way, one could run a script on a regular
> > > > > basis to check that all his installed packages still work correctly.
> > > >
> > > > I'm not sure I like this. I would prefer using the Ubuntu proposal (or
> > > > something similar) for package testing, and somehow plug the own library unit
> > > > tests into the distribution package framework. After all, the package
> > > > maintainer is basically who needs/is interested in package testing...
> > >
> > > Yes, the unit tests need to be ran while packaging. If unit tests are
> > > available for a library then this is great for the "package testing
> > > before upload". I don't think a user/developer is going to rerun the
> > > tests to find the same results as the maintainer has.
> >
> > This requires manual copying of the test scripts to the debian/tests dir
> > by the maintainer. This is error-prone, etc.
>
> Not necessarily. In fact, I would try to solve it in a generic way, with
> the CDBS or whatever.
>
> > The Ubuntu proposal is flexible enough to allow to store tests in a -dev
> > package. Also, another reason for not storing tests in the source package is
> > that the Ubuntu proposal is not implemented yet.
>
> I'm not sure I understand ("not storing tests in the source package"?).
> What are you against exactly? The only thing I'm saying is that I don't like
> implementing something on our own (the "command architecture to deal with our
> unit tests" you said; it's still quoted to give context, perhaps I didn't
> understand you). I would much prefer using something already done, and
> "standard", if good enough. And it's probably going to be easy, plugging the
> own package unit tests to the Ubuntu testing framework.
>
> > I think test scripts often can be useful as documentation, especially
> > when example scripts are not provided.
>
> Yes, perhaps in some cases, but, in how many of them? In any case, perhaps
> we could provide them as documentation, but not try to execute them or
> otherwise treat them as code.
>
> > Also, I personally would like to be able to run the tests on all ruby
> > packages on a regular basis.
>
> "all ruby packages" = "including those not packaged by you", right?
>
> > The fact that the package works at build
> > time doesn't mean that it will continue to work. For example, when rake
> > 0.6.0-1 was uploaded, it was working just fine. But when ruby 1.8.3 was
> > uploaded, a change in fileutils broke rake (see bug #336937). If rake
> > came with a test suite, and if somebody had been running it regularly,
> > this would have been easily picked up.
>
> I think that's our duty. I don't see the point of letting _everyone_ (as
> opposed to each package maintainer) continually run the tests. If you see
> something bad, and you know what packages are involved, you can always run the
> tests once for those packages.
>
> One more question: if we follow the Ubuntu proposal and make all Ruby
> packages testable with it, it will probably be very easy to test all of them
> at once. Perhaps even Debian will have an automated testing infrastructure
> that will do what you describe, but in a centralized fashion... or do you see
> any difference?
Some context here: I'm part of the not-so-active "MOTURuby" Ubuntu team,
whose goal is to ensure that "ruby packages" (basically all packages
depending in ruby) work well in the Ubuntu releases. As you might know,
ruby in Ubuntu breezy has several annoying bugs just because nobody
cared for the ruby packages during the freeze. Our goal is to avoid this
situation for Ubuntu Dapper (due 04/2006).
Our job can involve backporting a specific version from Debian during
the freeze. Our problem is this case is to ensure that all other
packages (which were or weren't backported) are still working. That's
why I'm interested in having a way to regression-test "all ruby packages".
The Ubuntu Automated Test proposal is aimed at providing a common
framework for all packaegs, not just ruby ones. We will have to do some
work on our own anyway to integrate the Ubuntu proposal and test/unit.
Also, it seems to me that the proposal is generic enough to all tests to
be stored in the source package or in the binary package.
Maybe, for now, we could just stick with "the package maintainer decides
whether the package's tests are useful as documentation, and includes
them in a -dev or -doc binary package if appropriate" ? Let's wait for
Ian Jackson to actually release something before making any decision
about a better handling of tests.
--
| Lucas Nussbaum
| lucas@lucas-nussbaum.net http://www.lucas-nussbaum.net/ |
| jabber: lucas@nussbaum.fr GPG: 1024D/023B3F4F |
Reply to: