Re: 0ad beta 6 packages
On Sun, Aug 14, 2011 at 10:46 AM, Paul Wise <pabs@debian.org> wrote:
> [...]
To possibly clarify, the current development process is:
1) Artists (by which I mean audio too) create art in various programs
and save in various formats, with proprietary source formats (.max
(3ds Max), .psd (Photoshop), .psp (Paint Shop Pro), whatever
presumably expensive systems musicians use, etc) and open source
(.blend (Blender), .wav (for sound effects), etc), or sometimes only
save in 'flattened' formats like .png (particularly if they're
modifying someone else's art and don't have whatever program the
original source format was for).
2) Sometimes they put those original source formats on some FTP site
or some separate SVN repository, but not in a reliable or organised
way, and sometimes they lose them entirely.
3) They export to standard formats (.dae for 3D models and animations,
.png for textures, .ogg for audio; some old content uses .dds for
textures and the game engine's custom binary model/animation formats
(.pmd/.psa)) and put those exported files in the main SVN repository.
4) Some files in SVN are generated by running some tools (also in SVN)
on other files in SVN, then manually committing the output when
necessary. (In particular, constructing the font textures - I don't
remember any others.)
5) The game executable can be run in a mode that collects all the data
files from an SVN checkout, does some processing and compression and
conversion into more efficient file formats (.dae -> custom binary
formats (.pmd/.psa), .xml -> custom binary format (.xmb), .png (or any
other texture) -> .dds), and sticks them all into a single .zip file.
(If the game is run without first processing the data files and
packing them into a zip, e.g. if you run it directly from an SVN
checkout, it will do the same processing work at runtime when it loads
each individual file, so it works transparently (except for delays of
tens of seconds while it converts the files) - the zip is merely an
optimistion.)
6) When creating 0ad-*-unix-data.tar.xz files for releases, we run the
executable to pack all of binaries/data/mods/public/ (from SVN) into
public.zip and only include public.zip in the release tarball.
'Building' the package from that tarball just involves copying files.
It sounds like you're asking for:
1b) Artists create art only in DFSG-free applications and save in the
original lossless editable formats and put those files in the main SVN
repository.
2b) There is a build script that can automatically convert all those
files into formats the engine can load (which probably involves
invoking Blender and Gimp and Audacity etc from the command line to
export from their formats).
3b) (Maybe we have some system that uploads the converted output into
SVN so developers can develop without the huge pain and slowness of
having to run all those conversions themselves.)
4b) We release a tarball with those original source files and the
build script, then the distro package scripts run the build script
(eventually producing a public.zip the game can load, like in the
current system).
I think the fundamental problem is that 1b is never going to happen -
we'd have to throw out all our art and half our artists and start
again with open source tools, since very little is currently done in
that way. Using the new process only for a portion of the art doesn't
seem useful - either it's necessary for all art, or else it's not
necessary and there's no point spending time setting up a new pipeline
for only some of it.
(I'm not claiming the current process is ideal - I believe we ought to
be much more organised about storing original source files for art,
and ought to converge on fewer applications for creating it, since
that would make development smoother, but that'll be a gradual change
if it happens at all, and doesn't seem relevant from a DFSG-compliance
perspective.)
Am I understanding this correctly? Is the problem fatal?
It would certainly be technically possible to change the process to
get rid of step 4 and to make step 6 build public.zip as part of the
distro package build script, using a new release tarball that has the
files directly from SVN (as opposed to the current one that only
contains the pregenerated public.zip), though it would introduce a
(slightly circular) build-time dependency on the game executable
package (which contains the code to generate the .zip) and I don't
know if the change would really solve any problems.
> There is a ZIP file in there too. It seems a bit rediculous to put a
> compressed file inside the already compressed tarball and even more
> rediculous to put such a file in a VCS (didn't look if it was there
> too).
Assuming you mean public.zip, it's an uncompressed zip file (to avoid
interfering with tarball compression). It's not in the VCS - it's
generated by step 6 above.
> Also the ./public/art/LICENSE.txt file indicates that all the
> materials from CGTextures are missing their source form.
For most of the textures we can't usefully distribute the original
source version since it's either lost or in a format like .psd; for
the ones using CGTextures materials we couldn't distribute them in any
case since CGTextures doesn't allow distribution of their work in
unmodified form (but we got explicit permission to distribute the
derived bitmap textures as CC-BY-SA).
> I find it completely bizarre that they are using an Excel spreadsheet
> to store techtree info. Does the game really require a Excel file
> reading code??
That file is just documentation. Not sure what it's doing in SVN there
- should delete it some time (probably when implementing techtrees and
verifying that it's obsolete).
> There are two RAR files.
Those aren't used and should be deleted some time (after verifying
they don't contain anything useful).
> Those xmb files look like binary versions of the corresponding xml
> files.
Yep, via step 6.
> There are some weird binary formats. One of them contained a string:
> "God Knows", perhaps that should be replaced with "God Knows WTF these
> files are and how to modify them" ;)
I think those would be the .pmd/.psa files, which are generated either
by step 6 or step 3 (exported directly to the custom binary formats).
> I wonder about the DDS files. They can be opened in GIMP with gimp-dds
> but their names seem to indicate that there are TGA files that are the
> source form of them.
Indeed, they're from step 6 (conversions of the texture files in SVN
that are .tga or .png or .dds).
> IMO the DDS files should be created at
> runtime/build time from their source TGA files, or the game should
> simply switch to using the TGA files.
We need mipmapping, which is only really supported by DDS. Also we
want S3TC compression (saves >75% video memory) which needs DDS.
Creating the compressed DDS at runtime can take a couple of seconds
per texture - we do that when running the game from an SVN checkout,
but that's not good for normal players. So we do do it at build time
(but in the build step before creating the tarball).
> There are a bunch of pre-encoded Ogg files. I don't know what is the
> source but Ogg definitely is not it.
I think most audio files we only have in .ogg format, but we're
gradually replacing them and trying to be more organised with storing
.wav versions of the replacements. But storing .wav in the main SVN
repository sounds non-ideal (the music alone will add a few hundred
megabytes to the download), so we're currently sticking them in a
separate parallel SVN repository and ought to find a better way to
handle them eventually.
--
Philip Taylor
excors@gmail.com
Reply to: