5.1 updates wanted, and CD remastered size w/ hardlink script

[Klaus Knopper <knopper@knopper.net>]

Hi,

Since I'm still without broadband internet access while on a visit in
poland (which is entirely my fault, of course, should have thought of
ordering DSL), just a short all-in-one note from my side, addressing
some questions. Martin Öhler and others who have been kindly testing and
fixing bugs in the recent release are also monitoring this list, so
please report all new bugs, PLEASE also with a solution or patch, if you
can, right here for now.

Looks like we will issue a bugfix/update release soon (as in a few
days), not only because of the kicker bug, which really escaped all
pre-beta-testers. It seems that the problem is not so much kicker
crashing, but it's just not visible in all virtual desktops but the one
it has been started from. The new kicker from Debian/unstable may fix
this bug, however, kdebase grew another 6MB, so again some work to do in
order to shrink the CD to a reasonable size. :-/

Some bugs that we just can't fix quickly, unless someone provides an
ad-hoc-patch. For example, the CA french keyboard. I really don't know
why it worked for gilpel in 5.0.1. I'm not aware of any change
concerning keyboards in KDE. Maybe a path change in a recent Debian
package?

Also, there is a new and "up to 600 times faster" ntfs-3g version
available, new firefox, pardon, iceweasel. Also, kaffeine should be the
default multimedia player until mplayer catches up in Debian/unstable.

Now about the CD size (Mike): There are some more tricks, apart from
using -b in create_compressed_fs and shrinking the apt-get database with

# most important: delete old dowloaded packages
apt-get clean

# Zero lists, reduce dpkg-available to installed packages.
echo -n "Zeroing available Package lists ..."
for i in /var/lib/apt/lists/*_{Release,Packages,Sources}; do :>$i; touch -t 198103190000.00 $i; done
apt-get check
dpkg --clear-avail
rm -f /var/cache/apt/*.bin
apt-cache gencaches
sync-available

If you increase the blocksize for create_compressed_fs, you get better
compression, but at the cost of some overhead at runtime when exceed
data is decompressed. With cloop 2.05, there is a small decompressed
block cache, so the data decompressed in the first cycle MAY get reused
in consecutive reads, which actually improves performance when using
larger blocks, and reduces seek times. For 5.1, we doubled the block
size to 131072 bytes for that reason (and because the aforementioned
better compression).

A trick that, with some caution, also saves a gread deal of duplicate
data, is replacing all files with identical content, by hardlinks. See
the attached script, that first creates another shellscript with he
necessary commands (./Knoppix.hardlinks | tee /tmp/runme.sh) and also
tells you how many bytes you would save when doing an sh /tmp/runme.sh.
This can be in the range of 30-60MB for the CD! The script tries to
ensure that no backup or empty files are hardlinked, else you may run
into trouble when a program does not unlink a backup or log file before
recreating/rewriting it.

The final "dirty trick" for the CD is to manually remove stuff from
/usr/share/doc. Especially removing compressed docs and pictures gains a
lot of diskspace. I'm not sure if anybody is missing these docs, since
nobody (except for the Quantian maintainer) has ever complained about
documentation missing on the CD, but there is always the possibility to
remaster your own version re-including these docs by apt-get install
--reinstall packagename, or, just use the DVD instead which has all the
docs and kernel sources.

In fact, removing docs and even changelogs is still legal for the BINARY
distribution due to the GPL as long as the technical docs and changelogs
stay intact with the SOURCES that you deliver. Which reminds me that I
have to update the debian-knoppix repository as soon as I'm back to a
broadband connection.  License files must NOT be removed, independent
from the GPL, so deleting the entire /usr/share/doc would indeed be a
(legal) problem.

I'm thinking about removing the KDE HTML docs (only german and english
are still present there anyways) completely in order to get spanish
language back in the CD. What do you think?

With kind regards
-Klaus Knopper

#!/bin/bash

find /var /bin /sbin /lib /usr /opt -type f -size +0c -not -iname \*old -not -iname font\* -printf '%s %m %U %G %T@ %i %p\n' | egrep -v -e '(/var/lib/apt/lists)' | sort +0 -0 | \
( save=0
  old_link=""
  old_owner=""
  old_group=""
  old_size=""
  old_time=""
  old_perm=""
  old_inode=""
  old_file=""
  old_md5=""
  while read size perm owner group time inode file; do
   if [ "$size" = "$old_size" -a "$perm" = "$old_perm" -a "$owner" = "$old_owner" -a "$group" = "$old_group" -a "$inode" != "$old_inode" ]; then
    if cmp "$file" "$old_file" >/dev/null; then
     echo "rm -f '$file'; ln '$old_file' '$file'"
     inode="$old_inode"
     save="$(expr $save + $size)"
    fi
   fi
   old_size="$size"
   old_perm="$perm"
   old_owner="$owner"
   old_group="$group"
   old_time="$time"
   old_inode="$inode"
   old_file="$file"
  done
  echo "# Would save $save bytes."
)