Re: RFC: implementation of package pools
On Fri, 20 Oct 2000, Eray Ozkural wrote:
> You can make a system very flexible and fault tolerant if you want to.
> That's a matter of programming skill. Same as how your dollars don't get lost
> at banks. Period.
Ever tried? Ever worked for a company that has tried? It has almost
nothing to do with skill of the individuals, or individual minor choices
(such as hash functions) but *everything* to do with testing and
specification.
There is quite a bit of skill required to prepare a proper specification,
you need to cover every case and every contingency [IMHO most degree
programs don't teach this well]. Five 9's reliability (which is what
banks, telcos, etc expect) is a very serious non-academic exercise.
Notice that a telco for instance runs their buisness expecting their
software and hardware to fail. It is a calculated risk that is managed to
not seriously effect their earnings. Banks loose money, telcos misdirect
calls, but it is infrequent.
> These are all true for the archive. Archive is just a set of files,
> and all indices into the archive, auxiliary structures, etc. can be
> reconstructed/repaired at any time.
Only if you have a limited view of what can go wrong. Sure you can rebuild
most of the SQL database, but how are you going to deal with massive
filesystem corruption, or an accident in one of the scripts?
> The idea is to automate things. Automatons can be more reliable than
> humans.
Sure, but an automation that is infact able to handle every possible
failure case (like a well trained human) is generally uneconomical to
build.
Try this, design a B-Tree structure that stores a set of words. The
underlying hardware you are running on destroys 0.001% of all your
'pointers' in an unpredicable way. Design an algorithim to operate 100%
correctly and a seperate one to operate 99% correctly, but allow a human
to fix the tree structure if necessary. Which is longer? Which is slower?
If you could use another datastructure besides a B-Tree could you get 100%
reliability? (perhaps one that did not use 'pointers')
> It will be a magnitude more complicated (correct) and a magnitude
> more efficient, and at the same time be more reliable. How about that?
Actually, I've done benchmarkings on this case, I can get several
magnitudes speed gains with binary databases - how ever the time spent
loading the database is small enough that this can't justify the size of
code required to support a fault proof implementation.
Engineering trade off at work here. [ In most cases engineering decisions
are framed in physical sorts of things, you can make a chemcial processing
plant that is 100% safe, however that is not economically feasable, so you
make one that is 'just safe enough'. This results in something we can
actually afford to build. Same for software. We don't have 10 years to
wait while someone works out the complexities of fault recovery, and
perfects the ideal algorithims. ]
Further, remember, any monkey can fire off a hundred thousand line
program, complete with the most advanced algorithms and best
speed-enhancing techniques. That isn't really hard at all.
The real trick is producing software that will meet a need, last, adapt to
changes and remain viable years into the future. Not surprisingly a huge
amount of software out there (free and commercial) does poorly at
some of these facets :|
AJ was quite right to bring up examples like ext2 and the linux kernel -
they are good example of mixing both requirements, and how fine that line
is.
Anyhow, we can revisit the hash function, framed in the above. You contend
its distribution is imperfect and can be improved. However:
1) Changing it will introduce more complexity
2) Changing it will not significantly enhance performance
[ By your own stats we have about 400 directories/dir. However that
is about 3x smaller than the current worst case - which *still*
performs acceptably. ]
3) Changing it introduces a 'personnel problem' which has a very
real cost.
Given that, do you still think it is at all important?
> trouble. :< I insist that every package should have an ITP. This is
Eh? Why? Hello? ITPs are to prevent duplication of effort, that is why
there were introduced and that is their purpose. You can't duplicate
packaging a unique peice of software :P
Jason
Reply to: