Bug#563765: ITP: python-whisper -- fast, reliable storage of numeric data over time.
Package: wnpp
Severity: wishlist
Owner: Elliot Murphy <elliot@ubuntu.com>
This bug report is about my intent to package python-whisper,
which is a part of the graphite project:
http://graphite.wikidot.com/faq
* Package name : python-whisper
Version : 0.9.5
Upstream Author : Chris Davis <chrismd@gmail.com>
* URL : http://pypi.python.org/pypi/whisper/
* License : Apache-2.0
Programming Lang: Python
Description : fast, reliable storage of numeric data over time.
Whisper is a fixed size database, similar in design to RRD.
RRD is great, and initially Graphite did use RRD for storage..
Over time though, we ran into several issues inherent to RRD's design.
1. RRD can't take updates for a timestamp prior to its most recent update.
So for example, if you miss an update for some reason you have no simple
way of back-filling your RRD file by telling rrdtool to apply an update
to the past. Whisper does not have this limitation, and this makes
importing historical data into Graphite way way easier.
2. At the time whisper was written, RRD did not support compacting multiple
updates into a single operation. This feature is critical to Graphite's
scalability.
3. RRD doesn't like irregular updates. If you update an RRD but don't follow
up another update soon, your original update will be lost. This is the
straw that broke the camel's back, since Graphite is used for various
operational metrics, some of which do not occur regularly (randomly
occuring errors for instance) we started to notice that Graphite
sometimes wouldn't display data points which we knew existed because
we'd received alarms on them from other tools. The problem turned out
to be that RRD was dropping the data points because they were
irregular. Whisper had to be written to ensure that all data was
reliably stored and accessible.
Reply to: