Re: looking for a file format for time series data other software can easily consume
On Wed, Nov 1, 2017 at 1:00 PM, Ben Caradoc-Davies <ben@transient.nz> wrote:
> On 01/11/17 20:38, Dan Hitt wrote:
>>
>> The NetCDF does seem to do what i want. As far as i can tell, it
>> allows you to set metadata at the file level as well as attributes
>> attached to specific arrays in your data. It can be installed in
>> debian (packages libnetcdf-dev, netcdf-doc, netcdf-bin). The api
>> doesn't look too complicated, at least if i understand it correctly.
>> The packages come with some simple example programs. And the 'file'
>> command recognizes the data that it produces. And there's some kind
>> of mailing list. So if i can make a go of it, i think that's what
>> i'll be doing.
>
>
> What language would you like to use? There are bindings for many popular
> languages including Python, and a Java implementation for JVM languages.
>
> Timeseries representation is straightforward: a one-dimensional "coordinate
> variable" for time with an attribute like units="seconds since [your epoch
> goes here]" and data variables of the same shape. The udunits library will
> even convert units for you. If you make your time dimension UNLIMITED, new
> time values are written at the end of the file (rather than rewriting the
> whole file).
Well, my first step is to write a few test programs to make sure that
it can do what i think it can (write metadata, and write data). Looks
good so far, but i'm not done with this stage.
The people on the netcdf mailing list have been very helpful in that regard.
I'm writing these test programs in c.
The data generation i'll probably do in d, because it is so convenient
to program in. If there are d-bindings that would be nice, but not at
all essential, as i can wrap the parts i need pretty easily (knock on
wood).
For the data display, i'm sort of tentatively planning on using
matplotlib (python 3), again because of ease of use.
But for now this is all vapor ware until i'm a little more solidified. :)
dan
Reply to: