Parsing NOAA sources 2 Comments
For those that prefer Ruby over Java, I have good news: Sensorpedia scripts can now be written using JRuby. It uses the same API as the Java interface (both a positive and negative), so now you’ll be able to parse data sources using the convenience of Ruby regular expressions. In order to test this, I wrote a simple parser for the NOAA data sources (which consist of simple text files). Since a sensor can have multiple scripts associated with it, I also integrated Tables functionality by associating the NOAA sensors with the “Tables” tag (the Tables scripts were already written for the DASMet sources). Here’s a video demonstrating this functionality.
(Direct link: http://blip.tv/file/get/Jhorey-NOAADemo650.ogv)
In the video, the user first creates a pivot table to examine the “Air Temperature”. Not all the nodes have air temperature data, but some of those that do are in pretty cold climes (temperatures are in F). The user then writes a local function that filters nodes that register less than 20 F. Afterwards, the user creates a new pivot table to view which nodes were filtered. Then, the user constructs a collective function to take the average temperature and locations of all the filtered nodes. Finally, the user requests the final averaged data.
There’s still quite a bit of work to do with respect to the scripting layer. The most important probably being the key-value store I’m using to store and access all the data from the scripts. Right now it’s pretty fragile, but ultimately I want something that resembles map-reduce (both in terms of functionality and scalability).
Your article is very useful. Can you please post some new-found information about this? sincerely