Collecting RuuviTag measurements and displaying them with Grafana

[Scrin]

Background

I have been using Grafana and InfluxDB and other time-series databases before for drawing all kinds of graphs, so this was naturally the first project I did with the RuuviTags when I got my hands on them. Long story short, the graphs are viewable live here (there are also some additional dashboards both related to Ruuvi and my other projects found behind that link, for example what the climate in a sauna looks like)

Project

There are basically three key components:

• RuuviCollector application (such an innovative name…) which is an application I wrote that collects measurements from RuuviTags, parses them, and pushes to InfluxDB
• InfluxDB which is a time-series database for long-term storage for the measurements
• Grafana which renders pretty graphs about these measurements stored

RuuviCollector

Probably the “heart” of this project is this little collector application, it’s written in Java and originally put together in just a couple of hours, reusing parts of my other similar coding projects which utilized InfluxDB as storage as well. In simplicity, it uses hcitool and hcidump from the bluez and bluez-hcidump packages to scan for BLE traffic and dump the raw packets. Ruuvi beacons are detected from the traffic and the sensor data is parsed from them and finally pushed to InfluxDB after adding some metadata such as identification of the physical tag and the format of the sensor data.

At the moment of writing this, a lot of improvements, code cleanups and writing documentation is to be done, but for those who have some “techy knowledge” can head over to the github page and try the application.

The setup

In my setup, the RuuviCollector is running on a Raspberry Pi 3 running Raspbian, and InfluxDB and Grafana are running on my dedicated server. As both RuuviCollector and Grafana communicate with InfluxDB over http(s), it’s possible to have these set up in physically different locations. Any device, computer or home server can be used to run the collector, as long as bluez, bluez-hcidump and Java 8 are available on the system, and naturally a bluetooth adapter supporting BLE is present. It is also possible to run this entire setup completely all on one single Raspberry PI 3.

I use the latest weather-station firmware (found here) on the RuuviTags in high-precision mode, which also reports the accelerometer readings as well as the battery voltage. The ‘default’ mode in weather station firmware reports measurements at a reduced accuracy to avoid spamming nearby phones when the temperature changes 0.01⁰C. The high precision mode reports the measurements as “Manufacturer specific data” rather than the plain Eddystone-URL so the beacons won’t pop up on nearby phones.

Most likely I forgot to mention something relevant, so feel free to ask questions if you have any.

[mahead]

Nice graphics! How do you get the high precision data? I have similar setup, three ruuvitags broadcasting data that is collected by RaspberryPi. That also process and dumps the data into MySql db. From there, another RaspberryPi fetches it and displays on small display. Also on my MySql server there is couple of scripts for plotting the data as a graph. I take measurements every minute, then calculate averages per selected timeframe (hour, day, week, month, year) for both inside and outside tags (two outside tags, one on northern side and another on southern side of the apartment; only lower temperature is taken into graph). I also graph the min & max values. Result looks like this:

However, the temperatures are only at 1 ⁰C precision, so that’s painful flaw I would like to correct as soon as possible.

[Scrin]

I use the latest weather-station firmware (found here) on the RuuviTags in high-precision mode, which also reports the accelerometer readings as well as the battery voltage. The ‘default’ mode in weather station firmware reports measurements at a reduced accuracy to avoid spamming nearby phones when the temperature changes 0.01⁰C. The high precision mode reports the measurements as “Manufacturer specific data” rather than the plain Eddystone-URL so the beacons won’t pop up on nearby phones.

I’ll update the original post with these details too

[shuckle]

I have also program which reads the weather station data. How fo you read the sensortag data? Has it the same format? Do you know where the source code of sensortag code is?

[Scrin]

The sensortag uses Data Format 3, which is really close to the Data Format 2 used by the “default” weather station firmware. The sensortag firmware sources are linked in my previous post.

One major difference is that, as mentioned in my previous post, the high-precision data from the sensortag is not encoded into an Eddystone-URL, but rather put “as-is” into the raw data, so instead of grabbing the url from the raw data, base64 decoding the hash part of the url, and parsing data from that, you have to grab the raw data from the raw packet and parse that directly instead. In the end that’s actually easier to handle than the url because you don’t need to do any base64 decoding.

[shuckle]

Thanks!
Yes, it was too easy. Works now.

[tomgidden]

I have a similar setup, although I use EspruinoHub to capture the advertisement data, send it to an MQTT broker, which Node-RED then decodes (using some custom JS that I really should post here), and then rebroadcasts it via MQTT and pushes it into InfluxDB for Grafana.

I use MQTT and Node-RED extensively for my home automation, with several clock/status/weather displays, control keypads, smart bulbs, medical monitoring, power usage monitoring and control, and even linking to Alexa for Echo Dot and Google Home for voice recognition. Node-RED and MQTT form the backbone of the whole system with Node-RED being the logic and wiring, and MQTT being the communication system.

My one RuuviTag unfortunately picked the short straw, permanently stuck outside for the menial task of temperature, humidity and pressure, being the only one of many devices I have that’ll do that job reliably and efficiently. I can’t wait for orders to reopen so I can get another to properly play with.

[otso]

We created an image for Raspberry Pi 3 last weekend at Finnish hackerspace Helsinki Hacklab. The images uses @Scrin 's RuuviCollector to collect data from RuuviTags and sends the data to InfluxDB running locally.

Additionally Telegraf is used to collect stats about system itself. Grafana (and Chronograf) are used to display data on browser. Apache is used to direct port 80 (default http port) to port 3000 (Grafana).Kapacitor is installed, but it is not configured. Chronograf and Grafana have example dashboards which read and display data, you’ll need to configure the mac addresses of your tags.

All the services start up automatically, I’ll open a pull request for a script to start scrin’s collector soon. Additionally the Raspberry will attempt to register itself as "“raspberrypi.local” [edit], on Linux and MacOS machines you can just point your browser at “raspberrypi.local” [edit]. On Windows you’ll need to install Bonjour to find the Pi at “raspberrypi.local” [edit].

The image is available at Google Drive, it fits on a 8 GB SD-card. All the passwords are default, and SSH server is enabled. Please change at least Pi’s root password ASAP

• Pi: pi/raspberry
• Grafana: admin/admin
• InfluxDB: blank

Give it a try and let us know if you have any suggestions for improvement.

EDIT: Image link updated.
New install does not have Telegraf, Chronograf and Kapacitor installed, please download and install these from InfluxDB downloads.

[otso]

I made a quick Node-Red node which listens to BLE advertisements using noble and parses binary broadcasts of Weather Stations (no URL-mode support yet), code is here.

I’d love to see how your code pushes data into InfluxDB, I’m considering making a Node-Red Raspberry Pi image to collect data from RuuviTags next.

[tomgidden]

My Node-RED JavaScript code is:

var data = new Buffer(msg.payload.replace(/"/g,''), 'hex');

var battery = data.readUInt16BE(12)/1000;

var decoded64 = data.slice(0, 6);

if (decoded64[0] !== 3) {
    node.warn("Bad data: "+decoded);
    return null;
}

var humidity = decoded64[1] * 0.5;
var air_pressure = ((decoded64[4] << 8) + decoded64[5]) + 50000;

var uTemp = (((decoded64[2] & 127) << 8) | (decoded64[3]*256.0/99.0));
var tempSign = (decoded64[2] >> 7) & 1;
var temp = tempSign === 0 ? uTemp/256.0 : -1 * uTemp/256.0;

return [
    {
        "payload": parseFloat(temp.toFixed(5))
    },
    {
        "payload": humidity
    },
    {
        "payload": air_pressure/100
    }
];

(with 3 outputs set, to separate out the three main environmental sensors)

This function feeds into this flow:

This protects against spikes and spurious readings, and also removes duplicate values. This is so Grafana doesn’t get overloaded by points. It would be better to not filter out duplicates so code can distinguish between duplicate values and lack of data (eg. failure of the sensor / Bluetooth), but for my purposes the priority was reducing the point count to Grafana.

Once complete, on the right-most end, this links to another flow that does the InfluxDB work:

That’s using node-red-contrib-influxdb, so it’s very easy. The parseFloat function remodels the message for the InfluxDB node:

msg.measurement = msg.topic;

msg.payload = Number(parseFloat(msg.payload));

return msg;

I found that it was setting the value type to either integer or string, thereby either truncating all future values, or making them impossible to do numeric functions on. So, it seems best to manually insert a float, eg. 0.5, into the new measurement and then DELETE it, using the influx command-line. Alternatively, temporarily add 0.001 or something to the msg.payload on the first hit, just so it’s forced to be typed as float from then on.

[tomgidden]

Incidentally, @otso, I think it’s worth you pushing your custom node as node-red-contrib-ruuviweather, as it’s much more out-of-the-box than my cut-and-paste function. My code converts the Hex data into a Buffer sooner and does it mathematically, rather than via string manipulation and late conversion from Hex.

If I weren’t already using EspruinoHub for other things, I’d use your noble based pipeline along with your custom node. EspruinoHub is basically doing the same thing – monitoring hcitool – but sending results via MQTT rather than directly into Node-RED. That way I’m able to separate the physical-location-dependent Bluetooth job from the main Node-RED server, which is necessary in my case.

[otso]

Thank you for the detailed post, awesome work!

[tomgidden]

Right back at you! I actually started writing a node-red-contrib-ruuviweather last week the same way you have, but I got distracted by having to relocate all my code from a failing Pi. I suggest you publish your code to npm, as it’d make it all practically plug-and-play for Node-RED.

I figured node-red-contrib-ruuviweather was more suitable than node-red-contrib-ruuvitag, just because RuuviTags could be used for more than just the default weather firmware.

[otso]

I’ll look into publishing the code into NPM as soon as the dependencies of package are in good shape. I agree with you that “ruuviweather” would be a more descriptive name to current node, and since the message current node sends out does not contain format information from node any update to current node would break user applications.

Maybe we’ll publish the current node as RuuviWeather ASAP and support the current weather station firmware with it and make another development version node which would support upcoming features such as connecting to tags over GATT. Thank you for the suggestion

[mikewhitten]

Hi,
I spent some time with this node, trying it out with my intended use case - flowing sensor data into InfluxDB for monitoring lab environments. After looking at the InfluxDB node-red node I decided to try a little change and it’s worked out well. It leads to a quite simple node-red flow: scanBLE -> ruuvinode -> influxdb, with nothing else needed to get the sensor data logged the way I was looking to do it.

The InfluxDB node will take an array of objects - the first is a list of field values, the second is a list of tags. It made sense to me that the uuid of the Ruuvitag would be a good tag and all the other data would be good fields.

My goal is to have several groups of sensors monitoring discrete entities within one larger lab. So say three sensors in dome 1, two sensors in dome 2, a sensor in computer rack 1 for example.

The idea is to catch the data from all sensors into one measurement with a relatively short retention policy and have continuous queries run to sort the data by uuid into time-averaged measurements for dome1, dome2, and rack1. It looks like it’s going to work great.

The code change I made from the excellent node @otso supplied is simple:

  ruuviData.rssi = msg.rssi;
  // msg.payload = JSON.stringify(ruuviData);
  let ruuviUuid = {};
  ruuviUuid.uuid = msg.peripheralUuid;
  msg.payload = [ruuviData, ruuviUuid];
  node.send(msg);

That’s it. If it’s of any use feel free to pick it up. Thanks again for the node!
Mike

[Mikael]

hi otso, what files do I enter my ruuvi mac addresses into? And thanks for the image.iso

[otso]

Hello Mikael,

1. Start grafana: sudo systemctl enable grafana
2. Point your browser to your grafana installation, e.g. http://ruuvi.local or http://IP_ADDRESS:3000
3. Edit the mac values in grafana dashboards

EDIT: New image runs on http://raspberrypi.local or http://IP_ADDRESS, i.e. on port 80. Therefore there is no need to add :3000 anymore to URL

[Mikael]

Hi otso,

1. I tried: sudo systemctl enable grafana I get an error: Failed to execute operation: No such file or directory. I didn’t think I needed to be in a specific directory.
2. so I did: sudo systemctl enable grafana-server.service
3. systemctl list-unit-files now shows grafana-server.service is enabled

However, ruuvi.local in bonjour enabled computer returns Service Unavailable. Do I need to enable apache somewhere?
and through http://192.168.0.106:3000/ I get This site can’t be reached.

Any thoughts for next steps? Thanks.

[otso]

Hello, Apache is enabled by default, so it should be ok from the start.

Sorry for incorrect command for enabling Grafana, happy to see that you got it fixed.

I’m assuming your error is “ERR_CONNECTION_REFUSED”, which would mean that the Grafana is not accepting connections. Could you please run “systemctl status grafana-server.service” to check that Grafana is actually running? It might be that enabling the service will only start it after the next reboot. If so, you can manually start the service by “sudo systemctl start grafana-server.service”

[Mikael]

Hi otso, I ran status check: --> active (running)

in browser on pi3 just did ruuvi.local and it worked! Not sure what went on there. I rebooted system awhile ago and gave it time in case anything was slow to boot or start. I’m in grafana now and working to setup the mac addresses.
Thanks.