Collecting RuuviTag measurements and displaying them with Grafana

ejk · March 13, 2019, 4:21pm

I’m using ruuviberry image “2018_05_minimal”, where I have updated Grafana from 5.0.4 to 5.4.3 and InfluxDB from 1.5.2 to 1.7.4 manually.

How can I show in Grafana pressure difference between two different tags?

otso · March 13, 2019, 5:09pm

As far as I know Grafana / InfluxDB does not support this in an easy manner, you’d probably need some kind of continuous query which creates difference data . @Scrin would you happen to know an elegant way to do this?

Scrin · March 13, 2019, 6:57pm

As far as I know you are correct, InfluxDB does not support on-the-fly calculation between separate measurements (pressure values from two different measurements in this case), and as far as I know, the InfluxDB builtin Continuous Queries are restricted by this same limitation. The method recommended by InfluxData is to use Kapacitor, which is their data processing engine, to perform this kind of calculations.

I used to have Kapacitor tickscripts created for calculating some of the “extra values” (such as dew point and absolute humidity), but I have since implemented them straight into the collector itself because Kapacitor tickscripts are rather clumsy to work with compared to doing the same with Java straight in RuuviCollector.

PasiK · March 23, 2019, 2:25pm

Hi,
so, adding “temperature” in “field(value)” didn’t work.

We showed the project to our teacher and his opinion was that somehow the RuuviCollector isn’t sending data. Now we are trying to build the ruuvicollector again, but now the building is giving errors/failures.

We then tried modifying the values inside ruuvicollector properties file, and with modifying “influxBatchMaxSize=2000” to 1234 and “limitingStrategy.defaultWithMotionSensitivity.threshold=0.05” to 0.06. This got rid of two “failures”. The building worked without properties file, but that didn’t give any data last time so that’s why we are trying to get this to work with the properties file.

Also, continuity to my last problem, hcitool. We noticed/remembered that line "sudo setcap ‘cap_net_raw,cap_net_admin+eip’ which hcidump gave an usage error for setcap.

dgerman · March 24, 2019, 12:46pm

PasiK

Issue

hcidump |head -30

systemctl status ruuvicollector # NB may show last failure even if ruuvicollector is running OK

systemctl status influxdb

ps -el |grep hci # find all things bluetooth

ps -el |grep Ruuvi

ls -ltra /var/log | tail # show last several most recently updated logs

and copy / paste results here ( Not screen print which omits information)

influxdb takes a while to get ready to accept transactions (especially after a powerfail)
I have found it helpful to add this to RuuviCollectorStart.sh just before java -jar target/ruuvi-collector-*.jar

for i in 240 220 200 180 160 140 120 100 80 60 40 20
do
logger " ++ RuuviCollector sleeping $i more seconds to give influx a start."
sleep 20
done

To slow down influxdb restarts after failures add this to service section of /etc/systemd/system/influxd.service

RestartSec=600

dgerman · March 24, 2019, 4:14pm

The following script displays any ruuvi tags with hex values
(There is an occasionally incorrectly formated fmt05 movement & seq# )

hcidump -t --raw |                           #include time stamps
grep --line-buffered --invert-match '\-\-' |               
sed --unbuffered "N ;s/\n//; s/   / /" |
grep --line-buffered '99 04' |
sed --unbuffered "s/04 3E .. 02 01 .. 01 \(..\) \(..\) \(..\) \(..\) \(..\) \(..\)/ \6:\5:\4:\3:\2:\1  /"|    #  pkt:Meta Event/" |
sed --unbuffered "s/1. 02 01 0. 1. FF 99 04//"|
sed --unbuffered "s/10 \(..\) 03 72 75 75 2E 76 69 2F 23 42/TX:\1 https:\/\/ruuvi\/#B/"|
sed --unbuffered "s/10 \(..\) 03 72 75 75 2E 76 69 2F 23 41/TX:\1 https:\/\/ruuvi\/#A/"|
sed --unbuffered "s/52 75 75 76 69 42 6F 6F 74/RuuviBoot\//"|

sed --unbuffered "s/   03 \(..\) /fmt03: \1% /" |
sed --unbuffered "s/fmt03: \(...\) \(..\) \(..\) \(..\) \(..\) \(..\) \(..\) \(..\) \(..\)/fmt03:\1 \2.\3C  \4\5kpa   \6\7x \8\9y03/" |
sed --unbuffered "s/y03 \(..\) \(..\) \(..\) \(..\) 00 00 00 00 ../y \1\2z  \3\4v /"      |

sed --unbuffered "s/   05 \(..\) \(..\) /fmt05: \1\2C /" |
sed --unbuffered "s/fmt05: \(....C\) \(..\) \(..\) \(..\) \(..\) \(..\) \(..\) \(..\) \(..\)/fmt05:\1 \2\3% \4\5kpa   \6\7x \8\9y05/" |
sed --unbuffered "s/y05 \(..\) \(..\) \(..\) \(..\)/y \1\2z  \3\4vTX /"      |
sed --unbuffered "s/vTX  \(..\) \(..\) \(..\) \(..\)/vTX MV:\1 #\2\3 \4/"

PasiK · March 27, 2019, 11:21am

so for the first one (I wasn’t sure how much you wanted from it so I copied everything):

pi@raspberrypi:~ $ hcidump |head -30
bash: hcidump: komentoa ei löydy
pi@raspberrypi:~ $ systemctl status ruuvicollector
Unit ruuvicollector.service could not be found.
pi@raspberrypi:~ $ systemctl status influxdb
● influxdb.service - InfluxDB is an open-source, distributed, time series databa
Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset:
Active: active (running) since Sat 2019-03-23 14:39:11 EET; 3 days ago
Docs: InfluxDB OSS v2 Documentation
Main PID: 465 (influxd)
CGroup: /system.slice/influxdb.service
└─465 /usr/bin/influxd -config /etc/influxdb/influxdb.conf

maalis 27 10:39:39 raspberrypi influxd[465]: ts=2019-03-27T08:39:39.195924Z lvl=
maalis 27 10:39:39 raspberrypi influxd[465]: ts=2019-03-27T08:39:39.196382Z lvl=
maalis 27 11:09:39 raspberrypi influxd[465]: ts=2019-03-27T09:09:39.195830Z lvl=
maalis 27 11:09:39 raspberrypi influxd[465]: ts=2019-03-27T09:09:39.196198Z lvl=
maalis 27 11:39:39 raspberrypi influxd[465]: ts=2019-03-27T09:39:39.195891Z lvl=
maalis 27 11:39:39 raspberrypi influxd[465]: ts=2019-03-27T09:39:39.196299Z lvl=
maalis 27 12:09:39 raspberrypi influxd[465]: ts=2019-03-27T10:09:39.195862Z lvl=
maalis 27 12:09:39 raspberrypi influxd[465]: ts=2019-03-27T10:09:39.198984Z lvl=
maalis 27 12:39:39 raspberrypi influxd[465]: ts=2019-03-27T10:39:39.195894Z lvl=
maalis 27 12:39:39 raspberrypi influxd[465]: ts=2019-03-27T10:39:39.196274Z lvl=

[1]+ Pysäytetty systemctl status influxdb
pi@raspberrypi:~ $ ps -el |grep hci
1 S 0 584 1 0 80 0 - 521 - ? 00:00:00 hciattach
pi@raspberrypi:~ $ ps -el |grep Ruuvi
pi@raspberrypi:~ $ ls -ltra /var/log | tail
drwxr-xr-x 7 root root 4096 maali 27 06:25 .
-rw-r----- 1 root adm 32477 maali 27 06:25 syslog.1
-rw-r----- 1 root adm 20865 maali 27 12:49 user.log
-rw-r----- 1 root adm 357508 maali 27 12:49 messages
-rw-r----- 1 root adm 357322 maali 27 12:49 kern.log
-rw-r–r-- 1 root root 47259 maali 27 12:49 Xorg.0.log
-rw-r----- 1 root adm 106570 maali 27 12:55 debug
-rw-r----- 1 root adm 48961 maali 27 13:07 auth.log
-rw-r----- 1 root adm 16622 maali 27 13:07 syslog
-rw-r----- 1 root adm 560495 maali 27 13:07 daemon.log
pi@raspberrypi:~ $ ^C
pi@raspberrypi:~ $

I can’t get status for ruuvicollector as it cannot be compiled with the errors I’m having, so it’s not running. If I recall correctly it compiles without the properties file, but I want to have it with the properties file. The errors should be in my previous command. I still have them if you would prefer them in text. For the first command

hcidump |head -30

I have the bluez installed, but I don’t know why it doesn’t work, though in the end it gave me

pi@raspberrypi:~/Documents/bluez-5.50 $ systemctl status bluetooth
● bluetooth.service - Bluetooth service
Loaded: loaded (/lib/systemd/system/bluetooth.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2019-03-27 13:07:27 EET; 8s ago
Docs: man:bluetoothd(8)
Main PID: 3473 (bluetoothd)
Status: “Running”
CGroup: /system.slice/bluetooth.service
└─3473 /usr/local/libexec/bluetooth/bluetoothd --experimental

maalis 27 13:07:26 raspberrypi systemd[1]: Starting Bluetooth service…
maalis 27 13:07:27 raspberrypi bluetoothd[3473]: Bluetooth daemon 5.50
maalis 27 13:07:27 raspberrypi systemd[1]: Started Bluetooth service.
maalis 27 13:07:27 raspberrypi bluetoothd[3473]: Starting SDP server
maalis 27 13:07:27 raspberrypi bluetoothd[3473]: Bluetooth management interface 1.14 initialized
maalis 27 13:07:27 raspberrypi bluetoothd[3473]: Failed to set privacy: Rejected (0x0b)
maalis 27 13:07:27 raspberrypi bluetoothd[3473]: Endpoint registered: sender=:1.39 path=/A2DP/SBC/Source/1

Scrin · March 27, 2019, 1:31pm

~~This is actually a bug in the tests, for now you can workaround this issue by compiling the package without running the tests with the -DskipTests flag, for example mvn package -DskipTests~~

edit: this bug has been fixed now

It looks like you have bluez installed but not bluez-hcidump. You need both (as mentioned in the requirements), the latter one provides hcidump.

It looks like you have a standard raspbian, in which case you should be able to install it with:

sudo apt install bluez-hcidump

After that:

sudo setcap 'cap_net_raw,cap_net_admin+eip' `which hcidump`

should work without any “usage errors”. After these RuuviCollector should be able to get the data.

If not, you can try troubleshooting it by first stopping ruuvicollector and then running:

hcitool lescan --duplicates --passive

in one terminal window, it should print some lines as it receives various BLE data. While that is running in one terminal window, run this in another:

hcidump --raw

this should print out all the raw data that’s received (as long as hcitool is running in another window!). If you encounter any erros with these, you could try googling for help. Once those works, then RuuviCollector should work too, as RuuviCollector runs those two commands internally.

PasiK · March 27, 2019, 4:08pm

Hey, thanks. I must have missed that bluez-hcidump installation. I’ll be installing that asap. Thanks

dgerman · March 27, 2019, 6:49pm

Pasik: Why are you trying to revise RuuviCollector before you have the version Scrin made working?

PasiK · March 29, 2019, 10:20am

I’m trying to figure out why I’m not getting any data to grafana. I figured I could try with the properties file if there is something that could help me

ejk · March 29, 2019, 12:12pm

What databases, measurements and series do you have?

pi@localhost:~ $ influx
Connected to http://localhost:8086 version 1.7.4
InfluxDB shell version: 1.7.4
Enter an InfluxQL query
> show databases
name: databases
name
----
_internal
ruuvi
> use ruuvi
Using database ruuvi
> show measurements
name: measurements
name
----
ruuvi_measurements
> show series
key
---
ruuvi_measurements,dataFormat=3,mac=xxxxxxxxxxxx
ruuvi_measurements,dataFormat=3,mac=yyyyyyyyyyyy
> quit
pi@localhost:~ $

And you have setup a data source in Grafana which uses the same Influx database as the RuuvoCollector uses?

dgerman · March 29, 2019, 4:09pm

Pasik: See my post post #211 Mar 24, 12:14 PM

hcidump -t --raw | #include time stamps
grep --line-buffered --invert-match ‘--’ |
sed --unbuffered “N ;s/\n//; s/ / /” |
grep --line-buffered ‘99 04’ |
sed …

If you are not getting anything from ‘99 04’ manufacture ID packets from hcidump, RuuviCollector will not see anything!

Also if RuuviCollector is running and has forked

hcitool lescan

Another attempt to run hcitool lescan will fail with

Set scan parameters failed: Input/output error

To see if there is ANY ruuvi tag entries in the influx database use

influx -database ruuvi -precision rfc3339 
select mac, last(temperature) from ruuvi_measurements group by mac

results should be similar to:

name: ruuvi_measurements
tags: mac=C7103C6824C2
time                    mac          last
----                    ---          ----
2019-03-04T03:55:22.83Z C7103C6824C2 26.01

name: ruuvi_measurements
tags: mac=D3517872EC0F
time                     mac          last
----                     ---          ----
2019-03-29T16:02:32.745Z D3517872EC0F 11.65

PasiK · April 4, 2019, 1:44pm

Hi, so this is what’s happening:

I tried the first thing you had in this message, did I do it right? I just wrote the whole thing as you put it but it didn’t give anything. Screenshot_20190404_155311 The “sed” gives an error that the first character is an unknown command. I didn’t understand that message you are referencing to, care to open it a little for me? I probably need to use it but I don’t know what to do with it.

I used your influxdb command and it gave me https://i.imgur.com/esEZThJ.png I don’t understand how it doesn’t have any data from my ruuvitag(Just gives some random default data.) I have the mac address everywhere I could think of, for example in the ruuviCollector properties file under whitelist. The ruuvitag is using RAW mode, as I suppose it should be.

Any thoughts are appreciated, thanks.

PasiK · April 4, 2019, 2:32pm

pi@raspberrypi:~ $ influx
Connected to http://localhost:8086 version 1.5.2
InfluxDB shell version: 1.5.2
use ruuvi
Using database ruuvi
show measurements
name: measurements
name

ruuvi_measurements
show series
key

ruuvi_measurements,dataFormat=3,mac=AABBCCDDEEFF
ruuvi_measurements,dataFormat=3,mac=BBBBCCDDEEFF

I don’t know why my macs are abc’s.

Yeah the grafana has the same details as ruuvicollector

I haven’t changed any database related options from the properties file

PasiK · April 4, 2019, 2:50pm

So the hcitool works before running the ruuvicollector, like here https://i.imgur.com/pKF3dUJ.png But after running it, hcitool lescan gives an input/output error. For that I found another ruuviforum post and there someone suggested going to bluetoothctl console and powering on power, agent and scan, and reboot the machine after that. After doing all that,

hcitool lescan --duplicate --passive

works, but

hcidump --raw

stayed blank, before starting RuuviCollector.

After starting ruuvicollector, the “hcitool lescan” doesn’t work, but “hcidump --raw” gives the same blank window as before starting RuuviCollector. Am I missing something?

ejk · April 4, 2019, 2:53pm

dgerman) meant that run the command in his post which is number 211 in this thread. Here is link to the post: -> Collecting RuuviTag measurements and displaying them with Grafana

dgerman · April 4, 2019, 2:56pm

Pasik:

use

hcitool dev

if there are no devices shown, you don’t have a bluetooth device!
You should see something like

Devices:
hci0 B8:27:EB:E3:A4:6C

Add these 5 lines to /home/pi/RuuviCollector/RuuviCollector/.sh BEFORE the java statement

for i in 240 220 200 180 160 140 120 100 80 60 40 20
do
logger -p user.notice " ++ RuuviCollector sleeping $i more seconds to give InfluxDB a start."
sleep 20
done
Then

sudo shutdown --reboot now " add delay to Ruuvi collector"

`
4) When the system come back up:

hcitool lescan & hcidump -t --raw  # this starts an asynchronous lecscan and hcidump

You should see something like:

[1] 812
LE Scan …
HCI sniffer - Bluetooth packet analyzer ver 5.43
device: hci0 snap_len: 1500 filter: 0xffffffff
9C:9C:1D:0E:58:EB (unknown)
2019-04-04 10:51:51.507148 > 04 3E 0F 02 01 00 00 EB 58 0E 1D 9C 9C 03 02 01 06 A8
2019-04-04 10:51:51.508775 > 04 3E 24 02 01 04 00 EB 58 0E 1D 9C 9C 18 02 0A 00 14 09 44
65 6E 6E 69 73 20 48 65 61 72 69 6E 67 20 41 69 64 73 A8
59:B0:BE:CC:D6:9B (unknown)
2019-04-04 10:51:51.509917 > 04 3E 23 02 01 00 01 9B D6 CC BE B0 59 17 02 01 06 13 FF 4C
00 0C 0E 08 54 4D 28 78 89 DF 8F 09 AA 19 32 78 4D B4
2019-04-04 10:51:51.510526 > 04 3E 0C 02 01 04 01 9B D6 CC BE B0 59 00 B4
63:11:1F:F4:18:FD (unknown)
2019-04-04 10:51:51.542180 > 04 3E 1A 02 01 00 01 FD 18 F4 1F 11 63 0E 02 01 1A 0A FF 4C
00 10 05 13 1C C7 20 41 AE
4C:32:75:97:3B:AE (unknown)

+++
Note: once a process issues hcitool lescan subsequent attempts will file with the error. A shutdown may be needed to clear this issue.

Set scan parameters failed: Input/output error

++++
Please do not post screen shots, rather copy/paste the text from your system and post that.

Typpi · May 20, 2019, 4:03pm

Hi!

I’ve been trying to find solutions for reducing the influxdb size. If there’s already a solution then can you please point me to it
My issue is that I’ve been running raspberrypi with ruuvicollector, influxdb and grafana for ~10 months now. Just this morning my grafana interface stopped working and I noticed that raspberry has totally run out of space.
Is there something I can do to downsample the data? I’ve installed a virtual machine with ubuntu and tried messing up with a copy of the database but so far haven’t got any results…

Scrin · May 20, 2019, 6:25pm

For future samples, you can easily discard some measurements already in the collector by adjusting the measurementUpdateLimit configuration property in ruuvi-collector.properties.

Custom retention policies and continuous queries for downsampling the data is another more advanced approach which lets you “gradually” downsample data over time as it gets older.

For downsampling the existing data, you need to run an “INTO query” as mentioned in this post (obviously do this before setting the retention policies to drop the oldest “full accuracy” data if you don’t want to lose historic data. Though as always, remember to take backups before experimenting)