Collecting RuuviTag measurements and displaying them with Grafana

ssalonen · January 4, 2019, 6:05am

Excellent and thank you already in advance!

Looked at the code and I understand what you mean. Agree that this separation would indeed make the code cleaner as well as more testable.

Let me know if I can help in any way, review the code or similar. When you have separated parsers from the rest, let me know and I can have a look.

Btw, it might be that the maven central part is actually not so important for me personally since I probably have to distribute the parser .jar inside the openHAB/eclipse smarthome code base (maven <dependencies> are not used there for whatever OSGI/setup reason).

Best,
Sami

Scrin · January 4, 2019, 7:38am

Yeah now that I think about it probably the first step is to simply copy/refactor the code out into another repository as a “pre-release version”, then do the cleanup/documentation, write tests, etc. to eventually make it a “clean 1.0 version” of the library, and once all that is done then publish it to maven central or so and only then refactor RuuviCollector to use the library.

Slightly more work (and duplicate code while working on it) to do it in “individual steps over time” like this, but on the other hand it makes the library usable earlier on before the “stable 1.0 release” is ready and published to maven central. I might actually have time to do the “first step” during this weekend, if so I’ll let you know so you (or anyone else) can review, comment and/or contribute.

Scrin · January 6, 2019, 6:25pm

Alright I created the repository now: https://github.com/Scrin/ruuvitag-common-java/ and did some light and untested (well, I tested that the code compiles but nothing more ) refactoring and basically set up the base foundation for the library. In theory it might actually already work as it is, but there’s still much to do. Comments, suggestions and contributions are more than welcome (here on the forums, Ruuvi slack, or on Github)

URSALINK · January 8, 2019, 5:51am

trying to join in this discussion.

ssalonen · January 10, 2019, 5:36pm

Awesome! Will have a look as soon as I have a moment. Can contribute some tests…

ssalonen · February 3, 2019, 2:21pm

@Scrin, thanks for this!

Just wanted to update: it seems to work in real life as well – got this hooked up to openHAB.

Would appreciate if you can make a release so I would not have to use a snapshot version.

Scrin · February 16, 2019, 5:12pm

Sorry for the delay, I’ve been way too busy lately. I have now released the current version as 1.0.0

Motobiman · March 6, 2019, 6:54pm

All but given up on this.
Clearly its beyond my level of competence as the link to the raspberrypi website, here:

has several options I cant follow; I am not using a cable and whilst I can see the RuuviGW network but cant connect to and have no wif networks detected on the Rpi GUI, so I end up with no connection on the router.
Its the Rpi instructions I find difficult and obciouslly some of the changes have already been made but some are still needed, static ip addressing etc, so would it be possible for someone to clarify what specific changes to the Rpi are needed?
Thanks in advance

PasiK · March 9, 2019, 4:21pm

Hi. I don’t seem to be able to get any data to show on Grafana. The InfluxDB is on, RuuviCollector is on and listening. HCItool lescan gives Input/Output error which it didn’t give before I rebooted the machine. I tried to reset it with hciconfig hci0 reset, which worked yesterday in this same error but now it doesn’t. I made the image/built according to https://blog.ruuvi.com/rpi-gateway-6e4a5b676510 . I also tried with the latest Ruuviberry image I could find, but that also had somewhat similar problems.

I tried to fix the hcitool error with restarting bluetooth and dbus but the system went on to some weird limbo and I had to unplug the machine for it to work. I’m kind of in a dead end and would appreciate if anyone is able to suggest something.

Best,
Pasi

otso · March 11, 2019, 9:17am

Have you tried selecting a specific value inside field(value)? For example field(temperature).

ejk · March 13, 2019, 4:21pm

I’m using ruuviberry image “2018_05_minimal”, where I have updated Grafana from 5.0.4 to 5.4.3 and InfluxDB from 1.5.2 to 1.7.4 manually.

How can I show in Grafana pressure difference between two different tags?

otso · March 13, 2019, 5:09pm

As far as I know Grafana / InfluxDB does not support this in an easy manner, you’d probably need some kind of continuous query which creates difference data . @Scrin would you happen to know an elegant way to do this?

Scrin · March 13, 2019, 6:57pm

As far as I know you are correct, InfluxDB does not support on-the-fly calculation between separate measurements (pressure values from two different measurements in this case), and as far as I know, the InfluxDB builtin Continuous Queries are restricted by this same limitation. The method recommended by InfluxData is to use Kapacitor, which is their data processing engine, to perform this kind of calculations.

I used to have Kapacitor tickscripts created for calculating some of the “extra values” (such as dew point and absolute humidity), but I have since implemented them straight into the collector itself because Kapacitor tickscripts are rather clumsy to work with compared to doing the same with Java straight in RuuviCollector.

PasiK · March 23, 2019, 2:25pm

Hi,
so, adding “temperature” in “field(value)” didn’t work.

We showed the project to our teacher and his opinion was that somehow the RuuviCollector isn’t sending data. Now we are trying to build the ruuvicollector again, but now the building is giving errors/failures.

We then tried modifying the values inside ruuvicollector properties file, and with modifying “influxBatchMaxSize=2000” to 1234 and “limitingStrategy.defaultWithMotionSensitivity.threshold=0.05” to 0.06. This got rid of two “failures”. The building worked without properties file, but that didn’t give any data last time so that’s why we are trying to get this to work with the properties file.

Also, continuity to my last problem, hcitool. We noticed/remembered that line "sudo setcap ‘cap_net_raw,cap_net_admin+eip’ which hcidump gave an usage error for setcap.

dgerman · March 24, 2019, 12:46pm

PasiK

Issue

hcidump |head -30

systemctl status ruuvicollector # NB may show last failure even if ruuvicollector is running OK

systemctl status influxdb

ps -el |grep hci # find all things bluetooth

ps -el |grep Ruuvi

ls -ltra /var/log | tail # show last several most recently updated logs

and copy / paste results here ( Not screen print which omits information)

influxdb takes a while to get ready to accept transactions (especially after a powerfail)
I have found it helpful to add this to RuuviCollectorStart.sh just before java -jar target/ruuvi-collector-*.jar

for i in 240 220 200 180 160 140 120 100 80 60 40 20
do
logger " ++ RuuviCollector sleeping $i more seconds to give influx a start."
sleep 20
done

To slow down influxdb restarts after failures add this to service section of /etc/systemd/system/influxd.service

RestartSec=600

dgerman · March 24, 2019, 4:14pm

The following script displays any ruuvi tags with hex values
(There is an occasionally incorrectly formated fmt05 movement & seq# )

hcidump -t --raw |                           #include time stamps
grep --line-buffered --invert-match '\-\-' |               
sed --unbuffered "N ;s/\n//; s/   / /" |
grep --line-buffered '99 04' |
sed --unbuffered "s/04 3E .. 02 01 .. 01 \(..\) \(..\) \(..\) \(..\) \(..\) \(..\)/ \6:\5:\4:\3:\2:\1  /"|    #  pkt:Meta Event/" |
sed --unbuffered "s/1. 02 01 0. 1. FF 99 04//"|
sed --unbuffered "s/10 \(..\) 03 72 75 75 2E 76 69 2F 23 42/TX:\1 https:\/\/ruuvi\/#B/"|
sed --unbuffered "s/10 \(..\) 03 72 75 75 2E 76 69 2F 23 41/TX:\1 https:\/\/ruuvi\/#A/"|
sed --unbuffered "s/52 75 75 76 69 42 6F 6F 74/RuuviBoot\//"|

sed --unbuffered "s/   03 \(..\) /fmt03: \1% /" |
sed --unbuffered "s/fmt03: \(...\) \(..\) \(..\) \(..\) \(..\) \(..\) \(..\) \(..\) \(..\)/fmt03:\1 \2.\3C  \4\5kpa   \6\7x \8\9y03/" |
sed --unbuffered "s/y03 \(..\) \(..\) \(..\) \(..\) 00 00 00 00 ../y \1\2z  \3\4v /"      |

sed --unbuffered "s/   05 \(..\) \(..\) /fmt05: \1\2C /" |
sed --unbuffered "s/fmt05: \(....C\) \(..\) \(..\) \(..\) \(..\) \(..\) \(..\) \(..\) \(..\)/fmt05:\1 \2\3% \4\5kpa   \6\7x \8\9y05/" |
sed --unbuffered "s/y05 \(..\) \(..\) \(..\) \(..\)/y \1\2z  \3\4vTX /"      |
sed --unbuffered "s/vTX  \(..\) \(..\) \(..\) \(..\)/vTX MV:\1 #\2\3 \4/"

PasiK · March 27, 2019, 11:21am

so for the first one (I wasn’t sure how much you wanted from it so I copied everything):

pi@raspberrypi:~ $ hcidump |head -30
bash: hcidump: komentoa ei löydy
pi@raspberrypi:~ $ systemctl status ruuvicollector
Unit ruuvicollector.service could not be found.
pi@raspberrypi:~ $ systemctl status influxdb
● influxdb.service - InfluxDB is an open-source, distributed, time series databa
Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset:
Active: active (running) since Sat 2019-03-23 14:39:11 EET; 3 days ago
Docs: InfluxDB OSS v2 Documentation
Main PID: 465 (influxd)
CGroup: /system.slice/influxdb.service
└─465 /usr/bin/influxd -config /etc/influxdb/influxdb.conf

maalis 27 10:39:39 raspberrypi influxd[465]: ts=2019-03-27T08:39:39.195924Z lvl=
maalis 27 10:39:39 raspberrypi influxd[465]: ts=2019-03-27T08:39:39.196382Z lvl=
maalis 27 11:09:39 raspberrypi influxd[465]: ts=2019-03-27T09:09:39.195830Z lvl=
maalis 27 11:09:39 raspberrypi influxd[465]: ts=2019-03-27T09:09:39.196198Z lvl=
maalis 27 11:39:39 raspberrypi influxd[465]: ts=2019-03-27T09:39:39.195891Z lvl=
maalis 27 11:39:39 raspberrypi influxd[465]: ts=2019-03-27T09:39:39.196299Z lvl=
maalis 27 12:09:39 raspberrypi influxd[465]: ts=2019-03-27T10:09:39.195862Z lvl=
maalis 27 12:09:39 raspberrypi influxd[465]: ts=2019-03-27T10:09:39.198984Z lvl=
maalis 27 12:39:39 raspberrypi influxd[465]: ts=2019-03-27T10:39:39.195894Z lvl=
maalis 27 12:39:39 raspberrypi influxd[465]: ts=2019-03-27T10:39:39.196274Z lvl=

[1]+ Pysäytetty systemctl status influxdb
pi@raspberrypi:~ $ ps -el |grep hci
1 S 0 584 1 0 80 0 - 521 - ? 00:00:00 hciattach
pi@raspberrypi:~ $ ps -el |grep Ruuvi
pi@raspberrypi:~ $ ls -ltra /var/log | tail
drwxr-xr-x 7 root root 4096 maali 27 06:25 .
-rw-r----- 1 root adm 32477 maali 27 06:25 syslog.1
-rw-r----- 1 root adm 20865 maali 27 12:49 user.log
-rw-r----- 1 root adm 357508 maali 27 12:49 messages
-rw-r----- 1 root adm 357322 maali 27 12:49 kern.log
-rw-r–r-- 1 root root 47259 maali 27 12:49 Xorg.0.log
-rw-r----- 1 root adm 106570 maali 27 12:55 debug
-rw-r----- 1 root adm 48961 maali 27 13:07 auth.log
-rw-r----- 1 root adm 16622 maali 27 13:07 syslog
-rw-r----- 1 root adm 560495 maali 27 13:07 daemon.log
pi@raspberrypi:~ $ ^C
pi@raspberrypi:~ $

I can’t get status for ruuvicollector as it cannot be compiled with the errors I’m having, so it’s not running. If I recall correctly it compiles without the properties file, but I want to have it with the properties file. The errors should be in my previous command. I still have them if you would prefer them in text. For the first command

hcidump |head -30

I have the bluez installed, but I don’t know why it doesn’t work, though in the end it gave me

pi@raspberrypi:~/Documents/bluez-5.50 $ systemctl status bluetooth
● bluetooth.service - Bluetooth service
Loaded: loaded (/lib/systemd/system/bluetooth.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2019-03-27 13:07:27 EET; 8s ago
Docs: man:bluetoothd(8)
Main PID: 3473 (bluetoothd)
Status: “Running”
CGroup: /system.slice/bluetooth.service
└─3473 /usr/local/libexec/bluetooth/bluetoothd --experimental

maalis 27 13:07:26 raspberrypi systemd[1]: Starting Bluetooth service…
maalis 27 13:07:27 raspberrypi bluetoothd[3473]: Bluetooth daemon 5.50
maalis 27 13:07:27 raspberrypi systemd[1]: Started Bluetooth service.
maalis 27 13:07:27 raspberrypi bluetoothd[3473]: Starting SDP server
maalis 27 13:07:27 raspberrypi bluetoothd[3473]: Bluetooth management interface 1.14 initialized
maalis 27 13:07:27 raspberrypi bluetoothd[3473]: Failed to set privacy: Rejected (0x0b)
maalis 27 13:07:27 raspberrypi bluetoothd[3473]: Endpoint registered: sender=:1.39 path=/A2DP/SBC/Source/1

Scrin · March 27, 2019, 1:31pm

~~This is actually a bug in the tests, for now you can workaround this issue by compiling the package without running the tests with the -DskipTests flag, for example mvn package -DskipTests~~

edit: this bug has been fixed now

It looks like you have bluez installed but not bluez-hcidump. You need both (as mentioned in the requirements), the latter one provides hcidump.

It looks like you have a standard raspbian, in which case you should be able to install it with:

sudo apt install bluez-hcidump

After that:

sudo setcap 'cap_net_raw,cap_net_admin+eip' `which hcidump`

should work without any “usage errors”. After these RuuviCollector should be able to get the data.

If not, you can try troubleshooting it by first stopping ruuvicollector and then running:

hcitool lescan --duplicates --passive

in one terminal window, it should print some lines as it receives various BLE data. While that is running in one terminal window, run this in another:

hcidump --raw

this should print out all the raw data that’s received (as long as hcitool is running in another window!). If you encounter any erros with these, you could try googling for help. Once those works, then RuuviCollector should work too, as RuuviCollector runs those two commands internally.

PasiK · March 27, 2019, 4:08pm

Hey, thanks. I must have missed that bluez-hcidump installation. I’ll be installing that asap. Thanks

dgerman · March 27, 2019, 6:49pm

Pasik: Why are you trying to revise RuuviCollector before you have the version Scrin made working?