Gbcsdpd: daemon publishing measurements via MQTT (+dashboard setup)

Ruuvi 🇬🇧 Use Cases (examples, ideas, solutions)
1

Hi!

I’ve open-sourced a simple daemon that listens for Ruuvi advertisements and published them over MQTT protocol: github.com/p2004a/gbcsdpd

It’s target platform is Linux and the only dependency is Bluez D-Bus API.

The repository also contains instruction with Terraform configuration to setup a full monitoring dashboard on Google Cloud.

I’ve started this side project around June 2020, committed some code from time to time until October 2020 when it was finished end-to-end and I’ve used successfully since then. This week I’ve only polished it a bit by adding better comments, some docs etc.

Why? Primarily it was a fun project and I was able to try out a few things :slight_smile: . More reasons are:

  • I didn’t like dependencies other collectors brought with them, eg using deprecated hcitool (I see that now there is github.com/jiilaa/RuuviCore by @jommy) or requiring some expensive runtime. This collector depends only on actively maintained official Bluez D-Bus API.
  • I wanted it to have a small footprint and be very easy to deploy. My target device for this collector is my OpenWrt router: both memory and disk usage counts. Go allows for easy cross-compilation, and still has okish footprint to run it on my router: binary is 9.2MiB and it’s using ~12MiB of RAM (bluetooth Bluez + dbus daemons consume ~70MiB). (The alternative to Go I’ve considered was C with libdbus (or just use kernel interface directly), libmosquitto and libjose as dependencies)
  • I really wanted something very, very simple: just gather measurements and push them to some other place.

While developing it I also tried out a few things:

  • Play a bit with GCP: implemented monitoring dashboard on it with Infrastructure as Code approach with Terraform. If you decrease deamon publishing frequency to eg 1 in 2 min (it’s a feature in the daemon and it’s perfectly fine for my home monitoring) it fits fully in the free tier.
  • Try out Bazel in OSS project: looking back, IMHO not worth for such a small project. Was interesting to try out anyway.

Happy to hear any feedback :slight_smile: .

3 Likes
2

The link to your repository has a trailing period (dot) resulting in a 404

3

Whoops, fixed, thanks!

4

Hey, firstly I want to say thank you for coming up with a solution that does not rely on hcitools or lescan. While these deprecated packages may be useful for integration on older systems and microcontrollers like ESP32 and Pi, I was not able to get any of them working reliably on a new NUC10i5 running ESXi 7.0 U2 hypervisor and an Ubuntu 20.04 VM. There seems to be a lot of issues with newer BT hardware going offline after a short period of time, or outright not working with commands that rely on hcitool like lescan (I/O errors) . The NUC10 uses a combined Wifi/BT5.0 module AX201, and I was finally able to get reliable readings with it for more than a day now using your project.

I’ve tried things like bt-mqtt-gateway, which worked intermittently, but never for more than a few hours without taking the BT adapter down or losing readings. Then I tried out ruuvitag-discovery, which was also unreliable and uses Noble. My interim solution was using ruuvi-collector on an ESP32.

Do you by chance know how one would go about decoding the values this program transmits within node-red using a function block? I was also wondering if there is any provisions for a MAC address whitelist to improve scanning efficiency?

Thanks again!

1 Like
5

I’m happy that it works for you!

Do you by chance know how one would go about decoding the values this program transmits within node-red using a function block?

I’ve never used node-red but I will try. So, I assume that you have:

  • Set up gbcsdpd and it’s publishing measurements to some MQTT broker (e.g. localhost mosquitto)
  • You import those measurements to the node-red using Connect to an MQTT Broker : Node-RED

The default encoding of published messages is binary serialization of protobuf defined here climate.proto, but there is also an option in config to serialize to JSON instead which is easy to load in js. See here how to change format: test1/config.toml:14

The published json message looks like this:

{"measurements":[{"sensorMac":"f2:6c:11:34:1e:e1","temperature":22.83,"humidity":58.885,"pressure":974.82,"batteryVoltage":2.239}]}

Does it help?

I was also wondering if there is any provisions for a MAC address whitelist to improve scanning efficiency?

Not really. I set up the scanning with Bluez and it just publishes all changes to all bluetooth devices it received packets from. Current Bluez API doesn’t allow setting up filters on MAC.
I can add a filter option for a use-case where there are many Ruuvi tags and you would like to publish measurements only from a subset of them, if it’s what you would like to have. That will not improve scanning efficiency though.

Please reach out if you have any more questions/something doesn’t work as intended!

1 Like
6

Yup, I’m using mosquitto as the MQTT broker and read the value into node-red using a MQTT subscribe node. Currently I’m working with 3 ruuvitags and this is exactly what I’m looking for! I should be able to figure out how to parse the JSON message.

Great work and thanks so much for the help.

7

Hey p2004a, I was hoping you could provide a bit more detail on how to get this setup as a systemd service. When following the instructions in the /init/ folder, I am running into some errors including: gbcsdpd.service: Failed to execute command: Permission denied and gbcsdpd.service: Failed at step EXEC spawning /usr/local/bin/gbcsdpd: Permission denied when attempting to start the service using systemctl enable gbcsdpd.service and checking it with systemctl status gbcsdpd.service

I’m likely doing something wrong when moving files and directories.

Maybe you can help me clear up some of these instructions:

  1. Put the gbcsdpd binary in /usr/local/bin
    -When it is referring to binary, am I supposed to move the directory of gbcsdpd/cmd/gbcsdpd/ into /usr/local/bin/gbcsdpd/, or should I be moving the entire file structure of the program ie /gbcsdpd/
    -Are there any special commands to make this move? I have been trying to use the mv command

  2. Create a /usr/local/etc/gbcsdpd/config.toml configuration file
    -I had first created the folder via mkdir /usr/local/etc/gbcsdpd, then created a file using touch config.toml. Is this the correct method to create a file in this directory and do any special user permissions need to be associated with it using chmod?

  3. Copy gbcsdpd.service to /usr/local/lib/systemd/system
    -again created the folders systemd and system using mkdir, then moved the gbcsdpd.service using cp

I’m in the process of learning how to use a Linux terminal and any suggestions are much appreciated!

Edit: after trying to give permission to the folders using chmod, I’m still running into the permission errors, here is a bit more detail:

â—Ź gbcsdpd.service - gbcsdpd Daemon
     Loaded: loaded (/usr/local/lib/systemd/system/gbcsdpd.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2021-11-24 11:08:27 MST; 7s ago
    Process: 19825 ExecStart=/usr/local/bin/gbcsdpd -config /usr/local/etc/gbcsdpd/config.toml -logtime false (code=exited, status=203/EXEC)
   Main PID: 19825 (code=exited, status=203/EXEC)
1 Like
8

Hi! Thank you for the details on what you did, including the systemctl status output!

I think the issue is in the first point, where you copy the binary to /usr/local/bin. You should only copy the executable binary file that you’ve built with bazel/go build, not any directory.

I’ve updated the instructions in the repository gbcsdpd/init/systemd/README.md with exact commands and expected output. I’ve tested it on my machine end to end, I hope that it will also work for you.

1 Like
9

This definitely clears things up for me. The information you have been providing is invaluable!

1 Like
10

I got the systemd service up and running quickly with your new instructions. One thing I ran into while setting this up on a new machine, was an incompatible golang version. I resolved it by manually installing the required 1.15 version:

sudo wget https://golang.org/dl/go1.15.5.linux-amd64.tar.gz	
sudo tar -C /usr/local -xzf go1.15.5.linux-amd64.tar.gz	
export PATH=$PATH:/usr/local/go/bin	
source ~/.bashrc	
go version

Regarding integration into node-red, I managed to parse out the combined data using switch and change nodes. You can filter out the message by MAC address of the ruuvitag, and then pull the data useful to you as objects. Make sure the MQTT subscribe node is set to a parsed JSON object:

image
image811Ă—194 12.7 KB

NR code:

[
    {
        "id": "e52888a3e2a1fd78",
        "type": "mqtt in",
        "z": "549f295024452a8e",
        "name": "gbcsdpd daemon",
        "topic": "/measurements",
        "qos": "2",
        "datatype": "json",
        "broker": "c2a6015cc109d5c2",
        "nl": false,
        "rap": true,
        "rh": 0,
        "inputs": 0,
        "x": 240,
        "y": 380,
        "wires": [
            [
                "72b8371342cbe8cc"
            ]
        ]
    },
    {
        "id": "72b8371342cbe8cc",
        "type": "switch",
        "z": "549f295024452a8e",
        "name": "",
        "property": "payload.measurements[0].sensorMac",
        "propertyType": "msg",
        "rules": [
            {
                "t": "eq",
                "v": "e5:c1:3d:06:73:84",
                "vt": "str"
            },
            {
                "t": "eq",
                "v": "d0:1a:99:a2:78:b3",
                "vt": "str"
            },
            {
                "t": "eq",
                "v": "de:08:37:36:d6:38",
                "vt": "str"
            }
        ],
        "checkall": "true",
        "repair": false,
        "outputs": 3,
        "x": 450,
        "y": 380,
        "wires": [
            [
                "eb01f971b3e819a6"
            ],
            [
                "bef85132817d821a"
            ],
            [
                "c403d4d287d34d24"
            ]
        ]
    },
    {
        "id": "af715fc878430285",
        "type": "debug",
        "z": "549f295024452a8e",
        "name": "veg_tent_ruuvi",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "payload",
        "targetType": "msg",
        "statusVal": "",
        "statusType": "auto",
        "x": 820,
        "y": 380,
        "wires": []
    },
    {
        "id": "7e21926037607a2c",
        "type": "debug",
        "z": "549f295024452a8e",
        "name": "flower_room_ruuvi",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "payload",
        "targetType": "msg",
        "statusVal": "",
        "statusType": "auto",
        "x": 830,
        "y": 440,
        "wires": []
    },
    {
        "id": "eb01f971b3e819a6",
        "type": "change",
        "z": "549f295024452a8e",
        "name": "dtRuuvi",
        "rules": [
            {
                "t": "set",
                "p": "payload.temp",
                "pt": "msg",
                "to": "payload.measurements[0].temperature",
                "tot": "msg"
            },
            {
                "t": "set",
                "p": "payload.humidity",
                "pt": "msg",
                "to": "payload.measurements[0].humidity",
                "tot": "msg"
            },
            {
                "t": "set",
                "p": "payload.pressure",
                "pt": "msg",
                "to": "payload.measurements[0].pressure",
                "tot": "msg"
            },
            {
                "t": "set",
                "p": "payload.bat",
                "pt": "msg",
                "to": "payload.measurements[0].batteryVoltage",
                "tot": "msg"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 640,
        "y": 320,
        "wires": [
            [
                "5a27233abeb099b8"
            ]
        ]
    },
    {
        "id": "bef85132817d821a",
        "type": "change",
        "z": "549f295024452a8e",
        "name": "vtRuuvi",
        "rules": [
            {
                "t": "set",
                "p": "payload.temp",
                "pt": "msg",
                "to": "payload.measurements[0].temperature",
                "tot": "msg"
            },
            {
                "t": "set",
                "p": "payload.humidity",
                "pt": "msg",
                "to": "payload.measurements[0].humidity",
                "tot": "msg"
            },
            {
                "t": "set",
                "p": "payload.pressure",
                "pt": "msg",
                "to": "payload.measurements[0].pressure",
                "tot": "msg"
            },
            {
                "t": "set",
                "p": "payload.bat",
                "pt": "msg",
                "to": "payload.measurements[0].batteryVoltage",
                "tot": "msg"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 640,
        "y": 380,
        "wires": [
            [
                "af715fc878430285"
            ]
        ]
    },
    {
        "id": "c403d4d287d34d24",
        "type": "change",
        "z": "549f295024452a8e",
        "name": "frRuuvi",
        "rules": [
            {
                "t": "set",
                "p": "payload.temp",
                "pt": "msg",
                "to": "payload.measurements[0].temperature",
                "tot": "msg"
            },
            {
                "t": "set",
                "p": "payload.humidity",
                "pt": "msg",
                "to": "payload.measurements[0].humidity",
                "tot": "msg"
            },
            {
                "t": "set",
                "p": "payload.pressure",
                "pt": "msg",
                "to": "payload.measurements[0].pressure",
                "tot": "msg"
            },
            {
                "t": "set",
                "p": "payload.bat",
                "pt": "msg",
                "to": "payload.measurements[0].batteryVoltage",
                "tot": "msg"
            }
        ],
        "action": "",
        "property": "",
        "from": "",
        "to": "",
        "reg": false,
        "x": 640,
        "y": 440,
        "wires": [
            [
                "7e21926037607a2c"
            ]
        ]
    },
    {
        "id": "5a27233abeb099b8",
        "type": "debug",
        "z": "549f295024452a8e",
        "name": "dry_tent_ruuvi",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "payload",
        "targetType": "msg",
        "statusVal": "",
        "statusType": "auto",
        "x": 820,
        "y": 320,
        "wires": []
    },
    {
        "id": "c2a6015cc109d5c2",
        "type": "mqtt-broker",
        "name": "mosquitto",
        "broker": "localhost",
        "port": "1883",
        "clientid": "",
        "autoConnect": true,
        "usetls": false,
        "protocolVersion": "4",
        "keepalive": "60",
        "cleansession": true,
        "birthTopic": "",
        "birthQos": "0",
        "birthPayload": "",
        "birthMsg": {},
        "closeTopic": "",
        "closeQos": "0",
        "closePayload": "",
        "closeMsg": {},
        "willTopic": "",
        "willQos": "0",
        "willPayload": "",
        "willMsg": {},
        "sessionExpiry": ""
    }
]
1 Like