How to decode the raw value in MQTT?

I’ve setup my RuuviTag gateway to publish to my own MQTT server in order to get the data directly into Home-Assistant, but I have some trouble decoding the data…

If I do:

from ruuvi_decoders import Df5Decoder

decoder = Df5Decoder()
data = decoder.decode_data('0201061BFF99040505C64D62CD62026C0320FFF4A2961DFBD7CF69C153A401')
print(data)

Looks like the payload their is not exactly the same as it would get from BT directly? I am unsure how to proceed from here, maybe someone can point me in the right direction? :wink:

ruuvi_decoders seem to only want the data after the companyID
This should work:

from ruuvi_decoders import Df5Decoder

decoder = Df5Decoder()
data = '0201061BFF99040505C64D62CD62026C0320FFF4A2961DFBD7CF69C153A401'
data = data.split("FF9904")[1]
sensor = decoder.decode_data(data)
print(sensor)

Edit:
You can see more info about the data here Bluetooth advertisements - docs

2 Likes

That works like a charm! Thanks for the fast response!

Hi, how did you make the Home Assistant integration? I can receive similar data by adding these lines into configuration.yaml:

sensor:

  • platform: mqtt
    name: ruuvi
    state_topic: “ruuvi/#”

The data looks like this:

{ “gw_mac”: “xx:xx:xx:xx:xx:xx”, “rssi”: -35, “aoa”: [], “gwts”: “1631217043”, “ts”: “1631217043”, “data”: “02010415FF9904034A1C14B706016CFFC0FC440BCB00000000”, “coords”: “” }

By adding ruuvi/sensor_mac state_topic into config file does not work.

BR,
Ilkka

So, what I did is use pyscript (it’s a custom integration that you can install with HACS that let’s you write your own python automations). In python I then read the MQTT data, decode it and send it back to MQTT for Hass to pickup as sensors (with auto-discovery).

If you know you way around pyscript I’m happy to send you the code.

1 Like

Hi

Yes, lease send the code. I will study the pyscript integration in the evening. Have never made similar things but anyway I can make sw… Thank you very much and have a nice day,

Ilkka

I received my gateway yesterday and am successfully sending encoded sensor data via MQTT to Mosquitto. If you’d be willing to share your pyscript code with me as well I’d really appreciate it - thanks!

Here it is… it’s rough around the edges, no guarantees other than “works on my machine”…

from ruuvi_decoders import Df5Decoder

decoder = Df5Decoder()


def sensor_exists(mac):
    try:
        sensor = f"sensor.{mac}"
        return state.get(sensor) != "unavailable"
    except (NameError):
        return False


@mqtt_trigger("ruuvi/#")
def mqtt_msg(topic=None, payload=None, payload_obj=None):
    if payload_obj.get("data") is not None:
        try:
            raw_data = payload_obj.get("data")
            clean_data = raw_data.split("FF9904")[1]
            data = decoder.decode_data(clean_data)
            # log.info(data)
            MAC = data.get("mac")
            TEMPERATURE = data.get("temperature")
            if sensor_exists(MAC):
                STATE_TOPIC = f"homeassistant/sensor/{MAC}/state"
                mqtt.publish(topic=STATE_TOPIC,
                             payload=TEMPERATURE, retain='True')
            else:
                TOPIC = (f"homeassistant/sensor/{MAC}/config")
                PAYLOAD = (f'{{"name": "{MAC}", "device_class": "temperature",'
                           '"unit_of_measurement": "°C",'
                           f'"uniq_id":"{MAC}",'
                           f' "state_topic": "homeassistant/sensor/{MAC}/state"}}')

                mqtt.publish(topic=TOPIC, payload=PAYLOAD, retain='True')
        except (AttributeError, ValueError, TypeError):
            log.warning(f'Error --> {payload}')

2 Likes

Sorry for the late reply… it’s here in the forum now!

1 Like

Awesome, thanks!!

I made a few updates based on my own use case (of course) this afternoon. It’s a little hacky, I’ll try to clean it up a bit if I get a chance. The gist is that I was using balda’s awesome ruuvitag-discovery addon previously, which has really neat features like friendly names and the ability to perform math on the metrics (i.e. convert temps to F because the US is stuck in the middle ages). I also have a few tags running old firmware that are hard to reach, so I have a mix of format type 3 and type 5. Here’s the script I’m using now, works perfectly on my system. Many thanks for sharing, I never would have been able to do this in a couple of hours without your original script to work from. :smile:

from ruuvi_decoders import Df3Decoder, Df5Decoder
from json import dumps


MQTT_TOPIC_PREFIX = "ruuvi"
TEMPERATURE_UNITS = "F" # C for Celsius or F for Fahrenheit
DECIMAL_PLACES = 2
BATTERY_RANGE = {"min":2500,"max":3000}
SENSOR_FRIENDLY_NAMES = {"f33c5fc10393":"Living Room",
                         "dc757bb86337":"Middle Bedroom",
                         "f9fd6a322a01":"Front Bedroom",
                         "dc2c241a4b24":"Kitchen",
                         "ef336477a31a":"Back Bedroom"}
MEASUREMENT_CONFIG = {"temperature": {"stat_t":"homeassistant/sensor/<MAC_ADDRESS>/temperature/state",
                                      "json_attr_t":"homeassistant/sensor/<MAC_ADDRESS>/temperature/attributes",
                                      "name":"<SENSOR_FRIENDLY_NAME> Temperature",
                                      "unit_of_meas":f"°{TEMPERATURE_UNITS}",
                                      "dev_cla":"temperature",
                                      "uniq_id":"ruuvitag_<MAC_ADDRESS>_temperature",
                                      "device":{"ids":"ruuvitag-<MAC_ADDRESS>",
                                                "mf":"Ruuvi Innovations Ltd",
                                                "mdl": "RuuviTag",
                                                "name": "<SENSOR_FRIENDLY_NAME>"}},
                      "humidity": {"stat_t":"homeassistant/sensor/<MAC_ADDRESS>/humidity/state",
                                   "json_attr_t":"homeassistant/sensor/<MAC_ADDRESS>/humidity/attributes",
                                   "name":"<SENSOR_FRIENDLY_NAME> Humidity",
                                   "unit_of_meas":"%",
                                   "dev_cla":"humidity",
                                   "uniq_id":"ruuvitag_<MAC_ADDRESS>_humidity",
                                   "device":{"ids":"ruuvitag-<MAC_ADDRESS>",
                                             "mf":"Ruuvi Innovations Ltd",
                                             "mdl": "RuuviTag",
                                             "name": "<SENSOR_FRIENDLY_NAME>"}},
                      "battery_level": {"stat_t":"homeassistant/sensor/<MAC_ADDRESS>/battery_level/state",
                                        "json_attr_t":"homeassistant/sensor/<MAC_ADDRESS>/battery_level/attributes",
                                        "name":"<SENSOR_FRIENDLY_NAME> Battery Level",
                                        "unit_of_meas":"%",
                                        "dev_cla":"battery",
                                        "uniq_id":"ruuvitag_<MAC_ADDRESS>_battery",
                                        "device":{"ids":"ruuvitag-<MAC_ADDRESS>",
                                                  "mf":"Ruuvi Innovations Ltd",
                                                  "mdl": "RuuviTag",
                                                  "name": "<SENSOR_FRIENDLY_NAME>"}}
                     }


def sensor_exists(mac):
    try:
        sensor = f"sensor.{mac}"
        return state.get(sensor) != "unavailable"
    except (NameError):
        return False


@mqtt_trigger(f"{MQTT_TOPIC_PREFIX}/#")
def mqtt_msg(topic=None, payload=None, payload_obj=None):
    if payload_obj.get("data") is not None:
        try:
            raw_data = payload_obj.get("data")
            clean_data = raw_data.split("FF9904")[1]

            # decode the data based on the format field
            format = clean_data[0:2]
            data = {}
            if format == "03":
                decoder = Df3Decoder()
                data = decoder.decode_data(clean_data)
            else:
                decoder = Df5Decoder()
                data = decoder.decode_data(clean_data)

            # get the mac from the topic (data format 3 doesn't include the mac)
            raw_mac = topic.split('/')[-1]
            mac = raw_mac.lower().replace(':','')

            measurements = {}
            if "temperature" in MEASUREMENT_CONFIG:
                # get the temperature
                temperature = round(data.get("temperature"), DECIMAL_PLACES)
                if TEMPERATURE_UNITS == "F":
                    temperature = (temperature * (9/5) + 32)
                measurements["temperature"] = round(temperature, DECIMAL_PLACES)

            if "humidity" in MEASUREMENT_CONFIG:
                # get the humidity
                measurements["humidity"] = round(data.get("humidity"), DECIMAL_PLACES)

            if "battery_level" in MEASUREMENT_CONFIG:
                # get the battery and convert it to a percentage
                battery_level = round((data.get("battery") - BATTERY_RANGE["min"]) * 100 / (BATTERY_RANGE["max"] - BATTERY_RANGE["min"]))
                battery_level = max(1, battery_level)
                battery_level = min(100, battery_level)
                measurements["battery_level"] = round(battery_level, DECIMAL_PLACES)

            for measurement,value in measurements.items():
                if not sensor_exists(mac):
                    config_topic = f"homeassistant/sensor/{mac}/{measurement}/config"
                    config_payload = dumps(MEASUREMENT_CONFIG[measurement]).replace("<MAC_ADDRESS>",mac)
                    config_payload = config_payload.replace("<SENSOR_FRIENDLY_NAME>",SENSOR_FRIENDLY_NAMES[mac])
                    mqtt.publish(topic=config_topic, payload=config_payload, retain='True')

                state_topic = f"homeassistant/sensor/{mac}/{measurement}/state"
                mqtt.publish(topic=state_topic, payload=measurements[measurement], retain='True')

        except (AttributeError, ValueError, TypeError):
            log.warning(f'Error --> {payload}')
3 Likes

Very nice improvements!! I think I am going to use this, to expand my script. One tip: pyscript has a nice option where it can read application configuration from config.yaml I would suggest putting all the static configuration in there, it makes for a cleaner python script.

Using “device” in the auto-discovery is really nice! Thanks!

I rewrote things a bit and stuck it on github. I added a basic evaluation function to be able to write rudimentary transformation logic in the measurement definitions. Originally I just used eval, but the Internet said that was “evil” (the Internet’s words, not mine) so I copied that ast-based solution straight off of stackoverflow.

Good callout on the config.yaml option, I’ll check it out.

Oh, one thing that’s quite strange that I haven’t figured out is that I moved the ruuvi_decoders to /config/pyscript/modules from a different directory, and somewhere the format 5 decoder broke on the get_mac function:

2021-09-15 01:47:58 ERROR (MainThread) [custom_components.pyscript.file.ruuvi-gateway.mqtt_msg] Exception in <file.ruuvi-gateway.mqtt_msg> line 75:

return ‘’.join(‘{:02x}’.format(x) for x in data[10:])

NotImplementedError: file.ruuvi-gateway.mqtt_msg: not implemented ast ast_generatorexp

I ended up disabling that function in the main data parsing function in the module for now, but eventually I need to figure that out.

1 Like

Hi, thank you very very much. It is working :slight_smile:

Thanks again for the tip on the config file @sj3fk3. I updated the script tonight to move the entire configuration to config/configuration.yaml; while I’m not a fan of YAML in general, it probably does make it more readable/accessible to people who don’t feel as comfortable editing python dictionaries to add measurements or tags.

I forgot to mention before that I also added some rudimentary arithmetic expression evaluation, which you can define in the config using a “transform” key for a particular measurement. This should make it a little more flexible…you can convert units pretty easily, or apply offsets if you wanted, to a measurement. And since all of that is in configuration.yaml now, it’s easier to play with. :wink:

Closing the loop a little on the above…here’s the script I wrote to parse the raw MQTT messages and send them back in to Home Assistant. All comments/suggestions/PRs happily considered. :smile:

On more tip: you can actually put ruuvi_decoders in requirements.txt in the pyscript root folder and it will automatically install the dependency. You don’t have to manually install it in modules/

Nice, documentation updated!

Sorry for kind of highjacking this thread, but did anybody manage to decode the mqtt message in node red? To be honest, that was my initial idea why I bought the ruuvi gateway.

Answering it myself … the ruuvi node was “recently” updated, so that it also can process the mqtt messages.
Node Red Ruuvi Node