How to decode the raw value in MQTT?

Iā€™ve setup my RuuviTag gateway to publish to my own MQTT server in order to get the data directly into Home-Assistant, but I have some trouble decoding the dataā€¦

If I do:

from ruuvi_decoders import Df5Decoder

decoder = Df5Decoder()
data = decoder.decode_data('0201061BFF99040505C64D62CD62026C0320FFF4A2961DFBD7CF69C153A401')
print(data)

Looks like the payload their is not exactly the same as it would get from BT directly? I am unsure how to proceed from here, maybe someone can point me in the right direction? :wink:

ruuvi_decoders seem to only want the data after the companyID
This should work:

from ruuvi_decoders import Df5Decoder

decoder = Df5Decoder()
data = '0201061BFF99040505C64D62CD62026C0320FFF4A2961DFBD7CF69C153A401'
data = data.split("FF9904")[1]
sensor = decoder.decode_data(data)
print(sensor)

Edit:
You can see more info about the data here Bluetooth advertisements - docs

2 Likes

That works like a charm! Thanks for the fast response!

Hi, how did you make the Home Assistant integration? I can receive similar data by adding these lines into configuration.yaml:

sensor:

  • platform: mqtt
    name: ruuvi
    state_topic: ā€œruuvi/#ā€

The data looks like this:

{ ā€œgw_macā€: ā€œxx:xx:xx:xx:xx:xxā€, ā€œrssiā€: -35, ā€œaoaā€: [], ā€œgwtsā€: ā€œ1631217043ā€, ā€œtsā€: ā€œ1631217043ā€, ā€œdataā€: ā€œ02010415FF9904034A1C14B706016CFFC0FC440BCB00000000ā€, ā€œcoordsā€: ā€œā€ }

By adding ruuvi/sensor_mac state_topic into config file does not work.

BR,
Ilkka

So, what I did is use pyscript (itā€™s a custom integration that you can install with HACS that letā€™s you write your own python automations). In python I then read the MQTT data, decode it and send it back to MQTT for Hass to pickup as sensors (with auto-discovery).

If you know you way around pyscript Iā€™m happy to send you the code.

1 Like

Hi

Yes, lease send the code. I will study the pyscript integration in the evening. Have never made similar things but anyway I can make swā€¦ Thank you very much and have a nice day,

Ilkka

I received my gateway yesterday and am successfully sending encoded sensor data via MQTT to Mosquitto. If youā€™d be willing to share your pyscript code with me as well Iā€™d really appreciate it - thanks!

Here it isā€¦ itā€™s rough around the edges, no guarantees other than ā€œworks on my machineā€ā€¦

from ruuvi_decoders import Df5Decoder

decoder = Df5Decoder()


def sensor_exists(mac):
    try:
        sensor = f"sensor.{mac}"
        return state.get(sensor) != "unavailable"
    except (NameError):
        return False


@mqtt_trigger("ruuvi/#")
def mqtt_msg(topic=None, payload=None, payload_obj=None):
    if payload_obj.get("data") is not None:
        try:
            raw_data = payload_obj.get("data")
            clean_data = raw_data.split("FF9904")[1]
            data = decoder.decode_data(clean_data)
            # log.info(data)
            MAC = data.get("mac")
            TEMPERATURE = data.get("temperature")
            if sensor_exists(MAC):
                STATE_TOPIC = f"homeassistant/sensor/{MAC}/state"
                mqtt.publish(topic=STATE_TOPIC,
                             payload=TEMPERATURE, retain='True')
            else:
                TOPIC = (f"homeassistant/sensor/{MAC}/config")
                PAYLOAD = (f'{{"name": "{MAC}", "device_class": "temperature",'
                           '"unit_of_measurement": "Ā°C",'
                           f'"uniq_id":"{MAC}",'
                           f' "state_topic": "homeassistant/sensor/{MAC}/state"}}')

                mqtt.publish(topic=TOPIC, payload=PAYLOAD, retain='True')
        except (AttributeError, ValueError, TypeError):
            log.warning(f'Error --> {payload}')

2 Likes

Sorry for the late replyā€¦ itā€™s here in the forum now!

1 Like

Awesome, thanks!!

I made a few updates based on my own use case (of course) this afternoon. Itā€™s a little hacky, Iā€™ll try to clean it up a bit if I get a chance. The gist is that I was using baldaā€™s awesome ruuvitag-discovery addon previously, which has really neat features like friendly names and the ability to perform math on the metrics (i.e. convert temps to F because the US is stuck in the middle ages). I also have a few tags running old firmware that are hard to reach, so I have a mix of format type 3 and type 5. Hereā€™s the script Iā€™m using now, works perfectly on my system. Many thanks for sharing, I never would have been able to do this in a couple of hours without your original script to work from. :smile:

from ruuvi_decoders import Df3Decoder, Df5Decoder
from json import dumps


MQTT_TOPIC_PREFIX = "ruuvi"
TEMPERATURE_UNITS = "F" # C for Celsius or F for Fahrenheit
DECIMAL_PLACES = 2
BATTERY_RANGE = {"min":2500,"max":3000}
SENSOR_FRIENDLY_NAMES = {"f33c5fc10393":"Living Room",
                         "dc757bb86337":"Middle Bedroom",
                         "f9fd6a322a01":"Front Bedroom",
                         "dc2c241a4b24":"Kitchen",
                         "ef336477a31a":"Back Bedroom"}
MEASUREMENT_CONFIG = {"temperature": {"stat_t":"homeassistant/sensor/<MAC_ADDRESS>/temperature/state",
                                      "json_attr_t":"homeassistant/sensor/<MAC_ADDRESS>/temperature/attributes",
                                      "name":"<SENSOR_FRIENDLY_NAME> Temperature",
                                      "unit_of_meas":f"Ā°{TEMPERATURE_UNITS}",
                                      "dev_cla":"temperature",
                                      "uniq_id":"ruuvitag_<MAC_ADDRESS>_temperature",
                                      "device":{"ids":"ruuvitag-<MAC_ADDRESS>",
                                                "mf":"Ruuvi Innovations Ltd",
                                                "mdl": "RuuviTag",
                                                "name": "<SENSOR_FRIENDLY_NAME>"}},
                      "humidity": {"stat_t":"homeassistant/sensor/<MAC_ADDRESS>/humidity/state",
                                   "json_attr_t":"homeassistant/sensor/<MAC_ADDRESS>/humidity/attributes",
                                   "name":"<SENSOR_FRIENDLY_NAME> Humidity",
                                   "unit_of_meas":"%",
                                   "dev_cla":"humidity",
                                   "uniq_id":"ruuvitag_<MAC_ADDRESS>_humidity",
                                   "device":{"ids":"ruuvitag-<MAC_ADDRESS>",
                                             "mf":"Ruuvi Innovations Ltd",
                                             "mdl": "RuuviTag",
                                             "name": "<SENSOR_FRIENDLY_NAME>"}},
                      "battery_level": {"stat_t":"homeassistant/sensor/<MAC_ADDRESS>/battery_level/state",
                                        "json_attr_t":"homeassistant/sensor/<MAC_ADDRESS>/battery_level/attributes",
                                        "name":"<SENSOR_FRIENDLY_NAME> Battery Level",
                                        "unit_of_meas":"%",
                                        "dev_cla":"battery",
                                        "uniq_id":"ruuvitag_<MAC_ADDRESS>_battery",
                                        "device":{"ids":"ruuvitag-<MAC_ADDRESS>",
                                                  "mf":"Ruuvi Innovations Ltd",
                                                  "mdl": "RuuviTag",
                                                  "name": "<SENSOR_FRIENDLY_NAME>"}}
                     }


def sensor_exists(mac):
    try:
        sensor = f"sensor.{mac}"
        return state.get(sensor) != "unavailable"
    except (NameError):
        return False


@mqtt_trigger(f"{MQTT_TOPIC_PREFIX}/#")
def mqtt_msg(topic=None, payload=None, payload_obj=None):
    if payload_obj.get("data") is not None:
        try:
            raw_data = payload_obj.get("data")
            clean_data = raw_data.split("FF9904")[1]

            # decode the data based on the format field
            format = clean_data[0:2]
            data = {}
            if format == "03":
                decoder = Df3Decoder()
                data = decoder.decode_data(clean_data)
            else:
                decoder = Df5Decoder()
                data = decoder.decode_data(clean_data)

            # get the mac from the topic (data format 3 doesn't include the mac)
            raw_mac = topic.split('/')[-1]
            mac = raw_mac.lower().replace(':','')

            measurements = {}
            if "temperature" in MEASUREMENT_CONFIG:
                # get the temperature
                temperature = round(data.get("temperature"), DECIMAL_PLACES)
                if TEMPERATURE_UNITS == "F":
                    temperature = (temperature * (9/5) + 32)
                measurements["temperature"] = round(temperature, DECIMAL_PLACES)

            if "humidity" in MEASUREMENT_CONFIG:
                # get the humidity
                measurements["humidity"] = round(data.get("humidity"), DECIMAL_PLACES)

            if "battery_level" in MEASUREMENT_CONFIG:
                # get the battery and convert it to a percentage
                battery_level = round((data.get("battery") - BATTERY_RANGE["min"]) * 100 / (BATTERY_RANGE["max"] - BATTERY_RANGE["min"]))
                battery_level = max(1, battery_level)
                battery_level = min(100, battery_level)
                measurements["battery_level"] = round(battery_level, DECIMAL_PLACES)

            for measurement,value in measurements.items():
                if not sensor_exists(mac):
                    config_topic = f"homeassistant/sensor/{mac}/{measurement}/config"
                    config_payload = dumps(MEASUREMENT_CONFIG[measurement]).replace("<MAC_ADDRESS>",mac)
                    config_payload = config_payload.replace("<SENSOR_FRIENDLY_NAME>",SENSOR_FRIENDLY_NAMES[mac])
                    mqtt.publish(topic=config_topic, payload=config_payload, retain='True')

                state_topic = f"homeassistant/sensor/{mac}/{measurement}/state"
                mqtt.publish(topic=state_topic, payload=measurements[measurement], retain='True')

        except (AttributeError, ValueError, TypeError):
            log.warning(f'Error --> {payload}')
3 Likes

Very nice improvements!! I think I am going to use this, to expand my script. One tip: pyscript has a nice option where it can read application configuration from config.yaml I would suggest putting all the static configuration in there, it makes for a cleaner python script.

Using ā€œdeviceā€ in the auto-discovery is really nice! Thanks!

I rewrote things a bit and stuck it on github. I added a basic evaluation function to be able to write rudimentary transformation logic in the measurement definitions. Originally I just used eval, but the Internet said that was ā€œevilā€ (the Internetā€™s words, not mine) so I copied that ast-based solution straight off of stackoverflow.

Good callout on the config.yaml option, Iā€™ll check it out.

Oh, one thing thatā€™s quite strange that I havenā€™t figured out is that I moved the ruuvi_decoders to /config/pyscript/modules from a different directory, and somewhere the format 5 decoder broke on the get_mac function:

2021-09-15 01:47:58 ERROR (MainThread) [custom_components.pyscript.file.ruuvi-gateway.mqtt_msg] Exception in <file.ruuvi-gateway.mqtt_msg> line 75:

return ā€˜ā€™.join(ā€˜{:02x}ā€™.format(x) for x in data[10:])

NotImplementedError: file.ruuvi-gateway.mqtt_msg: not implemented ast ast_generatorexp

I ended up disabling that function in the main data parsing function in the module for now, but eventually I need to figure that out.

1 Like

Hi, thank you very very much. It is working :slight_smile:

Thanks again for the tip on the config file @sj3fk3. I updated the script tonight to move the entire configuration to config/configuration.yaml; while Iā€™m not a fan of YAML in general, it probably does make it more readable/accessible to people who donā€™t feel as comfortable editing python dictionaries to add measurements or tags.

I forgot to mention before that I also added some rudimentary arithmetic expression evaluation, which you can define in the config using a ā€œtransformā€ key for a particular measurement. This should make it a little more flexibleā€¦you can convert units pretty easily, or apply offsets if you wanted, to a measurement. And since all of that is in configuration.yaml now, itā€™s easier to play with. :wink:

Closing the loop a little on the aboveā€¦hereā€™s the script I wrote to parse the raw MQTT messages and send them back in to Home Assistant. All comments/suggestions/PRs happily considered. :smile:

On more tip: you can actually put ruuvi_decoders in requirements.txt in the pyscript root folder and it will automatically install the dependency. You donā€™t have to manually install it in modules/

Nice, documentation updated!

Sorry for kind of highjacking this thread, but did anybody manage to decode the mqtt message in node red? To be honest, that was my initial idea why I bought the ruuvi gateway.

Answering it myself ā€¦ the ruuvi node was ā€œrecentlyā€ updated, so that it also can process the mqtt messages.
Node Red Ruuvi Node