Proposed next high-precision Data Format

Products & Services 🇬🇧 RuuviTag & RuuviTag Pro Sensors
1

As of now, the Data Format with the highest precision is Format 3, which still has some room for improvement. The format 3 uses 14 bytes out of 24 bytes maximum, so ten more bytes can be added to that. Based on discussions on Slack, some ideas that could be added/changed:

  • Add more precision to (relative) humidity, as it’s the least precise measurement at the moment, and is insufficient when used for calculating other values, such as absolute humidity and dew point.
  • Add a unique ID, as iOS devices don’t expose the MAC address of other devices so iOS devices can’t tell current format 3 tags from each other.
  • Add a uptime counter or packet sequence number counter, useful for packet de-duplication when there are multiple receivers for one database.
  • Add a movement counter. The accelerometer (LIS2DH12) supports sending interrupts from motion detection, so use these to increment a movement counter. Makes it possible to build alarms that react on even the shortest movements between actual measurements.

Based on these, I have created a proposed next data format:

edit: NOTE! Scroll down for updated drafts, this one here is outdated.

offset = description
0      = Data format (8bit) whatever the next number will be
1-2    = Temperature (16bit signed) in 0.005 degrees (-163.84 to 163.83 range)
3-4    = Humidity (16bit unsigned) in 0.0025% (0-163.84% range, though realistically 0-100%)
5-6    = Pressure (16bit unsigned) as it is in format 3
7-8    = Acceleration-X (16bit signed) as it is in format 3
9-10   = Acceleration-Y (16bit signed) as it is in format 3
11-12  = Acceleration-Z (16bit signed) as it is in format 3
13-14  = Battery voltage (16bit unsigned) as it is in format 3 (or higher precision if available? the current range goes up to 65 volts which is not realistic)
15     = Movement counter (8bit unsigned), incremented by interrupts from LIS2DH12
16-17  = Packet sequence number (16bit unsigned), each packet has this incremented by one, used for packet de-duplication and calculating packet loss
18-23  = Tag ID, the 48bit MAC address (rather than the hw ID on the nRF chip to provide compatibility with current datasets that differentiate tags based on the MAC)

Some notes/comments on this:

  • If a significantly large unique ID is to be added, there is not much room for large counters, so a time-based uptime counter is not really feasible, a running packet sequence number would be more useful.
  • Since the current format is really lossy for the temperature, it would be nice to change that and achieve more precision (and actually easier parsing in most situations).
  • Since the temperature and humidity would be changed dramatically, the old order serves no purpose, so I reordered temperature, humidity and pressure to match the order generally preferred (and also used when displayed on ruu.vi)

For sure I haven’t taken something into consideration, so please share your thoughts, comments, suggestions and ideas for this. :slight_smile:

At least one thing to discuss would be the battery voltage and the movement counter, if the battery voltage would be reduced to 1.5 bytes (12bits), it would have a range from 0V to 4.095V, which is sufficient for that, and then the remaining 0.5 byte (4 bits) could be added to the movement counter, to increase its range from 0-255 to 0-4095.
This would definitely add more complexity to parsing the values, as the value offsets would not be full bytes, but it would provide a longer time between the movement counter “resets”, but is it worth the extra complexity?

3 Likes
2

I strongly suggest you include some error flags or special values as sensors and ADC s can become “broken” either permanently or intermittently

3

Well thought out format, thanks.

One possible need would be to include TX power in the format if the TX power is adjustable in future. Maybe battery voltage or packet counter could be reduced to one byte to make room for the power?

How you would like to implement the movement counter? Number of interrupts since last transmission or something else?

4

I agree, TX power would be useful, even if it’s not directly configurable, as different builds can exist with different TX powers.

Perhaps use the last four bits of the Battery voltage for the TX power? at 1dBm increments, that would give a range from -11dBm to +4dBm (4 bits) for the TX power and 0V to 4.095V (12 bits) range for the battery voltage. And maybe rename “Battery voltage” to “Power info” or something, to make it more clear that it’s not only the voltage provided there.

I would prefer to have the movement counter as a counter that starts from 0 on tag startup, and gets incremented each time an interrupt is received, and when max value is reached, it will flip over back to 0 and continue from there.
Basically totally separate from transmissions, so that movements aren’t lost if one transmission is lost in between, and movements since last heard transmission would still be relatively easy to calculate, just subtract the previous value from the latest. (assuming the counter haven’t overflowed, though I’d imagine it’s unlikely if the transmission interval is short)
Only downside in that the movement counter is meaningless if only one packet is ever observed, though apps utilizing the non-Eddystone formats are more than likely to be listening to multiple packets per tag anyway.

5

It makes sense to have a running movement counter like you suggest.
I’d like to get a bit broader scale on TX power, I think nRF52840 can do -40 to +10 dBm. Maybe 4 dB steps, i.e. (-40 + 4 * power) rounded down?

6

Valid point. If I recall correctly, the limit for Bluetooth 5 is 20dBm, so 4 bits at 4dBm steps would nicely give a range of -40dBm to +20dBm, and I doubt anyone would ever need less than -40dBm on a Ruuvitag :thinking:

7

I think setting TX frequency with read-only characteristic would be nice.

8

I would like to suggest

  1. a battery status level (merged with the data format code) Although tracking the actual voltage is good for some evaluations a 4 bit field would be sufficient to indicate OK, notice, critical and emergency.
  2. delta X,Y,Z in mm since prior packet
  3. using a 4 byte TAG ID (2 bytes using the current battery voltage bytes)

This is detailed at http://mybeacons.info/packetFormats.html#highres

9
  1. I would prefer not to tamper with the format byte in such way, it will add unnecessary complexity in receiving systems, as they’d need to interpret multiple distinct bytes as the “same version”, and also not to exhaust available versions too soon.
  2. Assuming I understood correctly, I don’t understand how you would calculate that, since the individual measurements are from individual points in time, and the measured acceleration in proper acceleration, and not coordinate acceleration, in case I misunderstood, can you explain what you mean by “delta X,Y,Z in mm” a bit more?
  3. Assuming you mean the nRF hardware ID, the TAG ID would be a bit redundant here, if the unique MAC address is already included.

Perhaps another “status data format” could be added later that would include things like the full MAC address, full HW ID, tag revision, sensor statuses (missing/enabled/disabled/broken/something), and other “non-measurement” data that rarely changes. Would be completely optional, but firmwares and applications supporting this could then show additional information about the tag itself. Thus this format could be broadcasted (if wanted) for example once every 10 packets or even more rarely (to save battery), as that’s information that rarely changes.

10

How is the MAC acquired?
The definition of the MAC is in the format MM:MM:MM:SS:SS:SS where the first 3 bytes are the manufacture code. Since we are dealing with only one manufacture and the objective is to provide a unique device identification, only the 3 SS:SS:SS bytes are necessary.

The use of specific "out of range " values as error indicators requires no additional space and coding it really very simple. For example using final values greater than 100% for humidity. Adding an additional status packet is significantly more complicated and since as we know that packets are frequently not received this poses the risk of interpreting incorrect data as valid.

11

The MAC can be acquired for example by querying NRF_FICR->DEVICEADDR[] register (6 bytes), which contains the Static Random Bluetooth Address, which is randomly created and unique for every device, assigned during the nRF chip manufacturing process. The address is not a IEEE assigned Public Bluetooth Address, which I think is what you are referring to, so all the bytes (well, all but 2 bits) are random, and thus can’t be just discarded (without risking a collision).

Having the special values for errors still require special coding, on both sides (FW and client app). How do you intend to detect errors while querying the measurements?

An additional packet would be 100% optional, and thus not a requirement to be implemented (neither on the FW or on the client app), so being “more complicated” is not necessarily a bad thing in this case.

12

Thanks for info on DEVICEADDR.

LIS2DH12 can return error during init or during any read as in
firmware/drivers/LIS2DH12

BME280 likewise
or during SPI transfer
or during app_timer_create

It is a really bad thing to not process error information.

Regarding batter voltage: it can never go below 1.7 volts or system will shutdown, nor can it go above 3.6 volts lest system fries! SO it only requires 11bits (3600-1700= 0x76C ) as biased from 1.700 . This would allow 0-32 to represent TX power in 2 dB steps . (bit fields aren’t really that hard, just mask and shift)

13

Good point on the battery voltage.

So to sum up the latest ideas/comments/suggestions here and on Slack after the original post above, the latest proposed format would be something like:

offset = description
0      = Data format (8bit) whatever the next number will be
1-2    = Temperature (16bit signed) in 0.005 degrees (-163.84 to 163.83 range)
3-4    = Humidity (16bit unsigned) in 0.0025% (0-163.83% range, though realistically 0-100%)
5-6    = Pressure (16bit unsigned) as it is in format 3
7-8    = Acceleration-X (16bit signed) as it is in format 3
9-10   = Acceleration-Y (16bit signed) as it is in format 3
11-12  = Acceleration-Z (16bit signed) as it is in format 3
13-14  = Power info (11+5bit unsigned), first 11bits unsigned is the battery voltage above 1.6V, in millivolts (1.6V to 3.647V range). last 5 bits unsigned is the TX power above -40dBm, in 2dBm steps. (-40dBm to +20dBm range)
15     = Movement counter (8bit unsigned), incremented by interrupts from LIS2DH12
16-17  = Packet sequence number (16bit unsigned), each packet has this incremented by one, used for packet de-duplication and calculating packet loss and approximate transmission interval
18-23  = Tag ID, the 48bit MAC address (rather than the hw ID on the nRF chip to provide compatibility with current datasets that differentiate tags based on the MAC)

If special values are to be added, the maximum value for unsigned values and the minimum value for signed values would mean the reading is not available, either because it’s not supported or because the sensor can’t be read.

14

Does this look correct-ish?

/**
 *  Parses sensor values into propesed format. 
 *  Note: calling this function has side effect of incrementing packet counter
 *  Changes values in "environmental" as they're paresed to ruuvi format
 *
 *  @param data_buffer uint8_t array with length of 24 bytes
 *  @param environmental  Environmental data as data comes from BME280, i.e. uint32_t pressure, int32_t temperature, uint32_t humidity
 *  @param acceleration 3 x int16_t having acceleration along X-Y-Z axes in MG. Low pass and last sample are allowed DSP operations
 *  @param acceleration_events counter of acceleration events. Events are configured by application, "value exceeds 1.1 G" recommended.
 *  @param vbatt Voltage of battery in millivolts
 *  @param tx_pwr power in dBm, -40 ... 16
 *
 */
void encodeToRawFormat5(uint8_t* data_buffer, bme280_data_t* environmental, acceleration_t* acceleration, uint16_t acceleration_events, uint16_t vbatt, int8_t tx_pwr)
{
static uint16_t packet_counter = 0;
data_buffer[0] = RAW_FORMAT_2;
environmental->temperature *= 2; //Spec calls for 0.005 degree resolution, bme280 gives 0.01
data_buffer[1] = (environmental->temperature)>>8;
data_buffer[2] = (environmental->temperature)&0xFF;
//Convert humidity from 1/1024 to 1/400 - TODO check for overflows
environmental->humidity *= 1024;
environmental->humidity /= 400; 
data_buffer[3] = (environmental->humidity)>>8;
data_buffer[4] = (environmental->humidity)&0xFF;
environmental->pressure = (uint16_t)((environmental->pressure >> 8) - 50000); //Scale into pa, Shift by -50000 pa as per Ruu.vi interface.
data_buffer[5] = (environmental->pressure)>>8;
data_buffer[6] = (environmental->pressure)&0xFF;
data_buffer[7] = (acceleration->x)>>8;
data_buffer[8] = (acceleration->x)&0xFF;
data_buffer[9] = (acceleration->y)>>8;
data_buffer[10] = (acceleration->y)&0xFF;
data_buffer[11] = (acceleration->z)>>8;
data_buffer[12] = (acceleration->z)&0xFF;
//Bit-shift vbatt by 4 to fit TX PWR in
vbatt -= 1700; //Bias by 1700 mV
vbatt <<= 5;   //Shift by 5 to fit TX PWR in
data_buffer[13] = (vbatt)>>8;
data_buffer[14] = (vbatt)&0xFF; //Zeroes tx-pwr bits
tx_pwr += 40;
tx_pwr /= 2;
data_buffer[14] |= (tx_pwr)&0x1F; //5 lowest bits for TX pwr
data_buffer[15] = acceleration_events % 256;
data_buffer[16] = packet_counter>>8;
data_buffer[17] = packet_counter&0xFF;
packet_counter++;
data_buffer[18] = ((NRF_FICR->DEVICEADDR[1]>>16)&0xFF) | 0xC0; //2 MSB of address are set to 11 by BLE Stack, see Bluetooth Core v4.0, Vol 3, Part C, chapter 10.8.1.
data_buffer[19] = ((NRF_FICR->DEVICEADDR[1]>>24)&0xFF);
const uint32_t address0 = NRF_FICR->DEVICEADDR[0]; //FICR is volatile and therefore incompatible with memcpy
memcpy(&(data_buffer[20]), &address0, 4);
}
15

On a really quick look (without checking what kinda stuff the bme data and others actually contain), it looks correct-ish, though I’d prefer to have the voltage range from 1.6V to 3.647V rather than 1.7V to 3.747V, as I find the unlikely value of 1.6V more probable than 3.7V for the 3.3V battery. Not entirely sure how the packet counter would behave when it overflows though, but format-wise that looks correct-ish.

16

Thanks, I’ll adjust the voltage and make packet counter uint32_t just in case.
Do you already have settings for lis2dh12 interrupts? I was thinking about using interrupt 2 with high-pass filtering, maybe 0.1 G, both up and down.

17

uint32_t should be sufficient, based on my quick calculation, you’d have to run with 60ms interval for 8 years without reboots to overflow the counter. 16bit would overflow in less than 2 hours at 100ms interval. :stuck_out_tongue: I haven’t gotten that far with my adventures to the world of LIS2DH12 interrupts yet, so I don’t have anything usable yet.

18

Here’s a compiled, but not tested package:


Source:

It’s based on merge-drivers pull request. Changes:

  • BME280.h: Added bme280_environmental_t struct which contains raw values
  • LIS2DH12 driver: Added support for 2nd interrupt pin
  • sensortag.c/h added support for new format
    *main.c Configure interrupt, call updated format.

Raw data as sent by tag:
ADVERTISEMENT
(flags) 0x02 01 06
(data) 0x1B FF 99 04 05 13 7E 45 0A C3 C3 00 84 00 70 04 14 9D F6 01 00 BC FB 01 7A 3A 64 E9
Translates to:
Length 27 (base 10)
Type FF manufacturer specific
Manufacturer ID 0499 (Ruuvi)
Format 05 (Raw v2)
Temperature 137E (4990, * 0.005 = 24,95) // Ok
Humidity 450A (17675, * 0.0025 = 44,1875) // Most of my tags show 30 - 40, but maybe room I’m in is more humid?
Pressure C3C3 (50115, + 50000 = 100115) // Fits well the other tags
X-Y-Z acceleration 0084 0070 0414 // Bit over 1G, I have to check the scaling
Battery 9DF6 (04EF, 1263 + 1600 = 2863) // Ok
TX Power F6 (16, 22 *2 -40 = 4) // ok
Activity events 01 // Was hardcoded, fix in 2.1.1
Sequence number 00BC // Could be
ID FB 01 7A 3A 64 E9 // MAC is EA 17 E9 64 3A 7A, fix in 2.1.1

SCAN RESPONSE
(length) 03
data 0x19 00 00
type 0x19 (appearance)
value 00 00 // Generic

(length) 0x0A
(data) 0x09 52 75 75 76 69 29 35 38 63 37
type 09 //Full name
Value 52 75 75 76 69 29 35 38 63 37 // Ruuvi 58c7

I’ll post again once fixes above are done

p.s.
Accelerometer config:

//Clear memory
lis2dh12_reset();

//Wait for reboot
nrf_delay_ms(10);

//Enable XYZ axes
lis2dh12_enable();

lis2dh12_set_scale(LIS2DH12_SCALE2G);

//Sample rate 10 for activity detection
lis2dh12_set_sample_rate(LIS2DH12_RATE_10);
lis2dh12_set_resolution(LIS2DH12_RES10BIT);

//XXX If you read this, I'm sorry about line below.
#include "lis2dh12_registers.h"

//Configure activity interrupt - TODO: Implement in driver, add tests.
uint8_t ctrl[1];

//Enable high-pass for Interrupt function 2
//CTRLREG2 = 0x02
ctrl[0] = LIS2DH12_HPIS2_MASK;
lis2dh12_write_register(LIS2DH12_CTRL_REG2, ctrl, 1);

//Enable interrupt 2 on X-Y-Z HI/LO
//INT2_CFG = 0x7F
ctrl[0] = LIS2DH12_HPIS2_MASK;
lis2dh12_write_register(LIS2DH12_INT2_CFG, ctrl, 1);    
//Interrupt on 64 mg+ (highpassed, +/-)
//INT2_THS= 0x04 // 4 LSB = 64 mg @2G scale
ctrl[0] = 0x04;
lis2dh12_write_register(LIS2DH12_INT2_THS, ctrl, 1);
    
//Enable LOTOHI interrupt on nRF52    
err_code |= pin_interrupt_enable(INT_ACC2_PIN, NRF_GPIOTE_POLARITY_LOTOHI, lis2dh12_int2_handler);

//Enable Interrupt function 2 on LIS interrupt pin 2 (stays high for 1/ODR)
lis2dh12_set_interrupts(LIS2DH12_I2C_INT2_MASK, 2);
19

Looks very promising. I’ll test this as well when I get home

20

2.1.1 Has acceleration and MAC address fixed, sources & release are at Github
[EDIT] 2.1.2 has bugfix in accelerometer config