Project status: Ruuvi Gateway

Ruuvi 🇬🇧 Ruuvi Announcements
1

Many RuuviTag users could benefit an easy to use gateway which would enable reliable sensor data routing and logging.

We’ve been looking for suitable simple and super reliable WiFi-BLE gateways for a long time but have found no proper options which we could nicely bundle with RuuviTags. The problem we’ve faced pretty much with every gateway is that they even have no certifications, bad antennas or they’re too expensive.

So, seems that it’s better to build one :grin:

We have been brainstorming this idea a while and now we would like to ask for your feedback. Here are some proposed specs:

Preliminary Technical Specs:

  • USB dongle form
  • WiFi connectivity
  • Bluetooth Low Energy reception
  • High-sensitivity BLE (and WiFi) antenna to maximise RuuviTags’ range - this is important!
  • Integrated sensors
  • Expansion port - add more sensors
  • LEDs (or a tiny OLED display) - instant package reception feedback from nearby RuuviTags

We already have some ideas of which parts to use on the board but we would also like hear if you’ve suggestions to share. What kind of features you would like to have?

Preliminary Software Specs:

  • Arduino/Python/etc SDKs - Allow easy firmware hacking
  • Ruuvi MQTT server
  • Server keeps the most recent data available
  • Ruuvi Station mobile apps and 3rd party services can request data from the server

Just like with the hardware, we would be happy to receive feedback also about the software side. I also welcome @otso to tell more about the software ideas we have.

Note: Even it looks like a USB dongle that would normally need a computer to work, normal operation requires only USB power!

9 Likes
2

can you expand thoughts on the planned stand-alone capability of the dongle?
Does it only need 5V from the USB-connector, or is the idea that there is (always, or optionally?) a host computer running server/database software upstream of the USB?

slightly offtopic: can the same hardware receive data from sports/health devices using BT or ANT+ protocols? e.g. indoor cycling nerds will have power, speed, cadence, heartrate data at least streaming in during training. using a suunto movescount tiny usb-dongle currently, but an open-hardware alternative would be cool :wink:

3

By default, it will not require USB data connection but just 5V power. So, one can easily power it using a cell phone charger or any other 5V power adapter.

We’re also considering to include Ethernet (RJ45) connection and Power over Ethernet. This would make it suitable for deployments where 100% uptime is required. However, before going forward with adding Ethernet connectivity, we first need to understand on how many would actually use it over WiFi.

@awallin, the product will be 100% open source and easy to hack with so there will be no limits on what kind of BLE devices it’ll be able to listen :sunglasses: What you described is exactly what we would like it to be. A generic gateway which would support listening RuuviTags out of the box but at the same time to be a fully customisable to allow also other uses.

4

Will there also be javascript?

5

Greate idea! Some notes:

  • Powering the dongle via an external power supply or a battery is a must for me
  • Integrated sensors are not that important, as we have the RuuviTags as sensors
  • WiFi connectivity is important, but not Ethernet (at least for the first iteration)
  • Ruuvi MQTT server: Not important for me to have this integrated as I’m running my own (Mosquitto)

Keep us posted, I’m very interested in this project :smiling_face_with_three_hearts:

1 Like
6

Is there a price target?

7

@dgerman Not yet. What you think could be suitable price target?

8

I think that at least lower than the price of a Raspberry PI, otherwise it would be more viable to just buy a Raspberry PI instead as it can be set up to do the same thing, with the same USB power, ethernet, wifi and bluetooth.

1 Like
9

First: I accept the statement that a reliable gateway with great range needs a product to go with the “out of the box” Ruuvi Tag experience. The app is limited by the phone coverage, and your phone or tablet just can’t be permanently placed in ideal place.

I have some doubts that there isn’t a suitable hardware for this, but if you see the market, I’d be happy if there was a low cost product designed for this. Raspberry Pi is great, but the 3B+ is expensive, and Zero W continues to have poor availability.

For the price, somewhere at the range of below Raspberry Pi, ideally around Pi Zero W cost; 20€ with case for an USB powered device would be great IMO, otherwise probably better instead build on a software distribution for Raspberry.

I have an existing implementation of similar gateway in Go running on Alpine Linux (Raspberry Pi). I use Prometheus instead of MQTT. From your description, sounds like an easy modification to your implementation.

Thinking about a nice user experience for “off the shelf” use of Ruuvi Tags, would be really nice if the device could have storage to persist at least 24 hours so that phone doesn’t need to be in range, but still cloud is not needed. And obviously that would need to be supported natively in the Ruuvi apps.

Ethernet (and PoE) has no benefit for me. That sounds like a niche enterprise use case.

Being able to use multiple receivers, and mesh of receivers would be convenient in future, but currently I can work within range of a single Raspberry Pi Zero W reception range. Should be software only improvement to support many receivers?

Automatic software updates and automatic updates for tags would be nice. (The tags could optionally pair with the gateway/app?)

2 Likes
10

A bit more details about the firmware, what do you think?

First boot

On the first boot, the dongle will be configured in browser over LAN connection. The configuration options are: WiFi credentials, MQTT QoS level, address of the MQTT broker and minimum update interval of each tag data.

After the dongle has been configured, it will store the configuration to persistent memory and attempt to connect to the configured server with given connection credentials.

In summary, on first boot:

  • Dongle must be connectible over LAN
    • Dongle may act as a WiFi access point for connection
  • Dongle must provide a HTTP page which allows configuration to be set
  • The page shall have Ruuvi’s MQTT broker and QoS 0 as default
  • The page must have field for WIFi configuration

After the configuration, dongle must not provide access to configuration interface.

Operation

After initial configuration, Dongle will fetch current time from a NTP server. Once current time has been retrieved, the dongle will start scanning for BLE advertisements. The time will be refreshed approximately daily.

Advertisement data is sent via MQTT with topic “Ruuvi/{MAC_ADDRESS}/”. The payload will be in format {TIMESTAMP}:{HEXSTRING}.

Status of the gateway is indicated via LEDs, for example: blue -> waiting for configuration, green -> normal operation, red -> error.

Tags are scanned continuously, and tag data is stored if the previous data point does not exist or is older than minimum update interval. Data is stored to RAM, and storage is flushed after it has been sent on QoS 0 and 1 or after receipt of data has been acknowledged on QoS 2.

If error is detected, for example WiFi connection fails, error is indicated via LEDs but no other action is taken autonomously.

In summary:

  • The dongle must fetch current real time from NTP server
  • The dongle must scan for BLE advertisements from Ruuvi Devices
    • Advertisements may be identified via sequence FF9904
  • The dongle must store new data to RAM buffer if previous data point is older than minimum update interval
  • The dongle must transmit data buffer to MQTT broker with topic “Ruuvi/{MAC_ADDRESS}
  • The data shall be in format {TIMESTAMP}:{PAYLOAD}
  • Status of dongle shall be indicated via LEDs

Factory reset

The dongle must return to first boot state if user triggers factory reset via button.

Firmware update

  • The firmware must be updateable without special equipment.
  • Update may require wired USB connection, or it may be implemented over the internet.
  • Update must be triggered only after user action requiring physical access to device, i.e. button press.

Updates to specs

For up-to-date document, please see check out:

2 Likes
11

Day’s worth of data is actually quite a bit even if downsampled, and would probably require a Raspberry Pi or similar, so it may be easier to relay data via cloud

It should be easy enough to deduplicate data, especially on RAWv2 which has a measurement sequence counter.

This would cause a whole lot of security concerns, in general our opinion is that reconfiguration or updates should require physical access to device. Maybe it would be better to implement in mobile phone app, “tap with NFC to update”?

12
  1. I assume the timestamp is a “unix timestamp” in milliseconds?
  2. What exactly does the HEXSTRING contain? The entire advertisement payload? (ie. starting FF 99 04 05 for data format 5, or just the “net payload” starting with the data format byte? )
  3. Will/can the dongle support other advertisement data (with the default firmware)? ie. manufacturer specific data with a different company identifier or even different eddystone packets?

Maybe the error situations should be split into (at least) 2 categories; “fatal” and “non-fatal” where only fatal errors would prevent autonomous actions/recovery attempts, for example in case WiFi connection is lost, it should be retried rather than fully giving up straight away.

1 Like
13

  • PAYLOAD: Would this be the low level, binary encoded RAW format 3/5 data directly from the advertisement, or some higher level representation (e.g. JSON)?

  • at least for large scale deployments it would probably be useful to have a ‘DongleID’ in the data, in addition to timestamp and payload. Ideally the DongleID would configurable in the ‘first boot’ phase.

What about support for MQTTs LWT functionality?

14

It would seem best choise to me, but I’m open to all suggtestions.

I was thinking “net payload” would be best, but to support other beacons maybe the data should contain entire advertisement. This way the dongle could support iBeacons, Eddystone beacons, other manufacturers and so on.

Good idea, I’ll add the support to optional specs. What the LWT should contain in your opinion?

15

:+1: I prefer unix timestamp in milliseconds, it’s the most common “standard” that can present any reasonable timestamp with a reasonable accuracy.

Either that, or split it to {TIMESTAMP}:{HEADER}:{PAYLOAD} where the header would be ie. “FF 99 04” and the payload would be the “net payload” starting with the data format byte. That way it would be easy to parse just the net payload if you are interested in that only. Though it may be extra clutter as the “header” would practically always be 3 bytes so should be easy to parse either way. :thinking:

1 Like
16

I would like to suggest HEADER be first.
This would ALLOW alternate formats perhaps shortening timestamp length to one that had less precision (or more since according to https://www.unixtimestamp.com unixtimestamp value is in seconds.)

Although I agree that some applications may require finer - perhaps even micro second precision - some may only require second or multi second .

What is your thoughts on the length of the timestamp field?

PS Many communications packet formats have HDR first.

19

Clarification: LWT is Last Will and Testament
MQTT is used in unreliable networks, … clients will … disconnect ungracefully. …use LWT to notify other clients about an ungracefully disconnected client

from https://www.hivemq.com/blog/mqtt-essentials-part-9-last-will-and-testament/

20

By the way, can the dongle report a RSSI for the received advertisements? If so, maybe the rssi should also be part of the MQTT payload (along with timestamp etc.)?

LWT topic probably will have to include a way to identify the dongle (another use-case for a “DongleID”, see above). Payload seems less important for LWT, or follow the pattern described in ‘Best Practices’ in the blogpost helpfully linked by @dgerman.

Hhm… do you really want all your neighbours SmartTVs, or every occasionally passing smartphone, to show up as topics on your MQTT broker? Maybe you don’t even want all ruuvi tags to show up, but just the ones you own?
Don’t get me wrong, I would love this to be usable for generic BLE scanning; I just think MQTT isn’t a good fit for this use-case / sending this kind of data upstream. Instead, i could think of… oh stop, that begins to look like yet another firmware (;

21

Seems like
whitelist of tag IDs to be processed and/or blacklist of tagIDs NOT to be processed as well as

Whitelist company identifiers (example 0499, Ruuvi Innovations, 05BF, Real-World-Systems) and/or blacklist some company identifiers

should be options in the configuration.

22

Regarding NTP
consider SNPT(https://github.com/lettier/ntpclient, see issues) and beware of timeout
pool.ntp.org.
Rather than polling every 24 hours perhaps some algorithm based on the discrepancy between local time and just received from NTP time would be approiate