Python3, Raspi: All local data collector and visualiser


#1

Hi,

Yet another take on a Python 3 data collector and webapp on RasPi…

Current features:

  • different time scales for the graphs selectable
  • auto-resample to daily min max if time series gets too dense
  • filter by sensor or metric
  • DB is sqlite via dataset and sqlalchemy, so switching to a bigger DB should be easy
  • pretty Google fonts and icons :slight_smile:
  • mock time series generator
  • can add sensors without stopping data collector

Overall, emphasis is on using existing APIs and avoiding overengineering as I don’t have too much time for side projects.

Stuff used

Screenshots show mock time series.

It’s on my Github but very very much pre-alpha, so I’m hesitant to link right now. If someone wants to give it a try, I’m happy to add some instructions and keep you posted.

Best,
David


#2

Nice! I have been using ruuvitag-sensor package and InfluxDB and Grafana to display the information. However, I’m interested to try your solution as I’m currently evaluating which solution to proceed with.


#3

To be honest, I’m considering to give up on my little project as started to play with InfluxDB and Grafana, too. When I started to build my stuff, I just didn’t know that it runs so well on the Raspi. I still don’t like the way Grafana displays single values, but the charts are really nice.

What I’ll definitely keep is my own Python-based data collector, which I cloned to stream to InfluxDB instead of using sqlite. It runs well and I added some functionality to add new Ruuvis on the fly.

If you’re still interested in trying my solution, let me know. But I will probably not maintain it, and take it as a learning experience.

Best,
David


#4

Hi!
Tanks, I’d like to have a look since I’m also interested how you made your data collector. I’m not fully pleased with how I collect my data from the tags, I involuntary drop some packets.


#5

Hi,

Understood. This is the file you are probably interested in:

Its my data collector for Influx, and it’s relatively robust against unexpected situations and quite well behaved. I have experienced hickups, though, when I’m running a second script (in parallel) that uses the Ruuvitag API.

Note that it will only stream to the DB the sensors which are named in the file sensors.yml to avoid flooding your DB with your neighbors Ruuvis. It will, however, report all MACs it sees, and it monitors sensory.yml for changes. So you can add new sensors to the yaml file without stopping data collection. If you happen to save the yaml in a corrupt state it wont reload.

Ignore the mocking stuff. I wrote it to simulate measurements before I had my Raspi.

Hope that helps. If you have questions, let me know.

Best,
David


#6

Thanks a lot! It works as intended.
I have tested variants of my code where I want to write to the database whenever a new packet has arrived, i.e. not on a fixed timeout basis. Not sure if I need that feature any longer but I may use your code as a better starting point for that implementation.
One question though. I noted that you import by calling from ruuvi instead of from ruuvitag_sensor.ruuvi
Have you installed the RuuviTag Sensor Python Package differently (pip install ruuvitag_sensor) or what version of the package are you running?


#7

Good catch – that’s actually an unintended import from another file from my project. Relied too much on Pycharm… Your import is the correct one actually.

To write absolutely everything the sensor sends, you can have a look at the file I linked below. I wrote that for discovering new Ruuvis until I found it to cause trouble when running in parallel to the collector. It either floods all readings to console, or it prints new MACs when it sees them.

What are you monitoring to require so many measurements?


#8

I’ve made a new code version of the RuuviTag that broadcast every 10 seconds. However, if it detects changes is the acceleration it starts to broadcast a few times per second. The idea was to be able to detect sudden movements, e.g. a burglar, and on the server trigger a push message to the user. Even unexpected changes of the other sensors could be of interest.