Corrupted logs (duplicates and 100x humidity)

I was looking at some old data in logs, and found something interesting. Below are excerpts of two logs uploaded from iOS. The first one was uploaded on January 31 and the second one August 1.

There are clearly duplicate pairs, 5 minutes apart, and within the pairs the values are one second apart. I guess the pairs were caused by some duplicate sync.

In the old upload the pairs have identical values for temperature, air pressure, and humidity. In the recent upload the first humidity value does not have the decimal point, leading to 100x values, i.e., 9202 instead of 92.02.

I am puzzled at how the old data, ok during an upload in January, has been corrupted in an upload a few days ago

Jan 31

18.12.2020 18:41:54 2.19 92.02 1010.76
18.12.2020 18:41:55 2.19 92.02 1010.76
18.12.2020 18:46:55 2.2 92.03 1010.76
18.12.2020 18:46:56 2.2 92.03 1010.76
18.12.2020 18:51:56 2.21 92.02 1010.81
18.12.2020 18:51:57 2.21 92.02 1010.81
18.12.2020 18:56:57 2.23 92.03 1010.81
18.12.2020 18:56:58 2.23 92.03 1010.81

Aug 1

18.12.2020 18:41:54 2.19 1010.76 9202
18.12.2020 18:41:55 2.19 1010.76 92.02
18.12.2020 18:46:55 2.2 1010.76 9203
18.12.2020 18:46:56 2.2 1010.76 92.03
18.12.2020 18:51:56 2.21 1010.81 9202
18.12.2020 18:51:57 2.21 1010.81 92.02
18.12.2020 18:56:57 2.23 1010.81 9203
18.12.2020 18:56:58 2.23 1010.81 92.03

There are also other oddities in the log uploaded recently

  • There are long streaks of 100x humidity values, without a single normal value. Temp and air pressure are ok. To me these values are received measurement values, not synced values
  • There are many recent duplicate pairs where both humidity values are ok, as below
1.8.2021 20:00:40 18.78 998.47 73.86
1.8.2021 20:00:40 18.78 998.47 73.86
1.8.2021 20:00:41 18.78 998.47 73.88
1.8.2021 20:00:41 18.78 998.47 73.88
1.8.2021 20:00:43 18.78 998.47 73.88
1.8.2021 20:00:43 18.78 998.47 73.88
1.8.2021 20:00:44 18.78 998.47 73.89
1.8.2021 20:00:45 18.78 998.47 73.89
1.8.2021 20:00:47 18.78 998.47 73.89
1.8.2021 20:00:47 18.78 998.47 73.89
1.8.2021 20:00:50 18.78 998.47 73.9
1.8.2021 20:00:50 18.78 998.47 73.9

I am not able to figure out why this happens. I hope you can.

Thank you for the report. @Rinat Please take a look

A few additional comments

  • Not all humidity values have these 100x duplicates: there are long streaks of good values, and also long streaks of single 100x humidity values, and then there were also these good values, in pairs, with identical timestamps
  • I have done several firmware upgrades and downgrades with my RuuviTags (but I guess it should not impact the data already copied to the App)

I you need a copy of the full file, please let me know.

Hi Mikko! I’m sorry, but we actually had this bug with *100 humidity values.

Finally we decided to store humidity as a fraction of one.

I’m wondering how have you managed to get data for December 2020? iOS app stores last ten days of data, are you using longmem firmware without reboot?

My opinion is: you’ve used old Ruuvi Station iOS app, where humidity was stored in percents (like 42), and with new version of the app we are storing fraction of one (like 0.42) in Database.

This mix of approaches should have been solved by pruning data older than 10 days. User might have experienced weird charts for 10 days maximum.

Hi Rinat,

Good that you have settled on the storage format for humidity.

The old data is something that was from older syncs and measurement reception between RuuviTags and Ruuvi Station App. I thought it was supposed to stay in the App forever, and not wiped after 10 days. I thought the App could be a permanent storage for the data until I upload it to somewhere.

I have no idea what longmem is. I have used the 3.28, 3.29, 3.30 firmware, and maybe others that are available in the same, as the official beta (?) for iOS. I have been upgrading and downgrading between firmware, due to problems with sync.

As for the iOS Apps, I usually update all my Apps within days of availability. So, I always run the latest Apps.

What is odd is that the App stored 100x and 1x values one second apart for some measurements, at least in 2020.