Thingspeakread and NumMinutes problem after clock change | ThingSpeak Apps | Forum

Avatar

Please consider registering
Guest

sp_LogInOut Log In sp_Registration Register

Register | Lost password?
Advanced Search

— Forum Scope —






— Match —





— Forum Options —





Minimum search word length is 3 characters - maximum search word length is 84 characters

sp_Feed sp_TopicIcon
Thingspeakread and NumMinutes problem after clock change
No permission to create posts
April 12, 2019
12:06 pm
Avatar
iotum
New Member
Members
Forum Posts: 4
Member Since:
April 12, 2019
sp_UserOfflineSmall Offline

Hi -
I've built my own speed camera and traffic monitor from a Rasp Pi. I'm using Node-red to upload data to Thingspeak using a http Post request every minute. It uploads a row for every vehicle the camera sees. (it's not a major road so not huge volume).

I'm new to ThingSpeak but have managed to build an app to run every hour to count the number of vehicles passing the camera over the last 60 minutes. I use
"data = thingSpeakRead(readChannelID, 'Fields',[1,2],'ReadKey', readAPIKey,'NumMinutes', 60, 'OutputFormat', 'timetable');"

All was working well until the UK (where I am) switched to summer time. Since then I started getting very high values for traffic volume. I eventually worked out that it was actually counting 2 hours of data rather than just 60 minutes. In fact if I change the NumMinutes parameter to say 3, then it counts 63 minutes of data. So it just adds 60 mins to whatever I specify. Clearly something amiss re timezones but not sure what I need to fix.

Any suggestions.?

Tim

April 12, 2019
4:01 pm
Avatar
cstapels
Moderator
Members


Moderators
Forum Posts: 801
Member Since:
March 7, 2017
sp_UserOfflineSmall Offline

I will look into the time zone discrepancy.

In the meantime, you can use the 'NumPoints' name value pair to read 60 points if your pi posting is relatively consistent.
https://www.mathworks.com/help/thingspeak/thingspeakread.html

You can get a lot more control using a MATLAB analysis to do your calculations. You still need to use thingSpeakRead() to read the data, but if you use the timetable format, you could use retime() to calculate the 60 minute average for up to 8000 points at once.

You could also count the rows or use duration values to make comparisons and be sure you have only one hour of data.

What are you using for image processing? We have a traffic monitor using Simulink and computer vision and image processing toolboxes in MATLAB to process data on a Rasberry Pi.
It looks out to Rte 9, a fairly busy highway in front of our headquarters.
https://thingspeak.com/channels/38629

April 13, 2019
10:34 am
Avatar
iotum
New Member
Members
Forum Posts: 4
Member Since:
April 12, 2019
sp_UserOfflineSmall Offline

Thanks for the info.

I'm using OpenCV on the Pi. it's allows me to measure individual vehicle speeds (as long as I don't have 2 cars passing in front of the camera at the same time which doesn't occur very often).

Although I post data every minute it is a variable number of data points. i send one data point per vehicle. This allows me to record individual speeds of the cars and work out average speeds as well as traffic density. So I can't use NumPoints = 60 as that wouldn't reliably give me an hour.

I've used the retime function on some of the code when I'm analysing a day's worth of data. But with the hourly process I'm trying to write the resultant hourly count to a new channel so i can then later on do further processing across a longer timescale. I'll have another look at retime and see if I can use that differently to what I've done before.

April 16, 2019
10:36 am
Avatar
cstapels
Moderator
Members


Moderators
Forum Posts: 801
Member Since:
March 7, 2017
sp_UserOfflineSmall Offline

Thanks for the info about your counter. There are almost always multiple cars in the frame for us, but we aren't getting speeds.

I tried setting my timezone to London and reading a channel with
data=thingSpeakRead(12397,'NumMinutes',3,'OutputFormat', 'timetable')

data =

3×8 timetable

Timestamps WindDirectionNorth0Degrees WindSpeedmph Humidity TemperatureF RainInchesminute PressureHg PowerLevelV LightIntensity
____________________ __________________________ ____________ ________ ____________ ________________ __________ ___________ ______________

16-Apr-2019 15:26:02 143 7.4 31 51.9 0 29.6 3.4 38
16-Apr-2019 15:27:04 99 8 31 52 0 29.6 3.4 38
16-Apr-2019 15:28:07 4 6 31 52.1 0 29.6 3.4 58

I get three entries, definitely not over an hour of data. Is your channel public? I can look at the data and try to read your channel to see if I can reproduce the error.

April 26, 2019
2:19 pm
Avatar
iotum
New Member
Members
Forum Posts: 4
Member Since:
April 12, 2019
sp_UserOfflineSmall Offline

Many apologies for slow response. I've been away and then the day job got in the way.

My raw data is captured in channel 673594 (https://thingspeak.com/channels/673954)
I then process that every hour to count the number of vehicles in the last 60 mins (using NumMinutes=60) and write the result to field 3 in channel 711054 https://thingspeak.com/channels/711054

I can confirm that the numbers being displayed in 711054 are for 2 hours of data rather than 1.

I've noticed that the data in 673594 is being shown with the wrong time of day. So Field 1 in this channel displays data with a timestamp of local time (British Summer Time) plus an extra 1 hour. So traffic captured right now at 19:13 BST will be displayed as captured at 20:13 (GMT +01:00). Could this be the cause of my problem ? I'm sending data to Thingspeak with local time timestamps which is my understanding of the correct approach but maybe I'm wrong.

April 29, 2019
9:38 am
Avatar
cstapels
Moderator
Members


Moderators
Forum Posts: 801
Member Since:
March 7, 2017
sp_UserOfflineSmall Offline

I was able to reproduce your issue. I read 60 minutes of data and got information from 8:30 to 10:30.
Now that I can reproduce the issue, I can report the problem.
In the meantime, you can use the following:
myTime=datetime('now');
myData=thingSpeakRead(673954,'daterange',[myTime-hours(1),myTime],'outputformat','timetable')

April 30, 2019
11:30 am
Avatar
cstapels
Moderator
Members


Moderators
Forum Posts: 801
Member Since:
March 7, 2017
sp_UserOfflineSmall Offline

After looking into the issue, we have deduced that your device is writing times in the future into the channel. Then the 60 minute read gets the last 60 minutes up to the present, but the preset is including future data that you have written. Can you make your device just send the timestamps in UTC? Then ThingSpeak should be able to correctly interpret your desired times using the timezone setting on your profile page.

May 6, 2019
6:02 am
Avatar
iotum
New Member
Members
Forum Posts: 4
Member Since:
April 12, 2019
sp_UserOfflineSmall Offline

Thank you. I amended the code to send timestamps in UTC and have been monitoring for a few days. Am now getting sensible numbers again. Many thanks for your help.

Forum Timezone: America/New_York

Most Users Ever Online: 166

Currently Online:
22 Guest(s)

Currently Browsing this Page:
1 Guest(s)

Top Posters:

rw950431: 272

Vinod: 240

piajola: 95

turgo: 70

vespapierre: 63

Adarsh_Murthy: 62

Member Stats:

Guest Posters: 1

Members: 8665

Moderators: 1

Admins: 2

Forum Stats:

Groups: 3

Forums: 14

Topics: 1600

Posts: 5760

Newest Members:

hernancrespo, hernancrespo89, Tommilar01, Alperian, rose22, lokesh4757

Moderators: cstapels: 460

Administrators: Hans: 405, lee: 457