Can't calculate max temperature for the last 30 days. Is there a limit? | MATLAB | Forum

Avatar

Please consider registering
Guest

sp_LogInOut Log In sp_Registration Register

Register | Lost password?
Advanced Search

— Forum Scope —






— Match —





— Forum Options —





Minimum search word length is 3 characters - maximum search word length is 84 characters

sp_Feed sp_TopicIcon
Can't calculate max temperature for the last 30 days. Is there a limit?
No permission to create posts
January 4, 2018
4:01 pm
Avatar
Deimos
New Member
Members
Forum Posts: 2
Member Since:
January 2, 2018
sp_UserOfflineSmall Offline

I've chosen a ThingSpeak example: "Calculate hourly max temperature"

I modified the code to calculate the max temperature for 30 days:

% Temperature for the past hour is read from a ThingSpeak channel and
% maximum temperature is written to another ThingSpeak channel.

% Channel 12397 contains data from the MathWorks Weather Station, located
% in Natick, Massachusetts. The data is collected once every minute. Field
% 4 contains temperature data.

% Channel ID to read data from
readChannelID = 300978;
% Temperature Field ID
TemperatureFieldID = 2;

% Channel Read API Key
% If your channel is private, then enter the read API
% Key between the '' below:
readAPIKey = '';

% To store the maximum temperature, write it to a channel other than
% the one used for reading data. To write to a channel, assign the write
% channel ID to the 'writeChannelID' variable, and the write API Key to the
% 'writeAPIKey' variable below. Find the write API Key in the right side pane
% of this page.

% TODO - Replace the [] with channel ID to write data to:
writeChannelID = [];
% TODO - Enter the Write API Key between the '' below:
writeAPIKey = '';

% Read temperature data for the last 60 minutes from the MathWorks Weather
% Station. Learn more about the THINGSPEAKREAD function by going to the Documentation
% tab on the right side pane of this page.

[tempF, timeStamp] = thingSpeakRead(readChannelID, 'Fields', TemperatureFieldID, 'numMinutes', 60*24*30, 'ReadKey', readAPIKey);

% Calculate the maximum temperature
[maxTempF, maxTempIndex] = max(tempF);

% Choose the timestamp at which the maximum temperature was measured
timeMaxTemp = timeStamp(maxTempIndex);

display(maxTempF, 'Maximum Temperature for the last hour is');

% Write the maximum temperature to another channel specified by the
% 'writeChannelID' variable

display(['Note: To successfully write data to another channel, ',...
'assign the write channel ID and API Key to ''writeChannelID'' and ',...
'''writeAPIKey'' variables above. Also uncomment the line of code ',...
'containing ''thingSpeakWrite'' (remove ''%'' sign at the beginning of the line.)'])

% Learn more about the THINGSPEAKWRITE function by going to the Documentation tab on
% the right side pane of this page.

% thingSpeakWrite(writeChannelID, maxTempF, 'timestamp', timeMaxTemp, 'Writekey', writeAPIKey);

The code previously was calculating data for 60 minutes, and it works fine.

I've changed it to 60*24*30 (trying to calculate for 30 days) and the result (max temperature) is not correct, the result may be true for 7 days but not for 30.

Doesn't matter what I use 60*24*7 or 60*24*30 or 60*24*90 I always get the same max temperature.

I assume that there is some limit (probably 8000 points).

P.S. I am new to MATLAB, maybe I am doing something wrong, please explain to me.

January 4, 2018
8:22 pm
Avatar
cstapels
Moderator
Members


Moderators
Forum Posts: 258
Member Since:
March 7, 2017
sp_UserOfflineSmall Offline

You are correct, ThingSpeak limits the number of points you can read with thingSpeakRead or a single REST API call to 8000.  However, you can read more by creating a loop if you need to.  Just be careful not to hit the code time maximum for your license type.

January 4, 2018
10:13 pm
Avatar
Vinod

MathWorks
Members
Forum Posts: 206
Member Since:
May 1, 2016
sp_UserOfflineSmall Offline

A much better approach than to read more than 8000 points in a loop is to downsample the data and store it in another channel. 

So, let's say you are getting raw data from your sensors every 15 seconds. You could set up a timecontrol that runs every hour and averages this data and saves it to another channel. If you needed averaging at a lower resolution, you could further resample the 1 hour averaged data to every day and write to another channel, etc.

The main advantage of this approach is that it allows you to operate on smaller amounts of data, which is much more efficient and quick than operating on large arrays of data.

To give an analogy, if you had to count the number of grains of sand in a bucket of sand, rather than count each grain, a better approach would be to count the grains in a small cup, and count the number of cups in the bucket. The second is an approximation, but it is a good enough approximation that you can hope to complete the task in a reasonable amount of time than spending a few years on something that doesn't need to be that precise.

January 5, 2018
8:09 am
Avatar
cstapels
Moderator
Members


Moderators
Forum Posts: 258
Member Since:
March 7, 2017
sp_UserOfflineSmall Offline

I agree, downsampling is a much better solution, and can be exact, not an approximation in the case of measuring a maximum (compared to counting sand).

January 6, 2018
4:54 pm
Avatar
turgo

Gold
Members
Forum Posts: 70
Member Since:
June 16, 2013
sp_UserOfflineSmall Offline

I've got a data channel with 8 million fifteen second entries.  I need to aggregate this into hourly averages.  How do I practically do this, given the 8000 point limit, and the 20 second calculation timeout? 

January 16, 2018
6:47 pm
Avatar
cstapels
Moderator
Members


Moderators
Forum Posts: 258
Member Since:
March 7, 2017
sp_UserOfflineSmall Offline

Turgo,

One method you could try would be to downsample the data bit by bit and use a timecontrol to run this code regularly.  The code below uses a new channel and reads its length to determine where to start the downsampling.  Put the code in a MATLAB analysis and trigger it with a timecontrol.  It will eventually reach the max or run out of data and stop.  Then you can turn off the timeControl.

You need to set up a new channel and provide its read and write API keys.  The variable oldChannel is the channel with many entries. hourlyChannel is the downsampled data.  If your old channel is a private channel, you will need to provide its read api key as well.

 

oldChannel=1234;
hourlyChannel=123456;
writeAPIKey='XXXXXXXXXXXXXXXX';
readAPIKey='YYYYYYYYYYYYYYYY';

sizeCheck=thingSpeakRead(hourlyChannel,'ReadKey',readAPIKey,'numDays',2000,'outputformat','timetable');
daySpot=round(height(sizeCheck)/24); % Hourly data
daysMax=1000; % 3 years of data
if (daySpot<daysMax) % Make sure you don't read forever
oneDayData=thingSpeakRead(oldChannel,'DateRange',[datetime('today')-days(daySpot+1),datetime('today')-days(daySpot)],'Outputformat','timetable');

if height(oneDayData)>0

hourlyData=retime(oneDayData,'hourly','mean');
thingSpeakWrite(hourlyChannel,hourlyData,'WriteKey',writeAPIKey);

end

end

 

Keep in mind that this method could consume a lot of messages (as many as 24,000).  

For future channels that you need long periods of data from, I suggest setting up a downsampled channel at design time. Hopefully that will save you the trouble next time.  

January 16, 2018
7:21 pm
Avatar
turgo

Gold
Members
Forum Posts: 70
Member Since:
June 16, 2013
sp_UserOfflineSmall Offline

Thanks for your effort to craft this solution. One note, it will take 5 day to run this script on each channel (11 total).  I'd best get going as I have 15 days left!

January 25, 2018
6:09 pm
Avatar
turgo

Gold
Members
Forum Posts: 70
Member Since:
June 16, 2013
sp_UserOfflineSmall Offline

I discovered that staples method result in a channel which is written backwards in groups. This results in a channel feed that is hard to use and read when downloaded. Here is a method that writes the channel from start to finish averaging hourly data into daily data.  Note the use of channel arrays to manage large numbers of channels.  I assemble the arrays from an excel spreadsheet.

% Aggregate daily from hourly (Channel 1)
myChannelName = 'Channel 1';
datetimeStart = datetime(2013,11,11); % UPDATE Start date of data
channelList = {'Hourly Channel 1';'Daily Channel 1';'Hourly Channel 2};
channelID = {####;####;####};
channelReadKey = {'XXXXXXXXXXX';'XXXXXXXXXX';'XXXXXXXXXXXX'};
channelWriteKey = {'XXXXXXXXXXX';'XXXXXXXXXX';'XXXXXXXXXXXX'};
channelTable = table(channelID,channelReadKey,channelWriteKey,'RowNames',channelList);
hourlyChannelID = channelTable.channelID{['Hourly ' myChannelName],:};
hourlyReadAPIKey = channelTable.channelReadKey{['Hourly ' myChannelName],:};
dailyChannelID = channelTable.channelID{['Daily ' myChannelName],:};
dailyReadAPIKey = channelTable.channelReadKey{['Daily ' myChannelName],:};
dailyWriteAPIKey = channelTable.channelWriteKey{['Daily ' myChannelName],:};

% determine the next day to read
aggregateProgress = thingSpeakRead(dailyChannelID, 'numPoints',1, 'ReadKey', dailyReadAPIKey,'outputFormat', 'timetable');
display(aggregateProgress);
if isempty(aggregateProgress) % New Hourly channel
display(['Daily channel empty ']);

datetimeStop = dateshift(datetimeStart,'start','day') + days(300);
else % Existing hourly channel
datetimeStart = dateshift(aggregateProgress.Timestamps(1),'start','hour') + hours(24);
datetimeStop = dateshift(aggregateProgress.Timestamps(1),'start','day') + days(300);
end
% Read the next day worth of data
display(datetimeStart);
display(datetimeStop);

data = thingSpeakRead(hourlyChannelID, 'DateRange',[datetimeStart,datetimeStop], 'ReadKey', hourlyReadAPIKey,'outputFormat','timetable');
% display(summary(data));
if height(data) > 5
display(data(1:5,:));
end
% Convert text numbers
strVars = varfun(@iscellstr,data,'output','uniform');
display(strVars);
for varnum = 1:length(strVars)
if strVars(varnum)
celldata = table2cell(data(:,varnum));
celldata = str2double(celldata);
data.(varnum)= celldata;
end
end
% error correction -999 to NaN for humidity data
negToNaNfunc = @(x) x .* (x>=-100)./(x>=-100);
display(negToNaNfunc(-1000));
data = varfun(negToNaNfunc,data);
if height(data) > 5
display(data(1:5,:));
end
if isempty(data)
display(['No data for that day period']);
else
dataDaily = retime(data,'daily','mean');
display(dataDaily(1:5,:));
thingSpeakWrite(dailyChannelID, dataDaily, 'Writekey', dailyWriteAPIKey);
end

Forum Timezone: America/New_York

Most Users Ever Online: 114

Currently Online:
25 Guest(s)

Currently Browsing this Page:
1 Guest(s)

Top Posters:

rw950431: 261

Vinod: 196

piajola: 85

turgo: 70

vespapierre: 63

Adarsh_Murthy: 62

Member Stats:

Guest Posters: 1

Members: 5703

Moderators: 0

Admins: 2

Forum Stats:

Groups: 4

Forums: 17

Topics: 1313

Posts: 4565

Newest Members:

JamesVex, meghanfr69, raqueldk69, soniafw2, vitalijDrine, JosephAmapy

Administrators: Hans: 387, lee: 457