MATLAB analysis and data inserts | General | Forum

Avatar

Please consider registering
Guest

sp_LogInOut Log In sp_Registration Register

Register | Lost password?
Advanced Search

— Forum Scope —






— Match —





— Forum Options —





Minimum search word length is 3 characters - maximum search word length is 84 characters

sp_Feed sp_TopicIcon
MATLAB analysis and data inserts
No permission to create posts
March 16, 2018
9:16 am
Avatar
heh

Gold
Members
Forum Posts: 27
Member Since:
June 30, 2017
sp_UserOfflineSmall Offline

Can running analysis script access data that was inserted by an outside source after the script was started?

If yes, then how to tell this script that it should ignore it, because another script instance will process this new data insert?

March 20, 2018
2:57 pm
Avatar
Hans
Natick, MA

Admin
Forum Posts: 400
Member Since:
January 18, 2011
sp_UserOfflineSmall Offline

So, the analysis script is going to run up to 20seconds for free accounts and 60sec for commercial accounts. If you your first line is a thingSpeakRead, you will only get data that is in the channel at that moment. If you do another read at the end of the time, you may get new data in the channel.

What are you looking to do?

April 11, 2018
1:15 am
Avatar
heh

Gold
Members
Forum Posts: 27
Member Since:
June 30, 2017
sp_UserOfflineSmall Offline

As I thought, your idea of using Matlab Analysis did not work for batch inserts. And likely won't work for individual record inserts either.

In my case, the script is triggered on each batch insert, but it runs with random delays. Sometimes it could be minutes between actual runs and then it may execute several run instances at the same time.

Because there is no reliable way to determine what recordset each such instance has to process, it often processes same data batch several times or misses to process a number of consecutive data batches.

I am trying to run data analysis and push alerts to external relay server. Matlab Analysis seems useless for this purpose, because there is no way to implement incremental processing of data. The script often misses some datapoints or throws a bunch of duplicates, and either is not good.

April 16, 2018
7:50 pm
Avatar
heh

Gold
Members
Forum Posts: 27
Member Since:
June 30, 2017
sp_UserOfflineSmall Offline

Hello, ThingSpeak development & support.

Did my previous post make any sense to you?

Are there any plans to resolve the issue?

Waiting for your reply ...

Regards

April 17, 2018
10:54 am
Avatar
Hans
Natick, MA

Admin
Forum Posts: 400
Member Since:
January 18, 2011
sp_UserOfflineSmall Offline

I would recommend having two channels: one that contains the raw data and one that will contain the processed data.

In your MATLAB analytics code, I will start by reading data from the processed data channel and get the last time stamp (if there is one). I would use that time stamp in a second thingSpeakRead in your analysis to get data from the raw channel. Use a a DateRange of Now to the last time stamped processed.

This technique should help greatly in getting consistent analysis and avoid analyzing duplicate data.

April 17, 2018
3:56 pm
Avatar
heh

Gold
Members
Forum Posts: 27
Member Since:
June 30, 2017
sp_UserOfflineSmall Offline

Thank you - I thought of a similar approach too,
but in order to implement this solution correctly
I need more information about your ThingSpeak
server concurrency model.

Are analysis jobs run sequentially or they are
completely asynchronous and run in parallel?

Thanks

April 17, 2018
11:07 pm
Avatar
Vinod

MathWorks
Members
Forum Posts: 215
Member Since:
May 1, 2016
sp_UserOfflineSmall Offline

Depending on a number of factors that include your data collection rate, the conditions that trigger the MATLAB Analysis job to run (if it is set up to be run by a React app) , and the length of time your MATLAB code takes to execute, among other factors, can result in your analysis jobs running in parallel.

The approach I would recommend that works best is for the raw data to be logged to a certain channel, and a TimeControl be used to schedule a MATLAB analysis job that runs at a frequency of once every 60 minutes. This Analysis reads the data from the "raw data" channel from the time the last datapoint data was written to the "derived data" channel until "now" and writes the processed data to the derived channel. If the processing takes >20s per evaluation (total time), you may need to purchase a Standard license that allows your code a window of 60s to complete execution.

I don't see why this needs any knowledge of the internals of ThingSpeak. I also would argue that any solution that requires this knowledge is brittle.

PS: Technical Support is just one of the benefits of purchasing a ThingSpeak license. If you need assistance on demand, I would strongly urge you to purchase a license.

April 18, 2018
11:36 am
Avatar
heh

Gold
Members
Forum Posts: 27
Member Since:
June 30, 2017
sp_UserOfflineSmall Offline

Vinod

I still think of whether I want to use ThingSpeak and buy a license. It has too many issues and I would not care about ThingSpeak internals if it worked reasonably well.

Unfortunately, ThingSpeak lacks many basic features and in its current state it is more a toy than a tool. Using it for anything useful requires odd tricks like creating separate channels for data processing history, or tagging data records and so on.

Please, consider my questions as a guide to what your framework needs in order to be attractive to potential users. In some sense it is me, who is helping you to make it a successful commercial product.

Back to my previous question - could you give me a straight answer about concurrency?

April 19, 2018
5:29 pm
Avatar
Vinod

MathWorks
Members
Forum Posts: 215
Member Since:
May 1, 2016
sp_UserOfflineSmall Offline

Heh,

We have a very large number of users for whom ThingSpeak works really well and who appreciate the features of the service. Many of our users pay for the features the ThingSpeak platform provides, so we must be doing something right 😉 We are also always open to constructive feedback on how we can improve the service and add features that will benefit our user community. Many of our users use this forum to do exactly that. But, all of that is unrelated to your question.

The answer to your question "...straight answer about concurrency" is "it depends". As I mentioned in my previous response, depending on a number of factors that include data ingestion rate, the conditions that trigger the MATLAB Analysis job, the length of time your MATLAB code takes to execute, among other things, two executions of your code could run concurrently. That doesn't mean it always does so. If your MATLAB code was written assuming it could be executed concurrently and your algorithm is robust to this, you should be fine.

By the way, I would not consider having a channel for raw data and another channel for processed data to be an odd trick. I recommend this to users so that they can always go back to the raw data and apply a different processing algorithm to get different insights into it.

Hope this helps,
-Vinod

April 20, 2018
12:15 pm
Avatar
heh

Gold
Members
Forum Posts: 27
Member Since:
June 30, 2017
sp_UserOfflineSmall Offline

Thank you for response, Vinod.
Here is a fresh example of something requiring odd tricks
(encounter these issues every day):

- HTTP API does not allow to specify range of dates to read.
- Matlab Analysis API call thingSpeakRead(), allows reading data in a DateRange, but does not return Status field value.

I understand that I can run these 2 requests and merge responses,
Or request 8000 point and throw away part (most) of the response I do not need.

Wouldn't that be odd, though?

How do you read status within a date range without jumping through these hoops?
Why don't you, for example, offer some way of reading a range of feed IDs?

Regards

April 20, 2018
12:30 pm
Avatar
Vinod

MathWorks
Members
Forum Posts: 215
Member Since:
May 1, 2016
sp_UserOfflineSmall Offline

>> HTTP API does not allow to specify range of dates to read.

Actually, it does. See documentation here: https://www.mathworks.com/help/thingspeak/readdata.html

If you wanted JSON data for the first 15 days of this month, the GET request would be like this:

https://api.thingspeak.com/channels/9/feeds.json?start=2018-04-01&end=2018-04-15

Note that if you specify a large date range, the result is limited to 8000 points, as documented on the page.

>> Matlab Analysis API call thingSpeakRead(), allows reading data in a DateRange, but does not return Status field value.

This is an item we have on our backlog of features to implement.

>> How do you read status within a date range without jumping through these hoops?

https://api.thingspeak.com/channels/9/status.json?start=2018-04-01&end=2018-04-15

>> Why don't you, for example, offer some way of reading a range of feed IDs?

We believe feed entry IDs are not something the end user should have to care about. Time series data, which is what the data stored on ThingSpeak is, should be queryable/addressable by date range as the above example demonstrates. The thingSpeakRead() function in MATLAB does what is needed to get data and stitch it together so a user doesn't have to worry about the 8000 point limitation.

Hope this helps you on your project. If there is something that isn't working as you expect, please do post your code/what you have tried and someone on the community will look at it and offer suggestions sooner or later.

April 20, 2018
5:41 pm
Avatar
cstapels
Moderator
Members


Moderators
Forum Posts: 341
Member Since:
March 7, 2017
sp_UserOfflineSmall Offline

You may be able to read data and status at the same time.
The read data API also has a status parameter:
https://api.thingspeak.com/channels/472367/feeds.json?start=2018-04-01&end=2018-04-15&status=true

April 23, 2018
6:13 pm
Avatar
heh

Gold
Members
Forum Posts: 27
Member Since:
June 30, 2017
sp_UserOfflineSmall Offline

>> We believe feed entry IDs are not something the end user should have to care about.
On the contrary, it could be useful if feed IDs that triggered React were passed to Analysis script that this React invoked.
Of course, then users would need some way to retrieve feeds by these IDs.

>> DateRange mapped to start / end
Thank you for pointing this out.
I was looking for DateRange and overlooked start / end having similar purpose.

Regards

Forum Timezone: America/New_York

Most Users Ever Online: 114

Currently Online:
26 Guest(s)

Currently Browsing this Page:
1 Guest(s)

Top Posters:

rw950431: 261

Vinod: 196

piajola: 85

turgo: 70

vespapierre: 63

Adarsh_Murthy: 62

Member Stats:

Guest Posters: 1

Members: 5703

Moderators: 0

Admins: 2

Forum Stats:

Groups: 4

Forums: 17

Topics: 1313

Posts: 4565

Newest Members:

BrightfuryAboth, jaxs, WilanashAboth, TygolarAboth, HossokAboth, DoluneAboth

Administrators: Hans: 387, lee: 457