Editing Thingspeak Data | ThingSpeak Projects | Forum

Avatar

Please consider registering
Guest

sp_LogInOut Log In sp_Registration Register

Register | Lost password?
Advanced Search

— Forum Scope —






— Match —





— Forum Options —





Minimum search word length is 3 characters - maximum search word length is 84 characters

sp_Feed sp_TopicIcon
Editing Thingspeak Data
No permission to create posts
March 31, 2019
11:40 am
Avatar
Metawatch
New Member
Members
Forum Posts: 4
Member Since:
January 18, 2018
sp_UserOfflineSmall Offline

Hi
I have a water level feed that is subject to receiving false positives, and would like to delete several hundred entries from the THingSpeak data stored.

Was looking for the easiest method.

Tried downloading the file CSV, deleted entries, and re-uploading but the process fails as it says timestamp already present.
Tried by modifying the contents of the CSV with correct values (Replacing the incorrect values) and the same error occurred.

Does one have to delete the entire database and re-upload it for the csv upload to function properly, or is it just possible to modify the false positive values in another fashion ?
Confused
Would be nice if there was a simple spreadsheet like editor where the values could simply be deleted or replaced.

April 1, 2019
8:20 am
Avatar
cstapels
Moderator
Members


Moderators
Forum Posts: 730
Member Since:
March 7, 2017
sp_UserOfflineSmall Offline

Unfortunately, you will have to delete all the channel data to re-upload points with the same timestamp. I would recommend setting up a MATLAB analysis to filter your data as it comes in and write the filtered data to another channel. You can use the excel sheet with the cleaned values to start the new channel. You can use TimeControl to call the MATLAB analysis at regular intervals, and process a collection of the data each time segment.

I have felt your pain about wanting to delete specific entries in a channel before. I will note your request in our tracking system, thanks for the feedback.

April 1, 2019
1:43 pm
Avatar
Vinod

MathWorks
Members
Forum Posts: 306
Member Since:
May 1, 2016
sp_UserOfflineSmall Offline

Also, when clearing your channel and re-uploading your data, you do want to first check that the channel has 0 entries in it before you initiate your upload. My guess is you are getting the error about timestamp already present because of the data that is not cleared from the channel.

April 2, 2019
9:10 pm
Avatar
Metawatch
New Member
Members
Forum Posts: 4
Member Since:
January 18, 2018
sp_UserOfflineSmall Offline

Thank you for your informative response and recommendations...
As nothing in life is easy here are a few more variables I need to deal with.

1) The data feed is in reality being used in real time to support knowledge about water levels for a local community. (Free)
2) The number of feeds currently active exceeds Thingspeak free level, and is operating under the grandfather clause.
3) Would like confirmation if the entire database were to be deleted that it would not prevent the continuiation of the grandfather clause.Surprised

4) Setting up an additional feed (Filtered) not feasable as
i) Max Feed Count exceeded
ii) False Positives are random based on local weather variables causing water infiltration (Pressure Transducer) during daytime melt conditions and subsequent freezing (Expansion/triggering) due to gound temp nighttime conditions. (Working on a solar based heating "defrost" but not available for this year)

From the sound of it if I do a complete channel clear and re-upload will require
a) complete download (Archive)
b) Merging Downloaded data with (Historical Pre-Cleansed Data) - within a 15 min timeframe (To avoid additional channel data upload data loss)

Any Constraints on a free account to upload approx. 30,000 entries ?
Sound like a viable, workable plan ?

Note: No room for failure as freshet flood period is "any day now"....

Very Nervous ---Risk may exceed potentia benefit....

April 9, 2019
3:03 pm
Avatar
cstapels
Moderator
Members


Moderators
Forum Posts: 730
Member Since:
March 7, 2017
sp_UserOfflineSmall Offline

Sounds like a very interesting project.

The grandfather clause is for the number of channels. Depending on your workflow, you may be able to condense data into more fields, and use fewer channels.
If you clear all the data in a channel, it will not effect the grandfather status for the number of channels in use. If you delete the channel, it WILL change the status with respect to the new channel limit policy.

With respect to point 4)ii) Why are you worried about historical false positives if they are random and are actually caused by real data? Can you explain again why you need to remove false positives from the historical record?
To solve concern b), I suggest you borrow a channel from a friend, and write the data you want to fix to that channel. Then you can make sure the process works, and just add new data as it comes in to that channel. Then when you are ready to do the whole switch, you can be sure it will work fine.

If you do decide to copy all 30 k entries, you will be limited to 960 messages per update. You will need at least 15 seconds between requests. If you estimate 20 seconds, you can complete this in 10 minutes. You cannot use a single MATLAB analysis with a loop, as the compute time is limited. You could manually press the run button more frequently. You could use a time control to run the job every 5 minutes, and the process would complete in a few days. Keep in mind that this will also consume messages, but you have 3M messages for the year. I recommend thingSpeakRead() and thingSpeakWrite() over the bulk write API as the syntax is a bit easier.

Until I understand your workflow better, it seems that creating a derived channel for the new data and processing it is your best option. You may also be able to purchase a license to increase the number of messages per write and the number of channels allowed. I hope the weather cooperates with you!

Forum Timezone: America/New_York

Most Users Ever Online: 166

Currently Online:
23 Guest(s)

Currently Browsing this Page:
1 Guest(s)

Top Posters:

rw950431: 272

Vinod: 240

piajola: 95

turgo: 70

vespapierre: 63

Adarsh_Murthy: 62

Member Stats:

Guest Posters: 1

Members: 8665

Moderators: 1

Admins: 2

Forum Stats:

Groups: 3

Forums: 14

Topics: 1600

Posts: 5760

Newest Members:

ssh47k, aaceitunog01, AlinaCutle, MARTO, simplicity1, jango6954

Moderators: cstapels: 460

Administrators: Hans: 405, lee: 457