Download Entire Channel Feed | Announcements | Forum

Avatar

Please consider registering
Guest

sp_LogInOut Log In sp_Registration Register

Register | Lost password?
Advanced Search

— Forum Scope —






— Match —





— Forum Options —





Minimum search word length is 3 characters - maximum search word length is 84 characters

sp_Feed sp_TopicIcon
Download Entire Channel Feed
No permission to create posts
April 1, 2014
11:24 pm
Avatar
lee

Admin
Forum Posts: 457
Member Since:
March 14, 2011
sp_UserOfflineSmall Offline

Based on many user requests, we just released a new feature that allows you to download all of your Channel data in CSV format. To do so, open your Channel in a web browser, click on the "Data Import / Export" tab, and you should see a button to download your data.

lee

April 4, 2014
7:55 pm
Avatar
turgo

Gold
Members
Forum Posts: 70
Member Since:
June 16, 2013
sp_UserOfflineSmall Offline

Great Addition! Thanks.

A few observations about how it is working:

1. The download is incomplete. Downloading four channels each with about a million points, I'm only getting the first 83,000 to 158,000 points. Interestingly, the last point is an even number in the thousands.
2. The download slows to a crawl. It starts out at 35 k/sec, and falls to over about two minutes to 500 bytes/ sec. Could this be related to the incomplete downloading? Is the data throttled to the point that it "times out"?

April 4, 2014
11:14 pm
Avatar
lee

Admin
Forum Posts: 457
Member Since:
March 14, 2011
sp_UserOfflineSmall Offline

Hi,

I'll take a look at this, what is one of the Channel IDs that is not downloading completely?

lee

April 5, 2014
1:03 am
Avatar
turgo

Gold
Members
Forum Posts: 70
Member Since:
June 16, 2013
sp_UserOfflineSmall Offline

Channel 5140.

April 5, 2014
2:21 am
Avatar
lee

Admin
Forum Posts: 457
Member Since:
March 14, 2011
sp_UserOfflineSmall Offline

Ok I see the problem now for Channels with a large number of feeds--as the download progresses our database has to progressively scan larger and larger numbers of table rows to get the next chunk of data, which increasingly slows things down.

I'll work on optimizing this process so that you can download large Channel feeds at a constant instead of diminishing rate, and will reply back here when it's ready.

lee

April 5, 2014
3:13 am
Avatar
turgo

Gold
Members
Forum Posts: 70
Member Since:
June 16, 2013
sp_UserOfflineSmall Offline

Awesome, lee!

April 7, 2014
3:11 am
Avatar
lee

Admin
Forum Posts: 457
Member Since:
March 14, 2011
sp_UserOfflineSmall Offline

Hi,

You should be able to download your entire Channel feed at a nearly constant rate now. Please let me know if it's working ok for you.

lee

April 9, 2014
1:35 am
Avatar
turgo

Gold
Members
Forum Posts: 70
Member Since:
June 16, 2013
sp_UserOfflineSmall Offline

Hi Lee,
It's working better. Most downloads have successfully retrieved the whole channel, though a few have not. It takes quite a while at 30KB/sec.

It would be especially helpful if the download file were named after the channel, and if the header file had the actual names of the fields listed, instead of field1, field2, etc.

Thanks for this new feature.

April 10, 2014
12:23 am
Avatar
turgo

Gold
Members
Forum Posts: 70
Member Since:
June 16, 2013
sp_UserOfflineSmall Offline

I'm still having problems retrieving all the data. For channel 6278 with 833564 Entries, I tried twice. I downloaded 577000 entries the first time, and 605000 the second. Why does it stop after an even thousand entries?

April 10, 2014
12:51 pm
Avatar
lee

Admin
Forum Posts: 457
Member Since:
March 14, 2011
sp_UserOfflineSmall Offline

Hi,

The download errors are most likely due to random internet communication errors that we have no control over, but that are more likely to happen the longer a single connection is open. The downloads are streamed in chunks of 1000 entries from our database, so that's why if there's a problem the last entry will most likely be a multiple of 1000.

lee

April 13, 2014
2:16 am
Avatar
CorBee
Netherlands

Gold
Members
Forum Posts: 37
Member Since:
December 10, 2012
sp_UserOfflineSmall Offline

Hi

I like this new addition but to avoid the issues people have now I would like you to consider the following:

Why not create a file at your server based on the request of a user and inform the user the moment the file is available for download ? The latter could be done in 2 ways:
a) user gets a mail message that file X is ready for download or
b) user will see a file ready for download when logging in to his account.

regards
Cor

April 15, 2014
12:36 am
Avatar
turgo

Gold
Members
Forum Posts: 70
Member Since:
June 16, 2013
sp_UserOfflineSmall Offline

Another useful feature might be to divide the history files into separate years. A years worth of 15 second data is more than 100 MB. This will get unwieldy in five years.

December 19, 2015
1:43 pm
Avatar
frazelle09
Mexicali, Baja California

Silver
Members
Forum Posts: 5
Member Since:
December 16, 2015
sp_UserOfflineSmall Offline

i realize that this thread is a little outdated but it's almost on the same topic

For how long is data kept on ThingSpeak? One of the previous posters mentioned 1,000,000 data points. Like, gee, that's a bunch.

Have a wonderful day and thanks for ThingSpeak! 🙂

"The earth is one country and mankind its citizens."
Bahá'u'lláh
"La tierra es un sólo país y la humanidad sus ciudadanos."

December 23, 2015
6:36 am
Avatar
rw950431

Top
Members
Forum Posts: 261
Member Since:
January 30, 2014
sp_UserOfflineSmall Offline

Apart from the 15 second limit on updates there are no limits that thingspeak imposes (not sure how long that can last). 1 million data points may seem like a lot but its less than 6 months of 15 second data.

February 27, 2016
8:59 am
Avatar
johnmx
New Member
Members
Forum Posts: 3
Member Since:
February 27, 2016
sp_UserOfflineSmall Offline

I cannot export anything from my channels, always receive an error.

Using the Export option "Download all of this Channel's feeds in CSV format." in the export tab does not work either.
It downloads a csv file with the time stamps of all points, but without the data values.

This means I lost all my data?

February 29, 2016
9:38 am
Avatar
Adarsh_Murthy

MathWorks
Members
Forum Posts: 62
Member Since:
August 25, 2015
sp_UserOfflineSmall Offline

Hi,

Is your channel private or public? Also, do you see the data in your channel page: https://thingspeak.com/channels//private_show?

Thanks,
Adarsh

March 2, 2016
6:45 pm
Avatar
johnmx
New Member
Members
Forum Posts: 3
Member Since:
February 27, 2016
sp_UserOfflineSmall Offline

Both my channels are private.
But the export still doesn't work after changing them to public.

When I click in your link I only see the following message: "This channel is not public.".

March 3, 2016
2:11 pm
Avatar
Adarsh_Murthy

MathWorks
Members
Forum Posts: 62
Member Since:
August 25, 2015
sp_UserOfflineSmall Offline

Sorry, the link has a typo - the channel ID was missing. In the below link, replace the channel ID 12397 with your channel ID:

https://thingspeak.com/channels/12397/private_show

In the private view of your channel, do you see data in your channels? If you do see data, then another way to download data from the channel is the following url (note that you need to change the channelID 12397 to your channel ID):

https://thingspeak.com/channels/12397/feeds.csv?

This will download a csv with 100 rows in it. You can change this by using the info on this link:

https://www.mathworks.com/help/thingspeak/get-a-channel-feed.html

On clicking on the button to "Download all of this Channel's feeds in CSV format" in the "Data Import/Export" tab of a private channel I am able to download data. Are you able to download your data now?

-Adarsh

March 3, 2016
6:57 pm
Avatar
johnmx
New Member
Members
Forum Posts: 3
Member Since:
February 27, 2016
sp_UserOfflineSmall Offline

Today the export is working well on both channels.
Even the JSON and the XML export are working. Before they returned an error (page not found in one and -1 error in the other).
I don't know what was the problem.
Anyway, thank you for your help.

Forum Timezone: America/New_York

Most Users Ever Online: 114

Currently Online:
17 Guest(s)

Currently Browsing this Page:
1 Guest(s)

Top Posters:

rw950431: 261

Vinod: 196

piajola: 85

turgo: 70

vespapierre: 63

Adarsh_Murthy: 62

Member Stats:

Guest Posters: 1

Members: 5703

Moderators: 0

Admins: 2

Forum Stats:

Groups: 4

Forums: 17

Topics: 1313

Posts: 4565

Newest Members:

celinabw2, FitoSprayscroma, LunnonGor, lancevl16, bernicenk60, veronicadq18

Administrators: Hans: 387, lee: 457