Download Entire Channel Feed

The ThingSpeak community site has been upgraded to a new site. This site is currently in read-only mode. You can ask questions or post and read discussions on the new site.

Download Entire Channel Feed

April 1, 2014
11:24 pm

lee

Admin

Forum Posts: 457

Member Since:
March 14, 2011

Offline

Based on many user requests, we just released a new feature that allows you to download all of your Channel data in CSV format. To do so, open your Channel in a web browser, click on the "Data Import / Export" tab, and you should see a button to download your data.

lee

April 4, 2014
7:55 pm

turgo

Gold

Members

Forum Posts: 77

Member Since:
June 16, 2013

Offline

Great Addition! Thanks.

A few observations about how it is working:

1. The download is incomplete. Downloading four channels each with about a million points, I'm only getting the first 83,000 to 158,000 points. Interestingly, the last point is an even number in the thousands.
2. The download slows to a crawl. It starts out at 35 k/sec, and falls to over about two minutes to 500 bytes/ sec. Could this be related to the incomplete downloading? Is the data throttled to the point that it "times out"?

April 4, 2014
11:14 pm

lee

Admin

Forum Posts: 457

Member Since:
March 14, 2011

Offline

Hi,

I'll take a look at this, what is one of the Channel IDs that is not downloading completely?

lee

April 5, 2014
1:03 am

turgo

Gold

Members

Forum Posts: 77

Member Since:
June 16, 2013

Offline

Channel 5140.

April 5, 2014
2:21 am

lee

Admin

Forum Posts: 457

Member Since:
March 14, 2011

Offline

Ok I see the problem now for Channels with a large number of feeds--as the download progresses our database has to progressively scan larger and larger numbers of table rows to get the next chunk of data, which increasingly slows things down.

I'll work on optimizing this process so that you can download large Channel feeds at a constant instead of diminishing rate, and will reply back here when it's ready.

lee

April 5, 2014
3:13 am

turgo

Gold

Members

Forum Posts: 77

Member Since:
June 16, 2013

Offline

Awesome, lee!

April 7, 2014
3:11 am

lee

Admin

Forum Posts: 457

Member Since:
March 14, 2011

Offline

Hi,

You should be able to download your entire Channel feed at a nearly constant rate now. Please let me know if it's working ok for you.

lee

April 9, 2014
1:35 am

turgo

Gold

Members

Forum Posts: 77

Member Since:
June 16, 2013

Offline

Hi Lee,
It's working better. Most downloads have successfully retrieved the whole channel, though a few have not. It takes quite a while at 30KB/sec.

It would be especially helpful if the download file were named after the channel, and if the header file had the actual names of the fields listed, instead of field1, field2, etc.

Thanks for this new feature.

April 10, 2014
12:23 am

turgo

Gold

Members

Forum Posts: 77

Member Since:
June 16, 2013

Offline

I'm still having problems retrieving all the data. For channel 6278 with 833564 Entries, I tried twice. I downloaded 577000 entries the first time, and 605000 the second. Why does it stop after an even thousand entries?

April 10, 2014
12:51 pm

lee

Admin

Forum Posts: 457

Member Since:
March 14, 2011

Offline

Hi,

The download errors are most likely due to random internet communication errors that we have no control over, but that are more likely to happen the longer a single connection is open. The downloads are streamed in chunks of 1000 entries from our database, so that's why if there's a problem the last entry will most likely be a multiple of 1000.

lee

April 13, 2014
2:16 am

CorBee

Netherlands

Gold

Members

Forum Posts: 37

Member Since:
December 10, 2012

Offline

I like this new addition but to avoid the issues people have now I would like you to consider the following:

Why not create a file at your server based on the request of a user and inform the user the moment the file is available for download ? The latter could be done in 2 ways:
a) user gets a mail message that file X is ready for download or
b) user will see a file ready for download when logging in to his account.

regards
Cor

April 15, 2014
12:36 am

turgo

Gold

Members

Forum Posts: 77

Member Since:
June 16, 2013

Offline

Another useful feature might be to divide the history files into separate years. A years worth of 15 second data is more than 100 MB. This will get unwieldy in five years.

December 19, 2015
1:43 pm

frazelle09

Mexicali, Baja California

Silver

Members

Forum Posts: 5

Member Since:
December 16, 2015

Offline

i realize that this thread is a little outdated but it's almost on the same topic

For how long is data kept on ThingSpeak? One of the previous posters mentioned 1,000,000 data points. Like, gee, that's a bunch.

Have a wonderful day and thanks for ThingSpeak! 🙂

"The earth is one country and mankind its citizens."
Bahá'u'lláh
"La tierra es un sólo país y la humanidad sus ciudadanos."

December 23, 2015
6:36 am

rw950431

Top

Members

Forum Posts: 279

Member Since:
January 29, 2014

Offline

Apart from the 15 second limit on updates there are no limits that thingspeak imposes (not sure how long that can last). 1 million data points may seem like a lot but its less than 6 months of 15 second data.

February 27, 2016
8:59 am

johnmx

New Member

Members

Forum Posts: 3

Member Since:
February 27, 2016

Offline

I cannot export anything from my channels, always receive an error.

Using the Export option "Download all of this Channel's feeds in CSV format." in the export tab does not work either.
It downloads a csv file with the time stamps of all points, but without the data values.

This means I lost all my data?

February 29, 2016
9:38 am

Adarsh_Murthy

MathWorks

Members

Forum Posts: 62

Member Since:
August 25, 2015

Offline

Hi,

Is your channel private or public? Also, do you see the data in your channel page: https://thingspeak.com/channels//private_show?

Thanks,
Adarsh

March 2, 2016
6:45 pm

johnmx

New Member

Members

Forum Posts: 3

Member Since:
February 27, 2016

Offline

Both my channels are private.
But the export still doesn't work after changing them to public.

When I click in your link I only see the following message: "This channel is not public.".

March 3, 2016
2:11 pm

Adarsh_Murthy

MathWorks

Members

Forum Posts: 62

Member Since:
August 25, 2015

Offline

Sorry, the link has a typo - the channel ID was missing. In the below link, replace the channel ID 12397 with your channel ID:

https://thingspeak.com/channels/12397/private_show

In the private view of your channel, do you see data in your channels? If you do see data, then another way to download data from the channel is the following url (note that you need to change the channelID 12397 to your channel ID):

https://thingspeak.com/channels/12397/feeds.csv?

This will download a csv with 100 rows in it. You can change this by using the info on this link:

https://www.mathworks.com/help/thingspeak/get-a-channel-feed.html

On clicking on the button to "Download all of this Channel's feeds in CSV format" in the "Data Import/Export" tab of a private channel I am able to download data. Are you able to download your data now?

-Adarsh

March 3, 2016
6:57 pm

johnmx

New Member

Members

Forum Posts: 3

Member Since:
February 27, 2016

Offline

Today the export is working well on both channels.
Even the JSON and the XML export are working. Before they returned an error (page not found in one and -1 error in the other).
I don't know what was the problem.
Anyway, thank you for your help.

Forum Timezone: America/New_York

Most Users Ever Online: 166

Currently Online:
35 Guest(s)

Currently Browsing this Page:
1 Guest(s)