Download Entire Channel Feed | Announcements | Forum

Avatar

Please consider registering
Guest

Search

— Forum Scope —






— Match —





— Forum Options —





Minimum search word length is 3 characters - maximum search word length is 84 characters

Register Lost password?
sp_Feed sp_TopicIcon
Download Entire Channel Feed
Avatar
lee

Forum Posts: 457
sp_UserOfflineSmall Offline
1
April 1, 2014 - 11:24 pm
sp_Permalink sp_Print

Based on many user requests, we just released a new feature that allows you to download all of your Channel data in CSV format. To do so, open your Channel in a web browser, click on the "Data Import / Export" tab, and you should see a button to download your data.

lee

Avatar
turgo

Gold
Forum Posts: 64
sp_UserOfflineSmall Offline
2
April 4, 2014 - 7:55 pm
sp_Permalink sp_Print

Great Addition! Thanks.

A few observations about how it is working:

1. The download is incomplete. Downloading four channels each with about a million points, I'm only getting the first 83,000 to 158,000 points. Interestingly, the last point is an even number in the thousands.
2. The download slows to a crawl. It starts out at 35 k/sec, and falls to over about two minutes to 500 bytes/ sec. Could this be related to the incomplete downloading? Is the data throttled to the point that it "times out"?

Avatar
lee

Forum Posts: 457
sp_UserOfflineSmall Offline
3
April 4, 2014 - 11:14 pm
sp_Permalink sp_Print

Hi,

I'll take a look at this, what is one of the Channel IDs that is not downloading completely?

lee

Avatar
turgo

Gold
Forum Posts: 64
sp_UserOfflineSmall Offline
4
April 5, 2014 - 1:03 am
sp_Permalink sp_Print

Channel 5140.

Avatar
lee

Forum Posts: 457
sp_UserOfflineSmall Offline
5
April 5, 2014 - 2:21 am
sp_Permalink sp_Print

Ok I see the problem now for Channels with a large number of feeds--as the download progresses our database has to progressively scan larger and larger numbers of table rows to get the next chunk of data, which increasingly slows things down.

I'll work on optimizing this process so that you can download large Channel feeds at a constant instead of diminishing rate, and will reply back here when it's ready.

lee

Avatar
turgo

Gold
Forum Posts: 64
sp_UserOfflineSmall Offline
6
April 5, 2014 - 3:13 am
sp_Permalink sp_Print

Awesome, lee!

Avatar
lee

Forum Posts: 457
sp_UserOfflineSmall Offline
7
April 7, 2014 - 3:11 am
sp_Permalink sp_Print

Hi,

You should be able to download your entire Channel feed at a nearly constant rate now. Please let me know if it's working ok for you.

lee

Avatar
turgo

Gold
Forum Posts: 64
sp_UserOfflineSmall Offline
8
April 9, 2014 - 1:35 am
sp_Permalink sp_Print

Hi Lee,
It's working better. Most downloads have successfully retrieved the whole channel, though a few have not. It takes quite a while at 30KB/sec.

It would be especially helpful if the download file were named after the channel, and if the header file had the actual names of the fields listed, instead of field1, field2, etc.

Thanks for this new feature.

Avatar
turgo

Gold
Forum Posts: 64
sp_UserOfflineSmall Offline
9
April 10, 2014 - 12:23 am
sp_Permalink sp_Print

I'm still having problems retrieving all the data. For channel 6278 with 833564 Entries, I tried twice. I downloaded 577000 entries the first time, and 605000 the second. Why does it stop after an even thousand entries?

Avatar
lee

Forum Posts: 457
sp_UserOfflineSmall Offline
10
April 10, 2014 - 12:51 pm
sp_Permalink sp_Print

Hi,

The download errors are most likely due to random internet communication errors that we have no control over, but that are more likely to happen the longer a single connection is open. The downloads are streamed in chunks of 1000 entries from our database, so that's why if there's a problem the last entry will most likely be a multiple of 1000.

lee

Avatar
CorBee
Netherlands

Gold
Forum Posts: 37
sp_UserOfflineSmall Offline
11
April 13, 2014 - 2:16 am
sp_Permalink sp_Print

Hi

I like this new addition but to avoid the issues people have now I would like you to consider the following:

Why not create a file at your server based on the request of a user and inform the user the moment the file is available for download ? The latter could be done in 2 ways:
a) user gets a mail message that file X is ready for download or
b) user will see a file ready for download when logging in to his account.

regards
Cor

Avatar
turgo

Gold
Forum Posts: 64
sp_UserOfflineSmall Offline
12
April 15, 2014 - 12:36 am
sp_Permalink sp_Print

Another useful feature might be to divide the history files into separate years. A years worth of 15 second data is more than 100 MB. This will get unwieldy in five years.

Avatar
frazelle09
Mexicali, Baja California

Silver
Forum Posts: 5
sp_UserOfflineSmall Offline
13
December 19, 2015 - 1:43 pm
sp_Permalink sp_Print

i realize that this thread is a little outdated but it's almost on the same topic

For how long is data kept on ThingSpeak? One of the previous posters mentioned 1,000,000 data points. Like, gee, that's a bunch.

Have a wonderful day and thanks for ThingSpeak! 🙂

"The earth is one country and mankind its citizens."
Bahá'u'lláh
"La tierra es un sólo país y la humanidad sus ciudadanos."

Avatar
rw950431

Forum Posts: 252
sp_UserOfflineSmall Offline
14
December 23, 2015 - 6:36 am
sp_Permalink sp_Print

Apart from the 15 second limit on updates there are no limits that thingspeak imposes (not sure how long that can last). 1 million data points may seem like a lot but its less than 6 months of 15 second data.

Avatar
johnmx
New Member
Forum Posts: 3
sp_UserOfflineSmall Offline
15
February 27, 2016 - 8:59 am
sp_Permalink sp_Print

I cannot export anything from my channels, always receive an error.

Using the Export option "Download all of this Channel's feeds in CSV format." in the export tab does not work either.
It downloads a csv file with the time stamps of all points, but without the data values.

This means I lost all my data?

Avatar
Adarsh_Murthy

Forum Posts: 59
sp_UserOfflineSmall Offline
16
February 29, 2016 - 9:38 am
sp_Permalink sp_Print

Hi,

Is your channel private or public? Also, do you see the data in your channel page: https://thingspeak.com/channels//private_show?

Thanks,
Adarsh

Avatar
johnmx
New Member
Forum Posts: 3
sp_UserOfflineSmall Offline
17
March 2, 2016 - 6:45 pm
sp_Permalink sp_Print

Both my channels are private.
But the export still doesn't work after changing them to public.

When I click in your link I only see the following message: "This channel is not public.".

Avatar
Adarsh_Murthy

Forum Posts: 59
sp_UserOfflineSmall Offline
18
March 3, 2016 - 2:11 pm
sp_Permalink sp_Print

Sorry, the link has a typo - the channel ID was missing. In the below link, replace the channel ID 12397 with your channel ID:

https://thingspeak.com/channels/12397/private_show

In the private view of your channel, do you see data in your channels? If you do see data, then another way to download data from the channel is the following url (note that you need to change the channelID 12397 to your channel ID):

https://thingspeak.com/channels/12397/feeds.csv?

This will download a csv with 100 rows in it. You can change this by using the info on this link:

https://www.mathworks.com/help/thingspeak/get-a-channel-feed.html

On clicking on the button to "Download all of this Channel's feeds in CSV format" in the "Data Import/Export" tab of a private channel I am able to download data. Are you able to download your data now?

-Adarsh

Avatar
johnmx
New Member
Forum Posts: 3
sp_UserOfflineSmall Offline
19
March 3, 2016 - 6:57 pm
sp_Permalink sp_Print

Today the export is working well on both channels.
Even the JSON and the XML export are working. Before they returned an error (page not found in one and -1 error in the other).
I don't know what was the problem.
Anyway, thank you for your help.

Forum Timezone: America/New_York

Most Users Ever Online: 114

Currently Online:
17 Guest(s)

Currently Browsing this Page:
1 Guest(s)

Top Posters:

rw950431: 252

Vinod: 164

cstapels: 96

piajola: 75

vespapierre: 63

Adarsh_Murthy: 59

Newest Members:

winnieeq69

AutoCheernpen

Bostermet

engladNed

iivenul

olewaeayafaku

Forum Stats:

Groups: 4

Forums: 17

Topics: 1313

Posts: 4565

 

Member Stats:

Guest Posters: 1

Members: 5703

Moderators: 0

Admins: 2

Administrators: Hans, lee