Sending data to ThingSpeak API from device but channel is not updated -- what changed? | Announcements | Forum

Avatar

Please consider registering
Guest

sp_LogInOut Log In sp_Registration Register

Register | Lost password?
Advanced Search

— Forum Scope —






— Match —





— Forum Options —





Minimum search word length is 3 characters - maximum search word length is 84 characters

sp_Feed sp_TopicIcon
Sending data to ThingSpeak API from device but channel is not updated -- what changed?
No permission to create posts
January 3, 2019
6:11 pm
Avatar
Vinod

MathWorks
Members
Forum Posts: 258
Member Since:
May 1, 2016
sp_UserOfflineSmall Offline

During a planned update we noticed that some devices are sending data to ThingSpeak and closing the TCP connection without providing sufficient time for ThingSpeak API servers to respond, or confirming the data made its way to the ThingSpeak servers.

This is not a recommended approach as the assumption that the data being sent is accepted may not always be valid. For example, if your device tried to update a channel 50 times a second, that is faster than your allowed update rate. If the device did check the response, it would notice that it got a '0' response back indicating that the channel was not updated.

In the case of closing the connection before ThingSpeak sent a response code back, a HTTP 499 status code would be sent back, but, because the device wasn't leaving the connection open, this was not known to the device.

It is strongly recommended that you consider using the ThingSpeak library from https://github.com/mathworks/thingspeak-arduino for updating ThingSpeak channels from devices like Arduino, ESP8266, Particle Photon, Particle Electron, etc. The library provides a higher level API and manages the connections for you.

If you are noticing that your channels are no longer being updated by your Arduino or ESP8266 device, consider:

1) Modifying your code to use the library

OR

2) Modifying your code to wait till it gets a HTTP response code from ThingSpeak, or, at the very least waiting for 1 second before closing the connection and/or putting the device to deep sleep mode.

January 3, 2019
11:21 pm
Avatar
estave
New Member
Members
Forum Posts: 3
Member Since:
January 3, 2019
sp_UserOfflineSmall Offline

What about the devices that we have out in the field that have been reporting fine until today? These are going to be hard to get back and get reprogrammed! I have devices that report 1 time per minute and they have all stopped working now.

January 4, 2019
2:27 am
Avatar
danlorek
New Member
Members
Forum Posts: 2
Member Since:
January 4, 2019
sp_UserOfflineSmall Offline

This is a pretty severe change, is it possible to revert this?

I have several sensors that stopped reporting as a result of this. They are battery operated, therefore I try to reduce their WIFI connection time as much as possible and was not waiting for a response.

January 4, 2019
2:59 am
Avatar
mikekgr
New Member
Members
Forum Posts: 3
Member Since:
January 4, 2019
sp_UserOfflineSmall Offline

Dear Sirs,
the same it happens to me too. I think this change will be affect many thingspeak users unfortunately...
I use ESP8266 with Arduino core in case of this matter...

Thanks and Best Regards,
Mike Kranidis

Vinod said

During a planned update we noticed that some devices are sending data to ThingSpeak and closing the TCP connection without providing sufficient time for ThingSpeak API servers to respond, or confirming the data made its way to the ThingSpeak servers.

This is not a recommended approach as the assumption that the data being sent is accepted may not always be valid. For example, if your device tried to update a channel 50 times a second, that is faster than your allowed update rate. If the device did check the response, it would notice that it got a '0' response back indicating that the channel was not updated.

In the case of closing the connection before ThingSpeak sent a response code back, a HTTP 499 status code would be sent back, but, because the device wasn't leaving the connection open, this was not known to the device.

It is strongly recommended that you consider using the ThingSpeak library from https://github.com/mathworks/thingspeak-arduino for updating ThingSpeak channels from devices like Arduino, ESP8266, Particle Photon, Particle Electron, etc. The library provides a higher level API and manages the connections for you.

If you are noticing that your channels are no longer being updated by your Arduino or ESP8266 device, consider:

1) Modifying your code to use the library

OR

2) Modifying your code to wait till it gets a HTTP response code from ThingSpeak, or, at the very least waiting for 1 second before closing the connection and/or putting the device to deep sleep mode.  

January 4, 2019
7:17 am
Avatar
Vinod

MathWorks
Members
Forum Posts: 258
Member Since:
May 1, 2016
sp_UserOfflineSmall Offline

estave said

What about the devices that we have out in the field that have been reporting fine until today? These are going to be hard to get back and get reprogrammed! I have devices that report 1 time per minute and they have all stopped working now.  

We'll be looking into changing some timeout settings to see if it can reduce the likelihood of this. Can you tell me what channel numbers are so we can monitor it as we change the load balancer settings ?

January 4, 2019
7:20 am
Avatar
Vinod

MathWorks
Members
Forum Posts: 258
Member Since:
May 1, 2016
sp_UserOfflineSmall Offline

danlorek said

This is a pretty severe change, is it possible to revert this?

I have several sensors that stopped reporting as a result of this. They are battery operated, therefore I try to reduce their WIFI connection time as much as possible and was not waiting for a response.  

If the sole reason to close the connection early is to optimize the battery life, I would strongly suggest using the MQTT API. The protocol has much less overhead than the REST API and the MQTT service we have takes care of doing retries, etc.

If that is not possible, even a small delay of a few hundred milliseconds between writing the data and closing the connection, say doing other cleanup tasks on the embedded device, should be sufficient.

January 4, 2019
7:23 am
Avatar
Vinod

MathWorks
Members
Forum Posts: 258
Member Since:
May 1, 2016
sp_UserOfflineSmall Offline

mikekgr said

Dear Sirs,
the same it happens to me too. I think this change will be affect many thingspeak users unfortunately...
I use ESP8266 with Arduino core in case of this matter...

Thanks and Best Regards,
Mike Kranidis

  

Hi Mike,

We've been monitoring the amount of traffic impacted and would have reverted the change if it impacted a significant number of users. As it stands this only appears to impact a small percentage of requests.

Can you share your arduino sketch to vcherian mathworks com and I may be able to make some suggestions to resolve this on your devices.

January 4, 2019
9:06 am
Avatar
Zipgun
New Member
Members
Forum Posts: 3
Member Since:
January 4, 2019
sp_UserOfflineSmall Offline

All of my devices (9) have stopped as of 15:00z yesterday! Several devices are over 1000 miles away and I have no way do an OTA update to them.

Does closing early impact Mathworks? If I chose to operate on a send-and-pray methodology and a few data deposits are not registered, I have side-band monitoring to alert me.

in a perfect world, we would all program elegant code respecting try/catch rules with timers and interrupts. Consider that these are tiny, limited function SoC devices that, at least in my case, are nearly maxed out on memory. Timer interrupts are precious, and I have to prioritize data capture in sub-millisecond cycles for accurate averaging.

January 4, 2019
9:41 am
Avatar
mikekgr
New Member
Members
Forum Posts: 3
Member Since:
January 4, 2019
sp_UserOfflineSmall Offline

Dear @Vinod
I need to heavily modified my arduino sketch because in there, I am doing many other things like Blynk communication and calls among others so it is not easy at all. If you need, my channels are: 169516 and 179823

Vinod said

Hi Mike,

We've been monitoring the amount of traffic impacted and would have reverted the change if it impacted a significant number of users. As it stands this only appears to impact a small percentage of requests.

Can you share your arduino sketch to vcherian mathworks com and I may be able to make some suggestions to resolve this on your devices.  

January 4, 2019
10:01 am
Avatar
estave
New Member
Members
Forum Posts: 3
Member Since:
January 3, 2019
sp_UserOfflineSmall Offline

Here are a couple of channel #'s for you:

569373
558892
660884

January 4, 2019
10:20 am
Avatar
Vinod

MathWorks
Members
Forum Posts: 258
Member Since:
May 1, 2016
sp_UserOfflineSmall Offline

It appears from analysis that most, if not all, code experiencing this had this pattern:

if (client.connect(server,80)) {
String postStr = apiKey;
postStr +="&field3=";
postStr += String(t,1);
postStr +="&field4=";
postStr += String(h,1);
postStr += "

";

client.print("POST /update HTTP/1.1
");
client.print("Host: api.thingspeak.com
");
client.print("Connection: close
");
client.print("X-THINGSPEAKAPIKEY: "+apiKey+"
");
client.print("Content-Type: application/x-www-form-urlencoded
");
client.print("Content-Length: ");
client.print(postStr.length());
client.print("

");
client.print(postStr);
}
client.stop();

There are two problems with this code pattern:
1) These two lines could result in a malformed HTTP header with incorrect Content-Length. This gets rejected.

client.print("Content-Length: ");
client.print(postStr.length());

they really should be

client.print("Content-Length: ");
client.print(String(postStr.length()));

2) The closing of the connection immediately after the request was sent with the client.stop();

This send-and-pray that the data got posted approach used to work. However, the protection mechanisms we need in place on ThingSpeak to protect the service from bad actors sees the connection closed by the client and treats this as an abandoned request, and does not forward the request to our servers. So, sadly at this time we have not even seen the request to be able to store it.

A simple workaround that I have confirmed works is to wait ~200 milliseconds before client.stop(); However, the right approach is to read the response code to confirm the request actually was actioned.

If this 200 mSec is too much battery use for the device, and a truly fire-and-forget solution is what is desired, MQTT is what I would recommend.

January 4, 2019
10:40 am
Avatar
estave
New Member
Members
Forum Posts: 3
Member Since:
January 3, 2019
sp_UserOfflineSmall Offline

Notification in advance that this type of communication was going to stop working would have been the proper thing to do. Now I am left with many devices that will need to be retrieved and reprogrammed. I pay for a license for the extra data that we use, it would be nice to have some notification.

Here is the code in all of our sketches...

void loop()
{
if (client.connect(server, 80)) // "184.106.153.149" or api.thingspeak.com
{
sensors.requestTemperatures(); // Send the command to get temperatures
Serial.println("Temperature is: ");
Serial.println(sensors.getTempFByIndex(0)); // Why "byIndex"? You can have more than one IC on the same bus. 0 refers to the first IC on the wire
float Temp=sensors.getTempFByIndex(0);

String postStr = apiKey;
postStr += "&field1=";
postStr += String(Temp);
postStr += "

";

client.print("POST /update HTTP/1.1
");
client.print("Host: api.thingspeak.com
");
client.print("Connection: close
");
client.print("X-THINGSPEAKAPIKEY: " + apiKey + "
");
client.print("Content-Type: application/x-www-form-urlencoded
");
client.print("Content-Length: ");
client.print(postStr.length());
client.print("

");
client.print(postStr);

Serial.print("Temperature: ");
Serial.print(Temp);
Serial.println(". Send to Thingspeak.");
}
else
{
ESP.restart();
}
client.stop();

Serial.println("Waiting...");

delay(60000);

And also:

void loop()
{
if (client.connect(server, 80)) // "184.106.153.149" or api.thingspeak.com
{

String postStr = apiKey;
postStr += "&field1=";
postStr += String(feeder_rev);
postStr += "&field2=";
postStr += String(rssi);
postStr += "&field3=";
postStr += String(feeder_id);
postStr += "

";

client.print("POST /update HTTP/1.1
");
client.print("Host: api.thingspeak.com
");
client.print("Connection: close
");
client.print("X-THINGSPEAKAPIKEY: " + apiKey + "
");
client.print("Content-Type: application/x-www-form-urlencoded
");
client.print("Content-Length: ");
client.print(postStr.length());
client.print("

");
client.print(postStr);

rssi = WiFi.RSSI();
Serial.print("RSSI: ");
Serial.println(rssi);
Serial.print("Feeder Revolutions: ");
Serial.print(feeder_rev);
Serial.println(". Send to Thingspeak.");
}
else
{
ESP.restart();
}
client.stop();

Serial.println("Waiting...");

feeder_rev = 0;

delay(60000);

January 4, 2019
11:19 am
Avatar
Zipgun
New Member
Members
Forum Posts: 3
Member Since:
January 4, 2019
sp_UserOfflineSmall Offline

estave said

Notification in advance that this type of communication was going to stop working would have been the proper thing to do. Now I am left with many devices that will need to be retrieved and reprogrammed. I pay for a license for the extra data that we use, it would be nice to have some notification.

Here is the code in all of our sketches...
 

I agree with estave -- all of my devices store their IP address a channel field so I can find them (as they reboot their address can change). Since they are not reporting, I have no idea what their IP is and thus can't telnet or OTA to update them. Some advanced notice would have been the polite thing to do.

January 4, 2019
1:33 pm
Avatar
Vinod

MathWorks
Members
Forum Posts: 258
Member Since:
May 1, 2016
sp_UserOfflineSmall Offline

Based on the feedback in this thread, and in consideration that the devices are remote and may not be accessible, as of 1:15 US ET on Jan 4th, 2019, we temporarily reverted the config change. All systems should be operating just as they were as of 10:00 AM US ET on Jan 3rd, 2019.

For any code changes anyone made based on suggestions on this thread, the changes are compatible and the right thing to do for the long term. The behavior will return to what it was during the last 24 hours on Jan 30th, 2019.

Thank you for the feedback and patience as we reverted configuration.

To reiterate, If you have code pattern that immediately closes the client connection after the write, please make the changes to your code to
1) Use the ThingSpeak support library linked above OR
2) Check the return status code after the request is made OR
3) Add a delay of 200 milliseconds or more between the send and the closing of the connection

The above list is in highest to lowest preference solution.

It is requested you make the changes as soon as possible, definitely no later than Jan 28th, 2019.

January 4, 2019
1:51 pm
Avatar
danlorek
New Member
Members
Forum Posts: 2
Member Since:
January 4, 2019
sp_UserOfflineSmall Offline

Vinod said

If the sole reason to close the connection early is to optimize the battery life, I would strongly suggest using the MQTT API. The protocol has much less overhead than the REST API and the MQTT service we have takes care of doing retries, etc.

If that is not possible, even a small delay of a few hundred milliseconds between writing the data and closing the connection, say doing other cleanup tasks on the embedded device, should be sufficient.  

Yes, fair point and this has been on my list of things to do. I appreciate you reverting the change and will finally transition to MQTT soon now that I have a deadline. 🙂

January 4, 2019
1:53 pm
Avatar
Zipgun
New Member
Members
Forum Posts: 3
Member Since:
January 4, 2019
sp_UserOfflineSmall Offline

Thanks, I'll redeploy those that I can reach ASAP

January 4, 2019
2:46 pm
Avatar
mikekgr
New Member
Members
Forum Posts: 3
Member Since:
January 4, 2019
sp_UserOfflineSmall Offline

Thanks a lot Sir. I will make the necessary changes ASAP

Forum Timezone: America/New_York

Most Users Ever Online: 166

Currently Online:
21 Guest(s)

Currently Browsing this Page:
1 Guest(s)

Top Posters:

rw950431: 272

Vinod: 240

piajola: 95

turgo: 70

vespapierre: 63

Adarsh_Murthy: 62

Member Stats:

Guest Posters: 1

Members: 8665

Moderators: 1

Admins: 2

Forum Stats:

Groups: 3

Forums: 14

Topics: 1600

Posts: 5760

Newest Members:

Richard, naturepur, Nimper, kaplan@sor.cz, projectid030, mrlee234

Moderators: cstapels: 460

Administrators: Hans: 405, lee: 457