Sending data to ThingSpeak API from device but channel is not updated -- what changed? | Announcements | Forum

Avatar

Please consider registering
Guest

sp_LogInOut Log In sp_Registration Register

Register | Lost password?
Advanced Search

— Forum Scope —






— Match —





— Forum Options —





Minimum search word length is 3 characters - maximum search word length is 84 characters

sp_Feed sp_TopicIcon
Sending data to ThingSpeak API from device but channel is not updated -- what changed?
No permission to create posts
January 3, 2019
6:11 pm
Avatar
Vinod

MathWorks
Members
Forum Posts: 302
Member Since:
May 1, 2016
sp_UserOfflineSmall Offline

During a planned update we noticed that some devices are sending data to ThingSpeak and closing the TCP connection without providing sufficient time for ThingSpeak API servers to respond, or confirming the data made its way to the ThingSpeak servers.

This is not a recommended approach as the assumption that the data being sent is accepted may not always be valid. For example, if your device tried to update a channel 50 times a second, that is faster than your allowed update rate. If the device did check the response, it would notice that it got a '0' response back indicating that the channel was not updated.

In the case of closing the connection before ThingSpeak sent a response code back, a HTTP 499 status code would be sent back, but, because the device wasn't leaving the connection open, this was not known to the device.

It is strongly recommended that you consider using the ThingSpeak library from https://github.com/mathworks/thingspeak-arduino for updating ThingSpeak channels from devices like Arduino, ESP8266, Particle Photon, Particle Electron, etc. The library provides a higher level API and manages the connections for you.

If you are noticing that your channels are no longer being updated by your Arduino or ESP8266 device, consider:

1) Modifying your code to use the library

OR

2) Modifying your code to wait till it gets a HTTP response code from ThingSpeak, or, at the very least waiting for 1 second before closing the connection and/or putting the device to deep sleep mode.

January 3, 2019
11:21 pm
Avatar
estave
New Member
Members
Forum Posts: 3
Member Since:
January 4, 2019
sp_UserOfflineSmall Offline

What about the devices that we have out in the field that have been reporting fine until today? These are going to be hard to get back and get reprogrammed! I have devices that report 1 time per minute and they have all stopped working now.

January 4, 2019
2:27 am
Avatar
danlorek
New Member
Members
Forum Posts: 2
Member Since:
January 4, 2019
sp_UserOfflineSmall Offline

This is a pretty severe change, is it possible to revert this?

I have several sensors that stopped reporting as a result of this. They are battery operated, therefore I try to reduce their WIFI connection time as much as possible and was not waiting for a response.

January 4, 2019
2:59 am
Avatar
mikekgr

Silver
Members
Forum Posts: 5
Member Since:
January 4, 2019
sp_UserOfflineSmall Offline

Dear Sirs,
the same it happens to me too. I think this change will be affect many thingspeak users unfortunately...
I use ESP8266 with Arduino core in case of this matter...

Thanks and Best Regards,
Mike Kranidis

Vinod said

During a planned update we noticed that some devices are sending data to ThingSpeak and closing the TCP connection without providing sufficient time for ThingSpeak API servers to respond, or confirming the data made its way to the ThingSpeak servers.

This is not a recommended approach as the assumption that the data being sent is accepted may not always be valid. For example, if your device tried to update a channel 50 times a second, that is faster than your allowed update rate. If the device did check the response, it would notice that it got a '0' response back indicating that the channel was not updated.

In the case of closing the connection before ThingSpeak sent a response code back, a HTTP 499 status code would be sent back, but, because the device wasn't leaving the connection open, this was not known to the device.

It is strongly recommended that you consider using the ThingSpeak library from https://github.com/mathworks/thingspeak-arduino for updating ThingSpeak channels from devices like Arduino, ESP8266, Particle Photon, Particle Electron, etc. The library provides a higher level API and manages the connections for you.

If you are noticing that your channels are no longer being updated by your Arduino or ESP8266 device, consider:

1) Modifying your code to use the library

OR

2) Modifying your code to wait till it gets a HTTP response code from ThingSpeak, or, at the very least waiting for 1 second before closing the connection and/or putting the device to deep sleep mode.  

January 4, 2019
7:17 am
Avatar
Vinod

MathWorks
Members
Forum Posts: 302
Member Since:
May 1, 2016
sp_UserOfflineSmall Offline

estave said

What about the devices that we have out in the field that have been reporting fine until today? These are going to be hard to get back and get reprogrammed! I have devices that report 1 time per minute and they have all stopped working now.  

We'll be looking into changing some timeout settings to see if it can reduce the likelihood of this. Can you tell me what channel numbers are so we can monitor it as we change the load balancer settings ?

January 4, 2019
7:20 am
Avatar
Vinod

MathWorks
Members
Forum Posts: 302
Member Since:
May 1, 2016
sp_UserOfflineSmall Offline

danlorek said

This is a pretty severe change, is it possible to revert this?

I have several sensors that stopped reporting as a result of this. They are battery operated, therefore I try to reduce their WIFI connection time as much as possible and was not waiting for a response.  

If the sole reason to close the connection early is to optimize the battery life, I would strongly suggest using the MQTT API. The protocol has much less overhead than the REST API and the MQTT service we have takes care of doing retries, etc.

If that is not possible, even a small delay of a few hundred milliseconds between writing the data and closing the connection, say doing other cleanup tasks on the embedded device, should be sufficient.

January 4, 2019
7:23 am
Avatar
Vinod

MathWorks
Members
Forum Posts: 302
Member Since:
May 1, 2016
sp_UserOfflineSmall Offline

mikekgr said

Dear Sirs,
the same it happens to me too. I think this change will be affect many thingspeak users unfortunately...
I use ESP8266 with Arduino core in case of this matter...

Thanks and Best Regards,
Mike Kranidis

  

Hi Mike,

We've been monitoring the amount of traffic impacted and would have reverted the change if it impacted a significant number of users. As it stands this only appears to impact a small percentage of requests.

Can you share your arduino sketch to vcherian mathworks com and I may be able to make some suggestions to resolve this on your devices.

January 4, 2019
9:06 am
Avatar
Zipgun
New Member
Members
Forum Posts: 3
Member Since:
January 4, 2019
sp_UserOfflineSmall Offline

All of my devices (9) have stopped as of 15:00z yesterday! Several devices are over 1000 miles away and I have no way do an OTA update to them.

Does closing early impact Mathworks? If I chose to operate on a send-and-pray methodology and a few data deposits are not registered, I have side-band monitoring to alert me.

in a perfect world, we would all program elegant code respecting try/catch rules with timers and interrupts. Consider that these are tiny, limited function SoC devices that, at least in my case, are nearly maxed out on memory. Timer interrupts are precious, and I have to prioritize data capture in sub-millisecond cycles for accurate averaging.

January 4, 2019
9:41 am
Avatar
mikekgr

Silver
Members
Forum Posts: 5
Member Since:
January 4, 2019
sp_UserOfflineSmall Offline

Dear @Vinod
I need to heavily modified my arduino sketch because in there, I am doing many other things like Blynk communication and calls among others so it is not easy at all. If you need, my channels are: 169516 and 179823

Vinod said

Hi Mike,

We've been monitoring the amount of traffic impacted and would have reverted the change if it impacted a significant number of users. As it stands this only appears to impact a small percentage of requests.

Can you share your arduino sketch to vcherian mathworks com and I may be able to make some suggestions to resolve this on your devices.  

January 4, 2019
10:01 am
Avatar
estave
New Member
Members
Forum Posts: 3
Member Since:
January 4, 2019
sp_UserOfflineSmall Offline

Here are a couple of channel #'s for you:

569373
558892
660884

January 4, 2019
10:20 am
Avatar
Vinod

MathWorks
Members
Forum Posts: 302
Member Since:
May 1, 2016
sp_UserOfflineSmall Offline

It appears from analysis that most, if not all, code experiencing this had this pattern:

if (client.connect(server,80)) {
String postStr = apiKey;
postStr +="&field3=";
postStr += String(t,1);
postStr +="&field4=";
postStr += String(h,1);
postStr += "

";

client.print("POST /update HTTP/1.1
");
client.print("Host: api.thingspeak.com
");
client.print("Connection: close
");
client.print("X-THINGSPEAKAPIKEY: "+apiKey+"
");
client.print("Content-Type: application/x-www-form-urlencoded
");
client.print("Content-Length: ");
client.print(postStr.length());
client.print("

");
client.print(postStr);
}
client.stop();

There are two problems with this code pattern:
1) These two lines could result in a malformed HTTP header with incorrect Content-Length. This gets rejected.

client.print("Content-Length: ");
client.print(postStr.length());

they really should be

client.print("Content-Length: ");
client.print(String(postStr.length()));

2) The closing of the connection immediately after the request was sent with the client.stop();

This send-and-pray that the data got posted approach used to work. However, the protection mechanisms we need in place on ThingSpeak to protect the service from bad actors sees the connection closed by the client and treats this as an abandoned request, and does not forward the request to our servers. So, sadly at this time we have not even seen the request to be able to store it.

A simple workaround that I have confirmed works is to wait ~200 milliseconds before client.stop(); However, the right approach is to read the response code to confirm the request actually was actioned.

If this 200 mSec is too much battery use for the device, and a truly fire-and-forget solution is what is desired, MQTT is what I would recommend.

January 4, 2019
10:40 am
Avatar
estave
New Member
Members
Forum Posts: 3
Member Since:
January 4, 2019
sp_UserOfflineSmall Offline

Notification in advance that this type of communication was going to stop working would have been the proper thing to do. Now I am left with many devices that will need to be retrieved and reprogrammed. I pay for a license for the extra data that we use, it would be nice to have some notification.

Here is the code in all of our sketches...

void loop()
{
if (client.connect(server, 80)) // "184.106.153.149" or api.thingspeak.com
{
sensors.requestTemperatures(); // Send the command to get temperatures
Serial.println("Temperature is: ");
Serial.println(sensors.getTempFByIndex(0)); // Why "byIndex"? You can have more than one IC on the same bus. 0 refers to the first IC on the wire
float Temp=sensors.getTempFByIndex(0);

String postStr = apiKey;
postStr += "&field1=";
postStr += String(Temp);
postStr += "

";

client.print("POST /update HTTP/1.1
");
client.print("Host: api.thingspeak.com
");
client.print("Connection: close
");
client.print("X-THINGSPEAKAPIKEY: " + apiKey + "
");
client.print("Content-Type: application/x-www-form-urlencoded
");
client.print("Content-Length: ");
client.print(postStr.length());
client.print("

");
client.print(postStr);

Serial.print("Temperature: ");
Serial.print(Temp);
Serial.println(". Send to Thingspeak.");
}
else
{
ESP.restart();
}
client.stop();

Serial.println("Waiting...");

delay(60000);

And also:

void loop()
{
if (client.connect(server, 80)) // "184.106.153.149" or api.thingspeak.com
{

String postStr = apiKey;
postStr += "&field1=";
postStr += String(feeder_rev);
postStr += "&field2=";
postStr += String(rssi);
postStr += "&field3=";
postStr += String(feeder_id);
postStr += "

";

client.print("POST /update HTTP/1.1
");
client.print("Host: api.thingspeak.com
");
client.print("Connection: close
");
client.print("X-THINGSPEAKAPIKEY: " + apiKey + "
");
client.print("Content-Type: application/x-www-form-urlencoded
");
client.print("Content-Length: ");
client.print(postStr.length());
client.print("

");
client.print(postStr);

rssi = WiFi.RSSI();
Serial.print("RSSI: ");
Serial.println(rssi);
Serial.print("Feeder Revolutions: ");
Serial.print(feeder_rev);
Serial.println(". Send to Thingspeak.");
}
else
{
ESP.restart();
}
client.stop();

Serial.println("Waiting...");

feeder_rev = 0;

delay(60000);

January 4, 2019
11:19 am
Avatar
Zipgun
New Member
Members
Forum Posts: 3
Member Since:
January 4, 2019
sp_UserOfflineSmall Offline

estave said

Notification in advance that this type of communication was going to stop working would have been the proper thing to do. Now I am left with many devices that will need to be retrieved and reprogrammed. I pay for a license for the extra data that we use, it would be nice to have some notification.

Here is the code in all of our sketches...
 

I agree with estave -- all of my devices store their IP address a channel field so I can find them (as they reboot their address can change). Since they are not reporting, I have no idea what their IP is and thus can't telnet or OTA to update them. Some advanced notice would have been the polite thing to do.

January 4, 2019
1:33 pm
Avatar
Vinod

MathWorks
Members
Forum Posts: 302
Member Since:
May 1, 2016
sp_UserOfflineSmall Offline

Based on the feedback in this thread, and in consideration that the devices are remote and may not be accessible, as of 1:15 US ET on Jan 4th, 2019, we temporarily reverted the config change. All systems should be operating just as they were as of 10:00 AM US ET on Jan 3rd, 2019.

For any code changes anyone made based on suggestions on this thread, the changes are compatible and the right thing to do for the long term. The behavior will return to what it was during the last 24 hours on Jan 30th, 2019.

Thank you for the feedback and patience as we reverted configuration.

To reiterate, If you have code pattern that immediately closes the client connection after the write, please make the changes to your code to
1) Use the ThingSpeak support library linked above OR
2) Check the return status code after the request is made OR
3) Add a delay of 200 milliseconds or more between the send and the closing of the connection

The above list is in highest to lowest preference solution.

It is requested you make the changes as soon as possible, definitely no later than Jan 28th, 2019.

January 4, 2019
1:51 pm
Avatar
danlorek
New Member
Members
Forum Posts: 2
Member Since:
January 4, 2019
sp_UserOfflineSmall Offline

Vinod said

If the sole reason to close the connection early is to optimize the battery life, I would strongly suggest using the MQTT API. The protocol has much less overhead than the REST API and the MQTT service we have takes care of doing retries, etc.

If that is not possible, even a small delay of a few hundred milliseconds between writing the data and closing the connection, say doing other cleanup tasks on the embedded device, should be sufficient.  

Yes, fair point and this has been on my list of things to do. I appreciate you reverting the change and will finally transition to MQTT soon now that I have a deadline. 🙂

January 4, 2019
1:53 pm
Avatar
Zipgun
New Member
Members
Forum Posts: 3
Member Since:
January 4, 2019
sp_UserOfflineSmall Offline

Thanks, I'll redeploy those that I can reach ASAP

January 4, 2019
2:46 pm
Avatar
mikekgr

Silver
Members
Forum Posts: 5
Member Since:
January 4, 2019
sp_UserOfflineSmall Offline

Thanks a lot Sir. I will make the necessary changes ASAP

January 23, 2019
4:42 am
Avatar
Supawan
New Member
Members
Forum Posts: 3
Member Since:
January 23, 2019
sp_UserOfflineSmall Offline

Hello Vinod,

I got this e-mail from you.
………………………………………………………..
Hello,

During a recent planned update to the ThingSpeak server infrastructure, we noticed that one or more of your devices are sending data to ThingSpeak and closing the TCP connection without providing sufficient time for ThingSpeak API servers to respond, or confirming the data made its way to the ThingSpeak servers. This is not a recommended approach and may result in loss of data as the assumption that the data being sent is accepted may not always be valid. For example, if your device tried to update a channel 50 times a second, that is faster than your allowed update rate and the data may have been rejected. If the device did check the response, it would notice that it got a '0' response back indicating that the channel was not updated. In the case of closing the connection before ThingSpeak sent a response code back, a HTTP 499 status code would be sent back, but, because the device wasn't leaving the connection open, this was not known to the device.

Analysis of code patterns leads us to believe that the custom code that did this does not use the ThingSpeak library. It is strongly recommended that you use the ThingSpeak library from https://github.com/mathworks/thingspeak-arduino for updating ThingSpeak channels from devices like Arduino, ESP8266, Particle Photon, Particle Electron, etc. The library provides a higher level API and manages the connections for you. The library also contains new examples for newer devices that you may be using or considering.

If you do not wish to use the library, at the very least, you will need to delay for ~200 milliseconds after making the request to ThingSpeak API server, to ensure that the data made its way to our servers. You can also take a look at some of the examples published here as a starting point on how to update your device code.

Based on feedback from users, we temporarily rolled back our changes to the server infrastructure in order to give users time to update the code on their devices. However, we will be re-introducing the change on Feb 1st, 2019.

If you do not make changes to the code on your embedded device before Feb 1st 2019, requests from those devices may no longer make it to the ThingSpeak API servers and your data will no longer be collected on ThingSpeak.

Please refer to, or respond to, this forum post if you need more information.

Sincerely,
-Vinod
…………………………………………………

I have modified my code as you requirement but i'm not sure that i made the right way or not. I'm new for this so i got what you mean to is to add delay time around 200 ms after sending "String str="GET https://api.thingspeak.com/update?api_key=N11IT99RLTDPLENG&field1="+String(VBatt)+"&field2="+String(current)+"&field3="+String(temp)+"&field4="+String(humid);" by "mySerial.println(str)". And I add delay time after " mySerial.println((char)26);" 2500 ms. I can see entries number that send back to my device 53363 and 0 and 53364 and 0.... You can see my code below. For the total time in one loop it take around 10 sec. that is faster than allowed update rate . I know and i plan to buy license soon.

If I understand the wrong thing. Can you please reply me and give any suggestion. Thank you.

My code

........................................................
#include
#include

SoftwareSerial mySerial(7, 8); //RX,TX SIM
SoftwareSerial mySerial1(10, 11); //RX,TX Humid

String A, B;
String C = "CONNECT";
float humid, temp, VBatt, current;
String h, t, v, i;

void setup()
{
mySerial.begin(9600);
mySerial1.begin(9600);
Serial.begin(9600);
pinMode(13, OUTPUT);
delay(1000);

}

void loop()
{
mySerial1.listen();
delay(2);

Serial.println("Pass");
String data = mySerial1.readString();
delay(100);

int startsign = data.indexOf(':');

if(startsign >= 0){
int commaIndex = data.indexOf(',',startsign + 1);
int secondCommaIndex = data.indexOf(',', commaIndex +1);
int thirdCommaIndex= data.indexOf(',', secondCommaIndex +1);

humid = data.substring(startsign+1, commaIndex).toFloat();
temp = data.substring(commaIndex + 1, secondCommaIndex).toFloat();
VBatt = data.substring(secondCommaIndex + 1, thirdCommaIndex).toFloat();
current = data.substring(thirdCommaIndex + 1).toFloat();
delay(2);

Send2Pachube();

if (mySerial.available())
Serial.write(mySerial.read());

}
else{

}
}
void Send2Pachube()
{
mySerial.listen();
delay(2);
mySerial.println("AT");//Test command. Reply id "0"
mySerial.println("AT+CPIN?");
mySerial.println("AT+CREG?");// Check if GPRS is attached? n=1 --> +CGATT:n
mySerial.println("AT+CGATT?");
mySerial.println("AT+CIPSHUT");
mySerial.println("AT+CIPSTATUS");
mySerial.println("AT+CIPMUX=0");// "0" means single connection
mySerial.println("AT+CSTT=\"internet\"");//start task and setting the APN,
ShowSerialData();
mySerial.println("AT+CIICR");//bring up wireless connection
ShowSerialData();
mySerial.println("AT+CIFSR");//get local IP adress
ShowSerialData();//important
mySerial.println("AT+CIPSPRT=0");
delay(1000);
ShowSerialData();//important
mySerial.println("AT+CIPSTART=\"TCP\",\"api.thingspeak.com\",\"80\"");//start up the connection
delay(2000);
A = mySerial.readString();
B = A.substring(8, 15);

if(B == C){
digitalWrite(13, HIGH);
delay(500);
digitalWrite(13, LOW);
}

mySerial.println("AT+CIPSEND");//begin send data to remote server
delay(500);
ShowSerialData();
String str="GET https://api.thingspeak.com/update?api_key=N11IT99RLTDPLENG&field1="+String(VBatt)+"&field2="+String(current)+"&field3="+String(temp)+"&field4="+String(humid);
mySerial.println(str);//begin send data to remote server
delay(200);
//ShowSerialData();
mySerial.println((char)26);//sending
delay(2500);//waitting for reply, important! the time is base on the condition of internet
mySerial.println();
ShowSerialData();
mySerial.println("AT+CIPSHUT");//close the connection
delay(100);
ShowSerialData();

}

void ShowSerialData()
{
while(mySerial.available()!=0)
Serial.write(mySerial.read());

}
.................................................................

January 23, 2019
5:28 am
Avatar
TonvanW
Nederland
New Member
Members
Forum Posts: 1
Member Since:
April 13, 2012
sp_UserOfflineSmall Offline

Hello Vinod,

Thank you for your notification about this and for giving me (a little) time to fix the problem.

I will have a go at it soon enough.

January 23, 2019
10:36 am
Avatar
Vinod

MathWorks
Members
Forum Posts: 302
Member Since:
May 1, 2016
sp_UserOfflineSmall Offline

Supawan said

Hello Vinod,

I got this e-mail from you.

I have modified my code as you requirement but i'm not sure that i made the right way or not. I'm new for this so i got what you mean to is to add delay time around 200 ms after sending "String str="GET https://api.thingspeak.com/update?api_key=___REDACTED___&field1="+String(VBatt)+"&field2="+String(current)+"&field3="+String(temp)+"&field4="+String(humid);" by "mySerial.println(str)". And I add delay time after " mySerial.println((char)26);" 2500 ms. I can see entries number that send back to my device 53363 and 0 and 53364 and 0.... You can see my code below. For the total time in one loop it take around 10 sec. that is faster than allowed update rate . I know and i plan to buy license soon.

If I understand the wrong thing. Can you please reply me and give any suggestion. Thank you.

 

You need to do a couple of things:
1) Purchase a ThingSpeak license OR reduce your data rate to less than one GET request every 15s.
2) Verify that your GRPS module, and the commands you are using, do not automatically cause the TCP connection to be dropped. If you specify which GPRS module you are using, we may be able to provide some suggestions.

PS: Please consider changing your channel's Write API key and reprogramming your device. Never post your write API key on a forum as it is visible to the world and indexed by search engines, etc.

No permission to create posts
Forum Timezone: America/New_York

Most Users Ever Online: 166

Currently Online:
32 Guest(s)

Currently Browsing this Page:
1 Guest(s)

Top Posters:

rw950431: 272

Vinod: 240

piajola: 95

turgo: 70

vespapierre: 63

Adarsh_Murthy: 62

Member Stats:

Guest Posters: 1

Members: 8665

Moderators: 1

Admins: 2

Forum Stats:

Groups: 3

Forums: 14

Topics: 1600

Posts: 5760

Newest Members:

Matthewdupreez1, sbhunu, pabcstar, johnhutcheson1, madhuhada, dayne

Moderators: cstapels: 460

Administrators: Hans: 405, lee: 457