Testing bulk update feature with Arduino MKR GSM | Arduino | Forum

Avatar

Please consider registering
Guest

sp_LogInOut Log In sp_Registration Register

Register | Lost password?
Advanced Search

— Forum Scope —






— Match —





— Forum Options —





Minimum search word length is 3 characters - maximum search word length is 84 characters

sp_Feed sp_TopicIcon
Testing bulk update feature with Arduino MKR GSM
No permission to create posts
April 13, 2018
4:35 pm
Avatar
eplantes

Silver
Members
Forum Posts: 11
Member Since:
April 12, 2018
sp_UserOfflineSmall Offline

Hello.

I have been testing the bulk update feature using Arduino MKR 1400 GSM and I have experimented some problems. For some strange reason, I only can send bulk updates when the packet size is about 256 bytes or less, but not longer. Every time I send my packet with size longer than 256 bytes, I receive an error. I don't know if this error is caused by the library (MKRGSM) or is a limitation on the GSM chip. This supposes a very serious inconvenience for me, because I need to send about 180 measurements (2,2Kbytes); and the between consecutive bulk updates must be 15 seconds or more.

Can someone tell me what could be the reason of this limitation?

This is my arduino code. (For requests with packet size of 256 bytes works fine, but with longer size not).

#####################################################

 // include the GSM library
#include

// PIN number if necessary
#define PINNUMBER ""

// APN information obrained from your network provider
#define GPRS_APN "?????????" // replace with your GPRS APN
#define GPRS_LOGIN "" // replace with your GPRS login
#define GPRS_PASSWORD "" // replace with your GPRS password

// initialize the library instances
GSMClient client;
GPRS gprs;
GSM gsmAccess(true);

 

// ThingSpeak Settings
char thingSpeakAddress[] = "api.thingspeak.com";
String writeAPIKey = "MY_API_KEY_WR"; //replace with your API Key
const int updateThingSpeakInterval = 15 * 1000; // Time interval in milliseconds to update ThingSpeak (number of seconds * 1000 = interval)

// Variable Setup
long lastConnectionTime = 0;
boolean lastConnected = false;
int failedCounter = 0;

char superstring[] = "{\"write_api_key\":\"MY_API_KEY_WR\",\"updates\":[{\"delta_t\":0,\"field8\":-1000000},{\"delta_t\":1,\"field8\":-1000000},{\"delta_t\":1,\"field8\":-1000000},{\"delta_t\":1,\"field8\":-1000000},{\"delta_t\":1,\"field8\":-1000000},{\"delta_t\":1,\"field8\":-1000000},{\"delta_t\":1,\"field8\":-9999999},{\"delta_t\":1,\"field8\":-1000000},{\"delta_t\":1,\"field8\":-9999999}]}";

 

void setup() {
Serial.setTimeout(5);
Serial.begin(57600); // opens serial port, sets data rate to 9600 bps
StartGSM();
}

 

void loop() {
if (client.available()) {
char c = client.read();
Serial.print(c);
}

// Disconnect from ThingSpeak
if (!client.connected() && lastConnected){
Serial.println("...disconnected");
Serial.println();

client.stop();
}

// Update ThingSpeak
if(!client.connected() && (millis() - lastConnectionTime > updateThingSpeakInterval)) {
String ToThingSpeak = superstring;
Serial.println(ToThingSpeak);
updateThingSpeak(ToThingSpeak);
}

if (failedCounter > 3 ) {
StartGSM();
}

lastConnected = client.connected();
}

void updateThingSpeak(String tsData)
{
if (client.connect(thingSpeakAddress, 80))
{
Serial.println("[-] Enviando datos !!!! ");
client.println("POST /channels/MY_CHANNEL/bulk_update.json HTTP/1.1");
client.println("Host: api.thingspeak.com");
client.println("User-Agent: mw.doc.bulk-update (Arduino ESP8266)");
client.println("Connection: close");
client.println("Content-Type: application/json");
client.print("Content-Length: ");
client.print(tsData.length());
client.print("

");

client.print(tsData);

lastConnectionTime = millis();

if (client.connected()) {
Serial.println("Connecting to ThingSpeak...");
Serial.println();

failedCounter = 0;
} else {
failedCounter++;

Serial.println("Connection to ThingSpeak failed ("+String(failedCounter, DEC)+")");
Serial.println();
}

} else {
failedCounter++;

Serial.println("Connection to ThingSpeak Failed ("+String(failedCounter, DEC)+")");
Serial.println();

lastConnectionTime = millis();
}
}

void StartGSM() {
char server[] = "api.thingspeak.com"; // the base URL
int port = 80; // the port, 80 for HTTP

Serial.println("Starting Arduino web client.");
// connection state
boolean notConnected = true;

// Start GSM shield
// pass the PIN of your SIM as a parameter of gsmAccess.begin()
while(notConnected) {
if((gsmAccess.begin(PINNUMBER)==GSM_READY) &
(gprs.attachGPRS(GPRS_APN, GPRS_LOGIN, GPRS_PASSWORD)==GPRS_READY)) {
notConnected = false;
} else {
Serial.println("Not connected");
delay(1000);
}
}

Serial.println("connecting...");

// if you get a connection, report back via serial:
if (client.connect(server, port)) {
Serial.println("connected");
} else {
// if you didn't get a connection to the server:
Serial.println("connection failed");
}

}

################################################

 Thank you
Best regards

April 15, 2018
10:28 pm
Avatar
cstapels
Moderator
Members


Moderators
Forum Posts: 341
Member Since:
March 7, 2017
sp_UserOfflineSmall Offline

I have run into buffer size limitations with Arduino for HTTP posts.  Instead of building ‘superstring ‘ in advance, I would store your field data in an array and pass the array to your posting function.  Then send one field at a time so you don’t have a big string.

 

“client.println("Content-Type: application/json");
client.print("Content-Length: ");
client.print(tsData.length());
client.print("”

 

client.print(“&field1=”+String(fieldArray[1]);

client.print(“&field2=”+String(fieldArray[2]);

client.print(“&field3=”+String(fieldArray[3]);

client.print(“&field4=”+String(fieldArray[4]);

client.println();

…and so on.  You can obviously use a loop here.

April 16, 2018
8:21 am
Avatar
eplantes

Silver
Members
Forum Posts: 11
Member Since:
April 12, 2018
sp_UserOfflineSmall Offline

Hello cstapels.

Thanks for your response but I have a doubt. According to the documentation, the time between consecutive bulk updates must be 15 seconds or more. So if I have to send 180 measurenments, I need a lot of time to send all data. In my case I don't need to send eight flieds every time, only 1 field; for this reason I thought creating a superstring it would be the best option. 

On another hand, looking at the MKRGSM library I have noticed in the GSMClient.cpp file, in the write method, that the max size packet is 256. I suposse it must be some reason of that, so I'm not sure if is not possible increase it. This could be the cause of my problem.

This is the part of the code affected:

**************************************

size_t GSMClient::write(const uint8_t* buf, size_t size)
{
  if (_writeSync) {
    while (ready() == 0);
  } else if (ready() == 0) {
    return 0;
  }

  if (_socket == -1) {
    return 0;
  }

  size_t written = 0;
  String command;

  while (size) {
    size_t chunkSize = size;

    if (chunkSize > 256) { # <----------------------------------------------------------- !!!!
      chunkSize = 256;
    }

    command.reserve(19 + chunkSize * 2);

    command += "AT+USOWR=";
    command += _socket;
    command += ",";
    command += chunkSize;
    command += ",\"";

    for (size_t i = 0; i < chunkSize; i++) {
      byte b = buf[i + written];

      byte n1 = (b >> 4) & 0x0f;
      byte n2 = (b & 0x0f);

      command += (char)(n1 > 9 ? 'A' + n1 - 10 : '0' + n1);
      command += (char)(n2 > 9 ? 'A' + n2 - 10 : '0' + n2);
    }

    command += "\"";

    MODEM.send(command);
    if (_writeSync) {
      if (MODEM.waitForResponse(10000) != 1) {
        break;
      }
    }

    written += chunkSize;
    size -= chunkSize;
  }

  return written;
}

 

******************************

thank you

April 16, 2018
11:09 am
Avatar
cstapels
Moderator
Members


Moderators
Forum Posts: 341
Member Since:
March 7, 2017
sp_UserOfflineSmall Offline

I think you have got the problem figured out now.  You don't need to increase the buffer size though,  Just send each measurement separately.

It the same thing for one field.  

Send an array of measurements and times to your posting functions. Then you will never exceed the 256 byte limit. Here is pseudo code, I have not checked all the syntax- you really have to be careful with all the quotes and slashes:

int deltaTs[measurements];

float values[measurements];

posting_function(deltaTs,values){

..headers and other stuff...

client.print("{\"write_api_key\":\"+String(MY_API_KEY_WR);

client.print("\",\"updates\":[");

for (int i=0;i<measurements;i++){

client.print("{\"delta_t\":"+String(deltaTs[i])+",\"field8\":-"+String(values[i])+"},");

}

}

April 16, 2018
11:49 am
Avatar
eplantes

Silver
Members
Forum Posts: 11
Member Since:
April 12, 2018
sp_UserOfflineSmall Offline

Thanks Cstapels.

I'm going to test it!!

Thank you very much.

April 16, 2018
2:01 pm
Avatar
eplantes

Silver
Members
Forum Posts: 11
Member Since:
April 12, 2018
sp_UserOfflineSmall Offline

Hei Cstapels!

I've tested it and it works!!

At the moment I can't send all data al the same time, I don't know the reason, but I can send over 80 measurements every time. This is fantastic for me, because I can send all data in three requests. I just need to wait the precise time between requests. Once I've finished my code and I've tested it, I'll post it in this thread; maybe it can be useful for someone.

Thank you very much!

April 20, 2018
5:48 pm
Avatar
cstapels
Moderator
Members


Moderators
Forum Posts: 341
Member Since:
March 7, 2017
sp_UserOfflineSmall Offline

Great news! Definitely post your finished code. And please share the final results of your project too if you can.

June 22, 2018
6:10 am
Avatar
eplantes

Silver
Members
Forum Posts: 11
Member Since:
April 12, 2018
sp_UserOfflineSmall Offline

Hello.

Once I have done a lot of tests (about 4000 requests or more) I think my program is very stable, so I would like to share my code with the community. First of all my program starts a monitoring process during 150 seconds, where it reads some parameters like electrical conductivity, temperature, humidity, etc.. when this process has finished, it calls to bulkUpdate methods to send these values to ThingSpeak platform. Once the program finished, it goes to sleep mode.

##########################################################################################

// Arduino Libraries
#include
#include
#include
#include "DHT.h"
#include
#include
#include
//#include

#include

// GSM data
const char GPRS_APN[] = SECRET_GPRS_APN;
const char GPRS_LOGIN[] = SECRET_GPRS_LOGIN;
const char GPRS_PASSWORD[] = SECRET_GPRS_PASSWORD;
const char PINNUMBER[] = SECRET_PIN;
boolean connected = false;
byte failedCounter = 0;

// GSM Classes
GSMClient client;
GPRS gprs;
GSM gsmAccess;

// ThingSpeak Settings
const char server[] = "api.thingspeak.com";
const byte port = 80;
const char API_KEY[] = SECRET_API_KEY;
const char CHANNEL[] = SECRET_CHANNEL;

// Leds, DHT
#define POWER_ON 2
#define K1 3
#define K2 4
#define DHTPIN 5
#define DHTTYPE DHT11

// Analog inputs
#define PSOLAR_INPUT A1
#define BAT_INPUT A2

// DHT Temp hum
DHT dht(DHTPIN, DHTTYPE);
float lectura;
float temp;
float hum;
float tbulbo;
float volSolar;
float volBat;

// CDA 16 bits ADS1115
Adafruit_ADS1115 ads(0x48);
int16_t adc0, adc1, adc2, adc3;

static char vol[10];

// Timers and injections controllers
const byte NMUESTRAS = 75;
const unsigned long __TIMEOUT__ = 10 * 1000;
const int TIME_SLEEP = 5 * 60 * 1000; // 12 * 60 * 1000;
unsigned long time_read = 0;

byte countReads = 0;
byte indexPot = 0;

// Control injections variables
static boolean toSleep = false;
static boolean injectionFinished = false;
static boolean initProcess = true;
static boolean posCycle = true;
static int counterCycles = 1;
const int MAX_CYCLES = 96;

// Vector de potenciales electricos
float potenciales[NMUESTRAS];

/**
* ///////////////////////////////////////////////////////////////
* reboot Method
* Reinicia el dispositivo
* //////////////////////////////////////////////////////////////
**/
void reboot(){
NVIC_SystemReset();
}

/**
* ///////////////////////////////////////////////////////////////
* Setup Method
* Se inicializa todos los parametros del dispositivo
* //////////////////////////////////////////////////////////////
**/
void setup() {
//Serial.setTimeout(5);
Serial.begin(9600);
Serial.println("[INIT] Iniciando dispositivo .... ");

//Init digital relays outputs
pinMode(DHTPIN, INPUT);
pinMode(K1, OUTPUT);
pinMode(K2, OUTPUT);
pinMode(POWER_ON, OUTPUT);

// Init DHT AD conversor
dht.begin();
ads.begin();

//Init process
connectedGSM();
initProcess = true;

}

void loop() {
mainProcess();
Serial.println("[FINISHED] Proceso terminado. Dispositivo en modo SLEEP ....");
LowPower.sleep(TIME_SLEEP);
}

/**
* ///////////////////////////////////////////////////////////////
* - mainProcess
* Proceso general del programa
* - Obtiene 75 lecturas
* - Envia valores calculados
* - El dispositivo entra en modo SLEEP
* //////////////////////////////////////////////////////////////
**/

void mainProcess() {

while (!toSleep) {

if (!injectionFinished) {
if (initProcess) {
initProcess = false;
injectionFinished = false;
initCycle();
}

if (!injectionFinished) {
if (millis() - time_read >= 2000) {
readData();
time_read = millis();
}
}

} else {
Serial.println("[INFO] Inyeccion finalizada mostrando datos ....");

// Finalizar ciclo posito/negativo
// Lectura parametros: Temperatura, humedad, Vbat ...
finishCycle();
calculateFinalValues();

//Conexion GSM y envio de datos
digitalWrite(POWER_ON, LOW);
bulkUpdate(0, 75);
toSleep = true;
initProcess = true;
}
}
injectionFinished = false;
toSleep = false;
}

/**
* ///////////////////////////////////////////////////////////////
* - readData
* Realiza una lectura
* //////////////////////////////////////////////////////////////
**/
void readData() {
adc0 = ads.readADC_SingleEnded(0);
adc1 = ads.readADC_SingleEnded(1);
adc2 = ads.readADC_SingleEnded(2);
adc3 = ads.readADC_SingleEnded(3);

lectura = (adc0 * 0.1875);
countReads++;

if (!posCycle) {
lectura = lectura * -1;
}

Serial.print("[INFO] Sonda Pot: ");
Serial.println(lectura, 3);

potenciales[indexPot] = lectura;
indexPot++;

if (countReads >= NMUESTRAS) {
injectionFinished = true;
}
}

/**
* ///////////////////////////////////////////////////////////////
* - getLastString
* Recupera los valores almacenados en el proceso, y prepara la
* ultima cadena a enviar a la plataforma ThingSpeak
* //////////////////////////////////////////////////////////////
**/
String getLastString() {

String StringToThingSpeak =",{\"delta_t\":1,\"field1\":" + String(0)
+ ",\"field2\":" + String(counterCycles)
+ ",\"field3\":" + String(potenciales[NMUESTRAS - 1], 3)
+ ",\"field4\":" + String(tbulbo, 1)
+ ",\"field5\":" + String(temp)
+ ",\"field6\":" + String(hum)
+ ",\"field7\":" + String(volBat)
+ ",\"field8\":" + String(volSolar)
+ "}";

Serial.print("[-] Ultima cadena a enviar: ");
Serial.println(StringToThingSpeak);

return StringToThingSpeak;
}

/**
* ///////////////////////////////////////////////////////////////
* - Iniciar ciclo positivo/negativo
* //////////////////////////////////////////////////////////////
**/
void initCycle() {
Serial.println("[INFO] Iniciando CICLO .... ");
digitalWrite(POWER_ON, HIGH);
delay(2000);

if (posCycle && digitalRead(K2) == LOW) {
digitalWrite(K1, HIGH);
Serial.println("[+] Proceso de ciclo positivo.");
} else if (!posCycle && digitalRead(K1) == LOW){
digitalWrite(K2, HIGH);
Serial.println("[-] Proceso de ciclo negativo.");
}

// Iniciar contadores
countReads = 0;
indexPot = 0;
failedCounter = 0;

// Iniciar temporizadores
time_read = millis();

}

/**
* ///////////////////////////////////////////////////////////////
* - Finalizar ciclo positivo/negativo
* //////////////////////////////////////////////////////////////
**/
void finishCycle() {
if (digitalRead(K1) == HIGH) {
digitalWrite(K1, LOW);
posCycle = false;
Serial.println("[+] Finaliza ciclo positivo.");
} else if (digitalRead(K2) == HIGH) {
digitalWrite(K2, LOW);
posCycle = true;
Serial.print("[-] Finaliza de ciclo negativo.");
}
time_read = 0;
}

/**
* //////////////////////////////////////////////////////////////
* bulkUpdate
* //////////////////////////////////////////////////////////////
**/
void bulkUpdate(int indexIni, int IndexFin) {
int ldata = getlengthData(indexIni, IndexFin);
String potencial = "";

client.stop();
if (client.connect(server, port)) {
Serial.println("[-] Enviando datos !!!! ");
client.println("POST /channels/" + String(CHANNEL)+"/bulk_update.json HTTP/1.1");
client.println("Host: api.thingspeak.com");
client.println("User-Agent: Arduino MKRGSM");
client.println("Connection: close");
client.println("Content-Type: application/json");
client.print("Content-Length: ");
client.print(ldata);
Serial.print("[>] Longitud cadena: ");
Serial.println(ldata);
client.print("

");

potencial = String(potenciales[indexIni], 3);
client.print("{\"write_api_key\":\"" + String(API_KEY) +"\",\"updates\":[{\"delta_t\":0,\"field3\":" + potencial + "}");

for (int i = indexIni +1 ; i < IndexFin-1; i++){
potencial = String(potenciales[i], 3);
client.print(",{\"delta_t\":2,\"field3\":" + potencial + "}");
}

client.print(getLastString());
client.print("]}");

//Comprobacion codigo respuesta servidor
delay(250);

String rs = "Respuesta bulkUpdate: ";
client.parseFloat();
rs += String(client.parseInt());
Serial.println(rs);

} else {
failedCounter++;
Serial.println("Connection to ThingSpeak Failed ("+String(failedCounter, DEC)+")");
Serial.println();
}
}

/**
* ///////////////////////////////////////////////////////////////
* Gestiona la conexion GSM usando los parametros configurados
* en el fichero secrets: PIN, APN, user, password
* //////////////////////////////////////////////////////////////
**/

bool connectedGSM() {
Serial.println("[INFO] Starting Arduino web client.");
// connection state
//boolean connected = false;
while (!connected) {
if ((gsmAccess.begin(PINNUMBER)) == GSM_READY) {
if (gprs.attachGPRS(GPRS_APN, GPRS_LOGIN, GPRS_PASSWORD) == GPRS_READY) {
connected = true;
Serial.println("[OK] Connected APN settings succesful");
}
} else {
Serial.println("[FAIL] Not connected to GSM... Trying again!");
delay(1000);
}

}
return connected;
}

int getlengthData(int indexIni, int indexFi) {

int size = 0;
String val = String(potenciales[0], 3);

size += String("{\"write_api_key\":\"").length();
size += String(API_KEY).length();
size += String("\",\"updates\":[{\"delta_t\":0,\"field3\":").length();
size += String(val).length();
size += String("}").length();

for (int i = indexIni+1; i < indexFi-1; i++){
size += String(",{\"delta_t\":2,\"field3\":").length();
val = String(potenciales[i], 3);
size += String(val).length();
size += String("}").length();
}

// Finalización de la cadena
size += String(getLastString()).length();
size += String("]}").length();

Serial.print("[INFO] Longitud calculada: ");
Serial.println(size);
return size;
}

/**
* //////////////////////////////////////////////////////////////
* Obtiene el voltaje de la bateria
* //////////////////////////////////////////////////////////////
**/
void getLevelBattery(){
int sensorValue = analogRead(ADC_BATTERY);
// Convert the analog reading (which goes from 0 - 1023)
// to a voltage (0 - 4.3V):
volBat = sensorValue * (4.3 / 1023.0);
}

/**
* //////////////////////////////////////////////////////////////
* Calcula la temperatura y humedad exterior
* a partir de la sonda DHT11
* //////////////////////////////////////////////////////////////
**/
void calculateTempHum() {
//Temperatura bulbo
int16_t adc1 = ads.readADC_SingleEnded(1);
tbulbo = (adc1 * 0.1875) / 10.0 ;

//Temperatura y humedad exterior
hum = dht.readHumidity();
temp = dht.readTemperature();

if (isnan(hum) || isnan(temp)) {
Serial.println("Failed to read from DHT sensor!");
temp = 99;
hum = 99;
return;
}
}

/**
* ///////////////////////////////////////////////////////////////
* - getVolPanelSolar
* Calcula el voltaje del panel solar
* //////////////////////////////////////////////////////////////
**/
void getVolPanelSolar() {
int sensor = analogRead(PSOLAR_INPUT);
volSolar = sensor * (5 / 1024.0);
}

/**
* ///////////////////////////////////////////////////////////////
* - calculateFinalValues
* Calcula los valores finales y los imprime por pantalla
* //////////////////////////////////////////////////////////////
**/
void calculateFinalValues() {
calculateTempHum();
getVolPanelSolar();
getLevelBattery();
printResults();
}

/**
* ///////////////////////////////////////////////////////////////
* - printResults
* Muestra lo valores finales calculados.
* //////////////////////////////////////////////////////////////
**/
String printResults() {
Serial.println("[-] Valores calculados finales");
Serial.print("\t- Field 1: "); Serial.println(String(0));
Serial.print("\t- Contador: "); Serial.println(counterCycles);
Serial.print("\t- Voltaje Panel solar: "); Serial.println(volSolar);
Serial.print("\t- Temp exterior: "); Serial.println(temp);
Serial.print("\t- Hum exterior: "); Serial.println(hum);
Serial.print("\t- Temp bulbo: "); Serial.println(tbulbo);
Serial.print("\t- Voltaje bateria: "); Serial.println(volBat);
}

###########################################################

Thanks

Forum Timezone: America/New_York

Most Users Ever Online: 114

Currently Online:
26 Guest(s)

Currently Browsing this Page:
1 Guest(s)

Top Posters:

rw950431: 261

Vinod: 196

piajola: 85

turgo: 70

vespapierre: 63

Adarsh_Murthy: 62

Member Stats:

Guest Posters: 1

Members: 5703

Moderators: 0

Admins: 2

Forum Stats:

Groups: 4

Forums: 17

Topics: 1313

Posts: 4565

Newest Members:

BrightfuryAboth, jaxs, WilanashAboth, TygolarAboth, HossokAboth, DoluneAboth

Administrators: Hans: 387, lee: 457