Problem with ThingHTTP / Help needed | ThingSpeak Apps | Forum

Avatar

Please consider registering
Guest

sp_LogInOut Log In sp_Registration Register

Register | Lost password?
Advanced Search

— Forum Scope —






— Match —





— Forum Options —





Minimum search word length is 3 characters - maximum search word length is 84 characters

sp_Feed sp_TopicIcon
Problem with ThingHTTP / Help needed
No permission to create posts
May 27, 2018
6:26 am
Avatar
Alexan
New Member
Members
Forum Posts: 1
Member Since:
May 27, 2018
sp_UserOfflineSmall Offline

Hello ThingSpeak Community,

Recently I bought an ESP32-F Module and I started experimenting with the wifi. I want to get data from a website and read them thru the ESP. My code is ok, connecting to my ThingHTTP link and reading the data. Problem is that I can not read data from a website using ThingHTTP. Let's see an example:

I want to get data from openweathermap (yes I know that they have already an API option but it's just an example)
When creating a ThingHTTP, at the URL I paste https://openweathermap.org/city/734077
Then using the inspect option I find the XPath of the humidity which is //*[@id="weather-widget"]/table/tbody/tr[4]/td[2]
Then I paste it at the Parse String, select GET and HTTP Version 1.1 and save it.

Then when I use the link that it says, I keep getting: Error parsing document, try a different parse string.

However in the second example:

URL:https://www.timeanddate.com/worldclock/greece/athens
Parse String: //*[@id="ct"]

And then I get the data that I want.

Why is that happening? I want to create a project that will get data from a website and display them on an LCD Screen. I also tried with receiving data from youtube without luck.

Any help is welcome!

P.S: I am new to the subject. I am sorry if I did something wrong. Please let me know.

May 27, 2018
10:26 pm
Avatar
cstapels
Moderator
Members


Moderators
Forum Posts: 867
Member Since:
March 7, 2017
sp_UserOfflineSmall Offline

What you have found is a common issue with ThingHTTP. You can only parse the content that is originally loaded with the page. If you inspect the source of the first page, you will see a weather-widget java script that is populating the table with the humidity data. That data is provided to the page by the script, but the data isn't there yet when thingHTTP attempts to parse the page for you. If you look at the source of the second page, you will find the element 2:58:45 am, which is populated into the page at runtime (though it may be updated later.)

At the present time, it is not possible for MATLAB or ThingSpeak to parse the final content of pages that have embedded code that populates elements after run time.

Fortunately there is the API option as you pointed out. There are also time APIs to read from the second page you described.

You can use MATLAB analysis and the webread() function to return data from API calls and provide it to other apps or write to your ThingSpeak channel.

Forum Timezone: America/New_York

Most Users Ever Online: 166

Currently Online:
24 Guest(s)

Currently Browsing this Page:
1 Guest(s)

Top Posters:

rw950431: 272

Vinod: 240

piajola: 95

turgo: 70

vespapierre: 63

Adarsh_Murthy: 62

Member Stats:

Guest Posters: 1

Members: 8665

Moderators: 1

Admins: 2

Forum Stats:

Groups: 3

Forums: 14

Topics: 1600

Posts: 5760

Newest Members:

Team_mafia, icXu, emorphistechno, VTVMART, manimozhi, adamhebrew

Moderators: cstapels: 460

Administrators: Hans: 405, lee: 457