download data from internet

Post

Posted
Rating:
#1 (In Topic #169)
Trainee
i am just starting with gambas, i had previously started some code in vb6 to download data from a website, in which i had successfully downloaded the data, decided to try to complete in gambas, but so far have not really understood how to login to the website then download the data from a selected page, i believe i should use the curl library but any examples for logging in first would be helpful
Online now: No Back to the top

Post

Posted
Rating:
#2
Avatar
Guru
cogier is in the usergroup ‘Guru’
Hi westconn and welcome to the forum.

There is the easy way to do this in Gambas using tools you probably already have installed on your system. The following example uses curl and wget to go and get your external IP address but one or both will need to be installed on your system.

Code (gambas)

  1. Public Sub Form_Open()
  2. Dim sResult As String
  3.  
  4. Shell "curl http://ipecho.net/plain" To sResult
  5. Print sResult
  6.  
  7. ''OR
  8.  
  9. Shell "wget -O- http://ipecho.net/plain" To sResult
  10. Print sResult
  11.  

Then there is the harder way, the following code requires gb.net.curl. To activate this component you will need to go to the Gambas menu Project>Properties>Components and activate gb.net.curl. This code does the same as the above.

Code (gambas)

  1. ''Needs gb.net.curl
  2. Public Sub Form_Open()
  3.  
  4. Print Get("http://ipecho.net/plain")                                          'Find this computers external IP address
  5.  
  6. Public Sub Get(sURL As String) As String                                      'To 'Get' stuff from the web
  7. Dim hClient As HttpClient                                                     'To create a HTTP Client
  8. Dim sResult As String                                                         'To store the result
  9.  
  10. hClient = New HttpClient As "hClient"                                         'Create a HTTP Client
  11. With hClient                                                                  'With the Client..
  12.   .URL = sURL                                                                 'Set up the URL
  13.   .Async = False                                                              'No Asynchronous transmission?
  14.   .TimeOut = 60                                                               'Don't hang around waiting for more than 60 seconds
  15.   .get                                                                        'Get the data
  16.  
  17. If Lof(hClient) Then sResult = Read #hClient, Lof(hClient)                    'When all the data is downloaded store it in sResult
  18.  
  19. Return sResult                                                                'Return the result
  20.  

If you go to the Gambas Farm and download Your_location (I need to get the downloads up! :o ) this may help you as it gets a CSV file from the net and displays the data.
Online now: Yes Back to the top

Post

Posted
Rating:
#3
Trainee
 i have no problem using wget or curl from a terminal, so i could shell out to either

my problem is that i need to login, then navigate to the correct url, unless there is some way to shortcut that, if i just try to go directly to the download i am redirected to the login page
can  i pass parameters to the url to bypass the login page?
Online now: No Back to the top

Post

Posted
Rating:
#4
Trainee
(I need to get the downloads up!
so far i have been unable to find any link to the farm, the search on gambas one does not seem to be working, the gambasfarm.org does not seem to have the samples
do you have a direct link to it?

also thanks for your reply and samples
Online now: No Back to the top

Post

Posted
Rating:
#5
Avatar
Guru
cogier is in the usergroup ‘Guru’
The Gambas Farm on Gambas.One is for viewing only. You need to start Gambas and select the Farm from the menu, note the 'Installed software' option, you may need it later: -
Image

(Click to enlarge)

Then select "All software"
Image

(Click to enlarge)

Then from the list select the software you want to download
Image

(Click to enlarge)

Note that the software will be available from the main menu under "Installed software"

Regarding adding login in details you need to add the details to the URL

Code

https://xxx.university.edu/portal/server.pt?userName=SomeUser&password=somePassword

If you need more help click here
https://www.google.com…QBQgkKAA&biw=1915&bih=936
Online now: Yes Back to the top

Post

Posted
Rating:
#6
Avatar
Regular
jornmo is in the usergroup ‘Regular’
What do you mean by login?

Logging in by a form on a website, or the "pop-up" box that comes with web browsers?

For the first scenario I believe you can use Curl's HTTP FORM component:
/comp/gb.net.curl/httpform - Gambas Documentation
(I am not sure if it can handle sessions. If it can't it will not work. It seems there might be a possibility to store the session in a cookie, and then load it again)

For the second you can use Curl's password property:
/comp/gb.net.curl/curl - Gambas Documentation

Online now: No Back to the top

Post

Posted
Rating:
#7
Trainee
  cogier , so easy with pictures lol, i did not think of the farm being built into gambas pogram

jornmo , yes a login form on the webpage, any examples anywhere on using httpform, the manual did not seem to help me much
Online now: No Back to the top

Post

Posted
Rating:
#8
Trainee

Code (gambas)

  1.   client.URL = myurl
  2.     client.async = False
  3.   data = "login_username=myusername&login_password=mypass&submit=Login"
  4.   z[1] = "content - length: " & Str(Len(data))
  5. '    client.get   ' this works
  6.      client.post("application/x-www-form-urlencoded", data, z)
  7.  
  8.     If client.Status < 0 Then
  9.         Return ""
  10.     Endif
  11.  
  12.     ' no data available
  13.     If Not Lof(client) Then
  14.         Return ""
  15.     Endif
  16.  
  17.     ' Reads the data from the server and returns it as a String
  18.     d = Read #client, Lof(client)
  19.     TextArea1.Text = d
i have spent some time to get this far, but no result, nothing is returned to client, if i use client.get instead, the html code of the web page is returned
Online now: No Back to the top

Post

Posted
Rating:
#9
Avatar
Regular
jornmo is in the usergroup ‘Regular’
 Since I do not know how the site is built, it is difficult to know what to do. But, when you have got a successfull login, you need somehow to store the session, and then load the URL where the data you want to retrieve is stored with the current session loaded. (I have never attempted such a thing myself)

However, I do not think this is the normal way this kind of operation normally is done. Your webiste should have some API that your program can talk to, and get a JSON/XML/CSV file from. But that again depends on if this is your website?

If it is your webiste, I would probly create a folder that is password protected with HTACCESS/HTPASSWD (if using Apache?) that has a .php file (or what ever language you are using) that spits out a JSON file.

Online now: No Back to the top

Post

Posted
Rating:
#10
Trainee
 if this is your website?
not my web site, i am trying to get the usage data from my isp, so i can display as graph, afaik they have no api or other to get the data
there is an option on the url xml=yes, but that only returns a summary of the data, not the detail and still requires to login first

i have successfully returned and parsed the data using vb, but now am trying to get the same working for linux
while there would be thousands of examples using several methods for vb, i can find almost nothing for gambas, or even other linux commands (curl or wget) i could shell to
though on further searching through the farm (and google) i have found several other working (or not) examples that are helping me with other parts
Online now: No Back to the top

Post

Posted
Rating:
#11
Avatar
Regular
stevedee is in the usergroup ‘Regular’

westconn said

… still requires to login first
…i have successfully returned and parsed the data using vb, but now am trying to get the same working for linux…

I think you are saying that you cannot provide the login details via Gambas. If so, can you post that part of your VB code that allows you to log-in? This may allow us to come up with a Gambas alternative.
Online now: No Back to the top

Post

Posted
Rating:
#12
Trainee
 i have at last got this working in gambas…………….     hooray

now to start on the next unknowns
parsing the data
graphing the data
creating a trayicon

such a lot for a slow learner………..

thank you all for replies and assistance
Online now: No Back to the top
1 guest and 0 members have just viewed this.