Get a download file size before download (gb.net.curl)

Post

Posted
Rating:
#1 (In Topic #569)
Guru
BruceSteers is in the usergroup ‘Guru’
Hi all

I'm downloading an archive using the gb.net.curl HttpClient object
Like the code below…

Code (gambas)

  1.  
  2. Public hFile As File
  3.  
  4.  
  5. Public Sub GetArchive()
  6.   Dim sFork As String = "gambas"
  7.   Dim sName As String = "gambas-stable.zip"
  8.   Dim sZip As String = "https://gitlab.com/gambas/gambas/-/archive/stable" &/ sName
  9.   If Message.Question("Download archive?", "Yes", "No") <> 1 Then Return
  10.   FMain.MeEnabled(False)
  11.   hClient = New HttpClient As "hClient"
  12.   hClient.Timeout = 20
  13.   hClient.URL = sZip
  14.   hFile = Open Parent &/ File.Name(sZip) For Write Create
  15.   hClient.Get()
  16.  
  17.  
  18. Public Sub hClient_Read()
  19.   Dim sSize As String, Data As Variant
  20.   Dim sTemp As String[]
  21.   Read #hClient, Data, -256
  22.  
  23.   Dim iGot As Integer = hClient.Downloaded
  24.  
  25.   If iGot < 1024 Then
  26.     sSize = Str(iGot) & " Bytes"
  27.   Else If iGot < (1024 * 1024) Then
  28.     sSize = Str(iGot / 1024)
  29.     If InStr(sSize, ".") Then sSize = sSize[0, InStr(sSize, ".") + 2]
  30.     sSize &= " Kilobytes"
  31.   Else
  32.     sSize = Str(iGot / (1024 * 1024))
  33.     If InStr(sSize, ".") Then sSize = sSize[0, InStr(sSize, ".") + 2]
  34.     sSize &= " Megabytes"
  35.  
  36.   Print "Downloading: " & sSize & "\r"
  37.  
  38.   Write #hFile, Data
  39.  
  40. Public Sub hClient_Finished()
  41. Wait 0.1
  42.  
  43.  If hClient Then hClient.Close()
  44.  If hFile Then hFile.Close()
  45.   Print ("\nDownload Complete.\n")
  46.  Wait 0.1
  47.  FMain.MeEnabled(True)
  48.  FMain.tvComp.Input("cd '" & Parent & "' && unzip  -ou " & File.Name(hClient.URL) & "\n")
  49.  End
  50.  

In the _Read() event i can get the download size but i can't find a way to get the file size before downloading.
I tried the HttpClient.Head() method but the list of headers did not have Content.Length ?
A bit puzzled.

And help appreciated :)
Bruce
Online now: No Back to the top

Post

Posted
Rating:
#2
Avatar
Guru
cogier is in the usergroup ‘Guru’
I have tried a few things, but I can't get the result you are looking for. If it helps there is a 'Size' property in 'Download' but I can't get it to work.

<IMG src="https://www.cogier.com/gambas/curldownload.png"> </IMG>
Online now: No Back to the top

Post

Posted
Rating:
#3
Guru
BruceSteers is in the usergroup ‘Guru’

cogier said

I have tried a few things, but I can't get the result you are looking for. If it helps there is a 'Size' property in 'Download' but I can't get it to work.

<IMG src="https://www.cogier.com/gambas/curldownload.png"> </IMG>

Thanks Charlie.
I'd suspect that Size property is just downloaded not total size.

Puzzling.
Online now: No Back to the top

Post

Posted
Rating:
#4
Avatar
Enthusiast
PJBlack is in the usergroup ‘Enthusiast’
found this

Code

Use the -I option to only retrieve the headers, and look for the “Content-Length” header. Add the -L option if necessary to follow redirects. For example:

$ curl -L -I https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-9.4.0-amd64-netinst.iso
HTTP/1.1 302 Found
Date: Mon, 18 Jun 2018 09:51:32 GMT
Server: Apache/2.4.29 (Unix)
Location: https://gensho.ftp.acc.umu.se/debian-cd/current/amd64/iso-cd/debian-9.4.0-amd64-netinst.iso
Cache-Control: max-age=300
Expires: Mon, 18 Jun 2018 09:56:32 GMT
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 200 OK
Date: Mon, 18 Jun 2018 09:51:32 GMT
Server: Apache/2.4.29 (Unix)
Last-Modified: Sat, 10 Mar 2018 11:56:52 GMT
Accept-Ranges: bytes
Content-Length: 305135616
Age: 228
Content-Type: application/x-iso9660-image

may it helps …
Online now: No Back to the top

Post

Posted
Rating:
#5
Guru
BruceSteers is in the usergroup ‘Guru’

PJBlack said

found this

Code

Use the -I option to only retrieve the headers, and look for the “Content-Length” header. Add the -L option if necessary to follow redirects. For example:

$ curl -L -I https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-9.4.0-amd64-netinst.iso
HTTP/1.1 302 Found
Date: Mon, 18 Jun 2018 09:51:32 GMT
Server: Apache/2.4.29 (Unix)
Location: https://gensho.ftp.acc.umu.se/debian-cd/current/amd64/iso-cd/debian-9.4.0-amd64-netinst.iso
Cache-Control: max-age=300
Expires: Mon, 18 Jun 2018 09:56:32 GMT
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 200 OK
Date: Mon, 18 Jun 2018 09:51:32 GMT
Server: Apache/2.4.29 (Unix)
Last-Modified: Sat, 10 Mar 2018 11:56:52 GMT
Accept-Ranges: bytes
Content-Length: 305135616
Age: 228
Content-Type: application/x-iso9660-image

may it helps …

It help yes but not as i'd hoped.
using curl -I i read the headers for the file in question (the gambas source archive)
https://gitlab.com/gambas/gambas/-/archive/master/gambas-master.zip
the header list i got was something i had also managed to view using httpclient but did not see any content-length info.

and Benoit says this on the M/l..

Benoit said

Content-Length is not a mandatory header. So if it is not present, you
can't know by advance the size of the download.

so looks to me like the file i am trying to get the size of is one of those that does not report it :(.

Sigh , oh well.

Thanks for the help guys :)
Bruce
Online now: No Back to the top

Post

Posted
Rating:
#6
Guru
BruceSteers is in the usergroup ‘Guru’
 Hey Charlie.
Just an observation.

Line 8 on my first post does not show as i wrote it.

written it is just
Dim sZip As String = "the/url/to/fie"

the post shows href= and class= stuff that should not be there.

Just a note m8. dunno if it anything to do with you or not.
Bruce.
Online now: No Back to the top

Post

Posted
Rating:
#7
Avatar
Guru
cogier is in the usergroup ‘Guru’
Hi Bruce,

Not sure what the issue is/was, seems OK to me or have I missed something?

Code (gambas)

  1. Dim sZip As String = "the/url/to/fie"
Online now: No Back to the top

Post

Posted
Rating:
#8
Guru
BruceSteers is in the usergroup ‘Guru’

cogier said

Hi Bruce,

Not sure what the issue is/was, seems OK to me or have I missed something?

Code (gambas)

  1. Dim sZip As String = "the/url/to/fie"

No the first post,  not the last where i typed a url so it could be seen as it is supposed to look.
the first where it is wrong.
Online now: No Back to the top

Post

Posted
Rating:
#9
Avatar
Guru
cogier is in the usergroup ‘Guru’
Ahh yes, we have run into this before. I have corrected your first post. You need to add this: -

<IMG src="https://www.cogier.com/gambas/URL%20Issue.png"> </IMG>
Online now: No Back to the top

Post

Posted
Rating:
#10
Guru
BruceSteers is in the usergroup ‘Guru’

cogier said

Ahh yes, we have run into this before. I have corrected your first post. You need to add this: -

<IMG src="https://www.cogier.com/gambas/URL%20Issue.png"> </IMG>

aah thanks m8, will remember that in future :)
Online now: No Back to the top
1 guest and 0 members have just viewed this.