[Scilab-Dev] SEP 88: New functions to get remote files

[Samuel Gougeon]

Hello Sylvestre,

Le 24/01/2013 18:14, Sylvestre Ledru a écrit :
> Hello,
>
> The attachment describes the SEP #88, targeting Scilab 5.5.0.
>
> The goal of this SEP is to propose the introduction of three new functions:
> * getURLcontent - Return the content of an URL (HTTP, HTTPS, FTP...)
> Profiles:
> filename = getURL(URL [, targetDir [, username [, password]]]]);
> filename = getURL(URL [, targetFile [, username [, password]]]]);
>
> * getURL - Download an URL (HTTP, HTTPS, FTP...)
> Profile:
> output = getURLcontent(URL [, username, [, password]]);
>
> * splitURL - Split a URL (HTTP, HTTPS, FTP...)
> Profile:
> [proto, server, path, query, username, port, fragment] = splitURL(URL);
>
> Don't hesitate to comment it,
> Sylvestre
Sorry for the delay. The good thing is that now there is more 
information in the help
of the beta-implementation available in the 5.5 branch; and it is 
possible to test them.

I developped some utilities under Scilab with curl ~3 years ago, and i 
had a look
on how to use these new functions instead of my owns. Here are some comments
and suggestions about their present implementation.

*getURL() and getURLcontent()*:
At the first glance, it is hard to understand the difference between 
getURL() and
getURLcontent(). Actually, it is very slight. Since it is only a matter 
of output and
that 90% of their respective jobs are the same, in my opinion, this does 
not
deserve 2 distinct functions: It would be nicer to add an option 
specifying the
output, or even more simply to have a second optionnal output argument:
[filename [, content]] = getURL(...)
It is always possible to specify a trash-file as filename.
or
output = getURL(...,"returnContent")
At least, for the moment, a single help page merging the description of 
both
functions would be nice.

_Present embarassing limitations w.r.t. curl features_:
The most frustrating one is that these functions allow only queries in 
GET method.
The POST method is not available,  whereas curl allows it with no 
problem and
very great utility.
=> So: It would be very useful to add a curl_options string optionnal 
parameter,
that would be passed to curl as is, allowing to post data with "-d 
param=value ...etc"
and any other curl usage.
By the way, specifying a file in which to dump the targetted content is 
possible
in this way. Proxy parameters can also be specified.

Presently, it is not natural to use atomsSetConfig() to specify a proxy 
to use for
getURL...().

In few words, it could be a good basic target to have a simple interface 
with curl,
such as a curlScilab() function.
The only problem i had when using curl through unix_#() functions was some
encoding issues in the output content. getURL...() greatly fix this point.
Otherwise, curl is so great that is is a big improvement to include it 
the Scilab
package for all plateforms (not only Windows).

Other advanced features could be useful, such as allowing to follow
  <META HTTP-EQUIV="Refresh" CONTENT="n ; url=new_target" >
within a limited nesting depth ; and other ones.
More elaborated functions could be posted to ATOMS or FileExchange.

*splitURL()*:
works fine :-) IMO, its help page should be located in the "paths - 
filenames"
subsection, together with fileparts() and other paths-processing functions.

Best regards
Samuel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.scilab.org/pipermail/dev/attachments/20130323/35089f7c/attachment.htm>