[Scilab-Dev] SEP: csv read write functions

[Serge Steer]

Le 16/03/2011 09:48, Michaël Baudin a écrit :
> Hello,
>
> Thank you for providing your interesting comments.
>
> Do not forget that this is an English mailing list.
>
> The idea that you suggest on csv_stringtodouble is interesting. We did
> not think about extending the current double function. The idea of
> Allan on this topic was to mimic the str2double function in Matlab:
>
> http://www.mathworks.com/help/techdoc/ref/str2double.html
>
> The main issue here is that the str2double function is extremely
> flexible: the range of complex doubles strings that Matlab can process
> is wide. It is rather difficult to entirely reproduce this. For
> example, the imaginary number i can be before or after the imaginary
> part, e.g. '123 + 45i' or '45i+123' or '45j+123' or 'j+123' or
> '-j-123' or '-45*j-123' or '-45e+12*j-123e-3', etc... Octave has
> troubles to reproduce these features. Part of the problem is that the
> Matlab function str2double makes use of the sscanf function, which can
> manage flexible formats.
probably Matlab does something like cvstr (parse an expression and
evaluate it) . It is more flexible but less efficient. If you are
looking for efficiency you should restrict to real or integer numbers.
The Matlab str2double may be emulated using cvstr. the "45j" or "45i"
feature is the most difficult to handle, but it can be solved by a
preprocessing that changes the regular expressions [0-9]i and [0-9]j to
add a *.
>
> The compatibility with Excel and the backward compatibility were our
> primary concern: be sure that we are well aware of these issues and
> tried to solve it as much as possible. I think that the currrent
> proposal is quite consistent and here is why.
>
> Notice the bug #8654 : In write_csv, the default separator and decimal
> mark are non-standard. There are three problems:
>  * The separator is the tab, while the Comma Separated Value data file
> format is clear about the fact that the separator is the comma.
>  * The decimal mark is the comma ",", while the standard is the dot ".".
>  * Such a file will not be read correctly by the default settings of
> read_csv.
>
> I emphasize that  the decimal point is the comma "," only in France:
> in most other countries (including U.S.), the decimal point is the dot
> ".". In Open Office for example, we can customize this with Tools -
> Options - Language Settings - Languages. In the proposed csv_read
> function, the decimal point is also customizable. A valid suggestion,
> though, would be to customize the decimal point depending on the
> locale. But this is beyond my technical knowledge. May be Sylvestre
> could improve this.
>
I am not sure that relying on the locale should be a good solution. One
may want to read data generated under an other locale
It seem better to make that customisable throught parameter as you proposed.
> I emphasize that the columns of a Comma Separated Value data files are
> separated by commas ",", as suggested by the name of this "format". If
> the separator was the Tab, this would be called a "TAB Separated Value".
>
> There is a link between the separator and the decimal point in the
> current function. The reason why the TAB was chosen in the current
> read_csv function, is because the comma "," was chosen as the decimal
> point. If the comma was also chosen to be the separator, we could not
> read the file anymore, because of the confusion with the decimal
> point. This is the reason why the current default for the separator is
> the TAB.
>
> The backward compatibility is a real concern.
>  * The backward compatibility can be broken, since the current state
> is inconsistent. What is the point in maintaining the current state,
> where a file written by write_csv cannot be read by read_csv ?
>  * The "forward" compatibility is improved: the comma is the default
> separator in Matlab and Octave.
>  * The backward compatibility will probably be broken, by creating new
> functions called "csv_read/csv_write", while the current functions are
> "read_csv/write_csv". The current functions "read_csv/write_csv" will
> be tagged as obsolete, and then removed in a later release.
>
Ok! I did not realize that you proposed a name change. Good solution!

Serge Steer
> Best regards,
>
> Michaël Baudin
>
>
> Le 15/03/2011 16:53, Serge Steer a écrit :
>> Le 15/03/2011 15:31, Michaël Baudin a écrit :
>>> Hello,
>>>
>>> This SEP describes the definition and the improvement of CSV import
>>> and export functions in Scilab.
>>>
>>> This is a work with Allan Cornet, who is the leader of this project.
>>>
>>> Best regards,
>>>
>>> Michaël Baudin
>>>
>>>
>>> -- 
>>> Michaël Baudin
>>> Ingénieur de développement
>>> michael.baudin at scilab.org
>>> -------------------------
>>> Consortium Scilab - Digiteo
>>> Domaine de Voluceau - Rocquencourt
>>> B.P. 105 - 78153 Le Chesnay Cedex
>>> Tel. : 01 39 63 56 87 - Fax : 01 39 63 55 94
>>>
>>>   
>> J'ai un petit souci avec ce SEP. le format csv était a priori prévus
>> pour les échanges avec excel d'où l'utilisation de la virgule au lieu
>> du point décimal et par conséquent de tab comme séparateur.... Il me
>> semble que cette spec permet de gerer ces deux choses, mais ce n'est
>> pas dit explicitement, de même que le défaut n'est pas précisé. Pour
>> la compatilibilité ascendante il serait bon que le défaut corresponde
>> a la situation actuelle...
>>
>> Par ailleurs je ne comprend pas pourquoi la fonction
>> csv_stringtodouble  est prefixée par csv, si je comprend bien elle
>> n'a rien a voir avec le format csv. Pourquoi ne pas l'appeler
>> stringtodouble ou tout simplement completer la semantique de double,
>> int8,int16,int32,....) (sachant que string fait l'inverse)
>>
>> Serge
>> Serge
>>
>
>
> -- 
> Michaël Baudin
> Ingénieur de développement
> michael.baudin at scilab.org
> -------------------------
> Consortium Scilab - Digiteo
> Domaine de Voluceau - Rocquencourt
> B.P. 105 - 78153 Le Chesnay Cedex
> Tel. : 01 39 63 56 87 - Fax : 01 39 63 55 94
>
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.scilab.org/pipermail/dev/attachments/20110316/941ad3ff/attachment.htm>