[Scilab-Dev] SEP: csv read write functions

[Michaël Baudin]

Hello,

Thank you for providing your interesting comments.

Do not forget that this is an English mailing list.

The idea that you suggest on csv_stringtodouble is interesting. We did 
not think about extending the current double function. The idea of Allan 
on this topic was to mimic the str2double function in Matlab:

http://www.mathworks.com/help/techdoc/ref/str2double.html

The main issue here is that the str2double function is extremely 
flexible: the range of complex doubles strings that Matlab can process 
is wide. It is rather difficult to entirely reproduce this. For example, 
the imaginary number i can be before or after the imaginary part, e.g. 
'123 + 45i' or '45i+123' or '45j+123' or 'j+123' or '-j-123' or 
'-45*j-123' or '-45e+12*j-123e-3', etc... Octave has troubles to 
reproduce these features. Part of the problem is that the Matlab 
function str2double makes use of the sscanf function, which can manage 
flexible formats.

The compatibility with Excel and the backward compatibility were our 
primary concern: be sure that we are well aware of these issues and 
tried to solve it as much as possible. I think that the currrent 
proposal is quite consistent and here is why.

Notice the bug #8654 : In write_csv, the default separator and decimal 
mark are non-standard. There are three problems:
  * The separator is the tab, while the Comma Separated Value data file 
format is clear about the fact that the separator is the comma.
  * The decimal mark is the comma ",", while the standard is the dot ".".
  * Such a file will not be read correctly by the default settings of 
read_csv.

I emphasize that  the decimal point is the comma "," only in France: in 
most other countries (including U.S.), the decimal point is the dot ".". 
In Open Office for example, we can customize this with Tools - Options - 
Language Settings - Languages. In the proposed csv_read function, the 
decimal point is also customizable. A valid suggestion, though, would be 
to customize the decimal point depending on the locale. But this is 
beyond my technical knowledge. May be Sylvestre could improve this.

I emphasize that the columns of a Comma Separated Value data files are 
separated by commas ",", as suggested by the name of this "format". If 
the separator was the Tab, this would be called a "TAB Separated Value".

There is a link between the separator and the decimal point in the 
current function. The reason why the TAB was chosen in the current 
read_csv function, is because the comma "," was chosen as the decimal 
point. If the comma was also chosen to be the separator, we could not 
read the file anymore, because of the confusion with the decimal point. 
This is the reason why the current default for the separator is the TAB.

The backward compatibility is a real concern.
  * The backward compatibility can be broken, since the current state is 
inconsistent. What is the point in maintaining the current state, where 
a file written by write_csv cannot be read by read_csv ?
  * The "forward" compatibility is improved: the comma is the default 
separator in Matlab and Octave.
  * The backward compatibility will probably be broken, by creating new 
functions called "csv_read/csv_write", while the current functions are 
"read_csv/write_csv". The current functions "read_csv/write_csv" will be 
tagged as obsolete, and then removed in a later release.

Best regards,

Michaël Baudin


Le 15/03/2011 16:53, Serge Steer a écrit :
> Le 15/03/2011 15:31, Michaël Baudin a écrit :
>> Hello,
>>
>> This SEP describes thedefinition and the improvement of CSV import 
>> and export functions in Scilab.
>>
>> This is a work with Allan Cornet, who is the leader of this project.
>>
>> Best regards,
>>
>> Michaël Baudin
>>
>>
>> -- 
>> Michaël Baudin
>> Ingénieur de développement
>> michael.baudin at scilab.org
>> -------------------------
>> Consortium Scilab - Digiteo
>> Domaine de Voluceau - Rocquencourt
>> B.P. 105 - 78153 Le Chesnay Cedex
>> Tel. : 01 39 63 56 87 - Fax : 01 39 63 55 94
>>
>>    
> J'ai un petit souci avec ce SEP. le format csv était a priori prévus 
> pour les échanges avec excel d'où l'utilisation de la virgule au lieu 
> du point décimal et par conséquent de tab comme séparateur.... Il me 
> semble que cette spec permet de gerer ces deux choses, mais ce n'est 
> pas dit explicitement, de même que le défaut n'est pas précisé. Pour 
> la compatilibilité ascendante il serait bon que le défaut corresponde 
> a la situation actuelle...
>
> Par ailleurs je ne comprend pas pourquoi la fonction 
> csv_stringtodouble  est prefixée par csv, si je comprend bien elle n'a 
> rien a voir avec le format csv. Pourquoi ne pas l'appeler 
> stringtodouble ou tout simplement completer la semantique de double, 
> int8,int16,int32,....) (sachant que string fait l'inverse)
>
> Serge
> Serge
>


-- 
Michaël Baudin
Ingénieur de développement
michael.baudin at scilab.org
-------------------------
Consortium Scilab - Digiteo
Domaine de Voluceau - Rocquencourt
B.P. 105 - 78153 Le Chesnay Cedex
Tel. : 01 39 63 56 87 - Fax : 01 39 63 55 94


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.scilab.org/pipermail/dev/attachments/20110316/2330fae2/attachment.htm>