[Bugzilla] [Bug 15491] New: csvWrite is done in 20sec, csvRead needs about 50min

[bugzilla.admin at scilab.org]

https://bugzilla.scilab.org/show_bug.cgi?id=15491
--------- Bug Summary -----------
csvWrite is done in 20sec, csvRead needs about 50min

---------- Changed by ------------
Ahab1983 at freenet.de

---------- What changed ----------
bug_id, short_desc, product, version, rep_platform, op_sys, bug_status, bug_severity, priority, component, assigned_to, reporter


            Bug ID: 15491
           Summary: csvWrite is done in 20sec, csvRead needs about 50min
           Product: Scilab software
           Version: 6.0.0
          Hardware: (not used)
                OS: Windows 10
            Status: NEW
          Severity: Medium
          Priority: (field not used)
         Component: Data import/export
          Assignee: bugzilla.admin at scilab.org
          Reporter: Ahab1983 at freenet.de

BUG DESCRIPTION:
----------------
I got a csv-dataset with 2e7 entries from a MSO4104(~700MByte). 

ERROR LOG:
----------
Runs without error msg

HOW TO REPRODUCE THE BUG:
-------------------------

I created a Matrix with a timestamp:

function [m]=create_data(count,timeres,row,timestart)
    if count>0 & row>0 then

        time=(1:count)';
        time=time*timeres+timestart;

        m=ones(count,row);
        m(:,1)=time;
    end
endfunction

k=create_data(2e7,2e-10,5,-2e-3);  //needs ~2 sec

csvWrite(k,"test2.csv",",",".");  //needs ~8 sec

u=csvRead("test2.csv",",",[],"double");   //needs ~50 min


OTHER INFORMATION:
------------------

I often used datasets of 1e4 entries from the same source/kind and there was no problem to get it in some secs via csvRead. For example Notepad++(64
Bit) can open the whole dataset in ~10s. To make progress I had to fragment into subsets of 1e6 entries to be loadable in excel.

(Trying to have a plot from the whole set or even one of these subsets caused a shutdown of Scilab without any error msg. Giving Java a heap of
2560MB, restart...reloading solved that.)

hint? On the task-manager the memory consumption seemed not be continuous growing. 
memory management issue?

-- 
You are receiving this mail because:
You are watching all bug changes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.scilab.org/pipermail/bugzilla/attachments/20180316/8a42f28b/attachment.htm>