[scilab-Users] Re: 500mo Ascii file

[Antoine Monmayrant]

Le 13/06/12 21:47, Tingsten a écrit :
> Thx you a lot Antoine,
>
> Indeed the script you send works fine with "few lines". Then on my computer
> it runs very slow (after 100000 lines it starts to be hard) and i need to
> read much more ;-). My computer is correct so I don't think another could do
> much better.
Oh, that's what I was expected, but reading your first post I thought it 
was working OK on your computer (I was a bit surprised to be honest).
>
> Moreover your script (don't see that as a critizme since mine was definitly
> not better) is also loading data at least twice in the memory.First time
> with mgetl in the variable data.Then in scan_data.
>
> As far as i understand in terms of performance the csv_read function is
> obtimized to load ascii file and would do the best job. Issue is how to
> remove the headers?
I am a bit new to csv_read but it seems to me that weirdly enough this 
feature is not present.
> Can we delete the first lines of the file in Scilab (with
> mputl maybe) or run csv_read starting from the 6th line?
Well, you can use mopen/mgetl/mputl to make a copy of your file starting 
from the 6th line.
I don't know how efficient this will be.
If you don't care about portability and you are lucky enough to work on 
a normal computer (ie one running linux or some kind of unix-like os), 
you can use scilab 'unix_g' function to call 'sed' or other commandline 
utilities that are really good at parsing big files.
This is way more efficient and I regularly parse/modify/sort multiMB 
text files in a snap.
>
> Thx you,
> Remi
>
> --
> View this message in context: http://mailinglists.scilab.org/500mo-Ascii-file-tp4024324p4024364.html
> Sent from the Scilab users - Mailing Lists Archives mailing list archive at Nabble.com.
>
> --
> To unsubscribe from this mailing-list, please send an empty mail to
> users-unsubscribe at lists.scilab.org
> To check the archives of this mailing list, see
> http://mailinglists.scilab.org/
>