[scilab-Users] Re: reading or loading big files

[Gary Nelson]

I have a similar issue with large wav files.
Is there any way to read in a large file, for example in buffers, process the buffer(s) as if the data are continuous, and write out results so that stacksize is not exceeded?
For example, suppose I want to simply filter a large (>100MB) file with a complex FIR filter 401 points long and save the results in another file.

One issue that I don't understand yet deals with data format. My convolution routine wants the filter kernel as a row vector and the data as a column vector. However, wav files seem to read in as row vectors after which I transpose them. This continuous read-filter-write process would render my present method unworkable. Can a loadwav incoming file be transposed  as it is read?

A related issue -- so far all my tests have been done with known-length files using length(file). But if we don't bring the entire file in before we begging filtering, can we know the length?

Thanks
Gary Nelson

On Jun 7, 2010, at 2:18 AM, Samuel Gougeon wrote:

> Hello,
> 
> ----- Message d'origine ----- 
> De : Hsu (David) 
> Date : 05/06/2010 08:20:
>> 
>> I am just learning SciLab.  I need to read enormous EEG files (electroencephalograms), for example, 20 channels sampled at 32,000 Hz for days on end.  This data may be saved as a matrix with 20 columns but a huge number of rows, so big that I run up against size limits if I try to load the whole thing as a matrix.  This data is usually in binary format.  I tried using stacksize('max') to maximize these size limits but am still running into size limits.
> A simple calculation:
> 20 channels x 32000 measurements/s/channel x 1 byte/measurement
> x 3600x24 s/day ~ 55 Gbytes/day
> If you want to load the whole data as a int8() or uint8() matrix
> (as minimal data format), you need at least ~ 60 Gbytes of RAM.
> Do you have it (assuming that Scilab or another software could handle it)?
> It looks unrealistic to load and handle these data as a whole into a matrix.
> You will likely need to use some detailled binary commands
> after mopen() : mseek(), mtell(), mget()... to read some piece of data.
> 
> Samuel
> 

Gary Nelson
gnelson at quantasonics.com



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.scilab.org/pipermail/users/attachments/20100607/380211cc/attachment.htm>