[scilab-Users] RE: Processing 280.000 WAV files - Scilab getting slower and slower to finally die

[Mathieu Dubois]

Hello Lode,

I have never used scilab to process wav files nor doing heavy audio 
processing so maybe my suggestions are pointless.

First of all, do you get any error message when scilab crashes? This may 
help you to debug your script.

Second, do you free unneeded variables after processing? For instance 
after processing one file you can delete the variable with the clear 
command.

Last but not least, if scilab is bad at processing directories with a 
large number of files one solution might be to use a combination of 
shell scripts and scilab functions.
For instance you can write a function which processes a single file 
(given as parameter) and then quit scilab (with the quit function). Then 
create a script which invokes scilab with the -nwni, -nb and -e options 
(see scilab -h) and pass the right parameter to the function. Note that 
in order for scilab to find your function you may have to load it in 
your startup file (see startup in scilab help).
This increases the time to process a file (because you start and stop 
scilab for each file) but you are sure that memory is freed and you can 
then use the shell to do simple parallelization (like starting n process 
with different args).
See attachment as a simple example.

Hope that helps,
Mathieu

Le 19/12/2009 07:08, Lode Vande Sande a écrit :
> Hello All,
>
> This night I tried to analyse 15.000 separate WAV files by correlating
> these recordings to another constant WAV file.
>
> I started the process with 4 cores in parallel using 4 separate consoles
> in the hope of finding the results in the morning ... only 5000 WAV
> files were processed and all 4 Scilab consoles were gone.
>
> In short SciLab does not seem to be able to process large amount of WAV
> files and that problem exists for Scilab v5.1.1. and the latest v5.2.0.
>
> I am new to Scilab and would appreciate some guidance in pinpointing the
> issue to get the scalability up of the solution.
>
> Lode.
>
> *From:* Lode Vande Sande [mailto:lode.vandesande at lvs-c.com]
> *Sent:* Wednesday, December 16, 2009 12:17 AM
> *To:* users at lists.scilab.org
> *Subject:* [scilab-Users] Processing 280.000 WAV files - Scilab getting
> slower and slower to finally die
>
> Hello,
>
> I had to position about 10 WAV files in about 28.000 WAV files over the
> weekend hence I wrote a simple correlation analysis script to open each
> of the 28.000 files and calculate the correlation with each of the 10
> position WAV files. I did this in parallel with 12 manually opened
> SciLab sessions to speed up things on a 12 core server.
>
> I found out that the further I got into the processing of the WAV files
> the more frequent did SciLab practically died but memory and stack
> reported to be fine. I also cleared the Command Line History with clc()
> after each processing of a WAV file to SciLab from slowing down on
> keeping the command history but still after processing a few thousand
> WAV files consistently did SciLab become very slow and finally die.
> Manually restarting got me a few WAV files further but the dieing
> frequency increased hence I moved some files into a ./processed
> directory and then the script did better.
>
> I finally got through the list with about 70 restarts of SciLab and
> restarting of the simple script. I then restarted to script to just do a
> verification loop over all the 28.000 WAV files and verify if each file
> was processed, as I wrote a xyz.wav.lock file before processing a WAV
> file to allow some simple parallel processing on the same machine. The
> script was not even able to verify all the 28.000 .lock files as it
> practically killed SciLab just running through the files. This gave me
> the impression that there are some internal issues in SciLab when
> processing files or looping through many files in a single directory.
>
> Does anybody have any idea what to change or improve as I am a
> relatively new user to SciLab? Is anybody interested to get this problem
> analysed to the bottom as it might increase the scalability of the
> SciLab solution for large volume file/WAV processing.
>
> Thanks,
>
> Lode.
>
> L V S .
>
> /LODE VANDE SANDE/ - Managing Partner
>
> M +32 476 690 448 | T +32 89 323 705 | F +32 89 323 709
> lode.vandesande at lvs-c.com <mailto:joeri.golsteyn at lvs-c.com> |
> www.lvs-c.com <http://www.lvs-c.com/>
>
> Dorpsstraat 62/2, 3665 As, Belgium
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test_lode.tar.gz
Type: application/x-gzip
Size: 343 bytes
Desc: not available
URL: <https://lists.scilab.org/pipermail/users/attachments/20091220/094b2d01/attachment.bin>