Processing 280.000 WAV files - Scilab getting slower and slower to finally die

[Lode Vande Sande]

Hello,


I had to position about 10 WAV files in about 28.000 WAV files over the weekend hence I wrote a simple correlation analysis script to open each of the 28.000 files and calculate the correlation with each of the 10 position WAV files. I did this in parallel with 12 manually opened SciLab sessions to speed up things on a 12 core server.

I found out that the further I got into the processing of the WAV files the more frequent did SciLab practically died but memory and stack reported to be fine. I also cleared the Command Line History with clc() after each processing of a WAV file to SciLab from slowing down on keeping the command history but still after processing a few thousand WAV files consistently did SciLab become very slow and finally die. Manually restarting got me a few WAV files further but the dieing frequency increased hence I moved some files into a ./processed directory and then the script did better.

I finally got through the list with about 70 restarts of SciLab and restarting of the simple script. I then restarted to script to just do a verification loop over all the 28.000 WAV files and verify if each file was processed, as I wrote a xyz.wav.lock file before processing a WAV file to allow some simple parallel processing on the same machine. The script was not even able to verify all the 28.000 .lock files as it practically killed SciLab just running through the files. This gave me the impression that there are some internal issues in SciLab when processing files or looping through many files in a single directory.

Does anybody have any idea what to change or improve as I am a relatively new user to SciLab? Is anybody interested to get this problem analysed to the bottom as it might increase the scalability of the SciLab solution for large volume file/WAV processing.


Thanks,
Lode.

 L   V   S   .
LODE VANDE SANDE - Managing Partner
M +32 476 690 448 | T +32 89 323 705 | F +32 89 323 709
lode.vandesande at lvs-c.com<mailto:joeri.golsteyn at lvs-c.com> | www.lvs-c.com<http://www.lvs-c.com/>
Dorpsstraat 62/2, 3665 As, Belgium

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.scilab.org/pipermail/users/attachments/20091216/0da16082/attachment.htm>