<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:odc="urn:schemas-microsoft-com:office:odc" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc="http://microsoft.com/officenet/conferencing" xmlns:D="DAV:" xmlns:Repl="http://schemas.microsoft.com/repl/" xmlns:mt="http://schemas.microsoft.com/sharepoint/soap/meetings/" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ppda="http://www.passport.com/NameSpace.xsd" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sub="http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec="http://www.w3.org/2001/04/xmlenc#" xmlns:sp="http://schemas.microsoft.com/sharepoint/" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcs="http://schemas.microsoft.com/data/udc/soap" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udcp2p="http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf="http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss="http://schemas.microsoft.com/office/2006/digsig-setup" xmlns:dssi="http://schemas.microsoft.com/office/2006/digsig" xmlns:mdssi="http://schemas.openxmlformats.org/package/2006/digital-signature" xmlns:mver="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mrels="http://schemas.openxmlformats.org/package/2006/relationships" xmlns:spwp="http://microsoft.com/sharepoint/webpartpages" xmlns:ex12t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:ex12m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:pptsl="http://schemas.microsoft.com/sharepoint/soap/SlideLibrary/" xmlns:spsl="http://microsoft.com/webservices/SharePointPortalServer/PublishedLinksService" xmlns:Z="urn:schemas-microsoft-com:" xmlns:st="" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Arial Rounded MT Bold";
panose-1:2 15 7 4 3 5 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page Section1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=NL-BE link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span lang=EN-US>Hello All,<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>This night I tried to analyse 15.000
separate WAV files by correlating these recordings to another constant WAV
file.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>I started the process with 4 cores in
parallel using 4 separate consoles in the hope of finding the results in the
morning ... only 5000 WAV files were processed and all 4 Scilab consoles were
gone.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>In short SciLab does not seem to be able to
process large amount of WAV files and that problem exists for Scilab v5.1.1. and
the latest v5.2.0.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>I am new to Scilab and would appreciate
some guidance in pinpointing the issue to get the scalability up of the
solution.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Lode.<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'>
<p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:
"Tahoma","sans-serif"'>From:</span></b><span lang=EN-US style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'> Lode Vande Sande
[mailto:lode.vandesande@lvs-c.com] <br>
<b>Sent:</b> Wednesday, December 16, 2009 12:17 AM<br>
<b>To:</b> users@lists.scilab.org<br>
<b>Subject:</b> [scilab-Users] Processing 280.000 WAV files - Scilab getting
slower and slower to finally die<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><span lang=EN-US>Hello,<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>I had to position about 10 WAV files in
about 28.000 WAV files over the weekend hence I wrote a simple correlation
analysis script to open each of the 28.000 files and calculate the correlation
with each of the 10 position WAV files. I did this in parallel with 12 manually
opened SciLab sessions to speed up things on a 12 core server.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>I found out that the further I got into the
processing of the WAV files the more frequent did SciLab practically died but
memory and stack reported to be fine. I also cleared the Command Line History
with clc() after each processing of a WAV file to SciLab from slowing down on
keeping the command history but still after processing a few thousand WAV files
consistently did SciLab become very slow and finally die. Manually restarting
got me a few WAV files further but the dieing frequency increased hence I moved
some files into a ./processed directory and then the script did better.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>I finally got through the list with about
70 restarts of SciLab and restarting of the simple script. I then restarted to
script to just do a verification loop over all the 28.000 WAV files and verify
if each file was processed, as I wrote a xyz.wav.lock file before processing a
WAV file to allow some simple parallel processing on the same machine. The
script was not even able to verify all the 28.000 .lock files as it practically
killed SciLab just running through the files. This gave me the impression that
there are some internal issues in SciLab when processing files or looping
through many files in a single directory.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Does anybody have any idea what to change
or improve as I am a relatively new user to SciLab? Is anybody interested to
get this problem analysed to the bottom as it might increase the scalability of
the SciLab solution for large volume file/WAV processing.<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Thanks,<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US>Lode.<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-family:"Arial Rounded MT Bold","sans-serif";
color:white;background:black;mso-highlight:black'> L </span><span
lang=EN-US style='font-family:"Arial Rounded MT Bold","sans-serif";color:white'> <span
style='background:black;mso-highlight:black'> V </span> <span
style='background:black;mso-highlight:black'> S </span> .
<o:p></o:p></span></p>
<p class=MsoNormal><i><span lang=EN-US style='font-size:10.0pt;font-family:
"Verdana","sans-serif";color:gray'>LODE VANDE SANDE</span></i><span lang=EN-US
style='font-family:"Verdana","sans-serif";color:gray'> - </span><span
lang=EN-US style='font-size:10.0pt;font-family:"Verdana","sans-serif";
color:gray'>Managing Partner<o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Verdana","sans-serif";
color:gray'>M +32 476 690 448 </span><span lang=EN-US
style='font-size:7.5pt;font-family:"Verdana","sans-serif";color:gray'>|</span><span
lang=EN-US style='font-size:10.0pt;font-family:"Verdana","sans-serif";
color:gray'> T +32 89 323 705 </span><span lang=EN-US style='font-size:7.5pt;
font-family:"Verdana","sans-serif";color:gray'>|</span><span lang=EN-US
style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:gray'> F +32
89 323 709<br>
</span><span lang=EN-US style='font-size:10.0pt;font-family:"Verdana","sans-serif";
color:#1F497D'><a href="mailto:joeri.golsteyn@lvs-c.com"
title="mailto:joeri.golsteyn@lvs-c.com
http://www.lvs-c.com/"><span
style='color:gray'>lode.vandesande@lvs-c.com</span></a></span><span lang=EN-US
style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:gray'> </span><span
lang=EN-US style='font-size:7.5pt;font-family:"Verdana","sans-serif";
color:gray'>|</span><span lang=EN-US style='font-size:10.0pt;font-family:"Verdana","sans-serif";
color:gray'> </span><span lang=EN-US style='font-size:10.0pt;font-family:"Verdana","sans-serif";
color:#1F497D'><a href="http://www.lvs-c.com/" title="http://www.lvs-c.com/"><span
style='color:gray'>www.lvs-c.com</span></a></span><span lang=EN-US
style='font-family:"Verdana","sans-serif";color:#1F497D'><o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US style='font-size:10.0pt;font-family:"Verdana","sans-serif";
color:gray'>Dorpsstraat 62/2, 3665 As, Belgium</span><span lang=EN-US
style='font-family:"Verdana","sans-serif";color:#1F497D'><o:p></o:p></span></p>
<p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p>
</div>
</body>
</html>