<html><head></head><body><div>Hi Steve.</div><div><br></div><div>In fact, this is an issue that seems to require more detail.</div><div>My 20GB of data its server's logs with weak correlation with each other.</div><div>So I can process a few MB at time without any penalty.</div><div><br></div><div>It takes about 1 hour in a notebook with a very simple memory optimization to reduce the disk access.</div><div><br></div><div>However, <b>I think in this debate we are forgetting about each kind of data requires a type of treatment.</b></div><div><br></div><div>Process 20GB or even 1 TB of data with very low correlation can be very easy.</div><div>However, process the same amount of data with high correlation, like a particle accelerator data or some financial data, and it can be a hell without an appropriate framework and hardware.</div><div><br></div><div><b>I think we should talk about the "ecosystem", not only about vectorization or algorithms.</b></div><div><b><br></b></div><div>-- Amanda Osvaldo</div><div><br></div><div><br></div><div>On Thu, 2017-03-02 at 16:25 -0500, shorne@energetiq.com wrote:</div><blockquote type="cite">
<p><font size="2" face="sans-serif">I've never had to deal with such large data sets. My first thought would be to use an ssd to improve swap times to the disk. Or add more ram.</font><br>
<font size="2" face="sans-serif">Or buy a bigger workstation. Is your algorithm parallelizable? Or, of course, just wait -- it all depends on what your real constraints are.</font><br>
<br>
<font size="2" face="sans-serif"> </font><br>
<script language="JavaScript" type="text/javascript">
<!--
function _dSectionExpand(sec) {
document.getElementById("cSec"+sec).style.display = "none";
document.getElementById("xSec"+sec).style.display = "block";
}
function _dSectionCollapse(sec) {
document.getElementById("xSec"+sec).style.display = "none";
document.getElementById("cSec"+sec).style.display = "block";
}
// -->
</script>
</p><div id="cSec1"><a href="javascript:_dSectionExpand('1')" onclick="return _dSectionExpand('1');"><img src="/icons/expand.gif" border="0" alt="Show details for Amanda Osvaldo ---03/02/2017 04:00:03 PM---Steve, how you handle with the memory limitations in your system ? For example, how you handle 20 GB"></a><a onclick="return _dSectionExpand('1');"><font size="2" color="#424282" face="sans-serif">Amanda Osvaldo ---03/02/2017 04:00:03 PM---Steve, how you handle with the memory limitations in your system ? For example, how you handle 20 GB</font></a></div><div id="xSec1"><a href="javascript:_dSectionCollapse('1')" onclick="return _dSectionCollapse('1');"><img src="/icons/collapse.gif" border="0" alt="Hide details for Amanda Osvaldo ---03/02/2017 04:00:03 PM---Steve, how you handle with the memory limitations in your system ? For example, how you handle 20 GB"></a><a onclick="return _dSectionCollapse('1');"><font size="2" color="#424282" face="sans-serif">Amanda Osvaldo ---03/02/2017 04:00:03 PM---Steve, how you handle with the memory limitations in your system ? For example, how you handle 20 GB</font></a><br>
<br>
<font size="1" color="#5F5F5F" face="sans-serif">From: </font><font size="1" face="sans-serif">Amanda Osvaldo <lambdasoftware@yahoo.es></font><br>
<font size="1" color="#5F5F5F" face="sans-serif">To: </font><font size="1" face="sans-serif">Users mailing list for Scilab <users@lists.scilab.org></font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Date: </font><font size="1" face="sans-serif">03/02/2017 04:00 PM</font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Subject: </font><font size="1" face="sans-serif">Re: [Scilab-users] Matlab vs Scilab perf; calling a fortran routine.</font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Sent by: </font><font size="1" face="sans-serif">"users" <users-bounces@lists.scilab.org></font><br>
<hr width="100%" size="2" align="left" noshade="" style="color:#8091A5; "><br>
<br>
<br>
<font size="3" face="serif">Steve, how you handle with the memory limitations in your system ?</font><br>
<br>
<font size="3" face="serif">For example, how you handle 20 GB of data in a machine with 4 GB of RAM ?</font><br>
<br>
<font size="3" face="serif">-- Amanda Osvaldo</font><br>
<br>
<br>
<font size="3" face="serif">On Thu, 2017-03-02 at 15:14 -0500, shorne@energetiq.com wrote:</font>
<ul style="padding-left: 36pt"><br>
<font size="2" face="sans-serif">I have done a lot of this kind of thing in scilab and other similar tools. A key point - If the algorithm is naturally vectorizable, there probably won't be much benefit in further optimization. However it it inherently is not vectorizable then the key calculation can be written as a callable fortran (my favorite) subroutine. Use Scilab for the I/O and visualization; write the compute intensive core in fortran. </font><font size="3" face="serif"><br>
</font><font size="2" face="sans-serif"><br>
While I'm on the subject, the old Intersci system was a very convenient way to automatically generate the interface routine between scilab and an arbitrary fortran subroutine. There seems<br>
not to be recent documentation on doing the same (specifically for fortran). Or am I missing something? I've had to use the "call" interface to use old code. Is there a better way?</font><font size="3" face="serif"><br>
</font><font size="2" face="sans-serif"><br>
Thanks - <br>
Steve</font>
<p><a href="javascript:_dSectionExpand('1')"></a><font size="2" color="#424282" face="sans-serif">Ricardo Fabbri ---03/02/2017 01:30:36 PM---Speaking from experience: It is worth mentioning that in many ways performance is not critical</font><br>
<a href="javascript:_dSectionCollapse('1')"></a><font size="2" color="#424282" face="sans-serif">Ricardo Fabbri ---03/02/2017 01:30:36 PM---Speaking from experience: It is worth mentioning that in many ways performance is not critical</font><font size="3" face="serif"><br>
</font><font size="1" color="#5F5F5F" face="sans-serif"><br>
From: </font><font size="1" face="sans-serif">Ricardo Fabbri <rfabbri@gmail.com></font><font size="1" color="#5F5F5F" face="sans-serif"><br>
To: </font><font size="1" face="sans-serif">Users mailing list for Scilab <users@lists.scilab.org></font><font size="1" color="#5F5F5F" face="sans-serif"><br>
Date: </font><font size="1" face="sans-serif">03/02/2017 01:30 PM</font><font size="1" color="#5F5F5F" face="sans-serif"><br>
Subject: </font><font size="1" face="sans-serif">Re: [Scilab-users] Matlab vs Scilab perf</font><font size="1" color="#5F5F5F" face="sans-serif"><br>
Sent by: </font><font size="1" face="sans-serif">"users" <users-bounces@lists.scilab.org></font><font size="3" face="serif"><br>
</font></p><hr width="100%" size="2" align="left" noshade=""><font size="3" face="serif"><br>
<br>
</font><tt><font size="2"><br>
Speaking from experience:<br>
<br>
It is worth mentioning that in many ways performance is not critical<br>
for a "lab" language like Scilab or Matlab. It is just an extremely<br>
simple language to test concepts and algorithms at a very small scale<br>
of granularity. The real crucial factor for Scilab or Matlab is the<br>
GUI for exploring data and developing algorithms interactively. Once<br>
you have a working solution, you'll fit it inside a bigger and more<br>
relevant<br>
system by porting promptly to a language like C++ for scalability and speed.<br>
<br>
Just use Scilab for what its worth, don't obsess with speed, even<br>
though it is important.<br>
<br>
Best,<br>
<br>
--<br>
Dr Ricardo Fabbri<br>
Professor of Computer Engineering<br>
GNU/Linux registered user #175401<br>
pt.wikipedia.org/wiki/IPRJ<br>
labmacambira.sf.net<br>
rfabbri.github.io<br>
<br>
<br>
On Thu, Mar 2, 2017 at 2:00 PM, Samuel Gougeon <sgougeon@free.fr> wrote:<br>
> Le 02/03/2017 à 16:29, Michael Benguigui a écrit :<br>
><br>
> Hi all,<br>
> As a Scilab user, I am currently fighting to reach Matlab performance..My<br>
> current scilab program takes 4x more than my matlab's vesion. Here are some<br>
> instructions requiring some optimizations.. If you have an idea... thanks a<br>
> lot!<br>
><br>
> All times are cumulative times after the execution (iterations)<br>
> I used the Matlab and Scilab profilers<br>
><br>
><br>
> Congratulations for using the new Scilab profiler :)<br>
> Could you post please the file of results? It is indeed hard to answer<br>
> without having more information either on a runnable part of your code, or<br>
> about results you got with the profiler.<br>
><br>
> Regards<br>
> Samuel<br>
><br>
><br>
> _______________________________________________<br>
> users mailing list<br>
> users@lists.scilab.org<br>
> </font></tt><a href="http://lists.scilab.org/mailman/listinfo/users"><tt><font size="2" color="#0000FF"><u>http://lists.scilab.org/mailman/listinfo/users</u></font></tt></a><tt><font size="2"><br>
><br>
_______________________________________________<br>
users mailing list<br>
users@lists.scilab.org</font></tt><tt><font size="2" color="#0000FF"><u><br>
</u></font></tt><a href="http://lists.scilab.org/mailman/listinfo/users"><tt><font size="2" color="#0000FF"><u>http://lists.scilab.org/mailman/listinfo/users</u></font></tt></a><font size="3" face="serif"><br>
</font><br>
<br>
<tt><font size="3">_______________________________________________<br>
users mailing list<br>
</font></tt><a href="mailto:users@lists.scilab.org"><tt><font size="3" color="#0000FF"><u>users@lists.scilab.org</u></font></tt></a><tt><font size="3"><br>
</font></tt><a href="http://lists.scilab.org/mailman/listinfo/users"><tt><font size="3" color="#0000FF"><u>http://lists.scilab.org/mailman/listinfo/users</u></font></tt></a><tt><font size="3"><br>
</font></tt><tt><font size="2">_______________________________________________<br>
users mailing list<br>
users@lists.scilab.org<br>
</font></tt><tt><font size="2"><a href="http://lists.scilab.org/mailman/listinfo/users">http://lists.scilab.org/mailman/listinfo/users</a></font></tt><tt><font size="2"><br>
</font></tt><br>
</ul>
</div>
<script language="JavaScript" type="text/javascript">
<!--
_dSectionExpand('1');
// -->
</script>
<pre>_______________________________________________
users mailing list
<a href="mailto:users@lists.scilab.org">users@lists.scilab.org</a>
<a href="http://lists.scilab.org/mailman/listinfo/users">http://lists.scilab.org/mailman/listinfo/users</a>
</pre></blockquote></body></html>