[Scilab-users] Matlab vs Scilab perf; calling a fortran routine.
Amanda Osvaldo
lambdasoftware at yahoo.es
Fri Mar 3 13:18:33 CET 2017
Hi Steve.
In fact, this is an issue that seems to require more detail.My 20GB of
data its server's logs with weak correlation with each other.So I can
process a few MB at time without any penalty.
It takes about 1 hour in a notebook with a very simple memory
optimization to reduce the disk access.
However, I think in this debate we are forgetting about each kind of
data requires a type of treatment.
Process 20GB or even 1 TB of data with very low correlation can be very
easy.However, process the same amount of data with high correlation,
like a particle accelerator data or some financial data, and it can be
a hell without an appropriate framework and hardware.
I think we should talk about the "ecosystem", not only about
vectorization or algorithms.
-- Amanda Osvaldo
On Thu, 2017-03-02 at 16:25 -0500, shorne at energetiq.com wrote:
> I've never had to deal with such large data sets. My first thought
> would be to use an ssd to improve swap times to the disk. Or add
> more ram.
>
> Or buy a bigger workstation. Is your algorithm parallelizable?
> Or, of course, just wait -- it all depends on what your real
> constraints are.
>
>
>
>
>
>
>
> <!--
> function _dSectionExpand(sec) {
> document.getElementById("cSec"+sec).style.display = "none";
> document.getElementById("xSec"+sec).style.display = "block";
> }
> function _dSectionCollapse(sec) {
> document.getElementById("xSec"+sec).style.display = "none";
> document.getElementById("cSec"+sec).style.display = "block";
> }
> // -->
>
>
> Amanda Osvaldo ---03/02/2017 04:00:03 PM---Steve, how you handle with
> the memory limitations in your system ? For example, how you handle
> 20 GB
> Amanda Osvaldo ---03/02/2017 04:00:03 PM---Steve, how you handle with
> the memory limitations in your system ? For example, how you handle
> 20 GB
>
>
>
> From: Amanda Osvaldo <lambdasoftware at yahoo.es>
>
> To: Users mailing list for Scilab <users at lists.scilab.org>
>
> Date: 03/02/2017 04:00 PM
>
> Subject: Re: [Scilab-users] Matlab vs Scilab perf; calling a
> fortran routine.
>
> Sent by: "users" <users-bounces at lists.scilab.org>
>
>
>
>
>
>
>
> Steve, how you handle with the memory limitations in your system ?
>
>
>
> For example, how you handle 20 GB of data in a machine with 4 GB of
> RAM ?
>
>
>
> -- Amanda Osvaldo
>
>
>
>
>
> On Thu, 2017-03-02 at 15:14 -0500, shorne at energetiq.com wrote:
>
>
> I have done a lot of this kind of thing in scilab and other similar
> tools. A key point - If the algorithm is naturally vectorizable,
> there probably won't be much benefit in further optimization.
> However it it inherently is not vectorizable then the key
> calculation can be written as a callable fortran (my favorite)
> subroutine. Use Scilab for the I/O and visualization; write the
> compute intensive core in fortran.
>
>
>
> While I'm on the subject, the old Intersci system was a very
> convenient way to automatically generate the interface routine
> between scilab and an arbitrary fortran subroutine. There seems
>
> not to be recent documentation on doing the same (specifically for
> fortran). Or am I missing something? I've had to use the "call"
> interface to use old code. Is there a better way?
>
>
>
> Thanks -
>
> Steve
> Ricardo Fabbri ---03/02/2017 01:30:36 PM---Speaking from experience:
> It is worth mentioning that in many ways performance is not critical
>
> Ricardo Fabbri ---03/02/2017 01:30:36 PM---Speaking from experience:
> It is worth mentioning that in many ways performance is not critical
>
>
>
> From: Ricardo Fabbri <rfabbri at gmail.com>
>
> To: Users mailing list for Scilab <users at lists.scilab.org>
>
> Date: 03/02/2017 01:30 PM
>
> Subject: Re: [Scilab-users] Matlab vs Scilab perf
>
> Sent by: "users" <users-bounces at lists.scilab.org>
>
>
>
>
>
>
>
> Speaking from experience:
>
>
>
> It is worth mentioning that in many ways performance is not critical
>
> for a "lab" language like Scilab or Matlab. It is just an extremely
>
> simple language to test concepts and algorithms at a very small scale
>
> of granularity. The real crucial factor for Scilab or Matlab is the
>
> GUI for exploring data and developing algorithms interactively. Once
>
> you have a working solution, you'll fit it inside a bigger and more
>
> relevant
>
> system by porting promptly to a language like C++ for scalability and
> speed.
>
>
>
> Just use Scilab for what its worth, don't obsess with speed, even
>
> though it is important.
>
>
>
> Best,
>
>
>
> --
>
> Dr Ricardo Fabbri
>
> Professor of Computer Engineering
>
> GNU/Linux registered user #175401
>
> pt.wikipedia.org/wiki/IPRJ
>
> labmacambira.sf.net
>
> rfabbri.github.io
>
>
>
>
>
> On Thu, Mar 2, 2017 at 2:00 PM, Samuel Gougeon <sgougeon at free.fr>
> wrote:
>
> > Le 02/03/2017 à 16:29, Michael Benguigui a écrit :
>
> >
>
> > Hi all,
>
> > As a Scilab user, I am currently fighting to reach Matlab
> performance..My
>
> > current scilab program takes 4x more than my matlab's vesion. Here
> are some
>
> > instructions requiring some optimizations.. If you have an idea...
> thanks a
>
> > lot!
>
> >
>
> > All times are cumulative times after the execution (iterations)
>
> > I used the Matlab and Scilab profilers
>
> >
>
> >
>
> > Congratulations for using the new Scilab profiler :)
>
> > Could you post please the file of results? It is indeed hard to
> answer
>
> > without having more information either on a runnable part of your
> code, or
>
> > about results you got with the profiler.
>
> >
>
> > Regards
>
> > Samuel
>
> >
>
> >
>
> > _______________________________________________
>
> > users mailing list
>
> > users at lists.scilab.org
>
> > http://lists.scilab.org/mailman/listinfo/users
>
> >
>
> _______________________________________________
>
> users mailing list
>
> users at lists.scilab.org
>
> http://lists.scilab.org/mailman/listinfo/users
>
>
>
>
>
> _______________________________________________
>
> users mailing list
>
> users at lists.scilab.org
>
> http://lists.scilab.org/mailman/listinfo/users
>
> _______________________________________________
>
> users mailing list
>
> users at lists.scilab.org
>
> http://lists.scilab.org/mailman/listinfo/users
>
>
>
>
>
>
>
> <!--
> _dSectionExpand('1');
> // -->
>
>
>
> _______________________________________________
> users mailing list
> users at lists.scilab.org
> http://lists.scilab.org/mailman/listinfo/users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.scilab.org/pipermail/users/attachments/20170303/eb2dcdb8/attachment.htm>
More information about the users
mailing list