[scilab-Users] RE: Concatenation with hypermatrix

[Michaël Baudin]

Hi,

Do you have a simple script to reproduce your performance issue ?

You might be interested by "Programming in Scilab", especially the 
section "3.3 Hypermatrices" and the section "3.12 Comparison of data 
types" :

http://forge.scilab.org/index.php/p/docprogscilab/downloads/

In the Figure 19, I analyzed the current implementation of hypermatrices 
in Scilab v5. Here is one explanation for the potential lack of 
performances, associated with the current implementation. Hypermatrices 
are partially implemented with mlists, supported by both compiled source 
code and Scilab macros. In v5, the hypermatrix is internally stored as a 
n-by-1 matrix of doubles, where n is the product of the dimensions. The 
values are stored column-by-column.

So, if you extract one sub-column of this matrix, such as with 
A(i:j,p,q), this should be fast because data is stored locally, i.e. the 
next values in the extracted matrix is the neighbor of the previous 
values . If, on the other hand, you extract the 2nd or the 3d dimension 
(or more), for example, then I guess that this will be slower, because 
the hypermatrix has to gather values which are scattered in the internal 
n-by-1 matrix: there are many values in the internal matrix between two 
values in the extracted matrix.

For those interested, here is the detail in Scilab v5. For a hypermatrix 
of doubles (or ints), the extraction operator is implemented in the 
compiled C source code: data_structures/src/c/hmops.c: intehm. This code 
is used in the statements :

M = ones(3,3,3);
B = M(1,2:3,2);

For other types of values (e.g. strings), the extraction is implemented 
in overloading/macros/%hm_e.sci. This code is used in the statements:

M = hypermat([2,2,2],["a","a","a","a","a","a","a","a"]);
B = M(1,1:2,2);

This also may lead to performance differences, in the sense that an 
hypermatrix of doubles is probably much faster than an hypermatrix of 
strings.

In Scilab v6, the implementation is based on compiled source code for 
all types of contents. Anyway, data locality will limit the performances 
in v6 the same way as it limits the performances in v5: when we extract 
values, scattered values have to be gathered the same way as before.

A regular matrix of doubles (i.e. with 2 indices) will in general be 
much faster than an hypermatrix of doubles. All in all, Scilab was first 
designed for that...

Best regards,

Michaël Baudin



Le 01/06/2011 16:17, Mike Page a écrit :
> Hi,
>
> I think you have encountered a general problem with hypermatrices.  They are
> quite fast to operate on when you work across some dimensions, but across
> other dimensions they can be *VERY* slow.  I don't know why this is - maybe
> somebody who wrote the hypermatrix code could look at it.
>
> For instance, some time ago I had a need to extract a "slice" (that is a
> matrix) from a hypermatrix.  This took a very long time, but when I
> "transposed" the hypermatrix using the permute function, then it was much
> faster.  I think if you arrange your dimensions so that the data to be
> joined fits on the end of the existing storage then it will be faster.
>
> Regards,
> Mike.
>
>
> -----Original Message-----
> From: Orbeman [mailto:ludo.wag at laposte.net]
> Sent: 01 June 2011 14:29
> To: users at lists.scilab.org
> Subject: [scilab-Users] RE: Concatenation with hypermatrix
>
>
> I've finally found the command C=cat(3,C,x). It works but very slowly. For
> exemple it needs more than 2 min to create a 890 * 2 * 5 3D hypermatrix, but
> barely 1 sec to create the same data with 890 * (2+2+2+2+2) 2D matrix.
>
> In fact, I've analysed n experiences which data (2 rows and variable line)
> are written in one file per experience. All of the files are not the same
> length, so I add number 'tag' to egal the maximun number of line, here 890.
>
> After that I've created some others (hyper) matrix with different size of
> row : 890 * 1 * 5, 890 * 3 * 5,...
>
> I can imagine the increase of time for comptute.
>
> --
> View this message in context:
> http://mailinglists.scilab.org/Concatenation-with-hypermatrix-tp3010503p3010
> 714.html
> Sent from the Scilab users - Mailing Lists Archives mailing list archive at
> Nabble.com.
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 9.0.901 / Virus Database: 271.1.1/3670 - Release Date: 05/31/11
> 19:34:00
>


-- 
Michaël Baudin
Ingénieur de développement
michael.baudin at scilab.org
-------------------------
Consortium Scilab - Digiteo
Domaine de Voluceau - Rocquencourt
B.P. 105 - 78153 Le Chesnay Cedex
Tel. : 01 39 63 56 87 - Fax : 01 39 63 55 94