[Scilab-users] HDF5 save is super slow

[Clément David]

Hi Antoine,

That one point, vec2var has been defined to pass some datatypes from Scilab "ast" (C++ side, data pointers, refcounted) to Scilab "scicos" (C, raw memory allocated once and passed around). Some data structures might not be handled correctly, I was even surprised that mlists worked correctly.

Scilab Struct (or Cell) are missing as they are more complex datatypes to serialize. Handle are even harder (as you need to list the properties somewhere). Feel free to take a look at the code [1],

[1]: http://cgit.scilab.org/scilab/tree/scilab/modules/scicos/src/cpp/var2vec.cpp?h=6.0#n243

Cheers,
--
Clément

-----Original Message-----
From: antoine monmayrant <antoine.monmayrant at laas.fr> 
Sent: Thursday, October 18, 2018 2:47 PM
To: Clément DAVID <clement.david at scilab-enterprises.com>; Users mailing list for Scilab <users at lists.scilab.org>
Cc: Clément David <Clement.David at esi-group.com>
Subject: Re: [Scilab-users] HDF5 save is super slow



Le 18/10/2018 à 14:09, Clément DAVID a écrit :
> Hello,
>
> My 2cents, this is probably a poor man’s approach but Xcos offers vec2var / var2vec functions that encode in a double vector any Scilab datatypes passed as arguments. The encoding duplicates the data in memory so there might be some overhead.
Er, I tried var2vec, but it does not work with structures:
--> typeof(t)
  ans  =
  st

--> var2vec(t)
var2vec: Wrong type for input argument #1: Double, Integer, Boolean, String or List type.

Arghh... so var2vec does not work for any datatype right?

Antoine
>
> On my machine, I have these timings using the attached script (Antoine’s one edited):
> save list of syslins: 1.361704
> save list of vec[]: 0.056788
> save var2vec(list of syslins): 0.014411
>
> Discarding hdf5 groups creation is a huge performance win but remove any way to create clean hdf5 (eg. to address subgroups directly).
>
> Thanks,
>
> --
> Clément
>
> From: users <users-bounces at lists.scilab.org> On Behalf Of Arvid Rosén
> Sent: Tuesday, October 16, 2018 1:01 PM
> To: antoine.monmayrant at laas.fr; Users mailing list for Scilab 
> <users at lists.scilab.org>
> Subject: Re: [Scilab-users] HDF5 save is super slow
>
> From: users 
> <users-bounces at lists.scilab.org<mailto:users-bounces at lists.scilab.org>
> > on behalf of "amonmayr at laas.fr<mailto:amonmayr at laas.fr>" 
> <amonmayr at laas.fr<mailto:amonmayr at laas.fr>>
> Reply-To: 
> "antoine.monmayrant at laas.fr<mailto:antoine.monmayrant at laas.fr>" 
> <antoine.monmayrant at laas.fr<mailto:antoine.monmayrant at laas.fr>>, Users 
> mailing list for Scilab 
> <users at lists.scilab.org<mailto:users at lists.scilab.org>>
> Date: Tuesday, 16 October 2018 at 09:53
> To: "users at lists.scilab.org<mailto:users at lists.scilab.org>" 
> <users at lists.scilab.org<mailto:users at lists.scilab.org>>
> Subject: Re: [Scilab-users] HDF5 save is super slow
>
> Couldn't you create your own atom package that restore this raw memory dump for scilab 6.0?
> I understand why we moved away from this model, but it seems to be key for you.
> There is always a trade-off between portability (and robustness) and raw speed...
>
> Yeah, if that was possible, I would certainly do it. We already have a bunch of C/C++ binaries that we compile and link dynamically, but for that to be easy to implement, I guess the lists and structures need to be stored linearly in one consecutive chunk of memory. I don’t know if that is the case. Anyone? C++ integrations and gateways are very poorly documented at the moment.
> Otherwise, I would need to do some recursive implementation, that handles a bunch of different object types. Sounds painful.
>
> Cheers,
> Arvid