[Scilab-users] HDF5 save is super slow

[Stéphane Mottelet]

Hello again,

Le 18/10/2018 à 14:56, Clément David a écrit :
> Hi Antoine,
>
> That one point, vec2var has been defined to pass some datatypes from Scilab "ast" (C++ side, data pointers, refcounted) to Scilab "scicos" (C, raw memory allocated once and passed around). Some data structures might not be handled correctly, I was even surprised that mlists worked correctly.
>
> Scilab Struct (or Cell) are missing as they are more complex datatypes to serialize. Handle are even harder (as you need to list the properties somewhere). Feel free to take a look at the code [1],
>
> [1]: https://antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/cgit.scilab.org/scilab/tree/scilab/modules/scicos/src/cpp/var2vec.cpp?h=6.0#n243
Why is the code for structs (lines 242--74)  commented out ? Is it 
broken or else ?

> Cheers,
> --
> Clément
>
> -----Original Message-----
> From: antoine monmayrant <antoine.monmayrant at laas.fr>
> Sent: Thursday, October 18, 2018 2:47 PM
> To: Clément DAVID <clement.david at scilab-enterprises.com>; Users mailing list for Scilab <users at lists.scilab.org>
> Cc: Clément David <Clement.David at esi-group.com>
> Subject: Re: [Scilab-users] HDF5 save is super slow
>
>
>
> Le 18/10/2018 à 14:09, Clément DAVID a écrit :
>> Hello,
>>
>> My 2cents, this is probably a poor man’s approach but Xcos offers vec2var / var2vec functions that encode in a double vector any Scilab datatypes passed as arguments. The encoding duplicates the data in memory so there might be some overhead.
> Er, I tried var2vec, but it does not work with structures:
> --> typeof(t)
>    ans  =
>    st
>
> --> var2vec(t)
> var2vec: Wrong type for input argument #1: Double, Integer, Boolean, String or List type.
>
> Arghh... so var2vec does not work for any datatype right?
>
> Antoine
>> On my machine, I have these timings using the attached script (Antoine’s one edited):
>> save list of syslins: 1.361704
>> save list of vec[]: 0.056788
>> save var2vec(list of syslins): 0.014411
>>
>> Discarding hdf5 groups creation is a huge performance win but remove any way to create clean hdf5 (eg. to address subgroups directly).
>>
>> Thanks,
>>
>> --
>> Clément
>>
>> From: users <users-bounces at lists.scilab.org> On Behalf Of Arvid Rosén
>> Sent: Tuesday, October 16, 2018 1:01 PM
>> To: antoine.monmayrant at laas.fr; Users mailing list for Scilab
>> <users at lists.scilab.org>
>> Subject: Re: [Scilab-users] HDF5 save is super slow
>>
>> From: users
>> <users-bounces at lists.scilab.org<mailto:users-bounces at lists.scilab.org>
>>> on behalf of "amonmayr at laas.fr<mailto:amonmayr at laas.fr>"
>> <amonmayr at laas.fr<mailto:amonmayr at laas.fr>>
>> Reply-To:
>> "antoine.monmayrant at laas.fr<mailto:antoine.monmayrant at laas.fr>"
>> <antoine.monmayrant at laas.fr<mailto:antoine.monmayrant at laas.fr>>, Users
>> mailing list for Scilab
>> <users at lists.scilab.org<mailto:users at lists.scilab.org>>
>> Date: Tuesday, 16 October 2018 at 09:53
>> To: "users at lists.scilab.org<mailto:users at lists.scilab.org>"
>> <users at lists.scilab.org<mailto:users at lists.scilab.org>>
>> Subject: Re: [Scilab-users] HDF5 save is super slow
>>
>> Couldn't you create your own atom package that restore this raw memory dump for scilab 6.0?
>> I understand why we moved away from this model, but it seems to be key for you.
>> There is always a trade-off between portability (and robustness) and raw speed...
>>
>> Yeah, if that was possible, I would certainly do it. We already have a bunch of C/C++ binaries that we compile and link dynamically, but for that to be easy to implement, I guess the lists and structures need to be stored linearly in one consecutive chunk of memory. I don’t know if that is the case. Anyone? C++ integrations and gateways are very poorly documented at the moment.
>> Otherwise, I would need to do some recursive implementation, that handles a bunch of different object types. Sounds painful.
>>
>> Cheers,
>> Arvid
> _______________________________________________
> users mailing list
> users at lists.scilab.org
> https://antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/lists.scilab.org/mailman/listinfo/users


-- 
Stéphane Mottelet
Ingénieur de recherche
EA 4297 Transformations Intégrées de la Matière Renouvelable
Département Génie des Procédés Industriels
Sorbonne Universités - Université de Technologie de Compiègne
CS 60319, 60203 Compiègne cedex
Tel : +33(0)344234688
http://www.utc.fr/~mottelet