[Scilab-users] HDF5 save is super slow

[Stéphane Mottelet]

Le 18/10/2018 à 15:51, Clément DAVID a écrit :
> Hello Stephane,
>
> Probably commented out as we have no easy way to extract such data easily using only C constructs (from a Scicos block). It might be possible to uncomment and check the counterpart side (vec2var.cpp) to ensure it works correctly.
only single structs are supported, likely.

S.
>
> Thanks,
>
> --
> Clément
>
> -----Original Message-----
> From: users <users-bounces at lists.scilab.org> On Behalf Of Stéphane Mottelet
> Sent: Thursday, October 18, 2018 3:01 PM
> To: users at lists.scilab.org
> Subject: Re: [Scilab-users] HDF5 save is super slow
>
> Hello again,
>
> Le 18/10/2018 à 14:56, Clément David a écrit :
>> Hi Antoine,
>>
>> That one point, vec2var has been defined to pass some datatypes from Scilab "ast" (C++ side, data pointers, refcounted) to Scilab "scicos" (C, raw memory allocated once and passed around). Some data structures might not be handled correctly, I was even surprised that mlists worked correctly.
>>
>> Scilab Struct (or Cell) are missing as they are more complex datatypes
>> to serialize. Handle are even harder (as you need to list the
>> properties somewhere). Feel free to take a look at the code [1],
>>
>> [1]:
>> https://antispam.utc.fr/proxy/2/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/cgit.
>> scilab.org/scilab/tree/scilab/modules/scicos/src/cpp/var2vec.cpp?h=6.0
>> #n243
> Why is the code for structs (lines 242--74)  commented out ? Is it broken or else ?
>
>> Cheers,
>> --
>> Clément
>>
>> -----Original Message-----
>> From: antoine monmayrant <antoine.monmayrant at laas.fr>
>> Sent: Thursday, October 18, 2018 2:47 PM
>> To: Clément DAVID <clement.david at scilab-enterprises.com>; Users
>> mailing list for Scilab <users at lists.scilab.org>
>> Cc: Clément David <Clement.David at esi-group.com>
>> Subject: Re: [Scilab-users] HDF5 save is super slow
>>
>>
>>
>> Le 18/10/2018 à 14:09, Clément DAVID a écrit :
>>> Hello,
>>>
>>> My 2cents, this is probably a poor man’s approach but Xcos offers vec2var / var2vec functions that encode in a double vector any Scilab datatypes passed as arguments. The encoding duplicates the data in memory so there might be some overhead.
>> Er, I tried var2vec, but it does not work with structures:
>> --> typeof(t)
>>     ans  =
>>     st
>>
>> --> var2vec(t)
>> var2vec: Wrong type for input argument #1: Double, Integer, Boolean, String or List type.
>>
>> Arghh... so var2vec does not work for any datatype right?
>>
>> Antoine
>>> On my machine, I have these timings using the attached script (Antoine’s one edited):
>>> save list of syslins: 1.361704
>>> save list of vec[]: 0.056788
>>> save var2vec(list of syslins): 0.014411
>>>
>>> Discarding hdf5 groups creation is a huge performance win but remove any way to create clean hdf5 (eg. to address subgroups directly).
>>>
>>> Thanks,
>>>
>>> --
>>> Clément
>>>
>>> From: users <users-bounces at lists.scilab.org> On Behalf Of Arvid Rosén
>>> Sent: Tuesday, October 16, 2018 1:01 PM
>>> To: antoine.monmayrant at laas.fr; Users mailing list for Scilab
>>> <users at lists.scilab.org>
>>> Subject: Re: [Scilab-users] HDF5 save is super slow
>>>
>>> From: users
>>> <users-bounces at lists.scilab.org<mailto:users-bounces at lists.scilab.org
>>>> on behalf of "amonmayr at laas.fr<mailto:amonmayr at laas.fr>"
>>> <amonmayr at laas.fr<mailto:amonmayr at laas.fr>>
>>> Reply-To:
>>> "antoine.monmayrant at laas.fr<mailto:antoine.monmayrant at laas.fr>"
>>> <antoine.monmayrant at laas.fr<mailto:antoine.monmayrant at laas.fr>>,
>>> Users mailing list for Scilab
>>> <users at lists.scilab.org<mailto:users at lists.scilab.org>>
>>> Date: Tuesday, 16 October 2018 at 09:53
>>> To: "users at lists.scilab.org<mailto:users at lists.scilab.org>"
>>> <users at lists.scilab.org<mailto:users at lists.scilab.org>>
>>> Subject: Re: [Scilab-users] HDF5 save is super slow
>>>
>>> Couldn't you create your own atom package that restore this raw memory dump for scilab 6.0?
>>> I understand why we moved away from this model, but it seems to be key for you.
>>> There is always a trade-off between portability (and robustness) and raw speed...
>>>
>>> Yeah, if that was possible, I would certainly do it. We already have a bunch of C/C++ binaries that we compile and link dynamically, but for that to be easy to implement, I guess the lists and structures need to be stored linearly in one consecutive chunk of memory. I don’t know if that is the case. Anyone? C++ integrations and gateways are very poorly documented at the moment.
>>> Otherwise, I would need to do some recursive implementation, that handles a bunch of different object types. Sounds painful.
>>>
>>> Cheers,
>>> Arvid
>> _______________________________________________
>> users mailing list
>> users at lists.scilab.org
>> https://antispam.utc.fr/proxy/2/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/lists
>> .scilab.org/mailman/listinfo/users
>
> --
> Stéphane Mottelet
> Ingénieur de recherche
> EA 4297 Transformations Intégrées de la Matière Renouvelable Département Génie des Procédés Industriels Sorbonne Universités - Université de Technologie de Compiègne CS 60319, 60203 Compiègne cedex Tel : +33(0)344234688 https://antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/www.utc.fr/~mottelet
>
> _______________________________________________
> users mailing list
> users at lists.scilab.org
> https://antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/lists.scilab.org/mailman/listinfo/users
> _______________________________________________
> users mailing list
> users at lists.scilab.org
> https://antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/lists.scilab.org/mailman/listinfo/users


-- 
Stéphane Mottelet
Ingénieur de recherche
EA 4297 Transformations Intégrées de la Matière Renouvelable
Département Génie des Procédés Industriels
Sorbonne Universités - Université de Technologie de Compiègne
CS 60319, 60203 Compiègne cedex
Tel : +33(0)344234688
http://www.utc.fr/~mottelet