[Scilab-users] HDF5 save is super slow

[amonmayr at laas.fr]

Le 18/10/2018 à 15:00, Stéphane Mottelet a écrit :
> Hello again,
>
> Le 18/10/2018 à 14:56, Clément David a écrit :
>> Hi Antoine,
>>
>> That one point, vec2var has been defined to pass some datatypes from 
>> Scilab "ast" (C++ side, data pointers, refcounted) to Scilab "scicos" 
>> (C, raw memory allocated once and passed around). Some data 
>> structures might not be handled correctly, I was even surprised that 
>> mlists worked correctly.
>>
>> Scilab Struct (or Cell) are missing as they are more complex 
>> datatypes to serialize. Handle are even harder (as you need to list 
>> the properties somewhere). Feel free to take a look at the code [1],
>>
>> [1]: 
>> https://antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/cgit.scilab.org/scilab/tree/scilab/modules/scicos/src/cpp/var2vec.cpp?h=6.0#n243
> Why is the code for structs (lines 242--74)  commented out ? Is it 
> broken or else ?
If var2vec() / vec2var() could be extended to provide a universal way to 
serialize / deserialize really any scilab variable, that would be really 
nice.
Could we make a SEP or fill a bug as a wish ?

Antoine
>
>> Cheers,
>> -- 
>> Clément
>>
>> -----Original Message-----
>> From: antoine monmayrant <antoine.monmayrant at laas.fr>
>> Sent: Thursday, October 18, 2018 2:47 PM
>> To: Clément DAVID <clement.david at scilab-enterprises.com>; Users 
>> mailing list for Scilab <users at lists.scilab.org>
>> Cc: Clément David <Clement.David at esi-group.com>
>> Subject: Re: [Scilab-users] HDF5 save is super slow
>>
>>
>>
>> Le 18/10/2018 à 14:09, Clément DAVID a écrit :
>>> Hello,
>>>
>>> My 2cents, this is probably a poor man’s approach but Xcos offers 
>>> vec2var / var2vec functions that encode in a double vector any 
>>> Scilab datatypes passed as arguments. The encoding duplicates the 
>>> data in memory so there might be some overhead.
>> Er, I tried var2vec, but it does not work with structures:
>> --> typeof(t)
>>    ans  =
>>    st
>>
>> --> var2vec(t)
>> var2vec: Wrong type for input argument #1: Double, Integer, Boolean, 
>> String or List type.
>>
>> Arghh... so var2vec does not work for any datatype right?
>>
>> Antoine
>>> On my machine, I have these timings using the attached script 
>>> (Antoine’s one edited):
>>> save list of syslins: 1.361704
>>> save list of vec[]: 0.056788
>>> save var2vec(list of syslins): 0.014411
>>>
>>> Discarding hdf5 groups creation is a huge performance win but remove 
>>> any way to create clean hdf5 (eg. to address subgroups directly).
>>>
>>> Thanks,
>>>
>>> -- 
>>> Clément
>>>
>>> From: users <users-bounces at lists.scilab.org> On Behalf Of Arvid Rosén
>>> Sent: Tuesday, October 16, 2018 1:01 PM
>>> To: antoine.monmayrant at laas.fr; Users mailing list for Scilab
>>> <users at lists.scilab.org>
>>> Subject: Re: [Scilab-users] HDF5 save is super slow
>>>
>>> From: users
>>> <users-bounces at lists.scilab.org<mailto:users-bounces at lists.scilab.org>
>>>> on behalf of "amonmayr at laas.fr<mailto:amonmayr at laas.fr>"
>>> <amonmayr at laas.fr<mailto:amonmayr at laas.fr>>
>>> Reply-To:
>>> "antoine.monmayrant at laas.fr<mailto:antoine.monmayrant at laas.fr>"
>>> <antoine.monmayrant at laas.fr<mailto:antoine.monmayrant at laas.fr>>, Users
>>> mailing list for Scilab
>>> <users at lists.scilab.org<mailto:users at lists.scilab.org>>
>>> Date: Tuesday, 16 October 2018 at 09:53
>>> To: "users at lists.scilab.org<mailto:users at lists.scilab.org>"
>>> <users at lists.scilab.org<mailto:users at lists.scilab.org>>
>>> Subject: Re: [Scilab-users] HDF5 save is super slow
>>>
>>> Couldn't you create your own atom package that restore this raw 
>>> memory dump for scilab 6.0?
>>> I understand why we moved away from this model, but it seems to be 
>>> key for you.
>>> There is always a trade-off between portability (and robustness) and 
>>> raw speed...
>>>
>>> Yeah, if that was possible, I would certainly do it. We already have 
>>> a bunch of C/C++ binaries that we compile and link dynamically, but 
>>> for that to be easy to implement, I guess the lists and structures 
>>> need to be stored linearly in one consecutive chunk of memory. I 
>>> don’t know if that is the case. Anyone? C++ integrations and 
>>> gateways are very poorly documented at the moment.
>>> Otherwise, I would need to do some recursive implementation, that 
>>> handles a bunch of different object types. Sounds painful.
>>>
>>> Cheers,
>>> Arvid
>> _______________________________________________
>> users mailing list
>> users at lists.scilab.org
>> https://antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/lists.scilab.org/mailman/listinfo/users 
>>
>
>

-- 
+++++++++++++++++++++++++++++++++++++++++++++++++++++++

  Antoine Monmayrant LAAS - CNRS
  7 avenue du Colonel Roche
  BP 54200
  31031 TOULOUSE Cedex 4
  FRANCE

  Tel:+33 5 61 33 64 59
  
  email : antoine.monmayrant at laas.fr
  permanent email : antoine.monmayrant at polytechnique.org

+++++++++++++++++++++++++++++++++++++++++++++++++++++++