<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:-webkit-standard;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="FR" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hello Stephane,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><u><span lang="EN-US" style="mso-fareast-language:EN-US">TL ;DR ;</span></u><span lang="EN-US" style="mso-fareast-language:EN-US"> HDF5 is a cross-platform, cross-language, portable file format used in almost all scientific software these
days. Please use this sane <u>default !</u><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Writing a custom serialization scheme (like the one provided by vec2var / var2vec) might not be complicated to implement however the hard part is maintaining and describing a serialization
format to be used in the long term.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Using Scilab 5, the “stack” save and
<u>load</u> functions were almost trivial as they are directly mapped from memory to disk; the format used is “the stack” so it is known and used everywhere (even for custom string encoding). This vec2var serialization is only used internally (to pass block
parameters around), does not respect any described format nor validate against any documentation and is not portable; in the long term, I won’t promise it to be stable. Implementing your own serialization scheme will probably lead your software into trouble.
Really, it isn’t easy in the long term! The HDF5 format is described, its serialized data are browsable (through
<u>hdfview</u>) and does not cope with low-level requirements.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">To me, the issue is really a performance bug. We might find a way to fix it within Scilab rather than provide a workaround (with custom encodings). The hdf5 library is a bug one, maybe
with a clever understanding of its internal <u>serialization,</u> we might find a better execution path for this use-case (without changing the file format).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">--<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Clément<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> users <users-bounces@lists.scilab.org>
<b>On Behalf Of </b>Stéphane Mottelet<br>
<b>Sent:</b> Thursday, October 18, <u>2018</u> 2:39 PM<br>
<b>To:</b> users@lists.scilab.org<br>
<b>Subject:</b> Re: [Scilab-users] HDF5 save is super slow<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hello Clément,<br>
<br>
Le 18/10/2018 à 14:09, Clément DAVID <u>a écrit</u> :<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hello,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">My 2cents, this is probably a poor man’s approach but Xcos offers vec2var / var2vec functions that encode in a double vector any Scilab datatypes passed as arguments. The encoding duplicates
the data in memory so there might be some overhead.</span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal">Do you think it would be complicated to continuously write the serialized data on the disk ?
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">On my machine, I have these timings using the attached script (Antoine’s one edited):</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US" style="mso-fareast-language:EN-US">save
<u>list</u> of <u>syslins</u>: 1.361704</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US" style="mso-fareast-language:EN-US">save
<u>list</u> of vec[]: 0.056788</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US" style="mso-fareast-language:EN-US">save var2vec(list of
<u>syslins</u>): 0.014411</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Discarding hdf5 groups creation is a huge performance win but remove
<u>any way</u> to create clean hdf5 (eg. to address subgroups directly).</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Thanks,
</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">--</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Clément</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> users
<a href="mailto:users-bounces@lists.scilab.org"><users-bounces@lists.scilab.org></a>
<b>On Behalf Of </b>Arvid Rosén<br>
<b>Sent:</b> Tuesday, October 16, <u>2018</u> 1:01 PM<br>
<b>To:</b> <a href="mailto:antoine.monmayrant@laas.fr">antoine.monmayrant@laas.fr</a>; Users mailing list for Scilab
<a href="mailto:users@lists.scilab.org"><users@lists.scilab.org></a><br>
<b>Subject:</b> Re: [Scilab-users] HDF5 save is super slow</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><b><span lang="EN-US" style="font-size:12.0pt">From:
</span></b><span lang="EN-US" style="font-size:12.0pt">users <<a href="mailto:users-bounces@lists.scilab.org">users-bounces@lists.scilab.org</a>> on behalf of "<a href="mailto:amonmayr@laas.fr">amonmayr@laas.fr</a>" <<a href="mailto:amonmayr@laas.fr">amonmayr@laas.fr</a>><br>
<b>Reply-To: </b>"<a href="mailto:antoine.monmayrant@laas.fr">antoine.monmayrant@laas.fr</a>" <<a href="mailto:antoine.monmayrant@laas.fr">antoine.monmayrant@laas.fr</a>>, Users mailing list for Scilab <<a href="mailto:users@lists.scilab.org">users@lists.scilab.org</a>><br>
<b>Date: </b>Tuesday, 16 October 2018 at 09:53<br>
<b>To: </b>"<a href="mailto:users@lists.scilab.org">users@lists.scilab.org</a>" <<a href="mailto:users@lists.scilab.org">users@lists.scilab.org</a>><br>
<b>Subject: </b>Re: [Scilab-users] HDF5 save is super slow</span><o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span lang="EN-US"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:.5in"><span lang="SV" style="font-size:13.5pt;font-family:-webkit-standard;background:white">Couldn't you create your own atom package that
<u>restore</u> this raw memory dump for <u>scilab</u> 6.0?</span><span lang="SV" style="font-family:-webkit-standard"><br>
</span><span lang="SV" style="font-size:13.5pt;font-family:-webkit-standard;background:white">I understand why we moved away from this model, but it seems to be key for you.</span><span lang="SV" style="font-family:-webkit-standard"><br>
</span><span lang="SV" style="font-size:13.5pt;font-family:-webkit-standard;background:white">There is always a trade-off between portability (and robustness) and raw speed...</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="SV"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">Yeah, if that was possible, I would certainly do it. We already have a bunch of C/C++ binaries that we compile and link dynamically, but for that to be easy to implement, I guess the lists and structures need to be stored
linearly in one consecutive chunk of memory. I don’t know if that is the case. Anyone? C++ integrations and gateways are very poorly documented at the moment.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">Otherwise, I would need to do some recursive implementation, that handles a bunch of different object types. Sounds painful.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">Cheers,</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">Arvid</span><o:p></o:p></p>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>users mailing list<o:p></o:p></pre>
<pre><a href="mailto:users@lists.scilab.org">users@lists.scilab.org</a><o:p></o:p></pre>
<pre><a href="https://antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/lists.scilab.org/mailman/listinfo/users">https://antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/lists.scilab.org/mailman/listinfo/users</a><o:p></o:p></pre>
</blockquote>
<p><o:p> </o:p></p>
<pre>-- <o:p></o:p></pre>
<pre>Stéphane Mottelet<o:p></o:p></pre>
<pre>Ingénieur de recherche<o:p></o:p></pre>
<pre>EA 4297 Transformations Intégrées de la Matière Renouvelable<o:p></o:p></pre>
<pre>Département Génie des Procédés Industriels<o:p></o:p></pre>
<pre>Sorbonne Universités - Université de Technologie de Compiègne<o:p></o:p></pre>
<pre>CS 60319, 60203 Compiègne cedex<o:p></o:p></pre>
<pre>Tel : +33(0)344234688<o:p></o:p></pre>
<pre><a href="http://www.utc.fr/~mottelet">http://www.utc.fr/~mottelet</a><o:p></o:p></pre>
</div>
</body>
</html>