<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-text-html" lang="x-unicode">
<div class="moz-cite-prefix">Hello,<br>
<br>
I looked a little bit in the sources: the evident bottleneck is
the nested creation of an hdf5 group each time that a container
variable is met.<br>
For the given example, this is particularly evident. If you
replace the syslin structure by the corresponding [A,B;C,D]
matrix, then save is ten times faster:<br>
<br>
N = 4;<br>
n = 1000;<br>
filters = list();<br>
for i=1:n<br>
G=syslin('c', rand(N,N), rand(N,1), rand(1,N), rand(1,1));<br>
filters($+1) = G;<br>
end<br>
tic();<br>
save('filters.dat', 'filters');<br>
disp(toc());<br>
--> disp(toc());<br>
<br>
0.724754<br>
<br>
N = 4;<br>
n = 1000;<br>
filters = list()<br>
for i=1:n<br>
G=syslin('c', rand(N,N), rand(N,1), rand(1,N), rand(1,1));<br>
filters($+1) = [G.a G.b;G.c G.d];<br>
end<br>
tic();<br>
save('filters.dat', 'filters');<br>
disp(toc());<br>
--> disp(toc());<br>
<br>
0.082302<br>
<br>
Serializing container objects seems to be the solution, but it
goes towards an orthogonal direction w.r.t. the hdf5 portability
spirit.<br>
<br>
S.<br>
<br>
<br>
Le 15/10/2018 à 12:22, Antoine Monmayrant a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:7254d271-46d7-a83e-814a-400c106c7c11@laas.fr">
<div class="moz-cite-prefix">Le 15/10/2018 à 11:55, Arvid Rosén
a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:20412977-4292-4E2F-B992-7DFDF29B9746@softube.com">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hi,</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">Thanks for getting
back to me!</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">Unfortunately, we
used Scilab’s pretty cool way of doing object
orientation, so we have big nested tlist structures with
multiple instances of various lists of filters and other
structures, as in my example. Saving those structures in
some explicit manual way would be extremely complicated.
Or is there some way of writing explicit HDF5
saving/loading schemes using overloading? That would be
great! I am sure we could find the main culprits and do
something explicit for them, but as they can be located
wherever in a big nested structure, it would be painful
to do anything on the top level.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">Another, related I
guess, problem here is that the new file format uses
about 15 times as much disk space as the old format (for
a typical ill-behaved nested structure). That adds to
the save/load time too I guess, but is probably not the
main source here.</span></p>
</div>
</blockquote>
Argh, yes, I tested it and in your example, I have a file x8.5
bigger.<br>
I think that both increases in time and size are real issues and
should be reported as bugs.<br>
<br>
By the way, I rewrote your script to run it under both 6.0 and
5.5:<br>
<br>
/////////////////////////////////<br>
N = 4;<br>
n = 10000;<br>
filters = list();<br>
<br>
for i=1:n<br>
G=syslin('c', rand(N,N), rand(N,1), rand(1,N), rand(1,1));<br>
filters($+1) = G;<br>
end<br>
<br>
ver=getversion('scilab');<br>
<br>
if ver(1)<6 then<br>
tic();<br>
save('filters_old.dat', filters);<br>
ts1 = toc();<br>
else<br>
tic();<br>
save('filters_new.dat', 'filters');<br>
ts1 = toc(); <br>
end<br>
<br>
printf("Time for save %.2fs\n", ts1);<br>
/////////////////////////////////<br>
<br>
Hope it helps,<br>
<br>
Antoine<br>
<br>
<blockquote type="cite"
cite="mid:20412977-4292-4E2F-B992-7DFDF29B9746@softube.com">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US"></span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">I think I might have
reported this earlier using Bugzilla, but I’m not sure.
I’ll check and report it if not.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">Cheers,</span></p>
<p class="MsoNormal"><span lang="EN-US">Arvid</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:36.0pt"><b><span
style="font-size:12.0pt;color:black" lang="EN-US">From:
</span></b><span style="font-size:12.0pt;color:black"
lang="EN-US">users <a class="moz-txt-link-rfc2396E"
href="mailto:users-bounces@lists.scilab.org"><users-bounces@lists.scilab.org></a>
on behalf of <a class="moz-txt-link-rfc2396E"
href="mailto:amonmayr@laas.fr">"amonmayr@laas.fr"</a>
<a class="moz-txt-link-rfc2396E"
href="mailto:amonmayr@laas.fr"><amonmayr@laas.fr></a><br>
<b>Reply-To: </b><a class="moz-txt-link-rfc2396E"
href="mailto:antoine.monmayrant@laas.fr">"antoine.monmayrant@laas.fr"</a>
<a class="moz-txt-link-rfc2396E"
href="mailto:antoine.monmayrant@laas.fr"><antoine.monmayrant@laas.fr></a>,
Users mailing list for Scilab <a
class="moz-txt-link-rfc2396E"
href="mailto:users@lists.scilab.org"><users@lists.scilab.org></a><br>
<b>Date: </b>Monday, 15 October 2018 at 11:08<br>
<b>To: </b><a class="moz-txt-link-rfc2396E"
href="mailto:users@lists.scilab.org">"users@lists.scilab.org"</a>
<a class="moz-txt-link-rfc2396E"
href="mailto:users@lists.scilab.org"><users@lists.scilab.org></a><br>
<b>Subject: </b>Re: [Scilab-users] HDF5 save is super
slow</span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span
lang="EN-US"> </span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">Hello,<br>
<br>
I tried your code in 5.5.1 and the last nightly-build of
6.0: I see a slowdown of around 175 between old save in
5.5.1 and new (and only) save in 6.0.<br>
It's really related to the data structure, because we
use hdf5 read/write a lot here and did not experience
significant slowdowns using 6.0.<br>
I think the overhead might come to the translation of
your fairly complex variable (a long array of tlist) in
the corresponding hdf5 structure.<br>
In the old save, this translation was not necessary.<br>
Maybe you could try to save your data in a different
way.<br>
For example:<br>
3) you could save each element of "filters" in a
separate file.<br>
2) you could bypass save and directly write your data in
a hdf5 file by using h5open(), h5write() directly. It
means you need to write your own load() for your custom
file format. But this way, you can try to find the best
way to layout your data in hdf5 format.<br>
3) in addition to 2) you could try to save each entry of
your "filters" array as one dataset in a given hdf5
file.<br>
<br>
Did you search on bugzilla whether this bug was already
submitted?<br>
Could you try to report it?<br>
<br>
<br>
Antoine<br>
<br>
Le 15/10/2018 à 10:11, Arvid Rosén a écrit :</p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US">/////////////////////////////////</span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US">N = 4;</span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US">n = 10000;</span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US"> </span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US">filters = list();</span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US"> </span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US">for i=1:n</span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US"> G=syslin('c', rand(N,N), rand(N,1),
rand(1,N), rand(1,1));</span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US"> filters($+1) = G;</span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US">end</span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US"> </span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US">tic();</span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US">save('filters.dat', filters);</span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US">ts1 = toc();</span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US"> </span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US">tic();</span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US">save('filters.dat', 'filters');</span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US">ts2 = toc();</span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US"> </span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US">printf("old save %.2fs\n", ts1);</span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US">printf("new save %.2fs\n", ts2);</span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US">printf("slowdown %.1f\n", ts2/ts1);</span></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New""
lang="EN-US">/////////////////////////////////</span></p>
</blockquote>
<p style="margin-left:36.0pt"> </p>
<pre style="margin-left:36.0pt">-- </pre>
<pre style="margin-left:36.0pt">+++++++++++++++++++++++++++++++++++++++++++++++++++++++</pre>
<pre style="margin-left:36.0pt"> </pre>
<pre style="margin-left:36.0pt"> Antoine Monmayrant LAAS - CNRS</pre>
<pre style="margin-left:36.0pt"> 7 avenue du Colonel Roche</pre>
<pre style="margin-left:36.0pt"> BP 54200</pre>
<pre style="margin-left:36.0pt"> 31031 TOULOUSE Cedex 4</pre>
<pre style="margin-left:36.0pt"> FRANCE</pre>
<pre style="margin-left:36.0pt"> </pre>
<pre style="margin-left:36.0pt"> <a class="moz-txt-link-freetext" href="Tel:+33">Tel:+33</a> 5 61 33 64 59</pre>
<pre style="margin-left:36.0pt"> </pre>
<pre style="margin-left:36.0pt"> email : <a href="mailto:antoine.monmayrant@laas.fr">antoine.monmayrant@laas.fr</a></pre>
<pre style="margin-left:36.0pt"> permanent email : <a href="mailto:antoine.monmayrant@polytechnique.org">antoine.monmayrant@polytechnique.org</a></pre>
<pre style="margin-left:36.0pt"> </pre>
<pre style="margin-left:36.0pt">+++++++++++++++++++++++++++++++++++++++++++++++++++++++</pre>
<pre style="margin-left:36.0pt"> </pre>
</div>
</blockquote>
<p><br>
</p>
<pre class="moz-signature" cols="72">--
+++++++++++++++++++++++++++++++++++++++++++++++++++++++
Antoine Monmayrant LAAS - CNRS
7 avenue du Colonel Roche
BP 54200
31031 TOULOUSE Cedex 4
FRANCE
<a class="moz-txt-link-freetext" href="Tel:+33">Tel:+33</a> 5 61 33 64 59
email : <a class="moz-txt-link-abbreviated" href="mailto:antoine.monmayrant@laas.fr">antoine.monmayrant@laas.fr</a>
permanent email : <a class="moz-txt-link-abbreviated" href="mailto:antoine.monmayrant@polytechnique.org">antoine.monmayrant@polytechnique.org</a>
+++++++++++++++++++++++++++++++++++++++++++++++++++++++
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:users@lists.scilab.org">users@lists.scilab.org</a>
<a class="moz-txt-link-freetext" href="https://antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/lists.scilab.org/mailman/listinfo/users">https://antispam.utc.fr/proxy/1/c3RlcGhhbmUubW90dGVsZXRAdXRjLmZy/lists.scilab.org/mailman/listinfo/users</a>
</pre>
</blockquote>
<p><br>
</p>
<pre class="moz-signature" cols="72">--
Stéphane Mottelet
Ingénieur de recherche
EA 4297 Transformations Intégrées de la Matière Renouvelable
Département Génie des Procédés Industriels
Sorbonne Universités - Université de Technologie de Compiègne
CS 60319, 60203 Compiègne cedex
Tel : +33(0)344234688
<a class="moz-txt-link-freetext" href="http://www.utc.fr/%7Emottelet">http://www.utc.fr/~mottelet</a></pre>
</div>
</body>
</html>