<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Le 15/10/2018 à 11:55, Arvid Rosén a
écrit :<br>
</div>
<blockquote type="cite"
cite="mid:20412977-4292-4E2F-B992-7DFDF29B9746@softube.com">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Consolas",serif;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Thanks for getting back
to me!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Unfortunately, we used
Scilab’s pretty cool way of doing object orientation, so we
have big nested tlist structures with multiple instances of
various lists of filters and other structures, as in my
example. Saving those structures in some explicit manual way
would be extremely complicated. Or is there some way of
writing explicit HDF5 saving/loading schemes using
overloading? That would be great! I am sure we could find
the main culprits and do something explicit for them, but as
they can be located wherever in a big nested structure, it
would be painful to do anything on the top level.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Another, related I
guess, problem here is that the new file format uses about
15 times as much disk space as the old format (for a typical
ill-behaved nested structure). That adds to the save/load
time too I guess, but is probably not the main source here.</span></p>
</div>
</blockquote>
Argh, yes, I tested it and in your example, I have a file x8.5
bigger.<br>
I think that both increases in time and size are real issues and
should be reported as bugs.<br>
<br>
By the way, I rewrote your script to run it under both 6.0 and 5.5:<br>
<br>
/////////////////////////////////<br>
N = 4;<br>
n = 10000;<br>
filters = list();<br>
<br>
for i=1:n<br>
G=syslin('c', rand(N,N), rand(N,1), rand(1,N), rand(1,1));<br>
filters($+1) = G;<br>
end<br>
<br>
ver=getversion('scilab');<br>
<br>
if ver(1)<6 then<br>
tic();<br>
save('filters_old.dat', filters);<br>
ts1 = toc();<br>
else<br>
tic();<br>
save('filters_new.dat', 'filters');<br>
ts1 = toc(); <br>
end<br>
<br>
printf("Time for save %.2fs\n", ts1);<br>
/////////////////////////////////<br>
<br>
Hope it helps,<br>
<br>
Antoine<br>
<br>
<blockquote type="cite"
cite="mid:20412977-4292-4E2F-B992-7DFDF29B9746@softube.com">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I think I might have
reported this earlier using Bugzilla, but I’m not sure. I’ll
check and report it if not.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Arvid<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:36.0pt"><b><span
style="font-size:12.0pt;color:black" lang="EN-US">From:
</span></b><span style="font-size:12.0pt;color:black"
lang="EN-US">users <a class="moz-txt-link-rfc2396E" href="mailto:users-bounces@lists.scilab.org"><users-bounces@lists.scilab.org></a>
on behalf of <a class="moz-txt-link-rfc2396E" href="mailto:amonmayr@laas.fr">"amonmayr@laas.fr"</a> <a class="moz-txt-link-rfc2396E" href="mailto:amonmayr@laas.fr"><amonmayr@laas.fr></a><br>
<b>Reply-To: </b><a class="moz-txt-link-rfc2396E" href="mailto:antoine.monmayrant@laas.fr">"antoine.monmayrant@laas.fr"</a>
<a class="moz-txt-link-rfc2396E" href="mailto:antoine.monmayrant@laas.fr"><antoine.monmayrant@laas.fr></a>, Users mailing list for
Scilab <a class="moz-txt-link-rfc2396E" href="mailto:users@lists.scilab.org"><users@lists.scilab.org></a><br>
<b>Date: </b>Monday, 15 October 2018 at 11:08<br>
<b>To: </b><a class="moz-txt-link-rfc2396E" href="mailto:users@lists.scilab.org">"users@lists.scilab.org"</a>
<a class="moz-txt-link-rfc2396E" href="mailto:users@lists.scilab.org"><users@lists.scilab.org></a><br>
<b>Subject: </b>Re: [Scilab-users] HDF5 save is super
slow<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span
lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">Hello,<br>
<br>
I tried your code in 5.5.1 and the last nightly-build of
6.0: I see a slowdown of around 175 between old save in
5.5.1 and new (and only) save in 6.0.<br>
It's really related to the data structure, because we use
hdf5 read/write a lot here and did not experience
significant slowdowns using 6.0.<br>
I think the overhead might come to the translation of your
fairly complex variable (a long array of tlist) in the
corresponding hdf5 structure.<br>
In the old save, this translation was not necessary.<br>
Maybe you could try to save your data in a different way.<br>
For example:<br>
3) you could save each element of "filters" in a separate
file.<br>
2) you could bypass save and directly write your data in a
hdf5 file by using h5open(), h5write() directly. It means
you need to write your own load() for your custom file
format. But this way, you can try to find the best way to
layout your data in hdf5 format.<br>
3) in addition to 2) you could try to save each entry of
your "filters" array as one dataset in a given hdf5 file.<br>
<br>
Did you search on bugzilla whether this bug was already
submitted?<br>
Could you try to report it?<br>
<br>
<br>
Antoine<br>
<br>
Le 15/10/2018 à 10:11, Arvid Rosén a écrit :<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US">/////////////////////////////////</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US">N
= 4;</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US">n
= 10000;</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US">filters
= list();</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US">for
i=1:n</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US">
G=syslin('c', rand(N,N), rand(N,1), rand(1,N), rand(1,1));</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US">
filters($+1) = G;</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US">end</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US">tic();</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US">save('filters.dat',
filters);</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US">ts1
= toc();</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US">tic();</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US">save('filters.dat',
'filters');</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US">ts2
= toc();</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US">printf("old
save %.2fs\n", ts1);</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US">printf("new
save %.2fs\n", ts2);</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US">printf("slowdown
%.1f\n", ts2/ts1);</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt"><span
style="font-family:"Courier New"" lang="EN-US">/////////////////////////////////</span><o:p></o:p></p>
</blockquote>
<p style="margin-left:36.0pt"><o:p> </o:p></p>
<pre style="margin-left:36.0pt">-- <o:p></o:p></pre>
<pre style="margin-left:36.0pt">+++++++++++++++++++++++++++++++++++++++++++++++++++++++<o:p></o:p></pre>
<pre style="margin-left:36.0pt"><o:p> </o:p></pre>
<pre style="margin-left:36.0pt"> Antoine Monmayrant LAAS - CNRS<o:p></o:p></pre>
<pre style="margin-left:36.0pt"> 7 avenue du Colonel Roche<o:p></o:p></pre>
<pre style="margin-left:36.0pt"> BP 54200<o:p></o:p></pre>
<pre style="margin-left:36.0pt"> 31031 TOULOUSE Cedex 4<o:p></o:p></pre>
<pre style="margin-left:36.0pt"> FRANCE<o:p></o:p></pre>
<pre style="margin-left:36.0pt"><o:p> </o:p></pre>
<pre style="margin-left:36.0pt"> Tel:+33 5 61 33 64 59<o:p></o:p></pre>
<pre style="margin-left:36.0pt"> <o:p></o:p></pre>
<pre style="margin-left:36.0pt"> email : <a href="mailto:antoine.monmayrant@laas.fr" moz-do-not-send="true">antoine.monmayrant@laas.fr</a><o:p></o:p></pre>
<pre style="margin-left:36.0pt"> permanent email : <a href="mailto:antoine.monmayrant@polytechnique.org" moz-do-not-send="true">antoine.monmayrant@polytechnique.org</a><o:p></o:p></pre>
<pre style="margin-left:36.0pt"><o:p> </o:p></pre>
<pre style="margin-left:36.0pt">+++++++++++++++++++++++++++++++++++++++++++++++++++++++<o:p></o:p></pre>
<pre style="margin-left:36.0pt"><o:p> </o:p></pre>
</div>
</blockquote>
<p><br>
</p>
<pre class="moz-signature" cols="72">--
+++++++++++++++++++++++++++++++++++++++++++++++++++++++
Antoine Monmayrant LAAS - CNRS
7 avenue du Colonel Roche
BP 54200
31031 TOULOUSE Cedex 4
FRANCE
Tel:+33 5 61 33 64 59
email : <a class="moz-txt-link-abbreviated" href="mailto:antoine.monmayrant@laas.fr">antoine.monmayrant@laas.fr</a>
permanent email : <a class="moz-txt-link-abbreviated" href="mailto:antoine.monmayrant@polytechnique.org">antoine.monmayrant@polytechnique.org</a>
+++++++++++++++++++++++++++++++++++++++++++++++++++++++
</pre>
</body>
</html>