[Scilab-users] HDF5 save is super slow

[Arvid Rosén]

Dear Scilab list,

We have been using Scilab since version 3 at my company. Migrating from one version to another has always been some work, but going from 5 to 6 seems to be the most difficult so far.

One of the problems for us, is the new HDF5 format for loading and saving. We have a huge number of old data sets that we need to keep working on, and most of them contain large number of state-space systems stored in lists. However, loading of saving these data sets is extremely slow compared to the old binary format. So slow in fact, that using Scilab 6 is impossible for us at the moment. I have a short test case that demonstrates the problem:

/////////////////////////////////
N = 4;
n = 10000;

filters = list();

for i=1:n
  G=syslin('c', rand(N,N), rand(N,1), rand(1,N), rand(1,1));
  filters($+1) = G;
end

tic();
save('filters.dat', filters);
ts1 = toc();

tic();
save('filters.dat', 'filters');
ts2 = toc();

printf("old save %.2fs\n", ts1);
printf("new save %.2fs\n", ts2);
printf("slowdown %.1f\n", ts2/ts1);
/////////////////////////////////

Obviousle, the code above need to run in scilab 5, as it uses both new and old methods for saving the list of state-space filters. And to be fair, HDF5 saving a bit faster in scilab 6, but still orders of magnitude slower than the old format in scilab 5. As a reference, below is the output on my pretty fast Mac running scilab 5:

Warning: Scilab 6 will not support the file format used.
Warning: Please quote the variable declaration. Example, save('myData.sod',a) becomes save('myData.sod','a').
Warning: See help('save') for the rational.
Warning: file 'filters.dat' already opened in Scilab.
old save 0.03s
new save 20.93s
slowdown 775.0

So my questions:
Can and will this be addressed in future versions of Scilab?
Can I store large number of state-space systems in another way to make this faster?

Best Regards,
Arvid
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.scilab.org/pipermail/users/attachments/20181015/9b8ecb50/attachment.htm>