[Scilab-Dev] New profiling module (code coverage)
Samuel Gougeon
sgougeon at free.fr
Sat Apr 30 20:17:52 CEST 2016
Hello,
This message continues in a separate thread the discussion opened @
http://mailinglists.scilab.org/Release-Scilab-6-0-0-alpha-2-tp4033142p4033143.html
It aims to test and assess interfaces of the new Scilab code /coverage/
module, that has been implemented without any Scilab Enhancement Proposal.
This module aims to replace profiling functions
<https://help.scilab.org/docs/5.5.2/en_US/section_f8ae48740240f0c0b0770316ec9938aa.html>that
are removed in Scilab 6:
* add_profiling() => covStart()
* reset_profiling() => no equivalent
* remove_profiling() => covStop()
* profile() => no equivalent
* showprofile() => no equivalent
* plotprofile() => abandoned (actually, without displaying
the lines of code, it was hardly useful)
--------------------
* *"c**o**v**...**" naming*: "cov" rather makes thinking about
/cov/ariance rather than coverage, "code coverage" being an
expression quite specific to software engineering. It is not
explicit for users that would simply want to test and benchmark the
code of their "macros".
*"codebench()*" is suggested as an alternative name of a unique
function, with an action parameter: *codebench("start" ...)*,
codebench("write",..) etc.
By the way, replacing former profiling functions needs as well to
replace their *timing feature* (cumulated time spent to run each
line of code), not only their coverage one (the number of times that
each line has been run). So, the same profiling function should go
beyond coverage, without implementing extra specific functions.
*Time counters* have been implemented :
https://codereview.scilab.org/#/c/16397/
Timing features look to be already available in Scilab 6.0-b1,
despite this commit hereabove has not yet been included in the
distrib. Strange.
* *add_profiling()
<https://help.scilab.org/docs/5.5.2/en_US/add_profiling.html> /
covStart(paths)*:
o *covStart("isempty") does**not work* (and does not yield any
error message), contrarily to what /help covStart/ tells. It
should work. Currently, only /covStart(isempty)/ is able to
yields results. Profiling functions shall be as well
console-oriented: input arguments of text type should be
accepted as well, like with former profiling functions.
o *covStart(libraryname)* allows to set a whole library for
profiling. This is an improvement. With the former
add_profiling(), it was only possible to set functions one by
one. But:
--> covStart(differential_equationlib)
covStart: Wrong type for input argument #1: A two-columns string
matrix expected.
The documentation is not consistent with this requirement. And
this requirement is not consistent neither with the function
case (as isempty hereabove), nor with the following (that is
accepted):.
--> covStart differential_equationlib
-->
By the way, after that, no results are produced with
covWrite(..) (see below).
o *c**ovStart(paths)*: Since it is now possible to provide a file
path instead of a function name, we could think that it becomes
possible to get as well the /profile of scripts/ out of functions.
However, from first tests, nothing is produced by
covWrite("html",".") when the covStart() has been applied only
to some scripts. The fact that scripts.sce are excluded should
be explicitly documented, unless profiling is extended to them.
*
*
* *covWri**t**e(..)*:
o
showprofile(fun
<https://help.scilab.org/docs/5.5.2/en_US/showprofile.html>)
<https://help.scilab.org/docs/5.5.2/en_US/showprofile.html> has
no replacement:*.*
+ Displaying profiling results in the console is no longer
possible. 95% of time, i used to use this way to get and
read results. It was the most handy. It allowed to have the
code edited in Scinotes and related profiling scores
displayed in the console aside. Now, we have to switch
between the web browser and Scilab's IDE (since results can
now be displayed only in HTML, after having found the right
file of results and clicked on it...) . Display in console
was straightforward and efficient, to improve the code
"on-the-fly", without to have to manage any HTML files.
*imo, this limitation to HTML is **a critical regression*.
+ covWrite(..)outputs results in HTML /after/ each line of
code, instead of before each line number: *This is a
blocking regression*.
It prevents any use of new profiling functions. This trivial
bug #12807
<http://bugzilla.scilab.org/show_bug.cgi?id=12807> was as
well for a long time somewhat killing showprofile(). It is
back. Why?
Imagine that you manufacture suitcases, but the designer has
put the handle under them, between their wheels. Currently,
covWrite() is somewhat in this status.
See attached uman.html.zip, and please tell me where is the
handle.
+ outputs from covWrite() are unclear/undocumented (see
attached file, from the covWrite basic example):
# items in the header are not clearly defined: instrs,
coverage rates, ..
# Both percentages for each line have no legend
# Color codes (red/black) have no legend
# red "x 0" on some lines have no legend
# Spent times are displayed with an irrelevant relative
accuracy (too many digits)
o c = profile(fun)
<https://help.scilab.org/docs/5.5.2/en_US/profile.html> has no
replacement:
+ It was possible to get results in a matrix of text c. It is
no longer the case. *This is a medium regression*.
+ It was possible to get the profile of a single chosen
function (or of a subset of them with a for loop) among all
ones set for profiling.
It is no longer the case: We are now compelled either to
overwrite all previous results in the same given destination
directory, or to create a new destination directory
containing all results each time that we want new results
for a single function among all profiled ones.*This is a
major regression.
*
o *covWrite() does**not work on a library*:
covStop();
covStart("differential_equationlib"); // then run examples from
inttrap(), integrate() and intc() help pages
covWrite("html", ".") // produces only a ./report.html file
that is empty.
* reset_profiling()
<https://help.scilab.org/docs/5.5.2/en_US/reset_profiling.html> has
no replacement: It is no longer possible to reset counters of a
chosen function or of a set of chosen functions.
To do so, stopping and restarting *all of them* is now mandatory.
* remove_profiling(fun)
<https://help.scilab.org/docs/5.5.2/en_US/remove_profiling.html> /
*covStop()*:
o In one hand, remove_profiling(fun) did not allow to stop
profiling of all profiled functions. This was unhandy. covStop()
allows this. This is nice.
o In the other hand, covStop() does not allow to stop profiling
for a single function or a file or an explicit set of files or a
library. The only way to stop profiling is now to stop it for
*all* "started" functions.
It could be disturbing if profiling a function makes it slower,
or covWrite() cannot export resuls for a subset of "started"
functions. But does profiling a function make it slower? Tests
to be done.
IMO, covWrite() and covStop(..) shall have the same input modes
as allowed with covStart().
* Neither binary files of results nor covMerge() that processes them
have been tested. When simpler usages will work...
-----------------
*Partial (and personal) conclusions:*
* Removed profiling functions need replacements.
* The new profiling module has been designed without SEP. Just
matching features of former profiling functions would not have
needed any SEP. But this is not the situation at all. The new module
is clearly not ready. It is hardly an alpha release.
Other tests, comments and users opinions are somewhat needed and welcome.
Regards
Samuel Gougeon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.scilab.org/pipermail/dev/attachments/20160430/6762af4f/attachment.htm>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.scilab.org/pipermail/dev/attachments/20160430/6762af4f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: uman.html.zip
Type: application/x-zip-compressed
Size: 34585 bytes
Desc: not available
URL: <https://lists.scilab.org/pipermail/dev/attachments/20160430/6762af4f/attachment.bin>
More information about the dev
mailing list