In CSL REDUCE, the sources are in general compiled to byte code in contrast to native machine code. For efficiency reasons a limited subset of frequently used Lisp functions (currently 362) is determined via profiling the execution of the REDUCE test files. When building CSL reduce, this subset is automatically translated to C code, then compiled to native code, and finally linked with the binary CSL REDUCE.
In general, developers need not care about profiling at all. This is done by the CSL maintainers, and the profiling information is checked into the SVN tree. It is contained in the file
However, the tools for profiling are available to all developers. Below we describe how to possibly use them.
How to Profile, and Why
Profiling is performed as follows:
cd cslbuild/<host>/csl make profile
The above sequence runs all test files specified in
packages/package.map in order to collect profiling information into the file
cslbuild/profile.dat. The old file
profile.dat is moved to
profile.dat.old. By locally changing existing test files or locally adding further test files to
package.map, developers can verify if this influences the decision what to compile to C, and in the positive case locally tune their system to their particular needs.
Important note: It is agreed that private versions of
profile.dat are never checked into SVN without checking this with the CSL maintainers in advance.
In the current version there are some tests implemented to avoid infinite runs or large output during the the automatic build process. The test files should satisfy the following constraints:
- They do not consume more than 5 minutes CPU time.
- They do at most around 200 million
- They generate at most 10 MB output.
These constraints are implemented in
csl/cslbase/buildreduce.lsp using a general mechanism for such limits provided by
Documentation of profile.dat
profile.dat has a section for each package that has a test script. Consider for instance in the following entry for the Redlog package the data for
(redlog (num!-exponents 1902293640759984724 29 43299893) (noncomfp 258374047657212774 11 15469806) (rl_op 1797972776800570965 9 11385795) (reorder 920970208773136746 36 42733912) (quotf 1079448166529811008 15 17261200) ...
1079448166529811008is a checksum of the definition of
15is the length of the byte code compiled
17261200is the number of executed
quotfbyte codes during the Redlog test in a byte code-only CSL.
The first number admits to handle several functions with the same name. Furthermore, it admits via comparison of source code checksums to correctly handle byte code redefinitions of C-compiled functions. The final two numbers are used to rank the priority of things for conversion to C: Short functions that see heavy use come top. The order of the functions for each entry in
profile.dat reflects this ranking.
What is finally compiled to C?
When generating C there is first the top function from each package compiled, then the second from each etc. until a certain limit is reached. So each test script gets pretty much equal priority whether it is short or long. Notice that adding more computation to test files does not increase the chance to get more functions compiled to C. It is much more important to have test files that are representative with respect to the available algorithms, their expected use, and the characteristics of the expected input data.
A list of all functions actually compiled to C at the end can be found in
cslbuild/<host>/csl/buildlogs/c-code.log. The C-Code itself is located in
cslbuild/generated-c. The number of functions compiled to C is currently 362. This number is given in
For analyzing the usage of Lisp functions for certain computations there is mapstore available in bootstrapreduce. It yields a table of which Lisp-coded (byte-compiled) functions are most heavily used. For possibly speeding up code one could then either consider whether the calls to those routines are indeed essential or look at the algorithms in those code fragments.
- For further details on the profiling process see the well-commented files in the CSL distribution: