Ada Compiler Evaluation System
User's Guide
for
Version 2.1
FINAL
Contract Number F33600-92-D-0125
CDRL A0014
Prepared for:
High Order Language Control Facility
Business Software Support Branch
88 CG/SCTL
Wright-Patterson AFB OH 45433-5707
Prepared by:
CTA INCORPORATED
5100 Springfield Pike, Suite 100
Dayton, OH 45431
Availability of the Ada Compiler Evaluation System
The ACES software and documentation are available by anonymous FTP from the host "sw-eng.falls-church.va.us" in the directory "public/AdaIC/testing/aces/v2.1" and from other Ada-related hosts. Document files are included in PostScript format and as ASCII text. The ACES files are also accessible via the World-Wide Web. The home page URL is "http://sw-eng.falls-church.va.us/AdaIC/testing/aces/".
For further information about the ACES, contact the High Order Language Control Facility. As of 1 March 1996, the appropriate contact is:
Mr. Brian Andrews
88 CG/SCTL
3810 Communications, Suite 1
Wright-Patterson AFB, OH 45433-5707
(513) 255-4472
E-mail: andrewbp@email.wpafb.af.mil
This document is the User's Guide for the Ada Compiler Evaluation System (ACES) contract. The purpose of this document is to guide an ACES user in running the ACES benchmark test suite and supporting tools.
LIST OF APPENDICES
APPENDIX A - ACES COMMANDS AND THEIR SYSTEM TRANSLATIONS
APPENDIX B - PRETEST REPORT FORM
APPENDIX C - SYMBOLIC DEBUGGER ASSESSOR README FILE
APPENDIX D - DIAGNOSTIC ASSESSOR README FILE
APPENDIX E - PROGRAM LIBRARY ASSESSOR README FILE
APPENDIX F - CAPACITY ASSESSOR README FILE
APPENDIX G - TROUBLE SHOOTING GUIDE
LIST OF FIGURES
The following documents are referenced in this User's Guide.
ANSI/MIL-STD-1815A Reference Manual for the Ada Programming Language (LRM)
ISO/IEC 8652 (1995) Programming Language Ada, Language and Standard Libraries (RM 95)
Readers Guide for Ada Compiler Evaluation System (ACES), Version 2.1
High Order Language Control Facility
Business Software Support Branch
88 CG/SCTL
Wright-Patterson AFB OH
Version Description Document (VDD) for
Ada Compiler Evaluation System (ACES), Version 2.1
High Order Language Control Facility
Business Software Support Branch
88 CG/SCTL
Wright-Patterson AFB OH
Primer for Ada Compiler Evaluation System (ACES), Version 2.1
High Order Language Control Facility
Business Software Support Branch
88 CG/SCTL
Wright-Patterson AFB OH
"Improving a Poor Random Number Generator," C. Bays and
S. D. Durham, ACM Transactions on Mathematical Software,
Volume 2, Number 1, March 1976.
Ada Evaluation System AES/1
User Introduction to the Ada Evaluation System Release 2, Version 1,
Issue 1, Crown Copyright, 1990.
Ada Evaluation System AES/2
Reference Manual for the Ada Evaluation Compiler Tests, Release 2, Version 1, Issue 1, Crown Copyright, 1990.
Ada Evaluation System AES/3
System User Manual Parts 0 and 1
Introduction and General Information, Release 1, Version 1, Issue 2, Crown Copyright, 1990.
Introduction to the Theory of Statistics,
A. Mood and F. Grayhill, McGraw Hill, l963.
"Proposed Standard for a Generic Package of Elementary Functions for Ada," Ada Letters - A Special Edition from SIGAda, Volume XI, Number 7, Fall 1991.
Software Manual for the Elementary Functions,
W. Cody, Jr., and W. Waite, Prentice-Hall Series in Computational Mathematics, Prentice-Hall, Inc., 1980.
"The Need for an Industry Standard of
Accuracy for Elementary-Function Programs," C. Black, R. Burton, and T. Miller, ACM Transactions on Mathematical Software,
Volume 10, Number 4, December 1984.
ACES Version 2.1 is an update of ACES Version 2.0. The first version (1.0) of the ACES was a merged product between the Ada Compiler Evaluation Capability (ACEC) Version 3.1 (sponsored by the Ada Joint Programming Office (AJPO)) and the Ada Evaluation System (AES) Version 2.0 (sponsored by the Ministry of Defence of the United Kingdom).
The ACES is a set of tests, tools, and assessors to assist in the evaluation of an Ada compilation system. The test suite is designed to measure the performance of an Ada compilation system, emphasizing execution time, code size, and compilation speed, as well as the capabilities of its symbolic debugger, diagnostic messages, program library system, and system capacities. The ACES is contained in its entirety in the distribution files. The Primer, the User's and Reader's Guides and the Version Description Document are included in PostScript and ASCII formats. Fundamental instructions on the operation of the ACES are found in the Primer. Content of the ACES is listed file by file in the VDD. Instructions on how to use the ACES are contained in this User's Guide. Interpretation of the ACES and definitions of the purpose of tests and the analysis of test results are described in the Reader's Guide.
Users who want to quickly estimate execution speed should use the ACES Quick-Look facility. This subset of the ACES performance tests and support software produces data similar to that produced by the PIWG test suite. Typical time for downloading and executing the Quick-Look tests is less than one day. The Quick-Look facility is discussed in detail in Section 8 of the ACES Primer.
Figure 2-1 illustrates the paths through the ACES evaluation process. The first step is performing the Pretest activity. (The Pretest automation tool, ZP_SETUP, is useful here.) The results of the Pretest are recorded on the Pretest Report.
If the user wishes to run the performance tests, then the next step is the use of the Harness program (compiled during the Pretest activity). This program generates command scripts for compiling and executing the performance tests. The user then executes these scripts and captures the results in log files. The Harness program reads these log files and gives execution status information so that the user can choose to re-run selected tests or go on to other tests. This process is iterative.
When the user has run as many performance tests as desired, he/she runs the Analysis Menu program, first selecting the Condense tool. Condense reads the log files and produces the Analysis Database files. The user then selects either Comparative Analysis (in which case Analysis Database files for other systems must be available) or Single-System Analysis. These Analysis programs produce extensive reports, as directed by the user.
The user may elect to run the Assessors before, after, or independently of the performance tests, provided that at least part of the Pretest has been completed. The Assessors may be run in any order. The results from each are recorded on the appropriate Assessor Report form.
The following documents provide the user with the information necessary to set up the ACES, execute the ACES, and interpret the results generated by the ACES. The user's level of expertise with the ACES and the information needed will guide the user as to which document to reference.
Primer for ACES: All new users of the ACES are first directed to this document which provides an overview of the system. More importantly, it provides a quick start-up guide. This document provides the basic instructions for setting up the system, running the system, and interpreting the results.
User's Guide for ACES: Advanced users of the ACES and users who run into problems while working with the Primer are referred to this document to help in resolving problems and to gain more insight into the details of how the ACES is organized.
Reader's Guide for ACES: Users are referred to this document when attempting to interpret test results or attempting to find out the purposes of the ACES tests.
Version Description Document for ACES: Users are referred to this document when it is necessary to find out the components of a test. This document provides a variety of cross-reference tables that describe the contents of the ACES.
The Support Software Computer Software Configuration Item (CSCI) of the ACES consists of the automated support tools. It contains several separate tools, including:
1. Quick-Look - A set of programs and tools for obtaining execution speed measurements using a subset of the ACES performance tests.
2. Pretest - A set of programs and procedures to prepare for execution of the performance tests and analysis tools.
3. Include - A tool to perform text inclusion into an Ada source text file. It will assist in adapting programs to particular targets.
4. Harness - A program that provides an interactive interface for selecting performance tests, tracking their status, and generating command files which will compile, link, and run the selected tests.
5. Analysis Menu - An interactive interface for calling any of the analysis programs.
6. Condense - A tool to extract the timing data, the code expansion data, compile and link speed data, and certain ancillary data from the output produced by compiling and executing the Operational Software and to write this information in a format usable by the Comparative Analysis program and by the Single System Analysis program.
7. Comparative Analysis (CA) - A tool to statistically compare results of running the performance tests on various systems.
8. Single System Analysis (SSA) - A tool to analyze the performance results from a single system.
The Operational Software CSCI of the ACES consists of the following items:
1. Test Suite - A set of performance tests to assess execution time, code expansion size, and compilation and link time.
2. Symbolic Debugger Assessor - A set of programs and procedures to assess symbolic debuggers.
3. Diagnostic Assessor - A set of programs and procedures to assess the diagnostic message quality.
4. Library System Assessor - A set of programs and procedures to assess Ada program library systems.
5. System Capacity Assessor - A set of programs to assess Ada compilation and run-time capacity limits.
This Guide provides detailed instructions on using the ACES. It attempts to anticipate problems that may arise and to provide enough information to enable the user to resolve most problems. The Primer for ACES provides a much simpler view of the process. For most users, it is probably best to follow the Primer, using this User's Guide only when problems arise or when more detailed information is desired.
The User's Guide is organized into 11 informational sections and 7 appendices. The informational sections discuss the actual use of the ACES, while the appendices provide step-by-step instructions for the Assessors. In addition, a User Feedback section, a Notes section (containing a list of acronyms), and an Index are provided. We recommend that the user who is depending on the User's Guide read the section on the Pretest or selected Assessor before going to the step-by-step instructions in the relevant appendix.
Section 1, "Applicable Documents," describes the documents, other than the User's Guide itself, that are relevant to the ACES.
Section 2, "Overview of the ACES," gives a very high-level overview of the entire system and the evaluation process.
Section 3, "Introduction to the User's Guide," is the current section. It presents a high-level description of the contents of this document.
Section 4, "What You Need to Know Before Starting," outlines the prerequisites for ACES testing and describes the organization of the ACES.
Section 5, "Getting Started," describes the Pretest activity. It should be read before or in parallel with the "Pretest Readme File" appendix. This section describes the pretest activity when done manually. For automation support see the ACES Primer.
Section 6, "Running the Harness," describes the use of the Harness tool, which may be used to produce command scripts for running the performance tests and to track the run-time status of these tests.
Section 7, "Running the Performance Tests," discusses the actual testing process (not necessarily using the Harness tool).
Section 8, "Running the Assessors," discusses each of the four Assessors. This section should be read before or in parallel with the "Assessor Readme File" appendices.
Section 9, "Running the Analysis," discusses the use of the Analysis Menu, the Condense tool, the Comparative Analysis tool, and the Single System Analysis tool.
Section 10, "ACES User Feedback," provides contact information and forms for problem reports and change requests.
Section 11, "Notes," provides a list of acronyms and abbreviations used in the User's Guide.
Appendix A, "ACES Commands and their System Translations," describes the syntax and semantics of the language used in the template command script files. This appendix is most useful when a user must create her/his own command scripts, based on the distributed samples.
Appendix B, "Pretest Report Form," contains a copy of the file "zp_tmplt.txt", template for the Pretest report.
Appendix C, "Debugger Assessor Readme File," gives detailed instructions for running the Debugger Assessor. It should be read after or in parallel with the corresponding section of Section 8.
Appendix D, "Diagnostic Assessor Readme File," gives detailed instructions for running the Diagnostic Assessor. It should be read after or in parallel with the corresponding section of Section 8.
Appendix E, "Library Assessor Readme File," gives detailed instructions for running the Library Assessor. It should be read after or in parallel with the corresponding section of Section 8.
Appendix F, "Capacity Assessor Readme File," gives detailed instructions for running the Debugger Assessor. It should be read after or in parallel with the corresponding section of Section 8.
Appendix G, "Trouble Shooting Guide," provides descriptions of common problems, along with possible corrective actions.
This section provides the context for understanding the ACES and its use. It includes a discussion of the resources required and a description of the structure of the ACES.
In preparation for running the ACES, some resources must be acquired.
* A compilation system (host platform with compiler)
* A target system (if different from the host)
+ A host-targeted Ada compiler is necessary in this case, since some tools run on the host.
* A copy of the ACES software and documentation.
* An evaluation objective
+ At a minimum, it should be clear whether the objective is the comparison of two or more systems or the analysis of one particular system.
Resources necessary for the use of the ACES include an operable system with a compiler and a significant amount of disk space. How much space is required varies greatly between software implementations due to the usage characteristics of the host system, but the following information on disk space requirements can be used for planning purposes.
To run the ACES, disk space will be required to contain performance test problems, support tools, assessor programs, command files, readme files, and documentation present on the distribution tape. This totals over 35 megabytes. Space will also be required for the temporary files created by the zg_incld tool, the intermediate forms created by compiling files in the Ada program library, the executables created by linking test programs, the output files from running the test programs (log files for the performance tests and temporary files for some of the test problems which test the I/O facilities of a system), and both final and intermediate files from the analysis tools.
The sample command files and the command files generated by the Setup and Harness programs are set up to:
* Delete the temporary files created by zg_incld after they are compiled;
* Delete units from the program library when they will not be referenced again; and
* Delete the temporary files created by executed I/O test programs. Some of these are as large as five megabytes.
There are various options if disk space is limited. Users may decide to delete the executables for the performance tests after executing them to conserve disk space. However, because these executables may have to be run more than once, (for example, when the verification code fails) and the programs would have to be recompiled (which might be a slow process on some systems), the Harness has a user option governing whether the generated command files delete executables.
In addition to the disk space guidelines, the following information on time allotment to execute the ACES product can also be used for preliminary planning.
* Preparation time
Preparation is accomplished during the Pretest activity. The time required depends on the user's familiarity with the operating system(s) and compiler being used and their similarity to any of the default systems provided. Two days should normally be sufficient for downloading, decompression of files, and completion of the Pretest activity.
* Time for compilation and execution of the performance tests
After the timing loop code has been included, the performance tests consist of approximately half a million lines of code. For example, it took approximately 90 hours to compile/execute all the performance tests for one compilation system on a MicroVAX II, including the rerunning of some tests. The time may be MUCH slower for embedded targets due to downloading. The time varies greatly between systems and compilation options selected. If errors or restrictions are found in the compilation system being evaluated, then the time for the testing process may increase. As a general guideline, a user should allow 1 to 3 weeks to complete the full set of performance tests and analyze the results.
* Total time
In addition to the preparation time and the compilation/execution of performance tests, the time to execute the four ACES assessors must be factored in to get the total time investment to execute the ACES product. One week should be allowed for each of the assessors on the average. Therefore, as a rough estimate, it could take one programmer about 8 weeks to thoroughly evaluate a compilation system using the ACES test suite and assessors. The amount of time varies with the experience of the programmer, the reliability of the system being tested, and the amount of free disk space. Due to the organization of the performance tests into groups/subgroups and separate assessors, it is possible to run subsets of the ACES in less time.
The objective of this section is to describe how to start the ACES tool evaluation process.
All ACES Operational and Support files are divided into major categories by prime purpose. There are 21 execution-time performance test groups, four assessor groups, seven support groups, and the Quick-Look group. The names of all ACES source files and performance tests reflect the group to which they belong, so they can easily be identified and sorted.
The individual files in the Operational and Support files are prefixed by a standard block of comments, as shown in Figure 4-1.
-- Name : This will identify the test problem or tool
-- Prime Purpose : This states the prime goal of the problem
-- Optimization : Applicable optimizations
-- Related tests : List of similar and related tests
-- Author : Identify the author
-- Reviewer : Identify the reviewer of test
-- Date : Date of original test problem
-- Source : Citation of source of algorithms or test,
-- if appropriate
-- Dependencies : Mention use of system dependent features which
-- may require adaptation or warn of non-portability
-- Other Information : Anything else relevant
-- Revisions : Change history, if required
-- Author : Identify the author
-- Reviewer : Identify who reviewed the changes
-- Date : Date of revision
-- Change : Identify the change made
Each of the 21 major groups of performance test files in Figure 4-2 contains one or more subgroups of tests. This extra level makes it possible to do subgroup analysis. Most performance tests are in separate compilation units to isolate the potential effect on other test problems, in the event of a failure to compile a test problem. See the VDD Appendix B, "Test Problem to Source File Map", for subgroups and main programs that contain the tests.
The performance test groups are listed below.
* ap: Application group
* ar: Arithmetic group
* cl: Classical group
* do: Data Storage group
* dr: Data Structures group
* s: Delays and Timing group
* xh: Exception Handling group
* gn: Generics group
* io: Input Output group
* in: Interfaces
* ms: Miscellaneous group
* oo: Object Oriented
* op: Optimizations
* po: Program Organization group
* pt: Protected Types
* sr: Storage Reclamation group
* st: Statements group
* su: Subprograms group
* sy: Systematic Compile Speed group
* tk: Tasking group
* ud: User Defined
Command files that compile, link, and execute all the performance tests by group can be created using the Harness build command (see Section 6 "RUNNING THE HARNESS"). These command files can be concatenated together before execution for the more robust systems, or can be further broken down by self-contained subgroups to give the flexibility to execute only the selected subgroups.
* Dummy Versions - Before the "real" test is compiled, a "dummy" version of each problem which prints the problem name and a compile-time error code is compiled into the program library. There is a file for each of the subgroups containing "dummy" versions of the test problems for that subgroup. This file must be compiled before compiling the performance tests for that subgroup. Dummy files are created by the Harness.
* Common Packages - There are common packages at the subgroup level wherever needed. These packages must be compiled before compiling the performance tests for that subgroup.
* Main Programs - Test problems are grouped into main programs within subgroups and by compiler options. There is a default maximum of nine tests per main program. The packaging of multiple tests into programs eases the download task for embedded targets.
* Support Command Files - There are command files to:
+ Set the default Ada library.
+ Write a time stamp before and after a compile or link statement.
+ Copy an Ada source file from the source directory into the working directory.
+ Compile an Ada program without time stamps.
+ Compile an Ada program with time stamps.
+ Copy Timing Loop initialization parameters.
+ Include Timing Loop code into a test problem.
+ Delete files and library units.
These command files are created by the Setup program or (manually) by using their template files as guides. See Section 5 for more detail.
All source file names have eight characters or less and all suffixes have three characters or less. The first three characters (two-character group name abbreviation followed by an underscore) of any file name in the ACES test suite identify the group to which the file or test belongs. The execution-time groups do not have any two letter codes beginning with "y" (reserved for the assessor groups), "z" (reserved for the support groups) or "q" (reserved for the Quick-Look group)). All file names are unique throughout the ACES Operational and Support CSCI's.
All execution-time performance test groups are comprised of subgroups of tests. The subgroup name abbreviation is formed by the fourth and fifth characters in all the execution-time file and test names. This enables easy identification of those subsets of tests of particular interest and their associated support files.
The convention for the performance test file names is: a two-letter code for the group name, an underscore, a two-letter code for the subgroup name, two digits (representing the number of a file within that subgroup), and an underscore (indicating that this is an original release). The trailing underscore will be replaced with an "a" after the first revision. For example, under the Applications group and the Avionics subgroup, the first file has the name "ap_av01_.inc". The suffix ".inc" indicates that this is a file that must have the Timing Loop code "included" by Include before compilation.
* Dummy Versions - The convention for the dummy versions is: a two-letter group name code, an underscore, a two-letter subgroup name code, and the abbreviation, "dum". For example, the dummy file for the Avionics subgroup of the Applications group is named "ap_avdum.ada".
* Common Packages - The convention for the common packages is: a two-letter group name code, an underscore, a two-letter subgroup name code, and the abbreviation, "pkg". For example, the file containing the common packages for the Applications group and the Avionics subgroup is named "ap_avpkg.ada".
* Main Programs - The convention for the main programs is: a two-letter group name code, an underscore, a two-letter subgroup name code, an "m" (identifies the file as a main program), and then incremental numbering to identify the series of main programs within that subgroup. For example, the first main program for the Avionics subgroup of the Applications group is named "ap_avm01.ada".
* Command Files - The naming convention for the test suite support command files needed during the compilation/execution step is covered in Section 4.3.3.3 "Support Groups".
A list of all distributed files is contained in the VDD Appendix C.
The maximum length of any performance test name is 28 characters. All test names begin with the two letter code for the group name, an underscore, a two-letter code for the subgroup name, and then an underscore. The remaining characters are reflective of the test's prime purpose, if possible. For example, ap_av_arti_asum can be identified as a member of the Applications (ap) group and the Avionics (av) subgroup. Otherwise, an abbreviation of the subgroup name, plus an underscore, and two digits is used, for example, dr_ba_bool_arrays_01. This test can be identified as a member of the Data Structures (dr) group and the Boolean Arrays (ba) subgroup.
All two letter codes for the assessor group names begin with a "y" to distinguish them from the execution-time test group or the support group names.
* Debugger Assessor - The first three characters of a Debugger Assessor name are "yb_" and the next 5 characters identify the file's purpose, except for the test names, which are numbered incrementally. The sixth character of a Debugger Assessor file name is an underscore character, "_", or a "t"; a test containing tasking constructs has a "t". Where several files are required for one test, they are distinguished by an alphabetic character in the seventh position of the name. An example of a Debugger Assessor test name is:
yb_01_a.ada -- test 01, file a
yb_01ta.ada -- tasking version of test 01, file a
* Capacity Assessor - The first three characters of a Capacity Assessor name are "yc_" and the next five characters identify the file's purpose, except for the test names which are numbered incrementally. For example, "yc_serch.com" is a command file for the Capacity Assessor. The Capacity Assessor tests are divided into two subgroups with identifying characters in the fourth and fifth positions of the name as follows:
ct -- Compile-time tests
rt -- Run-time tests
An example of a Capacity Assessor compile-time test name is:
yc_ct01g.ada -- source generator for compile-time test 01
yc_ct01_.ada -- generated source code for compile-time test 01
* Diagnostic Assessor - The first 3 characters of a Diagnostic Assessor name are "yd_" and the next 5 characters identify the file's purpose, except for the test names, which are numbered incrementally. Where several files are required for one test, they are distinguished by an alphabetic character in the eighth position of the name. The file "yd_compl.com" is the Diagnostic Assessor's command file (VMS) to compile the tests. The Diagnostic Assessor tests are divided into four subgroups with identifying characters in the fourth and fifth positions of the name as follows:
cw -- Compiler Warnings
ce -- Compiler Errors
lt -- Link Time Errors
rt -- Run Time Errors
An example of a Diagnostic Assessor run-time test name is:
yd_rt01a.ada -- Run-time test 01, file a
* Program Library Assessor - The first three characters of a Library Assessor name are "yl_" and the next five characters identify the file's purpose, except for the test names. These contain the characters "ib" in the fourth and fifth positions, and then a number which increases incrementally in the sixth and seventh positions. Where several files are required for one test, they are distinguished by an alphabetic character in the eighth position of the name. The Library Assessor Summary Report Form file is "yl_tmplt.txt". An example of a Library Assessor test name is:
yl_ib14a.ada -- Library Assessor test 14, file a
All support group names begin with a "z". Here are examples from each of the seven major support groups:
* Analysis Group - The first three characters for all files in the Analysis group are "za_". Then some of the files in the Analysis group are further broken down into subgroups, one for common files and four for files related to a specific analysis tool. The files in the tool subgroups are identified by the next two characters in the file name:
co -- Common
ca -- Comparative Analysis (CA)
cn -- Condense
mn -- Menus
sa -- Single System Analysis (SSA)
The last three characters in these file names identify the file's purpose. An example of an analysis file name is "za_saopt.ssa" which is a template file used by SSA in generating its Main Report.
* Command File Templates - The first three characters for the templates for low-level command scripts and associated files are "zc_". The next characters (up to five) identify the file's purpose. An example of a test suite command file template name is "zc_adaop.tpl" which invokes an Ada compiler with a compiler option of optimization.
* Documentation - The first three characters for the distributed documentation files are "zd_". The next characters (up to five) identify the file's purpose. An example of a documentation file name is "zd_readg.txt" which is the Reader's Guide delivered as an ASCII text file.
* Global and Timing Loop Files - The first three characters for the Global and Timing Loop files are "zg_". The next characters (up to five) identify the file's purpose. There are four possible suffixes for some of the files in this group based on which measurement technique is desired for testing. The three-character extension is composed of two parts: the two left-most characters for time measurement ("el" for "elapsed time", "cp" for "cpu time") and the right-most character for code size measurements ("g" for the " ZG_GETAD" function, "l" for the "label'ADDRESS" attribute). An example of a Global package name is "zg_glob3.elg".
* Harness Group - The first three characters for the Harness group are "zh_". The next characters (up to five) identify the file's purpose or are numbered incrementally. An example of a Harness file is "zh_ap.txt", which contains test and file-name data for the Application (ap) group.
* Math Group - The first three characters for the Math group are "zm_". The next characters (up to five) identify the file's purpose. An example of a file name in the Math group is "zm_math.ada".
* Pretest Group - The first three characters for the Pretest files are "zp_". The next characters (up to five) identify the file's purpose. An example of a Pretest file name is "zp_stp05.tpl" which is the template for the Setup Step 5 command script.
The Quick-Look includes files from the support groups and the performance test groups. In the distribution, these shared files are duplicated in the Quick-Look subdirectory. Files that have special versions applicable to Quick-Look have ".ql" suffixes (e.g., "zg_glob3.ql"). Files that are unique to Quick-Look are identified by the first three characters ("ql_").
This section discusses the Pretest activity, the Pretest report form, and user adaptations that may be necessary for running the performance tests.
NOTE: For a greatly simplified process producing limited performance data, see the discussion of the Quick-Look facility (Primer, Section 8).
The ACES was designed to be portable, and the test suite should be ready to use with minimal system-specific adaptations. The Pretest allows the user to identify, perform, and test these adaptations in an organized, semi-automatic fashion. The Pretest process is described in detail in Section 3 of the ACES Primer. The current section provides discussions of possible problems, special actions that may be necessary for some systems, and examples of special units that may be required.
This section is organized into subsections, discussing special situations that may arise. For each such situation, the subsection title indicates both the problem area and the Pretest step in which it might arise.
Depending on the system, there are two alternative methods available for measuring code size expansion. If the system under evaluation supports the label'ADDRESS attribute, then this measurement is straightforward to compute by taking the difference between two addresses. An ACES user should review documentation on the Ada compilation system to see if the attribute is supported. If it is not, an assembly language function can be written which returns the address of its caller, and this function can be used to bracket code whose size is to be measured.
The ACES uses the label'ADDRESS attribute to collect code expansion size measurements. Not all implementations support this attribute correctly. Some systems accept the use of the attribute, but always return zero for a value, while other systems generate a compile time error message.
The Pretest program "zp_label.ada" helps the user decide which measurement technique to use throughout the Pretest and the execution of the performance tests. This test verifies that the system's label'ADDRESS attribute is working correctly. This test reports in eight-bit bytes the difference between two label'ADDRESS statements surrounding a one-parameter procedure call. The expected range for this value should be anywhere from four to 16 bytes. Then the user can compare this value with a system map or machine code listing in order to verify the accuracy.
If a system's label'ADDRESS attribute is not supported, or "zp_label.ada" shows that the code size value is outside the expected range, a sample assembly routine, "zg_getad.mar", is provided for the VMS targeted systems. It returns the address of its caller and can be adapted by the user to calculate code expansion size. If the user does not care about collecting code expansion sizes, the performance test timing measurements can still be collected. Simply adapt the ZG_GETAD function (in "zg_glob3.elg" or "zg_glob3.cpg") so that it always returns the same value. For Ada 95 implementations, the value "System.Null_Address" is appropriate.
An example that works on DEC Ada Version 2.3 under VMS is shown in Figure 5-1.
.TITLE ZG_GETAD
; This procedure returns the value of the calling
; module's call address in RO
;
.PSECT CODE PIC, SHR, NOWRT, LONG
.ENTRY ZG_GETAD ~M<>
;
; Move the PC contents to RO
;
MOVL 16(SP), RO
RET
.END
To use the "zg_getad.mar" function, select either the set of files with the suffix ".elg" which incorporates this modification for collecting code expansion size and elapsed time measurements, or the set with the suffix ".cpg" for CPU time. The assembly code function should be submitted to the assembler before Pretest Step 5. The command script for Step 5 may need to be modified to make the resulting object file available to the Ada library. It may also be necessary to modify the linker commands in zc_link, zc_linkd, and zc_lnk to specify the library where the assembler object exists.
Details of adapting this routine to other implementations depend on the provisions for calling assembler routines from Ada programs and are highly system dependent. If there is not enough system documentation to adapt the ZG_GETAD assembly routine, a user could write ZG_GETAD as an Ada function that returns a constant of type SYSTEM.ADDRESS. Then the difference between any two label results from ZG_GETAD will be 0. The constraints of this method are:
* The name of this function must be ZG_GETAD.
* The pragma interface to ZG_GETAD statements must be removed from one of the files listed below, depending on the user's choice of measurement technique (either elapsed or CPU time as determined in the next Pretest step).
+ zg_glob3.elg for measurement of elapsed time
+ zg_glob3.cpg for measurement of CPU time
To compute the code expansion size, the Include tool (zg_incld) inserts a unique label after the last line of the insertion for Startime (zg_start); inserts another unique label before the first line of Stoptime0 (zg_stop0); and generates an assignment statement setting the variable ZG_GLOB3.EXPANSION_SIZE to the difference between the addresses of these two labels for Stoptime2 (zg_stop2).
The SUBTRACT_ADDRESS function defined in "zp_label.ada" (Pretest Step 2) and zg_glob3 (Pretest Step 5) must be adapted for 16-bit machines due to their 16-bit addresses. Use a 16-bit unsigned integer arithmetic algorithm or whatever techniques are acceptable for the compiler being evaluated. If the compilation system defines a minus operator for address types (as Ada 95 requires), use that option.
The ACES timing measurements can be executed with either elapsed or CPU time. Two versions of the timing code are distributed. One calls CALENDAR.CLOCK, and the other calls an ACES function which must be adapted to call an operating system CPU time function. This test step verifies that the system clock is working accurately. The ACES is set up to measure elapsed time as a default. The user may wish to collect CPU time rather than elapsed time; this may be done if the target operating system provides a CPU time function.
The elapsed timing measurements are performed using the function CLOCK in the predefined package CALENDAR. CALENDAR must work accurately for the Timing Loop code to function. The Timing Loop files for elapsed time have a suffix of ".ell" or ".elg" which is determined by choosing label'ADDRESS or a ZG_GETAD routine for code size measurements. These files are compiled in Pretest Step 5. CALENDAR is tested with the programs "zp_tcal1.ada" and "zp_tcal2.ada" in this Pretest step.
On bare machine targets, elapsed time measurements are the appropriate metric to collect. On multiprogramming target systems, using the CPU time metric will permit the collection of measurements without having to shut the system down to eliminate contending jobs. Measurements of the IO performance tests using CPU times are not comparable with measurements on other systems using elapsed times.
An ACES user can choose to run the Timing Loops using CPU time rather than elapsed time. The Timing Loop files for CPU time have a suffix of ".cpl" or ".cpg" which is determined by choosing label'ADDRESS or a ZG_GETAD routine for code size measurements. The separate function CPU_TIME_CLOCK ("zg_cpu.dec") will need to be replaced, and renamed "zg_cpu.ada", with code which queries the Ada runtime environment and returns the CPU time as a value of the predefined and system-dependent type CALENDAR.DURATION. These files are compiled during Pretest Step 5. To use CPU time measurements, it is necessary that the target environment maintain job CPU time (See Section 5.1.3).
Figure 5-2 is an example of a function ("zg_cpu.dec") which will access CPU time.
-- This library unit contains the function that is called by the function,
-- CPU_TIME_CLOCK, in zg_glob3.CPG or zg_glob3.CPL. This enables the user
-- that wants CPU time measurements to only have to adapt this function
-- to their system dependencies one time, here in the function zg_cpu.
-- This function was developed by the ACM SlGAda, PIWG (Association for Computing
-- Machinery, Special Interest Group on Ada, Performance Issues Working Group).
-- It is their program A000012. This version is compatible with DEC VAX Ada,
-- calling on the VMS System Service routine "$GETJPI". Refer to VAX/VMS
-- System Services Reference Manual, Order No. AA--Z502C--TE for more information.
--
-- The Ada function has a return type of DURATION.
--
-- A common implementation technique introduced errors in using CPU time for
-- timing measurements. One field in the Task Control Block (TCB) will
-- represent cumulative CPU time, but is only updated on task scheduling.
-- A system call which returns the field from the TCB will ignore the time
-- the task has expended in the current quantum (that is, since last scheduled).
-- This would appear to a program as a clock which "stutters", keeping the
-- same value for a relatively long time and then updating itself by several
-- " ticks" at one time. Such a clock can keep long term accuracy, but programs
-- using it must accommodate substantial amounts of jitter. To compute current
-- CPU time, the time since last initiation of the task should be added to the
-- value stored in the TCB. If the built-in system call does not do this,
-- a user can. If not done, the ACES Timing Loop will compute a larger than
-- otherwise necessary value for the jitter compensation time, and the
-- time to execute the test suite will be longer than it needs to be.
-- Accuracy should not be seriously degraded. The VMS system call performs the
-- desired compensation.
with SYSTEM; use SYSTEM;
with CONDITION_HANDLING; use CONDITION_HANDLING;
with STARLET; use STARLET;
FUNCTION zg_cpu RETURN duration IS
CPUTIM: INTEGER;
pragma VOLATILE ( CPUTIM );
JPI_STATUS: COND_VALUE_TYPE;
JPI_ITEM_LIST: constant ITEM_LIST_TYPE :=
( ( 4, JPI_CPUTIM, CPUTIM'ADDRESS . ADDRESS_ZERO ),
(( 0 . 0, ADDRESS_ZERO, ADDRESS_ZERO ) );
CPU_TIME_AS_DURATION: DURATION;
BEGIN
-- Call GETJPI to set CPUTIM to total accumulated CPU time -- (in 1O-millisecond tics)
GETJPI ( STATUS =>, JPL_STATUS, ITMILST=> JPI_ITEM_LIST);
CPU_TIME_AS_DURATION:= DURATION(LONG_FLOAT (CPUTIM)/100.0));
return CPU_TIME_AS_DURATION;
END zg_cpu;
Every sixty seconds, for a period of fifteen minutes (or until the program is aborted by the user), "zp_tcal1.ada" and "zp_tcal2.ada" print out a count of elapsed minutes. Tests zp_tcal1 and zp_tcal2 should be run interactively. The ACES user should be ready with a stopwatch to verify that a line is generated every 60 seconds. Some error is tolerable, but most systems should show no discernible error. A one-second drift in two minutes is less than a 1% error. If CALENDAR.CLOCK doesn't work, the ACES user should get it fixed before proceeding. The first thing to check is that the Ada system has been properly installed.
Program "zp_tcal3.ada" is to be executed only if the user wants to verify CPU time. Programs zp_tcal1 and zp_tcal2 must be run first in order to verify that elapsed time is accurate. Program zp_tcal3 should be run interactively. The first time through, zp_tcal3 must be run as the only task in the system in order to compare that result against a run with contention.
Program zp_tcal3 computes the elapsed time and the CPU time for executing a loop multiple times. It reports the variation between the results. This test verifies that:
* The CPU time measurement is less than, and close to, the elapsed time when the program is run as the only task in the system. The reported difference between CPU and elapsed time is an estimate of operating system overhead in the form of background processing.
* The CPU time is consistent over multiple loop executions.
* The CPU time measurements are affected by contention from concurrent jobs. To test this, two copies of "zp_tcal3.ada" are executed concurrently. The CPU measurements should be shorter than they were the first time zp_tcal3 was executed in standalone mode.
Program "zp_tcal4.ada" is to be run only when users want to collect CPU measurements for the compilation/link process. Program "zp_tcal4.ada" tests whether the CPU function returns a value relative to the current process or the current program. The distinction is important if the user wants to measure CPU time for a compilation process since the measurement technique provided involves several separate programs.
Two programs, zp_tcal4 and zp_tcal5, are contained in "zp_tcal4.ada". These programs consume a significant amount of CPU time and write a value of CPU time as output when they complete. If the CPU time output from the second run (zp_tcal5) is roughly double the CPU output from the first run (zp_tcal4), then the user can conclude that the CPU time being returned by the system is job time. If the outputs are roughly equal, the CPU time is probably program time. To measure compilation CPU time, job time is desired. Operating systems which provide program time will often provide job time through a similar system call. This comparison is performed in the program, which will report whether job or program time is being used.
Parts of the ACES test suite and analysis tools require a math library of elementary functions. The ACES uses a compatible subset of the Association for Computing Machinery, Special Interest Group on Ada, Numerics Working Group (ACM SIGAda NUMWG) proposed specifications for an elementary math function library from Ada Letters - A Special Edition from SIGAda, "Proposed Standard for a Generic Package of Elementary Functions for Ada" pages 9-46. It is recommended that the vendor's math library be used where it is available. The ACES provides a portable implementation of a math library to permit execution on compilation systems which do not provide any math library support, or for which the functions are not sufficiently accurate.
For a detailed discussion on adapting the math library options to the compilation system, see Section 5.1.6.1 "Math Packages". The math library is tested in Pretest Step 8. The zp_mt* programs test the accuracy of the math library on a type declared with six digits of precision; the zp_dm* programs test the accuracy on a type declared with nine digits of precision. There are four approaches to adapting the Math packages, "zm_math.ada" (single precision), and "zm_dblma.ada" (double precision), listed below in decreasing order of preference:
* Use the Ada 95 math packages.
If the user has indicated that the system under test is an Ada 95 system, then Setup will use the required math packages. No user action is required in this case. If the system has not implemented the required packages, then modify the generated script for Step 5. Using the template file "zp_stp05.tpl" as a guide, replace the copy and compile statements with the portable math package options.
* Instantiate a system-provided version of the NUMWG package.
If a compilation system provides a NUMWG package, it can be directly instantiated. Where this alternative is available, execution times may be fast because the body of the package may have been tailored to the target hardware. "zm_math.ada" and "zm_dblma.ada" are the versions of Math and Double Math which assume NUMWG support.
* Interface with an implementation-provided non NUMWG math library.
If a compilation system provides a math library which is not compatible with NUMWG recommendations, adapt zm_math and zm_dblma by providing bodies for the functions which pass through calls to the provided non-NUMWG library. Where this alternative is available, execution times may be fast because the procedures may have been implemented as interfaces to highly optimized routines which are tailored to the target hardware.
* Use the ACES portable math package zm_genma with "zm_depen.por", which was developed for use on systems not providing a usable math library. Instantiating this package to implement zm_math should involve the least user effort on systems which do not implement the NUMWG recommendations. The program zp_dptst in Pretest Step 7 can be used to verify correct execution. The performance of this option will probably be slower than for systems where the implementors have provided math packages tailored to the target hardware.
The system timing characteristics are determined during this Pretest step. The initialization program is compiled and executed four times (once for each compilation option) in Step 5 for the selected measurement technique. THIS STEP MUST BE RUN WITHOUT CONTENTION ON THE SYSTEM TO INSURE THAT THE INITIALIZATION PARAMETERS ARE AS ACCURATE AS POSSIBLE. If there is a wide variance of results in these values, this command file should be rerun. This program, "zg_init.*" (*= ".cpg", ".cpl", ".elg", or ".ell") sets the value of the Timing Loop variables LOOP_TIME, TIME_PER_TICK, MIN_JITTER_COMPENSATION, OVERALL_MIN_TIME, OVERALL_MAX_TIME, and NULL_LOOP_SIZE for each compilation option. These values are output to a text file, "zg_cpy.*". (The character "*" in a file name represents one of the compilation options.) The four compilation options are:
* op => optimize time, suppress constraint checking
* no => no-optimize time, suppress constraint checking
* ck => optimize time, enable constraint checking
* sp => optimize space, suppress constraint checking
As part of the initialization of each performance test program, the initialization file, "zg_verfy.*" (*= ".cpg", ".cpl", ".elg", or ".ell") incorporates the appropriate copy parameter file, "zg_cpy.*", and executes some code to verify that these values are consistent with the target environment. This is done by executing a null statement within a Timing Loop and confirming that the measured time, using the supplied values for the Timing Loop parameters, is consistent with the variations in the Timing Loop observed during the Pretest. If the test program is not within an acceptable tolerance of the range, it halts with an error message.
If an optimizing compiler is able to translate the timing loop into a noop, then all the non-null ACES timing measurements will be larger than they should be because the compilation system will be executing the loop overhead instructions but subtracting off the computed timing loop overhead (of zero). This will not reflect the timing loop overhead in these cases. The "zg_init" program will write an error message if the measured loop overhead is not statistically significantly greater than zero. The error message informs the user that it may be necessary to adapt the timing loop by inserting a call on an assembler routine into the timing loop. The modification to the timing loop might be as simple as adding an unconditional call on an assembler language procedure as the first statement in "zg_glob3.proc_spoil". If testing shows that this is not sufficient to make the measured timing loop significantly greater than zero, it may be necessary to make all the calls on "zg_glob3.proc_spoil" unconditional by modifying the appropriate IF statements in "zg_init", "zg_stop0", and "zg_verfy". It will be necessary to go back and rerun the "zg_init" programs after these changes are made.
The ACES test suite and tools do not use all the functions and features of the NUMWG specifications. Unused functions are not provided in the portable package zm_genma (GEN_MATH). The NUMWG functions which are not required are:
* The hyperbolic (and inverse hyperbolic) functions.
* The trigonometric (and inverse) functions with a cycle parameter.
* The logarithm function with a base parameter.
* The COT function.
If the zm_genma package is instantiated with a constrained type, it will not operate as a NUMWG conformant would - it may raise an error when any intermediate variable in a computation is out-of-range, rather than only when initial or final values are out-of-range.
The functions required by the ACES test suite are:
* "**" - The power function, raising a real number to a real exponent.
* ARCCOS - The trigonometric arc cosine.
* ARCSIN - The trigonometric arc sine.
* ARCTAN - The trigonometric arc tangent.
* COS - The trigonometric cosine.
* EXP - The exponential function.
* LOG - The natural logarithm.
* SIN - The trigonometric sine function.
* SQRT - The square root function.
* TAN - The trigonometric tangent function.
The following sections discuss details of the alternative methods of providing a math package.
Where provided, a NUMWG package should be straightforward to instantiate and efficient to use.
This alternative involves providing a package which passes through a function call by interfacing to an implementor-provided routine, mapping name changes, exception processing, and argument definitions as required. The following code shows how this would appear for the ARCTAN function on DEC Ada.
with zg_glob1; use zg_glob1;
package zm_math is
base : CONSTANT := 2.0; --Binary floating-point machine
num_digits_in_mantissa : CONSTANT INTEGER := float6'MACHINE_MANTISSA
argument error : exception ;
...
function arctan (y: float6; x: float6 := 1.0) return float6;
...
end zm_math;
with zm_math_lib;
package body zm_math is
package vms_math_lib is new zm_math_lib( float6 ) ;
...
function arctan (y: float6; x: float6 := 1.0 ) return float6 is
begin
if x = 1.0 then
return vms_math _lib.atan(y);
else
return vms_math_lib.atan2(y,x);
end if;
exception
when others => raise argument error;
end arctan; .
..
end zm_math;
For a compilation system which provides a math library but does not contain all the functions the ACES test suite requires, an ACES user might: adapt the ACES portable math library to provide the missing functions (and access the implementor library for the functions which are provided); use only the portable math library; or use the implementor library and not run the test programs which use the unsupported functions. An implementor-provided math library might not handle exception conditions in a comparable manner - rather than raise an exception for an invalid argument (as the ACES portable math library does) it might crash the program. Such behavior complicates the task of interfacing the ACES test suite to an external library and can make the zp_mt* (MATH_TEST) programs, for example, impossible to execute without source code modifications.
An ACES user can construct a version of zm_math (or zm_dblma) with interfaces to a vendor-provided math library by writing a package zm_math (or zm_dblma). This package specifies the functions, and provides bodies for each function which return the value of a call on the appropriate vendor library function. An example of such an adaptation for the DEC Ada compilation system is distributed in the files "zm_math.dec" and "zm_dblma.dec".
The ACES portable math library "zm_genma.ada" (provided as a generic package) may be too large a generic unit for some systems to handle. If so, the compilation system being evaluated might accept the package if it were "de-genericized." To do this, edit the package source as follows:
* Remove the "generic" specification and the generic formals.
* Replace the generic package name (zm_genma) with the name of the desired instantiation (zm_math for the type ZG_GLOB1.FLOAT6; zm_dblma for the type ZG_GLOB6.FLOAT9). A text editor which can replace the string "zm_genma" everywhere it occurs will accomplish this (the string "zm_genma" is NOT embedded within other names in the source file).
* Replace the generic formal parameter name ("FLOAT_TYPE") with the name of the desired type. A text editor which can replace the string "FLOAT_TYPE" everywhere it occurs will accomplish this.
Comparing the performance of such a "de-genericized" version with a system which used the provided generic version may introduce an unfavorable bias, in that the system which supported the generic version without requiring special adaptation effort may run slower than it would if it had also been adapted. This is another reason why it is important that ACES users record all the modifications made to adapt the math library.
To provide a math library for target systems which do not provide one, the ACES distributes a portable version of a generic math package which can be readily adapted to additional targets.
The math package provided is based on the book Software Manual for the Elementary Functions by William J. Cody, Jr., and William Waite, published by Prentice-Hall in 1980.
The ACES math packages zm_math and zm_dblma depend on two generic packages:
* zm_depen
This is a representation-dependent generic package which provides functions permitting the manipulation of fields of floating point numbers. It is instantiated using the declared types. zm_depen is discussed in more detail later.
* zm_genma
This is a generic package which instantiates zm_depen and contains the algorithms for the supported elementary math functions.
The ACES contains the following versions of the file zm_depen (MATH_DEPENDENT).
.por portable version
.dec DEC Ada VAX host/VAX target
The file zm_depen provides access to the following three attributes of a real number:
* INTEXP ( x ) which returns the integer representation of the exponent in the normalized representation of its floating-point number parameter. For example, INTEXP ( 3.0 ) = 2 on binary machines because 3.0 = 0.75 * ( 2**2 ).
* ADX ( x, n ) which adds N to the integer exponent in the floating-point representation of X, thus scaling X by the N-th power of the radix. For example, ADX ( 1.0, 2 ) = 4.0 on binary machines because 1.0 = 0.5 * ( 2.0**1 ) and 4.0 = 0.5 * ( 2.0**3).
* SETEXP ( x, n ) which returns the floating-point representation of a number whose mantissa is the mantissa of the floating-point number X, and whose exponent is the integer N. For example, SETEXP ( 1.0, 3 ) = 4.0 on binary machines because 1.0 = 0.5 * 2.0**1 and 4.0 = 0.5 * ( 2.0**3 ).
There are two different approaches to implementing zm_depen. The first approach uses implementation-dependent features to manipulate fields with a floating point number, either by using record representation clauses or by calls on assembler routines. The second approach to implementing these functions is not implementation-specific, but relies on the definition of floating point numbers. It uses a table of powers of two. It searches the table for the INTEXP function, does multiplies on the ADX function, and calculates the SETEXP function by scaling the input (effectively dividing by INTEXP(X)) and then multiplying by the desired exponent value.
The portable version of zm_depen is coded by using operations on the floating point values without directly manipulating the bit patterns used to represent the values. To see how this is possible, consider the three functions exported by zm_depen in turn.
* Function ADX
This function is straightforward to implement by multiplying or dividing by an appropriate power of two. It can be tolerably efficient using a precomputed array containing the powers of two.
* Function SETEXP(X,N)
Using the function INTEXP to determine the power of two of a floating point value, the result of SETEXP can be computed as
RETURN ( X / 2.0 ** INTEXP(X) ) * 2.0 ** N;
which is representation independent. This formulation is intended for clarity and would be rather slow if coded as shown. The exponentiation can be efficiently calculated using an array of the powers of two.
* Function INTEXP
The implementation of this function is the key to the proposed representation-independent implementation. It is possible to directly determine the largest power of two greater than or equal to a floating point value (which will be the value of the binary exponent of the value) by searching an array containing powers of two instead of manipulating the bits of the floating point number representation. This will not be as efficient as a direct "bit manipulation" approach, but it is independent of where exponent fields are located in floating point numbers.
The functions must be coded carefully to avoid numeric overflow or underflow.
The Ada model numbers are defined in terms of binary radix. For a target machine with a non-binary radix, the error bounds produced by using the Ada model numbers will not be as tight as they are with binary radix targets. This will be most apparent in the zp_mt* (single precision) and zp_dm* (double precision) programs which verify the accuracy of the math library. These programs use attributes MACHINE_MANTISSA, MACHINE_EMAX, and MACHINE_EMIN to obtain the properties of the target machine used to calculate tolerable error bounds. For a non-binary radix machine representation, the value of these attributes will be the smallest binary value which is consistent with the actual representation. Using this definition in the zp_mt* programs will permit (slightly) more numeric errors in the implementation of the math functions before errors are reported.
In the following sections, two separate points are discussed. The first is what types of modification may be necessary to zm_depen, and the second is how various types of errors in the implementation of zm_depen would show up in zp_dptst (DEPTEST) results.
The package zm_depen must be adapted to reflect both the characteristics of the target machine floating point hardware and the facilities which the Ada compilation system provides to manipulate bit fields in floating point variables.
The size and location of the sign, exponent, and mantissa of a floating point number are critical, as are other representation details such as the encoding of the exponent field (biased, sign magnitude, or complement number representation). This information should be extracted from the documentation on the target machine. It is often included in Appendix F of the compiler's documentation.
Once information on the floating point representation is determined, there may still be a problem in coding zm_depen. The fundamental reason is that Ada is designed to be portable and system-dependent operations are not universally supported.
It is possible for an ACES user to implement a tailored version of zm_depen by interfacing with an assembly language routine. This might produce the fastest execution speeds.
There are several approaches to adapting zm_depen to a target system.
* The cleanest approach is the use of record representation clauses to treat the fields of a floating point number as integer (sub)types. Adaptation will involve modification to reflect the target representation. Remember that different compilers for the same target hardware may choose to number the bytes in a record differently (left-to-right versus right-to-left).
* To isolate fields in a floating point value, it is necessary to sidestep normal Ada type rules. This can be done by:
+ Defining several access types which point to integer and floating point objects and arranging for them all to point to the same actual objects. That is, instantiating UNCHECKED_CONVERSION between the access types (pointers to integers and pointers to float) and as part of system setup, initializing all the pointers to the same actual location.
+ Using instantiations of UNCHECKED_CONVERSION, either between floating point types and integer types, or between scalar types and record types. This approach was used for the DEC Ada version of zm_depen.
+ Bit field extraction can then be accomplished using record representation clauses or integer arithmetic:
- Divide and MOD to extract fields (however, a divide of a negative value in a machine using two's complement integer arithmetic is not a logical shift).
- Multiply by powers of two to shift left.
- Add to OR fields (after insuring that the field in one operand is zero).
- Negation to complement bits.
All these methods have disadvantages. The first is the most straightforward, but may not compile when the sizes are different (even though the conversion between different sized objects occurs in a piece of code which will not be executed when the sizes differ). The second alternative relies on the PRAGMA SUPPRESS being honored; however, a compiler which determines at compile time that a constraint violation would occur when a statement is executed may generate a compile time error (warning) and generate code which would raise the CONSTRAINT ERROR exception at execution time. SUPPRESS grants a compiler permission to omit checking, but the LRM and RM 95 explicitly allow compilers to ignore a PRAGMA SUPPRESS.
The program zp_dptst (DEPTEST) tests the functions in zm_depen (MATH_D) with a range of arguments which will expose many of the potential errors in the implementation of these functions. The program contains a series of statements which call on the functions and compare the results returned to the correct answer. In the zp_dptst output file, the discrepancies are flagged with a string "<<< ERROR >>>" starting in column 65, making them easy to detect.
Below are listed several symptoms of errors which might be reported by zp_dptst, and some candidates for what the underlying source of the errors might be:
* If INTEXP returns a constant for all values, the function is probably not extracting the correct bit field from the number. Perhaps parameters are grossly wrong (e.g., using record representation clauses, the bit numbering is backwards) or there are byte-ordering problems (machine architectures can number bytes from the high end or the low end; and when the target orders differently than the programmer expected it will appear that the bytes have been interchanged).
* If INTEXP returns a value which is a constant power of two off, the exponent bias is probably not set correctly.
* If SETEXP or ADX modify any bits of the mantissa, the computations to adjust the exponent field are wrong. When the computed results are not some power of two different from the expected results, the exponent field is not being isolated properly. Check for byte-interchange, bit numbering and field sizes.
* If ADX returns a value which is a constant multiple of the correct value, the exponent field location may be a bit or two off.
* If the result of SETEXP is off by a constant power of two, the exponent bias may be wrong. When ADX works properly an invalid bias is a likely cause of a constant factor error.
* Negative values, of either the floating point value or of the exponent field, are given special processing. If results for negative values are wrong, the code for processing negative values needs to be reviewed.
* Optimizing compilers may do strange things with these functions. Consider the following example which occurred on one compiler during development. The compiler noticed that the SETEXP function assigned to a float (through an access type), did an UNCHECKED_CONVERSION of the access to float to another access type and performed some manipulations on that second type, and finally returned the float value pointed to by the first access type. The compiler performed flow analysis and decided that there were no modifications to the value pointed to by the first access type before it was returned and so the load of the modified result could be "optimized" away as invariant. This transformed the SETEXP function into an identity function and made it useless. The compiler vendor agreed after inspection that the UNCHECKED_CONVERSION should have made the flow optimizer aware an alias had been created which could modify the values of the object pointed by the access type, and has modified the compiler to be aware of this fact. The immediate workaround to this problem was to insert an external procedure call between the assignment to the access type variables, which worked, but increased the execution time of the functions.
If zp_dptst does not initially work correctly, and the errors observed do not fit one of the patterns described above, the first thing a user should try is to compile the package zm_depen with no optimizations. On some systems, requesting support for a debugging option is a good way to suppress optimizations. The package may work then. If it does, the ACES user may decide to: isolate the difference optimization makes, perhaps by examining the listing of the machine code generated; or refer the package to the compilation system maintainers for correction; or simply not use any optimization options on that compilation system.
These functions must work properly for zm_genma to work. It is futile to try to verify that zm_genma is correct by running the zm_mt* programs until zm_depen has been verified with zp_dptst. It is much simpler and faster to isolate and correct errors in the zm_depen function using zp_dptst than using zm_genma. It is much easier to debug a function when the expected results are checked by a test program, than when a programmer must observe that a complex function (such as LOG) sometimes returns a wrong result and isolate the problem in that function to an error in a low level function which it calls upon. Most of the problems uncovered at execution time by the zp_mt* programs while transporting zm_genma onto new compilation systems have been due to errors in functions in the zm_depen package.
The "zp_mt*.ada" and "zp_dm*.ada" programs test the accuracy of the math packages. These should be run to insure that the math libraries are performing correctly. These test programs, executed in Pretest Steps 8 and 9 might reveal accuracy flaws in a vendor library, or flaws in adapting the ACES generic math package to a new target. For the ACES test problems, it is sufficient if the math library is not grossly inaccurate - say not losing more than 10 bits of accuracy. However, ACES users should carefully examine the accuracy requirements of their applications before using a math library which is not essentially accurate to target machine precision.
The NUMWG has recommended accuracy standards for the elementary math functions in terms of permissible maximal errors over various ranges – the zp_mt* and zp_dm* programs compare the observed errors against this standard and report if it is exceeded. The NUMWG also recommends that the value of functions for some specific values (usually for zero) be precise – the zp_mt* and zp_dm* programs compare the calculated results for these values to the recommend values. These programs test various identities and special cases for the elementary math functions, and output the number of bits in error in the computation of the function.
There are several groups of tests, covering ranges of the functions, and over each range, the program computes 2,000 sample points distributed at random (usually dividing the range into 2,000 intervals and selecting a point within each interval using a uniform random distribution). On each range, the program displays both the maximum relative error and the root-mean-square error in terms of the number of bits of precision lost. The root-mean-square is the square root of the sum of the squares of all the errors. It is commonly used as a measure of the "average" error in the set of numbers.
The programs zp_mt* and zp_dm* will write an error message whenever the maximum error is larger than that recommended by the NUMWG specifications, whenever some of the specific identities that the NUMWG recommends fail, and when selected examples which should (or should not) raise an exception, based on the NUMWG specifications, do (or do not).
These programs are an adaptation of the work of Cody and Waite. The interested reader is referred to their book, cited in Section 1.2, for details.
The zp_mt* programs require the package zm_ran (RANDOM), which contains a random number generator. There are two versions of zm_ran, one using 16-bit integers (zm_ran16) and another using 32-bit integers (zm_ran32). For systems which support 32-bit integer types, zm_ran32 should be used. This package uses a linear congruence, pseudo-random number generator and should be fairly fast. For compilation systems which do not support integer types with that range, zm_ran16 must be used. That package uses a Tausworth random number generator with a shuffling technique as described in "Improving a Poor Random Number Generator," by C. Bays and S. D. Durham, ACM Transactions on Mathematical Software , Volume 2, Number 1, March 1976. zm_ran16 should be fairly portable, although it assumes that it can perform an UNCHECKED_CONVERSION between a packed boolean array of 16 elements and an integer type.
The results of the zp_mt* programs should show very few bad tests, and the loss of significant bits should not be larger than the NUMWG recommendations.
An ACES user may be presented with a choice between using a fast, implementation-provided math library on which zp_mt* programs detect errors, and a zm_genma-based version which is slower, but with smaller errors and which processes exceptions as the NUMWG specifications recommend. This is not a trivial choice. The ACES test suite and support tools do not strongly rely on the NUMWG-recommended exception processing.
The treatment of exceptions may be the only discrepancy reported by zp_mt* and zp_dm* programs in testing an adaptation of a vendor-provided, nonNUMWG math library. In this case, users may elect to use the vendor library for the applications they develop. That is, easy portability to other NUMWG-based systems may not be a concern. Such users would probably not consider testing using anything but the supplied math library. If zp_mt* and zp_dm* detect large numeric errors, an ACES user must decide, based on the expected usage of the math library, which math library to use for testing.
It is possible that some systems may fail to compile and link the Harness program, though the development team believes that this program should operate correctly on any system that is compatible with either Ada 83 or Ada 95. If another compilation system is available for the host, it is worthwhile to attempt to compile and link Harness using the other system. Note that all six of the "zg_glob*" packages must be compiled before attempting to compile the Harness files. The particular version of "zg_glob3" is not important to the Harness, so the simplest one ("zg_glob3.ell") should be used.
If the Analysis tools cannot be compiled on the system under test, then performance testing can continue. However, to perform analysis of the results, another compilation system should be used to compile them. Note that both "zg_glob1" and "zm_math" must be compiled before attempting to compile the analysis tools. (The development team has observed at least one compilation system that cannot process the first three analysis tool files because of the size of the enumeration types.)
On most Ada compilation systems, the analysis programs can be linked and the analysis tools run from the Menu program. But, in case this first option is not possible due to capacity limitations, there are two options. The second option compiles one of the tools (Condense, CA, or SSA) and a dummy version of the other two tools, and then links these programs with the Menu program. The dummy versions are in the files "za_cndum.ada", "za_cadum.ada", and "za_sadum.ada". If that option is not possible, then the third option would be to compile and link Condense, CA, SSA, and Menu independently from each other. The common data modules need to be compiled regardless of which option is necessary. See Section 9 "RUNNING THE ANALYSIS" for more information.
The ACES Pretest provides a report form for collecting results. To fill in the report form, the user can invoke a system text editor and edit an on-line version of "zp_tmplt.txt" by replacing the blanks with YES/NO or the appropriate answer. Or, a hard copy can be printed and the blanks filled in by hand. There is space provided to insert comments onto the report form. The user should note all adaptations that were made, and general comments about system operations. When the Pretest is finished, the completed report form provides a summary report for future reference.
Several test problems may require system-dependent adaptation to operate. These fail to compile if attempted without performing the adaptation on the compilation systems where adaptation is required. Some users may want to perform the adaptation before attempting the problems. Files where adaptation may be required are:
Issue Files
------------------------------------------------------------------
interrupts tk_in01_.inc .. tk_in11_.inc and tk_lf20_.inc
FORM string io_dapkg.inc
time-slicing dt_dp16_.inc .. dt_dp21_.inc
pragma interface ms_il01_.inc
asynchronous IO io_tx01_.inc and io_tx02_.inc,
tk_la01_.inc .. tk_la03_.inc
Some projects developing code for embedded targets may not have target hardware available or accessible when an evaluation is desired. If a software simulator is accessible, it may still be possible to run the ACES with some adaptation. The ACES user must determine how the simulator treats time. If programs referencing CALENDAR.CLOCK running on the simulator return the actual time-of-day, then the ACES Timing Loop will provide measurements of the execution speed of the simulator. The speed of the simulator will rarely be of much interest; the primary concern is the execution speed on the target.
Some simulators provide estimates for what the eventual target speed would be (based on information on the speed of individual instructions). These estimates of "simulated" time may be accessible to programs running on the simulator. It would be easiest for ACES use if the "simulated" times were reflected in the values returned by CALENDAR.CLOCK (then the ACES might be run without modification). If "simulated" time is accessible from some other call, the ACES must be modified to use it. Perhaps the simplest way is to use the ACES CPU time option with the function returning the "simulated" time replacing the CPU time function. Also, the CPU time cutoff code in ZG_GLOB3.TERMINATE_TIMING_LOOP and ZG_GLOB3.STOPTIME2 would need to be removed because a simulator's measured CPU time would be more than the elapsed time.
If it is possible to measure simulated time, various values in the ACES must be adapted to minimize the testing time. The values of BASIC_ITERATION_COUNT_LOWER and BASIC_ITERATION_COUNT_UPPER should be set to small values. (NOTE: the program zp_basic (Pretest Step 6) will determine these, but as distributed this test itself will probably take an excessive time to run.) MIN_ITERATION_COUNT and MAX_ITERATION_COUNT should also be set to smaller values, both because simulated times should be more reliable and to limit the execution time (values of 2 and 3 might be appropriate). It is appropriate to attempt only a subset of ACES tests to reduce the testing time on a simulator.
If it is not possible to measure simulated time, it might still be useful to run some of the ACES performance test problems, not to collect timing data but simply to observe whether failures occur. This may be particularly helpful for the implicit storage reclamation tests (although, if successful, they will take a long time to complete on a simulator).
The following observations made during ACES testing provide some ways to work around problems with deleting library units:
* One compilation system created library units for local generic instantiations. The Harness command files do delete these units if the user has asked for library deletes. Of course, on systems that do not create these extra library units, this will result in delete requests for nonexistent units.
* One compilation system would not remove a subunit from the program library when a "zc_delbo" was used, even though that system identified the type of the unit as a body when a library directory was displayed. To work around this, an ACES user could modify the calls on "zc_delbo" for subunits to a command which will delete subunits. It may also be feasible to adapt the code within "zc_delbo" to delete all the library entries with the specified name, rather than simply deleting the body.
* A compilation system might create an explicit library unit for a specification for a library subprogram which was compiled with an implicit specification (that is, there was only one file which contained the subprogram definition and there was NOT a file compiled containing only the subprogram specification). It would be possible on such a system to adapt the "zc_delbo" command file to also delete specifications.
* One compilation system always created a library body entry, even for packages containing only a specification part. The sample command files assume this does not occur and will leave the "unexpected" body units in the program library. This might be worked around by making all the zc_del* files delete both specification and body units.
* One compilation system was very sensitive to the order of deleting units from the program library and would not delete a unit which has any other units in the library dependent on it (for example, it was improper to remove a package specification before removing the package body). The command files generated by the Harness try to be consistent with this restriction.
* There was one system which did not provide the capability to delete units from the program library. On this system it was necessary to delete the entire program library when it filled up and create a new one.
One adaptation alternative to the zc_del* files is to use library level deletes or sublibraries. Rather than deleting units as they become obsolete, the ACES users could periodically delete and create the program library and re-enter the basic units compiled by Pretest Steps 1, 3, and 5. It may be possible to avoid recompiling either by setting up a sublibrary structure (where the basic units are in a shared library and the sublibrary for the working units is periodically deleted and recreated) or by using facilities to copy intermediate forms of the basic units from a saved library into the newly created one.
In evaluating a compilation system, it is often useful to explore the performance of some implementation-specific extensions. Portability is often not a strict requirement of a project. There are existing ACES tests which might be easily modified to test for some areas where implementation-dependent features are often provided. These include:
1. Alternative interrupt mechanisms
Many implementations for embedded targets provide alternatives to the LRM-specified mechanism to interface interrupts to a task entry. One common approach is to specify a procedure which is to be called when an interrupt occurs, with various constraints placed on the procedure (typically it must be a library procedure which cannot contain any tasking constructions or DELAY statements). Variants can be much faster than the general LRM tasking construction, and constraints placed on it may be quite acceptable. The tests in the interrupt subgroup of the tasking group may be used as models for a set of interrupt tests using implementation-dependent mechanisms for communicating with hardware interrupts.
2. Linkages to other languages
The ACES provides one test for linkage to a procedure without parameters that is written in assembly language. Some systems provide linkages to other languages, and some projects may be very interested in using such linkages. The test ms_il_interface_lang_assem_01 may be used as a model for these tests.
3. Multiple program libraries
Some compilation systems provide for multiple program libraries. The compile and link time for programs exploiting this capability can be rather different from the times for problems using only one library. The tests in the Systematic Compile Speed group can be used as models for additional tests in this area.
4. Operating system services
Most Ada compilation systems provide for access to operating system services. Projects targeted to a specific operating system may want to test the performance of specific OS features. While not necessarily portable to all other operating systems, the performance of some of the features may be critical to some projects. Examples of features which the user may want to test include: windowing system calls; OS process spawning; transaction processing systems interfaces; and database system interfaces.
While there are not individual ACES tests which could serve as specific models for exercising these services, the general approach of the ACES test suite may be appropriate.
Users who wish to add new tests the ACES and include them in the analysis produced by Comparative Analysis should use the User-Defined (ud) group, as discussed in Section 5.5. It is also possible to add tests to the existing Harness and Analysis software and data files. See Section 9.1.6 "Adding Subgroups, Tests, and/or Main Programs" and Section 6.13 "ADDING NEW TESTS TO THE HARNESS".
Distribution directory "aces/tests/ud" (the User-Defined group) contains one file, "ud_tmplt.__". This file may be used as a template for up to 20 user-defined benchmarks (up to 10 for an Ada 83 implementation). The template file contains instructions for its use.
The Harness and Analysis Menu programs recognize two subgroups of the User-Defined group (the Ada 83 subgroup and the Ada 95 subgroup). When testing an Ada 83 implementation, only the Ada 83 subgroup is available. Ada 95 implementations may use both subgroups. Each subgroup has 10 predefined performance test names that are recognized by Harness and the Analysis tools.
WARNING: If the Harness user selects User-Defined test names that have not been created (by adapting the template file), then executing the generated script will produce errors (due to missing files).
This section contains some guidance on how the Harness might be used. It is essential that the System Name file ("zh_cosys.txt") be examined before beginning. It is highly recommended that users who will be evaluating more than one Ada compilation system adapt this file (at least by changing the system name) for each system to be evaluated. It is also recommended that the database file names (produced for each group) also be adapted for each system.
An algorithm for using the Harness is given below in Figure 6-1. This is certainly not the only way the Harness might be used, but it has proven helpful in previous work.
LOOP
select a group and its tests
LOOP
Build a command file for selected tests
submit it (this step must be done outside the Harness)
Update status
use Choose_by_status to select tests with non valid times
examine these test results in the log file more closely
EXIT WHEN out of time OR ELSE have accounted for all tests in this group
END LOOP
END LOOP
The Harness always begins with the Harness test selection methods screen. This allows the user to choose tests by group, subgroup, or test names, as provided in Harness version 1.0 through 2.0, or the user may select tests by performance topic. To select tests by performance topic, see Section 6.14 "SELECTING TESTS BY PERFORMANCE TOPICS". If the user chooses to select tools by group, subgroup, or test names, then the Harness displays the Groups display. (If the user has identified the system as supporting Ada 95, then all 21 groups are listed; otherwise only 18 groups whose tests are Ada 83-compatible will be available.) The first time the Harness is run this display will always show nothing selected and all tests in the No_data category. A group can be selected by typing its number. All the tests in this group can then be selected by entering "//all" (see Section 6.4 "BASIC SELECTION" and Section 6.5 "ADVANCED SELECTION" for a detailed discussion of how selections are made). After entering this, the display will show that the number of tests selected is equal to the number of tests in the group.
If "b" for Build is then typed, a screen like the one in Figure 6-22, Build Command - One Group - Compile Speed ON, will be displayed. See Section 6.10.7 "Build_com (B)", for more details on possible choices.
After Building the command files, the next step would be to run them (probably, but not necessarily in batch mode). Gather the results in a log file. Then, the Harness command, Update_status_from_log (U), can be used to read the execution time results from this log file into a Harness database file (names are specified in the system name file ("zh_cosys.txt"). Then, the totals in the Show_Groups display (see Figure 6-15) can be observed and the results for individual tests in the Show_Tests display (see Figure 6-17) can be examined. The Choose_by_status command is helpful if the user wishes to examine only the tests that do not have valid execution times. For example, the tests in the status range 4..14 (err_unreliable_time .. err_no_data) might be chosen. Or, if the user cares most about Ada failures, they might choose tests in the status range 8..14 (err_packaging .. err_no_data), since status codes 4..7 represent difficulties in gathering times, but not difficulties in compiling, linking, or running the test programs. The official status list is shown in Figure 6-19, Choose Tests - Sample Screen.
In using the Choose_by_status command you do need to remember that you lose the current selection information. The command works by selecting a subset of the already selected tests in the current group. If the current selections are important to you, you can save them by using the SAve_selection command (see Section 6.9.8 "SAve_selection (SA)"). Every time the Harness is exited, you have the option to save the current selection information in a file which will be automatically read when the Harness is restarted. However, you may also save selection information at any time, and you may read this selection information back in at any time (see Section 6.9.7 "Read_Selection (R)").
The Harness also saves execution time status information which has been read into a Harness database. This information may also be later used by the analysis programs: Single System Analysis (SSA) and Comparative Analysis (CA). The Harness does not gather Compile or Link times. For this purpose you must use Condense. The Harness only monitors and reports on execution time results.
There are 25 tests in the Systematic_Compile_Speed group for which execution time results are not produced. The tests are labeled in the Show_Tests display as library command or compile only. Since there are no execution results associated with these problems, the Harness will never display any results for them since the Harness only recognizes execution results, and not compilation times. Furthermore, you cannot select based on these categories. This is an inconsistency in the Harness. One helpful alternative would be to use the SET_status (SET) command to set the status of these tests (or any tests for which you do not care to gather data) to Not_applicable (NA).
The system name file ("zh_cosys.txt") is the interface between the Harness and the user file system for both input and output files. The subsections below explain each part of this file. Here, as with the analysis tools, this file, which has a name hard coded into the program, allows the user to easily change almost every other file name.
All output file names are concatenated with the output_path; input files must be in the current directory, or include the path in their name.
The database files (>exe_condensed) are input and output files; they are treated as input files in that they do NOT use the output_path. See Section 6.2.3 "Execution Time Database File(s)".
Many entries are REQUIRED, but all except the system_name have default values which are assigned in the program, and overwritten from the system name file if they are present.
The Harness needs the same Weight (structure) information that is needed by the analysis programs. However, because the Harness operates with data arrays that are limited in size (and therefore only contain one group at a time), this information is packaged in a separate file for each group. The file names are given in the system name file ("zh_cosys.txt") with entries that begin with ">weight_file" as in Figure 6-2. A single structure file could be used by the Harness, but this set of entries, one for each group, is not optional. If a single file were used, that file name would need to appear on the line for each group.
>weight_file - (ap) application := zh_ap.txt
>weight_file - (ar) arithmetic := zh_ar.txt
>weight_file - (cl) classical := zh_cl.txt
. . .
>weight_file - (sy) systematic_compile_speed := zh_sy.txt
>weight_file - (tk) tasking := zh_tk.txt
>weight_file - (ud) user_defined := zh_ud.txt
This entry is optional and may not be used by many Harness users. When the Harness starts up, if this log file is present in the current directory, then it is automatically read and the information is used to update the appropriate Harness database(s). The Update_status_from_log command is another option which may be used to interactively perform the same function.
>exe_log := test.log
Experience with the Harness has prompted the following recommendation: leave this field blank in the system name file (or make sure that no file with this name is present) and use the Update_status command to read new log files as they become available.
The execution time database(s) can be created by Harness or by Condense. The Harness will create a separate execution time database file for each group, while Condense creates one database file for execution time data and one for compile/link data. The Harness can use either a single file or one for each group, but the appropriate name(s) must be given for each group in the system name file ("zh_cosys.txt"). See Figure 6-3 for an example. The Harness will be slower if there is only one database file. Also, it will not be possible to set statuses and save this information. For these reasons, it is recommended that the Harness use the separate database files that it creates. Later, when the analysis tools are used, there is an option for Condense to merge these Harness database files so that the results can be used by the analysis tools.
Note: these file names could all be identical if Condense creates the database file.
>exe_condensed - (ap) application := test_ap.dbs
>exe_condensed - (ar) arithmetic := test_ar.dbs
>exe_condensed - (cl) classical := test_cl.dbs
. . .
>exe_condensed - (su) subprograms := test_su.dbs
>exe_condensed - (sy) systematic_compile_speed := test_sy.dbs
>exe_condensed - (tk) tasking := test_tk.dbs
>exe_condensed - (ud) user_defined := test_ud.dbs
The following files are REQUIRED by the Build command if a Build is requested for the Systematic_Compile_Speed group.
>sy_template_main := zh_sy.tpl
>sy_template_1000 := zh_sy000.tpl
>sy_template_cu := zh_sy_cu.tpl
The following files are REQUIRED by the Build command if Ada library deletions are requested.
>lib_delete - (ap) application := zh_ap.lib
>lib_delete - (ar) arithmetic := zh_ar.lib
>lib_delete - (cl) classical := zh_cl.lib
.
.
.
>lib_delete - (sr) storage_reclamation := zh_sr.lib
>lib_delete - (su) subprograms := zh_su.lib
--lib_delete - (sy) systematic_compile_speed := --------- NOT NEEDED
>lib_delete - (tk) tasking := zh_tk.lib
>lib_delete - (ud) user_defined := zh_ud.lib
The Harness writes the summary group status information whenever the Quit command is issued. During startup, this file, if present, is read and used to initialize the group totals which appear in the Show_Groups display. In this way, the Harness does not have to reread all of the database files to produce these totals. Whenever a group is selected, its database file is read and these status totals are brought up-to-date. The names for these files are given in the system_name file. The user does not ordinarily need to know what these file names are, since the process is transparent.
>harness_summary := <test>.sum
The default name is "test.sum". If present and corrupted, harness will not start. Delete the file and restart Harness. It will create a new file.
Whenever the Harness terminates, the user is asked if a selection summary file should be written. This file saves the current selection information on all groups, subgroups, and tests. If present, it will automatically be read when the Harness is restarted, and all selection information will be restored. The name for that file is given in the system_name file. The user does not ordinarily need to know what that file name is, since the process is transparent.
>harness_selection := <test>.sel
The default name is "test.sel". If present and corrupted, harness will not save any results. Delete the file and restart Harness. It will create a new file.
Build_com (B) - group abbreviation & script_suffix
Write_Groups (WG) - "groups.txt"
Write_Subgroups (WS) - group abbreviation & ".sub"
Write_Tests (WT) - group abbreviation & ".tst"
Write_Chosen (WC) - group abbreviation & ".cho"
Multiple commands cannot be entered on a line. However, multiple selections can be made without requiring that each selection command be entered on a separate line. Other Harness commands may be either alpha commands or numeric commands. Alpha commands are entered by typing the command or its abbreviation. Other numeric commands are entered by typing their number (which is associated with a description in the current menu) and (sometimes) an optional parameter (which is often a file name). Some of these commands are "toggle" commands which change a value between two choices.
An attempt was made to produce an interface that limits the user as little as possible. Any input should be accepted. Any command that is "possible" should be accepted at any point. Others will cause an error message to be generated, but will have no other effect. Some of the commands in the menus for operations like Write, Build, and Update, look like selection commands, but have other uses (see Section 6.8). In addition, only one toggle command can be entered per line and only one command that can take a parameter can be entered per line.
Note: If a user enters a command incorrectly, an error message will be displayed. This may cause the display to scroll past the point where it can be read. The display can be refreshed by typing the command for this menu. For example, if the user is looking at the groups menu, type Show_Groups (SG).
When groups, subgroups, tests, or statuses are displayed, the user may make a selection by typing the appropriate number (listed to the left). Preceding the number by a minus (-) sign will deselect. Ranges of numbers may also be entered in Ada style: "3..7". Ranges preceded by a minus sign will deselect: "- 3 .. 7" will undo "3..7". "all" or "+all" selects all. "-all" deselects all ("*" is a synonym for "all").
Only one group can be the current group, which is the group operated on by all but the Build command. The Build command operates on all currently selected groups. Selecting or deselecting a group does not select or deselect the subgroups or the tests in the group.
All subgroups can be selected at once. Selecting or deselecting a subgroup does not select or deselect tests in the subgroup.
All tests can be selected at once. You may select individual tests or ranges of tests. Selecting or deselecting tests does not change the selection status of the group or the subgroup of which they are a part.
All statuses can be selected at once. You may select individual statuses or ranges of statuses. Status selection is used to reduce the set of already selected tests to the subset consisting of those that have a given status (or have a status in a given range of statuses).
Only one status may be selected for this purpose. After typing the "Do" command, the existing status for all selected tests will be changed to the selected status.
This section discusses the effect of making group, subgroup, and test selections. The essential point to remember is that all of these selections are independent: selecting one has no effect on the selection status of any other part of the structure.
The last group selected is always the current group. This is important for assessing the behavior of other commands. ALMOST ALL ACTION REFERS TO THE CURRENT GROUP. Commands dealing with subgroups and tests, such as Write_Subgroups or Show_Tests assume the current group. If no current group has been specified, these commands cannot be invoked.
The Advanced Selection commands, discussed below, can sometimes operate independently of the current group. However, they operate only on the current group when no group is explicitly denoted in the command.
The only command in which the current group plays no special role is the Build command. This command operates on all selected groups (and within these groups, on all selected tests). In no other command does it matter which groups are selected, EXCEPT for the current group.
Subgroup selections are the least important selections. The reason subgroups are included is to ease the task of dealing with all of the performance tests. There is a current subgroup which is similar to the current group in that it is the last subgroup selected. However, it is never necessary to select any subgroup.
The Advanced Selection commands show how the user may use subgroups to facilitate the selection of tests in several ways. For example, users may easily select all of the tests in a subgroup, or the first five tests in every subgroup.
The performance tests are the core of the ACES. Ultimately, the goal of the Harness is to make it easy for the user to monitor the status of each test, and to select tests of interest for further action. The group and subgroup organization is designed to further that goal.
The Advanced Selection commands allow the user to select any test, or any combination of tests at any time, from the group menu, or any subgroup menu, or any test menu. These commands may be more powerful