VERSION-1

NATIONAL COMPUTER SECURITY CENTER

A GUIDE TO

UNDERSTANDING

SECURITY TESTING

AND

TEST DOCUMENTATION

IN

TRUSTED SYSTEMS

July 1993

Approved for Public Release:

Distribution Unlimited.

NCSC-TG-023
Library No. S-232.561
Version-1

FOREWORD

The National Computer Security Center is issuing A Guide to Understanding Security Testing
and Test Documentation in Trusted Systems as part of the "Rainbow Series" of documents our
Technical Guidelines Program produces. In the Rainbow Series, we discuss in detail the features
of the Department of Defense Trusted Computer System Evaluation Criteria (DoD 5200.28-STD)
and provide guidance for meeting each requirement. The National Computer Security Center,
through its Trusted Product Evaluation Program, evaluates the security features of commercially
produced computer systems. Together, these programs ensure that users are capable of protecting
their important data with trusted computer systems.

The specific guidelines in this document provide a set of good practices related to security testing
and the development of test documentation. This technical guideline has been written to help the
vendor and evaluator community understand what deliverables are required for test documentation,
as well as the level of detail required of security testing at all classes in the Trusted Computer System
Evaluation Criteria.

As the Director, National Computer Security Center, Invite your suggestions for revision to this
technical guideline. We plan to review this document as the need arises.

National Computer Security Center

Attention: Chief, Standard, Criteria and Guidelines Division

9800 Savage Road

Fort George G. Meade, MD 20755-6000

Patrick R. Gallagher, Jr. January, 1994

Director

National Computer Security Center

ACKNOWLEDGMENTS

Special recognition and acknowledgment for his contributions to this document are extended to
Virgil D. Gligor, University of Maryland, as primary author of this document.

Special thanks are extended to those who enthusiastically gave of their time and technical
expertise in reviewing this guideline and providing valuable comments and suggestions. The
assistance of C. Sekar Chandersekaran, IBM and Charles Bonneau, Honeywell Federal Systems,
in the preparation of the examples presented in this guideline is gratefully acknowledged.

Special recognition is extended to MAJ James P. Gordon, U.S. Army, and Leon Neufeld as
National Computer Security Center project managers for this guideline.

TABLE OF CONTENTS

FOREWORD i

ACKNOWLEDGMENTS iii

l. INTRODUCTION 1

1.1 PURPOSE 1

1.2 SCOPE 1

1.3 CONTROL OBJECTIVES 2

2. SECURITY TESTING OVERVIEW 3

2.1 OBJECTIVES 3

2.2 PURPOSE 3

2.3 PROCESS 4

2.3.1 System Analysis 4

2.3.2 Functional Testing 4

2.3.3 Security Testing 5

2.4 SUPPORTING DOCUMENTATION 5

2.5 TEST TEAM COMPOSITION 6

2.6 TEST SITE 17

3. SECURITY TESTING - APPROACHES, DOCUMENTATION, AND
EXAMPLES 8

3.1 TESTING PHILOSOPHY 8

3.2 TEST AUTOMATION 9

3.3 TESTING APPROACHES 11

3.3.1 Monolithic (Black-Box) Testing 11

3.3.2 Functional-Synthesis (White-Box) Testing 13

3.3.3 Gray-Box Testing 25

3.4 RELATIONSHIP WITH THE TCSEC SECURITY TESTING
REQUIREMENTS 18

3.5 SECURITY TEST DOCUMENTATION 21

3.5.1 Overview 21

3.5.2 Test Plan 22

3.5.2.1 Test Conditions 22

3.5.2.2 Test Data 24

3.5.2.3 Coverage Analysis 25

3.5.3 Test Procedures 27

3.5.4 Test Programs 27

3.5.5 Test Log 28

3.5.6 Test Report 28

3.6 SECURITY TESTING OF PROCESSORS' HARDWARE/FIRMWARE
PROTECTION MECHANISMS 28

3.6.1 The Need for Hardware/Firmware Security Testing 29

3.6.2 Explicit TCSEC Requirements for Hardware Security Testing 30

3.6.3 Hardware Security Testing vs. System Integrity Testing 31

3.6.4 Goals, Philosophy, and Approaches to Hardware Security Testing 31

3.6.5 Test Conditions, Data, and Coverage Analysis for Hardware Security
Testing 32

3.6.5.1 Test Conditions for Isolation and Noncircumventability Testing 32

3.6.5.2 Text Conditions for Policy-Relevant Processor Instructions 33

3.6.5.3 Tests Conditions for Generic Security Flaws 33

3.6.6 Relationship between Hardware/Firmware Security Testing and the TCSEC
Requirements 34

3.7 TEST PLAN EXAMPLES 36

3.7.1 Example of a Test Plan for "Access" 37

3.7.1.1 Test Conditions for Mandatory Access Control of "Access" 38

3.7.1.2 Test Data for MAC Tests 38

3.7.1.3 Coverage Analysis 39

3.7.2 Example of a Test Plan for "Open" 43

3.7.2.1 Test Conditions for "Open" 43

3.7.2.2 Test Data for the Access Graph Dependency Condition 44

3.7.2.3 Coverage Analysis 46

3.7.3 Examples of a Test Plan for "Read" 46

3.7.3.1 Test Conditions for "Read" 47

3.7.3.2 Test Data for the Access-Check Dependency Condition 47

3.7.3.3 Coverage Analysis 51

3.7.4 Examples of Kernel Isolation Test Plans 51

3.7.4.1 Test Conditions 51

3.7.4.2 Test Data 51

3.7.4.3 Coverage Analysis 53

3.7.5 Examples of Reduction of Cyclic Test Dependencies 54

3.7.6 Example of Test Plans for Hardware/Firmware Security Testing 57

3.7.6.1 Test Conditions for the Ring Crossing Mechanism 58

3.7.6.2 Test Data 58

3.7.6.3 Coverage Analysis 60

3.7.7 Relationship with the TCSEC Requirements 62

4. COVERT CHANNEL TESTING 66

4.1 COVERT CHANNEL TEST PLANS 66

4.2 AN EXAMPLE OF A COVERT CHANNEL TEST PLAN 67

4.2.1 Test Plan for the Upgraded Directory Channel 67

4.2.1.1 Test Condition 68

4.2.1.2 Test Data 68

4.2.1.3 Coverage Analysis 70

4.2.2 Test Programs 70

4.2.3 Test Results 70

4.3 RELATIONSHIP WITH THE TCSEC REQUIREMENTS 70

5. DOCUMENTATION OF SPECIFICATION-TO-CODE CORRESPONDENCE 72

APPENDIX 73

1 Specification-to-Code Correspondence 73

2 Informal Methods for Specification-to-Code Correspondence 74

3 An Example of Specification-to-Code Correspondence 76

GLOSSARY 83

REFERENCES 90

1. INTRODUCTION

The National Computer Security Center (NCSC) encourages the widespread availability of
trusted computer systems. In support of this goal the Department of Defense Trusted Computer
System Evaluation Criteria (TCSEC) was created as a metric against which computer systems could
be evaluated. The NCSC published the TCSEC on 15 August 1983 as CSC-STD-001-83. In
December 1985, the Department of Defense (DoD) adopted it, with a few changes, as a DoD
Standard, DoD 5200.28-STD. [13] DoD Directive 5200.28, "Security Requirements for Automatic
Data Processing (ADP) Systems," requires that the TCSEC be used throughout the DoD. The NCSC
uses the TCSEC as a standard for evaluating the effectiveness of security controls built into ADP
systems. The TCSEC is divided into four divisions: D, C, B, and A. These divisions are ordered in
a hierarchical manner with the highest division (A) being reserved for systems providing the best
available level of assurance. Within divisions C and B there are a number of subdivisions known
as classes. In turn, these classes are also ordered in a hierarchical manner to represent different
levels of security.

1.1 PURPOSE

Security testing is a requirement for TCSEC classes C1 though A1. This testing determines that
security features for a system are implemented as designed and that they are adequate for the
specified level of trust. The TCSEC also requires test documentation to support the security testing
of the security features of a system. The TCSEC evaluation process includes security testing and
evaluation of test documentation of a system by an NCSC evaluation team. A Guide to
Understanding Security Testing and Test Documentation for Trusted Systems will assist the
operating system developers and vendors in the development of computer security testing and testing
procedures. This guideline gives system developers and vendors suggestions and recommendations
on how to develop testing and testing documentation that will be found acceptable by an NCSC
Evaluation Team.

1.2 SCOPE

TCSEC classes C1 through A1 assurance is gained through security testing and the accompanying
test documentation of the ADP system. Security testing and test documentation ensures that the
security features of the system are implemented as designed and are adequate for an application
environment. This guideline discusses the development of security testing and test documentation
for system developers and vendors to prepare them for the evaluation process by the NCSC. This
guideline addresses, in detail, various test methods and their applicability to security and
accountability policy testing. The Trusted Computing Base (TCB) isolation, noncircumventability
testing, processor testing, and covert channel testing methods are examples.

This document provides an in-depth guide to security testing. This includes the definitions,
writing and documentation of the test plans for security and a brief discussion of the mapping
between the formal top-level specification (FTLS) of a TCB and the TCB implementation
specifications. This document also provides a standard format for test plans and test result
presentation. Extensive documentation of security testing and specification-to-code correspondence
arise both during a system evaluation and, more significantly, during a system life cycle. This
guideline addresses evaluation testing, not life-cycle testing. This document complements the
security testing guideline that appears in Section 10 of the TCSEC.

The scope and approach of this document is to assist the vendor in security testing and in particular
functional testing. The vendor is responsible for functional testing, not penetration testing. If
necessary, penetration testing is conducted by an NCSC evaluation team. The team collectively
identifies penetration vulnerabilities of a system and rates them relative to ease of attack and
difficulty of developing a hierarchy penetration scenario. Penetration testing is then conducted
according to this hierarchy, with the most critical and easily executed attacks attempted first [17].

This guideline emphasizes the testing of systems to meet the requirements of the TCSEC. A Guide
to Understanding Security Testing and Test Documentation for Trusted Systems does not address
the testing of networks, subsystems, or new versions of evaluated computer system products. It only
addresses the requirements of the TCSEC.

Information in this guideline derived from the requirements of the TCSEC is prefaced by the
word "shall." Recommendations that are derived from commonly accepted good practices are
prefaced by the word "should." The guidance contained herein is intended to be used when
conducting and documenting security functional testing of an operating system. The
recommendations in this document are not to be construed as supplementary requirements to the
TCSEC. The TCSEC is the only metric against which systems are to be evaluated.

Throughout this guideline there are examples, illustrations, or citations of test plan formats that
have been used in commercial product development. The use of these examples, illustrations, and
citations is not meant to imply that they contain the only acceptable test plan formats. The selection
of these examples is based solely on their availability in computer security literature. Examples in
this document are not to be construed as the only implementations that will satisfy the TCSEC
requirements. The examples are suggestions of appropriate implementations.

1.3 CONTROL OBJECTIVES

The TCSEC and DoD 5200.28-M [14] provide the control objectives for security testing and
documentation. Specifically these documents state the following:

"Component's Designated Approving Authorities, or their designees for this purpose . . .
will assure:. . .

"4. Maintenance of documentation on operating systems (O/S) and all modifications
thereto, and its retention for a sufficient period of time to enable tracing of security-related
defects to their point of origin or inclusion in the system.

"5. Supervision, monitoring, and testing, as appropriate, of changes in an approved ADP
System that could affect the security features of the system, so that a secure system is
maintained.

"6. Proper disposition and correction of security deficiencies in all approved ADP
Systems, and the effective use and disposition of system housekeeping or audit records,
records of security violations or security-related system malfunctions, and records of tests
of the security features of an ADP System.

"7. Conduct of competent system Security Testing and Evaluation (ST&E), timely review
of system ST&E reports, and correction of deficiencies needed to support conditional or
final approval or disapproval of an ADP system for the processing of classified
information.

"8. Establishment, where appropriate, of a central ST&E coordination point for the
maintenance of records of selected techniques, procedures, standards, and tests used in
testing and evaluation of security features of ADP systems which may be suitable for
validation and use by other Department of Defense components."

Section 5 of the TCSEC gives the following as the Assurance Control Objective:

"The third basic control objective is concerned with guaranteeing or providing confidence
that the security policy has been implemented correctly and that the protection critical
elements of the system do, indeed, accurately mediate and enforce the intent of that policy.
By extension, assurance must include a guarantee that the trusted portion of the system
works only as intended. To accomplish these objectives, two types of assurance are
needed. They are life-cycle assurance and operational assurance.

"Life-cycle assurance refers to steps taken by an organization to ensure that the system
is designed, developed, and maintained using formalized and rigorous controls and
standards. Computer systems that process and store sensitive or classified information
depend on the hardware and software to protect that information. It follows that the
hardware and software themselves must be protected against unauthorized changes that
could cause protection mechanisms to malfunction or be bypassed completely. For this
reason, trusted computer systems must be carefully evaluated and tested during the design
and development phases and reevaluated whenever changes are made that could affect
the integrity of the protection mechanisms. Only in this way can confidence be provided
that the hardware and software interpretation of the security policy is maintained
accurately and without distortion." [13]

2. SECURITY TESTING OVERVIEW

This section provides the objectives, purpose, and a brief overview of vendor and NCSC security
testing. Test team composition, test site location, testing process, and system documentation are
also discussed.

2.1 OBJECTIVES

The objectives of security testing are to uncover all design and implementation flaws that enable
a user external to the TCB to violate security and accountability policy, isolation, and
noncircumventability.

2.2 PURPOSE

Security testing involves determining (1) a system security mechanism adequacy for
completeness and correctness and (2) the degree of consistency between system documentation and
actual implementation. This is accomplished through a variety of assurance methods such as analysis
of system design documentation, inspection of test documentation, and independent execution of
functional testing and penetration testing.

2.3 PROCESS

A qualified NCSC team of experts is responsible for independently evaluating commercial
products to determine if they satisfy TCSEC requirements. The NCSC is also responsible for
maintaining a listing of evaluated products on the NCSC Evaluated Products List (EPL). To
accomplish this mission, the NCSC Trusted Product Evaluation Program has been established to
assist vendors in developing, testing, and evaluating trusted products for the EPL. Security testing
is an integral part of the evaluation process as described in the Trusted Product Evaluations-A
Guide For Vendors. [18]

2.3.1 System Analysis

System analysis is used by the NCSC evaluation team to obtain a complete and in-depth
understanding of the security mechanisms and operations of a vendor's product prior to conducting
security testing. A vendor makes available to an NCSC team any information and training to support
the NCSC team members in their understanding of the system to be tested. The NCSC team will
become intimately familiar with a vendor's system under evaluation and will analyze the product
design and implementation, relative to the TCSEC.

System candidates for TCSEC ratings B2 through A1 are subject to verification and covert channel
analyses. Evaluation of these systems begins with the selection of a test configuration, evaluation
of vendor security testing documentation, and preparation of an NCSC functional test plan.

2.3.2 Functional Testing

Initial functional testing is conducted by the vendor and results are presented to the NCSC team.
The vendor should conduct extensive functional testing of its product during development, field
testing, or both. Vendor testing should be conducted by procedures defined in a test plan. Significant
events during testing should be placed in a test log. As testing proceeds sequentially through each
test case, the vendor team should identify flaws and deficiencies that will need to be corrected.
When a hardware or software change is made, the test procedure that uncovered the problem should
then be repeated to validate that the problem has been corrected. Care should be taken to verify that
the change does not affect any previously tested procedure. These procedures also should be repeated
when there is concern that flaws or deficiencies exist. When the vendor team has corrected all
functional problems and the team has analyzed and retested all corrections, a test report should be
written and made a part of the report for review by the NCSC test team prior to NCSC security testing.

The NCSC team is responsible for testing vendor test plans and reviewing vendor test
documentation. The NCSC team will review the vendor's functional test plan to ensure it sufficiently
covers each identified security mechanism and explanation in sufficient depth to provide reasonable
assurance that the security features are implemented as designed and are adequate for an application
environment. The NCSC team conducts its own functional testing and, if appropriate, penetration
testing after a vendor's functional testing has been completed.

A vendor's product must be free of design and implementation changes, and the documentation
to support security testing must be completed before NCSC team functional testing. Functional
security testing is conducted on C1 through A1 class systems and penetration testing on B2, B3,
and A1 class systems. The NCSC team may choose to repeat any of the functional tests performed
by the vendor and/or execute its own functional test. During testing by the NCSC team, the team
informs the vendor of any test problems and provides the vendor with an opportunity to correct
implementation flaws. If the system satisfies the functional test requirements, B2 and above
candidates undergo penetration testing. During penetration testing the NCSC team collectively
identifies penetration vulnerabilities in the system and rates them relative to ease of attack and
difficulty in developing a penetration hierarchy. Penetration testing is then conducted according to
this hierarchy with the most critical and most easily executed attacks attempted first [17]. The vendor
is given limited opportunity to correct any problems identified [17]. When opportunity to correct
implementation flaws has been provided and corrections have been retested, the NCSC team
documents the test results. The test results are input which support a final rating, the publication of
the Final Report and the EPL entry.

2.3.3 Security Testing

Security testing is primarily the responsibility of the NCSC evaluation team. It is important to
note, however, that vendors shall perform security testing on a product to be evaluated using NCSC
test methods and procedures. The reason for vendor security testing is two-fold: First, any TCB
changes required as a result of design analysis or formal evaluation by the NCSC team will require
that the vendor (and subsequently the evaluation team) retest the TCB to ensure that its security
properties are unaffected and the required changes fixed the test problems. Second, any new system
release that affects the TCB must undergo either a reevaluation by the NCSC or a rating-maintenance
evaluation by the vendor itself. If a rating maintenance is required, which is expected to be the case
for the preponderant number of TCB changes, the security testing responsibility, including all the
documentation evidence, becomes a vendor's responsibility-not just that of the NCSC evaluation
team.

Furthermore, it is important to note that the system configuration provided to the evaluation team
for security testing should be the same as that used by the vendor itself. This ensures that consistent
test results are obtained. It also allows the evaluation team to examine the vendor test suite and to
focus on areas deemed to be insufficiently tested. Identifying these areas will help speed the security
testing of a product significantly. (An important implication of reusing the vendor's test suite is that
security testing should yield repeatable results.)

When the evaluation team completes the security testing, the test results are shown to the vendor.
If any TCB changes are required, the vendor shall correct or remove those flaws before TCB retesting
by the NCSC team is performed.

2.4 SUPPORTING DOCUMENTATION

Vendor system documentation requirements will vary, and depending on the TCSEC class a
candidate system will be evaluated for, it can consist of the following:

Security Features User's Guide. It describes the protection mechanisms provided by
the TCB, guidelines on their use, and how they interact with one another. This may be
used to identify the protection mechanisms that need to be covered by test procedures
and test cases.

Trusted Facility Manual. It describes the operation and administration of security
features of the system and presents cautions about functions and privileges that should
be controlled when running a secure facility. This may identify additional functions that
need to be tested.

Design Documentation. It describes the philosophy of protection, TCB interfaces,
security policy model, system architecture, TCB protection mechanisms, top level
specifications, verification plan, hardware and software architecture, system configuration
and administration, system programming guidelines, system library routines,
programming languages, and other topics.

Covert Channel Analysis Documentation. It describes the determination and maximum
bandwidth of each identified channel.

System Integrity Documentation. It describes the hardware and software features used
to validate periodically the correct operation of the on-site hardware and firmware
elements of the TCB.

Trusted Recovery Documentation. It describes procedures and mechanisms assuring
that after an ADP system failure or other discontinuity, recovery is obtained without a
protection compromise. Information describing procedures and mechanisms may also be
found in the Trusted Facility Manual.

Test Documentation. It describes the test plan, test logs, test reports, test procedures,
and test results and shows how the security mechanisms were functionally tested, covert
channel bandwidth, and mapping between the FTLS and the TCB source code. Test
documentation is used to document plans, tests, and results in support of validating and
verifying the security testing effort.

2.5 TEST TEAM COMPOSITION

A vendor test team should be formed to conduct security testing. It is desirable for a vendor to
provide as many members from its security testing team as possible to support the NCSC during
its security testing. The reason for this is to maintain continuity and to minimize the need for
retraining throughout the evaluation process. The size, education, and skills of the test team will
vary depending on the size of the system and the class for which it is being evaluated. (See Chapter
10 of the TCSEC, "A Guideline on Security Testing.")

A vendor security testing team should be comprised of a team leader and two or more additional
members depending on the evaluated class. In selecting personnel for the test team, it is important
to assign individuals who have the ability to understand the hardware and software architecture of
the system, as well as an appropriate level of experience in system testing. Engineers and scientists
with backgrounds in electrical engineering, computer science and software engineering are ideal
candidates for functional security testing. Prior experience with penetration techniques is important
for penetration testing. A mathematics or logic background can be valuable in formal specifications
involved in A1 system evaluation.

The NCSC test team is formed using the guidance of Chapter 10, in the TCSEC, "A Guideline
on Security Testing." This chapter specifies test team composition, qualifications and parameters.
Vendors may find these requirements useful recommendations for their teams.

2.6 TEST SITE

The location of a test site is a vendor responsibility. The vendor is to provide the test site. The
evaluator's functional test site may be located at the same site at which the vendor conducted his
functional testing. Proper hardware and software must be available for testing the configuration as
well as appropriate documentation, personnel, and other resources which have a significant impact
on the location of the test site.

3. SECURITY TESTING-APPROACHES, DOCUMENTATION,
AND EXAMPLES

3.1 TESTING PHILOSOPHY

Operating systems that support multiple users require security mechanisms and policies that
guard against unauthorized disclosure and modification of critical user data. The TCB is the principal
operating system component that implements security mechanisms and policies that must itself be
protected [13]. TCB protection is provided by a reference monitor mechanism whose data structures
and code are isolated, noncircumventable, and small enough to be verifiable. The reference monitor
ensures that the entire TCB is isolated and noncircumventable.

Although TCBs for different operating systems may contain different data structures and
programs, they all share the isolation, noncircumventability, and verifiability properties that
distinguish them from the rest of the operating system components. These properties imply that the
security functional testing of an operating system TCB may require different methods from those
commonly used in software testing for all security classes of the TCSEC.

Security testing should be done for TCBs that are configured and installed in a specific system
and operate in a normal mode (as opposed to maintenance or test mode). Tests should be done using
user-level programs that cannot read or write internal TCB data structures or programs. New data
structures and programs should also not be added to a TCB for security testing purposes, and special
TCB entry points that are unavailable to user programs should not be used. If a TCB is tested in the
maintenance mode using programs that cannot be run at the user level, the security tests would be
meaningless because assurance cannot be gained that the TCB performs user-level access control
correctly. If user-level test programs could read, write or add internal TCB data structures and
programs, as would be required by traditional instrumentation testing techniques, the TCB would
lose its isolation properties. If user-level test programs could use special TCB entry points not
normally available to users, the TCB would become circumventable in the normal mode of
operation.

Security testing of operating system TCBs in the normal mode of operation using user-level test
programs (which do not rely on breaching isolation and noncircumventability) should address the
following problems of TCB verifiability through security testing: (1) Coverage Analysis, (2)
Reduction of Cyclic Test Dependencies, (3) Test Environment Independence, and (4) Repeatability
of Security Testing.

(1) Coverage Analysis. Security testing requires that precise, extensive test coverage be obtained
during TCB testing. Test coverage analysis should be based on coverage of test conditions derived
from the Descriptive Top-Level Specification (DTLS)/Formal Top-Level Specification (FTLS), the
security and accountability model conditions, the TCB isolation and noncircumventability
properties, and the individual TCB-primitive implementation. Without covering such test
conditions, it would be impossible to claim reasonably that the tests cover specific security checks
in a demonstrable way. Whenever both DTLS and FTLS and security and accountability models
are unavailable or are not required, test conditions should be derived from documented protection
philosophy and resource isolation requirements [13]. It would be impossible to reasonably claim
that the implementation of a specific security check in a TCB primitive is correct without individual
TCB-primitive coverage. In these checks a TCB primitive may deal differently with different
parameters. In normal-mode testing, however, using user-level programs makes it difficult to
guarantee significant coverage of TCB-primitive implementation while eliminating redundant tests
that appear when multiple TCB primitives share the same security checks (a common occurrence
in TCB kernels).

The role of coverage analysis in the generation of test plans is discussed in Section 3.5.2, and
illustrated in Sections 3.7.1.3-3.7.3.3.

(2) Reduction of Cyclic Test Dependencies. Comprehensive security testing suggests that cyclic
test dependencies be reduced to a minimum or eliminated whenever possible. A cyclic test
dependency exists between a test program for TCB primitive A and TCB primitive B if the test
program for TCB primitive A invokes TCB primitive B, and the test program for TCB primitive B
invokes TCB primitive A. The existence of cyclic test dependencies casts doubts on the level of
assurance obtained by TCB tests. Cyclic test dependencies cause circular arguments and
assumptions about test coverage and, consequently, the interpretation of the test results may be
flawed. For example, the test program for TCB primitive A, which depends on the correct behavior
of TCB primitive B, may not discover flaws in TCB primitive A because such flaws may be masked
by the behavior of B, and vice versa. Thus, both the assumptions (1) that the TCB primitive B works
correctly, which must be made in the test program for TCB primitive A, and (2) that TCB primitive
A works correctly, which must be made in the test program for TCB primitive B, are incorrect. The
elimination of cyclic test dependencies could be obtained only if the TCB is instrumented with
additional code and data structures an impossibility if TCB isolation and noncircumventability are
to be maintained in normal mode of operation.

An example of cyclic test dependencies, and of their removal, is provided in Section 3.7.5.

(3) Test Environment Independence. To minimize test program and test environment
dependencies the following should be reinitialized for different TCB-primitive tests: user accounts,
user groups, test objects, access privileges, and user security levels. Test environment initialization
may require that the number of different test objects to be created and logins to be executed become
very large. Therefore, in practice, complete TCB testing cannot be carried out manually. Testing
should be automated whenever possible. Security test automation is discussed in Section 3.2.

(4) Repeatability of Security Testing. TCB verifiability through security testing requires that the
results of each TCB-primitive test be repeatable. Without test repeatability it would be impossible
to evaluate developers' TCB test suites independently of the TCB developers. Independent TCB
testing may yield different outcomes from those expected if testing is not repeatable. Test
repeatability by evaluation teams requires that test plans and procedures be documented in an
accurate manner.

3.2 TEST AUTOMATION

The automation of the test procedures is one of the most important practical objectives of security
testing. This objective is important for at least three reasons. First, the procedures for test
environment initialization include a large number of repetitive steps that do not require operator
intervention, and therefore, the manual performance of these steps may introduce avoidable errors
in the test procedures. Second, the test procedures must be carried out repeatedly once for every
system generation (e.g., system build) to ensure that security errors have not been introduced during
system maintenance. Repeated manual performance of the entire test suite may become a time
consuming, error-prone activity. Third, availability of automated test suites enables evaluators to
verify both the quality and extent of a vendor's test suite on an installed system in an expeditious
manner. This significantly reduces the time required to evaluate that vendor's test suite.

The automation of most test procedures depends to a certain extent on the nature of the TCB
interface under test. For example, for most TCB-primitive tests that require the same type of login,
file system and directory initialization, it is possible to automate the tests by grouping test procedures
in one or several user-level processes that are initiated by a single test-operator login. However,
some TCB interfaces, such as the login and password change interfaces, must be tested from a user
and administrator terminal. Similarly, the testing of the TCB interface primitives of B2 to Al systems
available to users only through trusted-path invocation requires terminal interaction with the test
operator. Whenever security testing requires terminal interaction, test automation becomes a
challenging objective.

Different approaches to test automation are possible. First, test designers may want to separate
test procedures requiring terminal interaction (which are not usually automated), from those that
do not require terminal interaction (which are readily amenable to automation). In this approach,
the minimization of the number of test procedures that require terminal interaction is recommended.

Second, when test procedures requiring human-operator interaction cannot be avoided, test
designers may want to connect a workstation to a terminal line and simulate the terminal activity
of a human test operator on the workstation. This enables the complete automation of the test
environment initialization and execution procedures, but not necessarily of the result identification
and analysis procedure. This approach has been used in the testing of the Secure XenixTM TCB.
The commands issued by the test workstation that simulates the human-operator commands are
illustrated in the appendix of reference [9].

Third, the expected outcome of each test should be represented in the same format as that assumed
by the output of the TCB under test and should be placed in files of the workstation simulating a
human test operator. The comparison between the outcome files and the test result files (transferred
to the workstation upon test completion) can be performed using simple tools for file comparisons
available in most current operating systems. The formatting of the outcome files in a way that allows
their direct comparison with the test program output is a complex process. In practice, the order of
the outcomes is determined only at the time the test programs are written, and sometimes only at
execution time. Automated analysis of test results is seldomly done for this reason. To aid analysis
of test results by human operators, the test result outputs can label and time-stamp each test.
Intervention by a human test operator is also necessary in any case of mismatches between obtained
test results and expected outcomes.

An approach to automating security testing using Prolog is presented in reference [20].

3.3 TESTING APPROACHES

All approaches to security functional testing require the following four major steps: (1) the
development of test plans (i.e., test conditions, test data including test outcomes, and test coverage
analysis) and execution for each TCB primitive, (2) the definition of test procedures, (3) the
development of test programs, and (4) the analysis of the test results. These steps are not independent
of each other in all methods. Depending upon how these steps are performed in the context of
security testing, three approaches can be identified: the monolithic (black-box) testing approach,
the functional-synthesis (white-box) testing approach, and a combination of the two approaches
called the gray-box testing approach.

In all approaches, the functions to be tested are the security-relevant functions of each TCB
primitive that are visible to the TCB interface. The definition of these security functions is given by:

Classes C1 and C2. System documentation defining a system protection philosophy,
mechanisms, and system interface operations (e.g., system calls).

Class B1. Informal interpretation of the (informal) security model and the system
documentation.

Classes b2 and B3. Descriptive Top-Level Specifications (DTLSs) of the TCB and by
the interpretation of the security model that is supposed to be implemented by the TCB
functions.

Class A1. Formal Top-Level Specifications (FTLSs) of the TCB and by the interpretation
of the security model that is supposed to be implemented by the TCB functions.

Thus, a definition of the correct security function exists for each TCB primitive of a system
designed for a given security class. In TCB testing, major distinctions between the approaches
discussed in the previous section appear in the areas of test plan generation (i.e., test condition, test
data, and test coverage analysis). Further distinctions appear in the ability to eliminate redundant
TCB-primitive tests without loss of coverage. This is important for TCB testing because a large
number of access checks and access check sequences performed by TCB kernels are shared between
different kernel primitives.

3.3.1 Monolithic (Black-Box) Testing

The application of the monolithic testing approach to TCBs and to trusted processes is outlined
in reference [2]. The salient features of this approach to TCB testing are the following: (1) the test
condition selection is based on the TCSEC requirements and include discretionary and mandatory
security, object reuse, labeling, accountability, and TCB isolation; (2) the test conditions for each
TCB primitive should be generated from the chosen interpretation of each security function and
primitive as defined above (for each security class). Very seldom is the relationship between the
model interpretation and the generated test conditions, data, and programs shown explicitly (3 and
4]. Without such a relationship, it is difficult to argue coherently that all relevant security features
of the given system are covered.

The test data selection must ensure test environment independence for unrelated tests or groups
of tests (e.g., discretionary vs. mandatory tests). Environment independence requires, for example,
that the subjects, objects, and access privileges used in unrelated tests or groups of tests must differ
in all other tests or group of tests.

The test coverage analysis, which usually determines the extent of the testing for any TCB
primitive, is used to delimit the number of test sets and programs. In the monolithic approach, the
test data is usually chosen by boundary-value analysis. The test data places the test program directly
above, or below, the extremes of a set of equivalent inputs and outputs. For example, a boundary is
tested in the case of the "read" TCB call to a file by showing that (1) whenever a user has the read
privilege for that file, the read TCB call succeeds; and (2) whenever the read privilege for that file
is revoked, or whenever the file does not exist, the read TCB call fails. Similarly, a boundary is
tested in the case of TCB-call parameter validation by showing that a TCB call with parameters
passed by reference (1) succeeds whenever the reference points to an object in the caller's address
space, and (2) fails whenever the reference points to an object in another address space (e.g., kernel
space or other user spaces).

To test an individual boundary condition, all other related boundary conditions must be satisfied.
For example, in the case of the "read" primitive above, the test call must not try to read beyond the
limit of a file since the success/failure of not reading/reading beyond this limit represents a different,
albeit related, boundary condition. The number of individual boundary tests for N related boundary
conditions is of the order 2N (since both successes and failures must be tested for each of the N
conditions). Some examples of boundary-value analysis are provided in [2] for security testing, and
in [5] and [6] for security-unrelated functional testing.

The monolithic testing approach has a number of practical advantages. It can always be used by
both implementors and users (evaluators) of TCBs. No specific knowledge of implementation details
is required because there is no requirement to break the TCB (e.g., kernel) isolation or to circumvent
the TCB protection mechanism (to read, modify, or add to TCB code). Consequently, no special
tools for performing monolithic testing are required. This is particularly useful in processor
hardware testing when only descriptions of hardware/firmware implemented instructions, but no
internal hardware/firmware design documents, are available.

The disadvantages of the monolithic approach are apparent. First, it is difficult to provide a precise
coverage assessment for a set of TCB-primitive tests, even though the test selection may cover the
entire set of security features of the system. However, no coverage technique other than boundary-
value analysis can be more adequate without TCB code analysis. Second, the elimination of
redundant TCB-primitive tests without loss of coverage is possible only to a limited extent; i.e., in
the case of access-check dependencies (discussed below) among TCB-primitive specifications.
Third, in the context of TCB testing, the monolithic approach cannot cope with the problem of
cyclic dependencies among test programs. Fourth, lack of TC code analysis precludes the possibility
of distinguishing between design and implementation code errors in all but a few special cases.
Also, it precludes the discovery of spurious code within the TCB-a necessary condition for Trojan
Horse analysis.

In spite of these disadvantages, monolithic functional testing can be applied successfully to TCB
primitives that implement simple security checks and share few of these checks (i.e., few or no
redundant tests would exist). For example, many trusted processes have these characteristics, and
thus this approach is adequate.

3.3.2 Functional-Synthesis (White-Box) Testing

Functional-synthesis-based testing requires the test of both functions implemented by each
program (e.g., program of a TCB primitive) as a whole and functions implemented by internal parts
of the program. The internal program parts correspond to the functional ideas used in building the
program. Different forms of testing procedures are used depending upon different kinds of
functional synthesis (e.g., control, algebraic, conditional, and iterative synthesis described in [1]
and [7]). As pointed out in [9], only the control synthesis approach to functional testing is suitable
for security testing.

In control synthesis, functions are represented as sequences of other functions. Each function in
a sequence transforms an input state into an output state, which may be the input to another function.
Thus, a control synthesis graph is developed during program development and integration with
nodes representing data states and arcs representing state transition functions. The data states are
defined by the variables used in the program and represent the input to the state transition functions.
The assignment of program functions, procedures, and subroutines to the state transition functions
of the graph is usually left to the individual programmer's judgment. Examples of how the control
synthesis graphs are built during the program development and integration phase are given in [1]
and [7].

The suitability of the control synthesis approach to TCB testing becomes apparent when one
identifies the nodes of the control synthesis graph with the access checks within the TCB and the
arcs with data states and outcomes of previous access checks. This representation, which is the dual
of the traditional control synthesis graphs [9], produces a kernel access-check graph (ACG). This
representation is useful because in TCB testing the primary access-check concerns are those of (1)
missing checks within a sequence of required checks, (2) wrong sequences of checks, and (3) faulty
or incomplete access checks. (Many of the security problems identified in the Multics kernel design
project existed because of these broad categories of inadequate access checks [8].) It is more suitable
than the traditional control-synthesis graph because major portions of a TCB, namely the kernel,
have comparatively few distinct access checks (and access-check sequences) and a large number
of object types and access privileges that have the same access-check sequences for different TCB
primitives [9]. (However, this approach is less advantageous in trusted process testing because
trusted processes-unlike kernels-have many different access checks and few shared access
sequences.) These objects cause the same data flow between access check functions and, therefore,
are combined as graph arcs.

The above representation of the control synthesis graph has the advantage of allowing the
reduction of the graph to the subset of kernel functions that are relevant to security testing. In
contrast, a traditional graph would include (1) a large number of other functions (and, therefore,
graph arcs), and (2) a large number of data states (and, therefore, graph nodes). This would be both
inadequate and unnecessary. It would be inadequate because the presence of a large number of
security-irrelevant functions (e.g., functions unrelated to security or accountability checks or to
protection mechanisms) would obscure the role of the security-relevant ones, making test coverage
analysis a complex and difficult task. It would be unnecessary because not only could security-
irrelevant functions be eliminated from the graph but also the flows of different object types into
the same access check function could be combined, making most object type-based security tests
unnecessary.

Any TCB-primitive program can be synthesized at the time of TCB implementations as a graph
of access-checking functions and data flow arcs. Many of the TCB-primitive programs share both
arcs and nodes of the TCB graph. To build an access-check graph, one must identify all access-
check functions, their inputs and outputs, and their sequencing. A typical input to an access-check
function consists of an object identifier, object type and required access privileges. The output
consists of the input to the next function (as defined above) and, in most cases, the outcome of the
function check. The sequencing information for access-check functions consists of (1) the ordering
of these functions, and (2) the number of arc traversals for each arc. An example of this is the
sequencing of some access check functions that depend on the object types.

Test condition selection in the control-synthesis approach can be performed so that all the above
access check concerns are satisfied. For example, test conditions must identify missing discretionary,
mandatory, object reuse, privilege-call, and parameter validation checks (or parts of those checks).
It also must identify access checks that are out of order, and faulty or incomplete checks, such as
being able to truncate a file for which the modify privilege does not exist. The test conditions must
also be based on the security model interpretation to the same extent as that in the monolithic
approach.

The test coverage in this approach also refers to the delimitation of the test data and programs
for each TCB primitive. Because many of the access-check functions, and sequences of functions,
are common to many of the kernel primitives (but not necessarily to trusted-process primitives), the
synthesized kernel (TCB) graph is fairly small. Despite this the coverage analysis cannot rely on
individual arc testing for covering the graph. The reason is that arc testing does not force the testing
of access checks that correspond to combinations of arcs and thus it does not force coverage of all
relevant sequences of security tests. Newer test coverage techniques for control synthesis graphs,
such as data-flow testing [9, 10, and 11] provide coverage of arc combinations and thus are more
appropriate than those using individual arc testing.

The properties of the functional-synthesis approach to TCB testing appear to be orthogonal to
those of monolithic testing. Consider the disadvantages of functional-synthesis testing. It is not as
readily usable as monolithic testing because of the lack of detailed knowledge of system internals.
Also, it helps remove very few redundant tests whenever few access check sequences are shared
by TCB primitives (as is the case with most trusted-process primitives).

Functional-synthesis-based testing, however, has a number of fundamental advantages. First, the
coverage based on knowledge of internal program structure (i.e., code structure of a kernel primitive)
can be more extensive than in the monolithic approach [1 and 7]. A fairly precise assessment of
coverage can be made, and most of the redundant tests can be identified. Second, one can distinguish
between TCB-primitive program failures and TCB-primitive design failures, something nearly
impossible with monolithic testing. Third, this approach can help remove cyclic test dependencies.
By removing all, or a large number of redundant tests, one removes most cyclic test dependencies
(example of Section 3.7.5).

TCB code analysis becomes necessary whenever a graph synthesis is done after a TCB is built.
Such analysis helps identify spurious control paths and code within a TCB-a necessary condition
for Trojan Horse discovery. (In such a case, a better term for this approach would be functional-
analysis-based testing.)

3.3.3 Gray-Box Testing

Two of the principal goals of security testing have been (1) the elimination of redundant tests
through systematic test-condition selection and coverage analysis, and (2) the elimination of cyclic
dependencies between the test programs. Other goals, such as test repeatability, which is also
considered important, can be attained through the same means as those used for the other methods.

The elimination of redundant TCB-primitive tests is a worthwhile goal for the obvious reason
that it reduces the amount of testing effort without loss of coverage. This allows one to determine
a smaller nucleus of tests that must be carried out extensively. The overall TCB assurance may
increase due to the judicious distribution of the test effort. The elimination of cyclic dependencies
among the TCB-primitive test programs is also a necessary goal because it helps establish a rigorous
test order without making circular assumptions of the behavior of the TCB primitives. Added
assurance is therefore gained.

To achieve the above goals, the gray-box testing approach combines monolithic testing with
functional-synthesis-based testing in the test selection and coverage areas. This combination relies
on the elimination of redundant tests through access-check dependency analysis afforded by
monolithic testing. It also relies on the synthesis of the access-check graph from the TCB code as
suggested by functional-synthesis-based testing (used for further elimination of redundant tests).
The combination of these two testing methods generates a TCB-primitive test order that requires
increasingly fewer test conditions and data without loss of coverage.

A significant number of test conditions and associated tests can be eliminated by the use of the
access-check graph of TCB kernels. Recall that each kernel primitive may have a different access-
check graph in principle. In practice, however, substantial parts of the graphs overlap. Consequently,
if one of the graph paths is tested with sufficient coverage for a kernel primitive, then test conditions
generated for a different kernel primitive whose graph overlaps with the first need only include the
access checks specific to the latter kernel primitive. This is true because by the definition of the
access-check graph, the commonality of paths means that the same access checks are performed in
the same sequence, on the same types of objects and privileges, and with the same outcomes (e.g.,
success and failure returns). The specific access checks of a kernel primitive, however, must also
show that the untested subpath(s) that has not been tested, of that kernel primitive, joins the tested
path.

(A subset of the access-check and access-graph dependencies for the access, open, read, write,
fcntl, ioctl, opensem, waltsem and slgsem primitives of UnixTM-like kernels are illustrated in
Figures 1 and 2, pages 23 and 24. The use of these dependencies in the development of test plans,
especially in coverage analysis, is illustrated in Sections 3.7.2.3 and 3.7.3.3; namely, in the test
plans for access, open, and read. Note that the arcs shown in Figure 2, page 24 include neither
complete flow-of-control information nor complete sets of object types, access-checks per call, and
call outcome.)

3.4 RELATIONSHIP WITH THE TCSEC SECURITY TESTING REQUIREMENTS

The TCSEC security testing requirements and guidelines (i.e., Part 1 and Section 10 of the TCSEC)
help define different approaches for security testing. They are particularly useful for test condition
generation and test coverage. This section reviews these requirements in light of security testing
approaches defined in Section 3.3.

Security Class C1

Test Condition Generation

"The security mechanisms of the ADP system shall be tested and found to work as claimed
in the system documentation." [TCSEC Part I, Section 2.1]

For this class of systems, the test conditions should be generated from the system documentation
which includes the Security Features User's Guide (SFUG), the Trusted Facility Manual (TFM),
the system reference manual describing each TCB primitive, and the design documentation defining
the protection philosophy and its TCB implementation. Both the SFUG and the manual pages, for
example, illustrate how the identification and authentication mechanisms work and whether a
particular TCB primitive contains relevant security and accountability mechanisms. The
Discretionary Access Control (DAC) and the identification and authentication conditions enforced
by each primitive (if any) are used to define the test conditions of the test plans.

Test Coverage

"Testing shall be done to assure that there are no obvious ways for an unauthorized user
to bypass or otherwise defeat the security protection mechanisms of the TCB." [TCSEC,
Part I, Section 2.1]

"The team shall independently design and implement at least five system-specific tests
in an attempt to circumvent the security mechanisms of the system." [TCSEC, Part II,
Section 10]

The above TCSEC requirements and guidelines define the scope of security testing for this
security class. Since each TCB primitive may include security-relevant mechanisms, security testing
shall include at least five test conditions for each primitive. Furthermore, because source code
analysis is neither required nor suggested for class C1 systems, monolithic functional testing (i.e.,
a black-box approach) with boundary-value coverage represents an adequate testing approach for
this class. Boundary-value coverage of each test condition requires that at least two calls of each
TCB primitive be made, one for the positive and one for the negative outcome of the condition.
Such coverage may also require more than two calls per condition. Whenever a TCB primitive refers
to multiple types of objects, each condition is repeated for each relevant type of object for both its
positive and negative outcomes. A large number of test calls may be necessary for each TCB
primitive because each test condition may in fact have multiple related conditions which should be
tested independently of each other.

Security Class C2

Test Condition Generation

"Testing shall also include a search for obvious flaws that would allow violation of
resource isolation, or that would permit unauthorized access to the audit and
authentication data." [TCSEC, Part I, Section 2.2]

These added requirements refer only to new sources of test conditions, but not to a new testing
approach nor to new coverage methods. The following new sources of test conditions should be
considered:

(1) Resource isolation conditions. These test conditions refer to all TCB primitives that
implement specific system resources (e.g., object types or system services). Test
conditions for TCB primitives implementing services may differ from those for TCB
primitives implementing different types of objects. Thus, new conditions may need to be
generated for TCB services. The mere repetition of test conditions defined for other TCB
primitives may not be adequate for some services.

(2) Conditions for protection of audit and authentication data. Because both audit and
authentication mechanisms and data are protected by the TCB, the test conditions for the
protection of these mechanisms and their data are similar to those which show that the
TCB protection mechanisms are tamperproof and noncircumventable. For example, these
conditions show that neither privileged TCB primitives nor audit and user authentication
files are accessible to regular users.

Test Coverage

Although class C1 test coverage already suggests that each test condition be covered for each
type of object, coverage of resource-specific test conditions also requires that each test condition
be covered for each type of service (whenever the test condition is relevant to a service). For example,
the test conditions which show that direct access to a shared printer is denied to a user shall be
repeated for a shared tape drive with appropriate modification of test data (i.e., test environments
set up, test parameters and outcomes-namely, the test plan structure discussed in Section 3.5).

Security Class B1

Test Condition Generation

The objectives of security testing ". . . shall be: to uncover all design and implementation
flaws that would permit a subject external to the TCB to read, change, or delete data
normally denied under the mandatory or discretionary security policy enforced by the
TCB; as well as to ensure that no subject (without authorization to do so) is able to cause
the TCB to enter a state such that it is unable to respond to communications initiated by
other users." [TCSEC, Part I, Section 3.1]

The security testing requirements of class B1 are more extensive than those of both classes C1
and C2, both in test condition generation and in coverage analysis. The source of test conditions
referring to users' access to data includes the mandatory and discretionary policies implemented
by the TCB. These policies are defined by an (informal) policy model whose interpretation within
the TCB allows the derivation of test conditions for each TCB primitive. Although not explicitly
stated in the TCSEC, it is generally expected that all relevant test conditions for classes C1 and C2
also would be used for a class B1 system.

Test Coverage

"All discovered flaws shall be removed or neutralized and the TCB retested to demonstrate
that they have been eliminated and that new flaws have not been introduced." [TCSEC,
Part I, Section 3.1]

"The team shall independently design and implement at least fifteen system specific tests
in an attempt to circumvent the security mechanisms of the system." [TCSEC, Part II,
Section 10]

Although the coverage analysis is still boundary-value analysis, security testing for class B1
systems suggests that at least fifteen test conditions be generated for each TCB primitive that
contains security-relevant mechanisms to cover both mandatory and discretionary policy. In
practice, however, a substantially higher number of test conditions is generated from interpretations
of the (informal) security model. The removal or the neutralization of found errors and the retesting
of the TCB requires no additional types of coverage analysis.

Security Class B2

Test Condition Generation

"Testing shall demonstrate that the TCB implementation is consistent with the descriptive
top-level specification." [TCSEC, Part I, Section 3.2]

The above requirement implies that both the test conditions and coverage analysis of class B2
systems are more extensive than those of class B1. In class B2 systems every access control and
accountability mechanism documented in the DTLS (which must be complete as well as accurate)
represents a source of test conditions. In principle the same types of test conditions would be
generated for class B2 systems as for class B1 systems, because (1) in both classes the test conditions
could be generated from interpretations of the security policy model (informal at B1 and formal at
B2), and (2) in class B2 the DTLS includes precisely the interpretation of the security policy model.
In practice this is not the case however, because security policy models do not model a substantial
number of mechanisms that are, nevertheless, included in the DTLS of class B2 systems. (Recall
that class B1 systems do not require a DTLS of the TCB interface.) The number and type of test
conditions can therefore be substantially higher in a class B2 system than those in a class B1 system
because the DTLS for each TCB primitive may contain additional types of mechanisms, such as
those for trusted facility management.

Test Coverage

It is not unusual to have a few individual test conditions for at least some of the TCB primitives.
As suggested in the gray-box approach defined in the previous section, repeating these conditions
for many of the TCB primitives to achieve uniform coverage can be both impractical and
unnecessary. Particularly this is true when these primitives refer to the same object types and
services. It is for this reason and because source-code analysis is required in class B2 systems to
satisfy other requirements that the use of the gray-box testing approach is recommended for the
parts of the TCB in which primitives share a substantial portion of their code. Note that the DTLS
of any system does not necessarily provide any test conditions for demonstrating the
tamperproofness and noncircumventability of the TCB. Such conditions should be generated
separately.

Security Class 83

Test Condition Generation

The only difference between classes B2 and B3 requirements of security testing reflects the need
to discover virtually all security policy flaws before the evaluation team conducts its security testing
exercise. Thus, no additional test condition requirements appear for class B3 testing. Note that the
DTLS does not necessarily provide any test conditions for demonstrating the TCB is tamperproof
and noncircumventable as with class B2 systems. Such conditions should be generated separately.

Test Coverage

"No design flaws and no more than a few correctable implementation flaws may be found
during testing and there shall be reasonable confidence that few remain." [TCSEC, Part
I, Section 3.3]

The above requirement suggests that a higher degree of confidence in coverage analysis is required
for class B3 systems than for class B2 systems. It is for this reason that it is recommended the gray-
box testing approach be used extensively for the entire TCB kernel, and data-flow coverage be used
for all independent primitives of the kernel (namely, the gray-box method in Section 3.3 above).

Security Class A1

The only differences between security testing requirements of classes B3 and A1 are (1) the test
conditions shall be derived from the FTLS, and (2) the coverage analysis should include at least
twenty-five test conditions for each TCB primitive implementing security functions. Neither
requirement suggests that a different testing method than that recommended for class B3 systems
is required.

3.5 SECURITY TEST DOCUMENTATION

This section discusses the structure of typical test plans, test logs, test programs, test procedures,
and test reports. The description of the test procedures necessary to run the tests and to examine
the test results is also addressed. The documentation structures presented are meant to provide the
system developers with examples of good test documentation.

3.5.1 Overview

The work plan for system testing should describe how security testing will be conducted and
should contain the following information:

· Test-system configuration for both hardware and software.

· Summary test requirements.

· Procedures for executing test cases.

· Step-by-step procedures for each test case.

· Expected results for each test step.

· Procedures for correcting flaws uncovered during testing.

· Expected audit information generated by each test case (if any).

See Section 3.7.7, "Relationship with the TCSEC Requirements."

3.5.2 Test Plan

Analysis and testing of mechanisms, assurances and/or documentation to support the TCSEC
security testing requirements are accomplished through test plans. The test plans should be
sufficiently complete to cover each identified security mechanism and should be conducted with
sufficient depth to provide reasonable assurance that any bugs not found lie within the acceptable
risk threshold for the class of the system being evaluated. A test plan consists of test conditions,
test data, and coverage analysis.

3.5.2.1 Test Conditions

A test condition is a statement of a security-relevant constraint that must be satisfied by a TCB
primitive. Test conditions should be derived from the system's DTLS/FTLS, from the interpretation
of the security and accountability models (if any), from TCB isolation and noncircumventability
properties, and from the specifications and implementation of the individual TCB primitive under
test. If neither DTLS/FTLS nor models are required, then test conditions should be derived from
the informal policy statements, protection philosophy and resource isolation requirements.

(1) Generation of Model or Policy-Relevant Test Conditions

This step suggests that a matrix of TCB primitives and the security model(s) or requirement
components be built. Each entry in the matrix identifies the security relevance of each primitive (if
any) in a security model or requirement area and the relevant test conditions. For example, in the
mandatory access control area of security policy, one should test the proper object labeling by the
TCB, the "compatibility" property of the user created objects, and the TCB implemented
authorization rules for subject access to objects. One should also test that the security-level
relationships are properly maintained by the TCB and that the mandatory access works
independently of, and in conjunction with, the discretionary access control mechanism. In the
discretionary access control area, one may include tests for proper user/group identifier selection,
proper user inclusion/exclusion, selective access distribution/revocation using the access control
list (ACL) mechanism, and access review.

Test conditions derived from TCB isolation and noncircumventability properties include
conditions that verify (1) that TCB data structures are inaccessible to user level programs, (2) that
transfer of control to the TCB can take place only at specified entry points, which cannot be bypassed
by user-level programs, (3) that privileged entry points into the TCB cannot be used by user level
programs, and (4) that parameters passed by reference to the TCB are validated.

Test conditions derived from accountability policy include conditions that verify that user
identification and authentication mechanisms operate properly. For example, they include
conditions that verify that only sufficiently complex passwords can be chosen by any user, that the
password aging mechanism forces reuse at stated intervals, and so on. Other conditions of
identification and authentication, such as those that verify that the user login level is dominated by
the user's maximum security level, should also be included. Furthermore, conditions that verify
that the user commands included in the trusted path mechanism are unavailable to the user program
interface of the TCB should be used. Accountability test conditions that verify the correct operation
of the audit mechanisms should also be generated and used in security testing.

The security relevance of a TCB primitive can only be determined from the security policy,
accountability, and TCB isolation and noncircumventability requirements for classes B1 to A1, or
from protection philosophy and resource isolation requirements for classes C1 and C2. Some TCB
primitives are security irrelevant. For example, TCB primitives that never allow the flow of
information across the boundaries of an accessible object are always security irrelevant and need
not be tested with respect to the security or accountability policies. The limitation of information
flow to user-accessible objects by the TCB primitives implementation, however, needs to be tested
by TCB-primitive-specific tests. A general example of security-irrelevant TCB primitives is
provided by those primitives which merely retrieve the status of user-owned processes at the security
level of the user.

(2) Generation of TCB-Primitive-Specific Test Conditions

The selection of test conditions used in security testing should be TCB-primitive-specific. This
helps remove redundant test conditions and, at the same time, helps ensure that significant test
coverage is obtained. For example, the analysis of TCB-primitive specifications to determine their
access-check dependencies is required whenever the removal of redundant TCB-primitive tests is
considered important. This analysis can be applied to all testing approaches. The specification of a
TCB primitive A is access-check dependent on the specification of a TCB primitive B if a subset
of the access checks needed in TCB primitive A are performed in TCB primitive B, and if a TCB
call to primitive B always precedes a TCB call to primitive A (i.e., a call to TCB primitive A fails
if the call to TCB primitive B has not been done or has not completed with a successful outcome).
In case of such dependencies, it is sufficient to test TCB primitive B first and then to test only the
access checks of TCB primitive A that are not performed in TCB primitive B. Of course, the
existence of the access-check dependency must be verified through testing.

As an example of access-check dependency, consider the fork and the exit primitives of the
Secure XenixTM kernel. The exit primitive always terminates a process and sends a return code to
the parent process. The mandatory access check that needs to be tested in exit is that the child's
process security level equals that of the parent's process. However, the specifications of the exit
primitive are access-check dependent on the specifications of the fork primitive (1) because an exit
call succeeds only after a successfully completed fork call is done by some parent process, and (2)
because the access check, that the child's process level always equals that of the parent's process
level, is already performed during the fork call. In this case, no additional mandatory access test is
needed for exit beyond that performed for fork. Similarly, the sigsem and the waitsem primitives
of some UnixTM based kernels are access-check dependent on the opensem primitive, and no
additional mandatory or discretionary access checks are necessary.

However, in the case of the read and the write primitives of UnixTM kernels, the specifications
of which are also access-check dependent on both the mandatory and the discretionary checks of
the open primitive, additional tests are necessary beyond those done for open. In the case of the
read primitive one needs to test that files could only be read if they have been opened for reading,
and that reading beyond the end of a file is impossible after one tests the dependency of read on
the specification of open. Additional tests are also needed for other primitives such as fcntl and
loctl; their specifications are both mandatory and discretionary access-check dependent on the open
primitives for files and devices. Note that in all of the above examples a large number of test
conditions and associated tests are eliminated by using the notion of access check dependency of
specifications because, in general, less test conditions are generated for access check dependency
testing than for the security testing of the primitive itself.

The following examples are given in references [3] and [4]: (1) of the generation of such
constraints from security models, (2) of the predicates, variables, and object types used in constraint
definition, and (3) of the use of such constraints in test conditions for processor instructions (rather
than for TCB primitives).

See Section 3.7.7, "Relationship with the TCSEC Requirements."

3.5.2.2 Test Data

"Test data" is defined as the set of specific objects and variables that must be used to demonstrate
that a test condition is satisfied by a TCB primitive. The test data consist of the definition of the
initialization data for the test environment, the test parameters for each TCB primitive, and the
expected test outcomes. Test data generation is as important as test condition generation because it
ensures that test conditions are exercised with appropriate coverage in the test programs, and that
test environment independence is established whenever it is needed.

To understand the importance of test data generation consider the following example. Suppose
that all mandatory tests must ensure that the "hierarchy" requirement of the mandatory policy
interpretation must be tested for each TCB primitive. (Expansion on this subject, i.e., the
nondecreasing security level requirement for the directory hierarchy can be found in [12].) What
directory hierarchy should one set up for testing this requirement and at the same time argue that
all possible directory hierarchies are covered for all tests? A simple analysis of this case shows that
there are two different forms of upgraded directory creation that constitute an independent basis
for all directory hierarchies (i.e., all hierarchies can be constructed by the operations used for one
or the other of the two forms, or by combinations of these operations). The first form is illustrated
in Figure 3a representing the case whereby each upgraded directory at a different level is upgraded
from a single lower level (e.g., system low). The second form is illustrated in Figure 3b and
represents the case whereby each directory at a certain level is upgraded from an immediately lower
level. A similar example can be constructed to show that combinations of security level definitions
used for mandatory policy testing cover all security level relationships.

Test data for TCB primitives should include several items such as the TCB primitive input data,
TCB primitive return result and success/failure code, object hierarchy definition, security level used
for each process/object, access privileges used, user identifiers, object types, and so on. This
selection needs to be made on a test-by-test basis and on a primitive-by-primitive basis. Whenever
environment independence is required, a different set of data is defined [2]. It is very helpful that
the naming scheme used for each data object helps identify the test that used that item. Different
test environments can be easily identified in this way. Note that the test data selection should ensure
both coverage of model-relevant test conditions and coverage of the individual TCB primitives.
This will be illustrated in an example in the next section.

See Section 3.7.7, "Relationship with the TCSEC Requirements."

3.5.2.3 Coverage Analysis

Test coverage analysis is performed in conjunction with the test selection phase of our approach.
Two classes of coverage analysis should be performed: model- or policy-dependent coverage and
individual TCB primitive coverage.

(1) Model- or Policy-Dependent Coverage

In this class, one should demonstrate that the selected test conditions and data cover the
interpretation of the security and accountability model and noncircumventability properties in all
areas identified by the matrix mentioned above. This is a comparatively simple task because model
coverage considerations drive the test condition and data selection. This kind of coverage includes
object type, object hierarchy, subject identification, access privilege, subject/object security level,
authorization check coverage, and so on. Model dependent coverage analysis relies, in general, on
boundary-value analysis.

(2) Individual TCB-Primitives Coverage

This kind of coverage includes boundary value analysis, data flow analysis of individual access-
check graphs of TCB primitives, and coverage of dependencies. The examples of reference [2]
illustrate boundary-value analysis. Other forms of TCB-primitive coverage will be discussed in
Section 3.7 of this guideline. For example, graph coverage analysis represents the determination
that the test conditions and data exercise all the data flows for each TCB-primitive graph. This
includes not only the traversal of all the graph access checks (i.e., nodes) but also of all the graph's
arcs and arc sequences required for each TCB primitive. (The example for access primitive of
UnixTM kernels included in Section 3.7 explains this form of coverage. Data flow coverage is also
presented in [10] and [11] for security-unrelated test examples.)

Coverage analysis is both a qualitative and quantitative assessment of the extent to which the test
shows TCB-primitive compliance with the (1) design documentation, (2) resource isolation, (3)
audit and authentication data protection, (4) security policy and accountability model conditions,
(5) DTLS/FTLS, as well as with those of the TCB isolation and noncircumventability properties.
To achieve significant coverage, all security-relevant conditions derived from a TCB model and
properties and DTLS/FTLS should be covered by a test, and each TCB-primitive test should cover
the implementation of its TCB primitive. For example, each TCB- primitive test should be performed
for all independent object types operated upon by that TCB primitive and should test all independent
security exceptions for each type of object.

See Section 3.7.7, "Relationship with the TCSEC Requirements."

3.5.3 Test Procedures

A key step in any test system is the generation of the test procedures (which are also known as
"test scripts"). The major function of the test procedure is to ensure that an independent test operator
or user is able to carry out the test and to obtain the same results as the test implementor. The
procedure for each test should be explained in sufficient detail to enable repeatable testing. The test
procedure should contain the following items to accomplish this:

(1) Environment Initialization Procedure. This procedure defines the login sequences and
parameters, the commands for object and subject cleanup operations at all levels involved in the
test, the choice of object names, the commands and parameters for object creation and initialization
at the required levels, the required order of command execution, the initialization at the required
levels, the initialization of different subject identifiers and access privileges (for the initialized
objects) at all required levels, and the specification of the test program and command names and
parameters used in the current test.

(2) Test Execution Procedure. The test procedure includes a description of the test execution from
a terminal including the list of user commands, their input, and the expected terminal, printer, or
file output.

(3) Result Identification Procedure. The test procedure should also identify the results file for a
given test, or the criteria the test operator must use to find the results of each individual test in the
test output file. The meaning of the results should also be provided.

See Section 3.7.7, "Relationship with the TCSEC Requirements."

Note: A system in which testing is fully automated eliminates the need for separate test procedure
documentation. In such cases, the environment initialization procedures and the test execution
procedures should be documented in the test data section of the test plans. Automated test operator
programs include the built-in knowledge otherwise contained in test procedures.

3.5.4 Test Programs

Another key step of any test system is the generation of the test programs. The test programs for
each TCB primitive consist of the Iogin sequence, password, and requested security level. The
security profile of the test operator and of the possible workstation needs to be defined a priori by
the system security administrators to allow logins and environment initialization at levels required
in the test plan. After login, a test program invokes several trusted processes (e.g., "mkdir," "rmdir,"
in some UnixYM systems) with predetermined parameters in the test plan and procedure to initialize
the test environment. A nucleus of trusted processes, necessary for the environment set up, are tested
independently of a TCB primitive under test whenever possible and are assumed to be correct.

After the test environment is initialized, the test program (which may require multiple logins at
different levels) issues multiple invocations to the TCB primitive under test and to other TCB
primitives needed for the current test. The output of each primitive issued by the test programs is
collected in a result file associated with each separate test and analyzed. The analysis of the test
results that are collected in the results file is performed by the test operator. This analysis is a
comparison between the results file and the expected outcome file defined by the test plan prior to
the test run. Whenever the test operator detects a discrepancy between the two files he records a
test error.

3.5.5 Test Log

A test log should be maintained by each team member during security testing. It is to capture
useful information to be included later in the test report. The test log should contain:

· Information on any noteworthy observations.

· Modifications to the test steps.

· Documentation errors.

· Other useful data recorded during the testing procedure test results.

3.5.6 Test Report

The test report is to present the results of the security testing in a manner that effectively supports
the conclusions reached from the security testing process and provides a basis for NCSC test team
security testing. The test report should contain:

· Information on the configuration of the tested system.

· A chronology of the security testing effort.

· The results of functional testing including a discussion of each flaw uncovered.

· The results of penetration testing covering the results of successful penetrations.

· Discussion of the corrections that were implemented and of any retesting that was
performed.

A sample test report format is provided in Section 3.7.

3.6 SECURITY TESTING OF PROCESSORS' HARDWARE/FIRMWARE
PROTECTION MECHANISMS

The processors of a computer system include the Central Processing Units (CPU), Input/Output
(I/O) processors, and application-oriented co-processors such as numerical co-processors and
signal-analysis co-processors. These processors may include mechanisms capabilities, access
privileges, processor-status registers, and memory areas representing TCB internal objects such as
process control blocks, descriptor, and page tables. The effects of the processor protection
mechanisms become visible to the system users through the execution of processor instructions and
I/O commands that produce transformations of processor and memory registers. Transformations
produced by every instruction or I/O command are checked by the processors protection
mechanisms and are allowed only if they conform with the specifications defined by the processor
reference manuals for that instruction. For few processors these transformations are specified
formally and for less processors a formal (or informal) model of the protection mechanisms is given
[3 and 4].

3.6.1 The Need for Hardware/Firmware Security Testing

Protection mechanisms of systems processors provide the basic support for TCB isolation,
noncircumventability, and process address space separation. In general, processor mechanisms for
the isolation of the TCB include those that (1) help separate the TCB address space and privileges
from those of the user, (2) help enforce the transfer of control from the user address space to the
TCB address space at specific entry points, and (3) help verify the validity of the user-level
parameters passed to the TCB during primitive invocation. Processor mechanisms that support TCB
noncircumventability include those that (1) check each object reference against a specific set of
privileges, and (2) ensure that privileged instructions which can circumvent some of the protection
mechanisms are inaccessible to the user. Protection mechanisms that help separate process address
spaces include those using base and relocation registers, paging, segmentation, and combinations
thereof.

The primary reason for testing the security function of a system's processors is that flaws in the
design and implementation of processor-supported protection mechanisms become visible at the
user level through the instruction set. This makes the entire system vulnerable because users can
issue carefully constructed sequences of instructions that would compromise TCB and user security.

(User visibility of protection flaws in processor designs is particularly difficult to deny. Attempts
to force programmers to use only high-level languages, such as PL1, Pascal, Algol, etc., which
would obscure the processor instruction set, are counterproductive because arbitrary addressing
patterns and instruction sequences still can be constructed through seemingly valid programs (i.e.,
programs that compile correctly). In addition, exclusive reliance on language compilers and on
other subsystems for the purpose of obscuring protection flaws and denying users the ability to
produce arbitrary addressing patterns is unjustifiable. One reason is that compiler verification is a
particularly difficult task; another is that reliance on compilers and on other subsystems implies
reliance on the diverse skills and interests of system programmers. Alternatively, hardware-based
attempts to detect instruction sequence patterns that lead to protection violations would only result
in severe performance degradation.)

The additional reason for testing the security function of a system's processor is that, in general,
a system's TCB uses at least some of the processor's mechanisms to implement its security policy.
Flawed protection mechanisms may become unusable by the TCB and, in some cases, the TCB
may not be able to neutralize those flaws (e.g., make them invisible to the user). It should be noted
that the security testing of the processor protection mechanisms is the most basic life-cycle evidence
available in the context of TCSEC evaluations to support the claim that a system's reference notion
is verifiable.

3.6.2 Explicit TCSEC Requirements for Hardware Security Testing

The TCSEC imposes very few explicit requirements for the security testing of a system's hardware
and firmware protection mechanisms. Few interpretations can be derived from these requirements
as a consequence. Recommendations for processor test plan generation and documentation,
however, will be made in this guideline in addition to explicit TCSEC requirements. These
recommendations are based on analogous TCB testing recommendations made herein.

Specific Requirements for Classes C1 and C2

The following requirements are included for security classes C1 and C2:

"The security mechanisms of the ADP system shall be tested and found to work as claimed
in the system documentation."

The security mechanisms of the ADP system clearly include the processor-supported protection
mechanisms that are used by the TCB and those that are visible to the users through the processor's
instruction set. In principle it could be argued that the TCB security testing implicitly tests at least
some processor mechanisms used by the TCB; therefore, no additional hardware testing is required
for these mechanisms. All processor protection mechanisms that are visible to the user through the
instruction set shall be tested separately regardless of their use by a tested TCB. In practice, nearly
all processor protection mechanisms are visible to users through the instruction set. An exception
is provided by some of the I/O processor mechanisms in systems where users cannot execute I/O
commands either directly or indirectly.

Specific Requirements for Classes B1 to B3

In addition to the above requirements of classes C1 and C2, the TCSEC includes the following
specific hardware security testing guidelines in Section 10 "A Guideline on Security Testing":

"The [evaluation] team shall have `hands-on' involvement in an independent run of the
test package used by the system developer to test security-relevant hardware and software.

The explicit inclusion of this requirement in the division B (i.e., classes B1 to B3) of the TCSEC
guideline on security testing implies that the scope and coverage of the security-relevant hardware
testing and test documentation should be consistent with those of the TCB security testing for this
division. Thus, the security testing of the processor s protection mechanisms for division B systems
should be more extensive that for division C (i.e., C1 and C2) systems.

Specific Requirement for Class A1

In addition to the requirements for divisions C and B, the TCSEC includes the following explicit
requirements for hardware and/or firmware testing:

"Testing shall demonstrate that the TCB implementation is consistent with the formal
top-level specifications." [Security Testing requirement] and

"The DTLS and FTLS shall include those components of the TCB that are implemented
as hardware and/or firmware if their properties are visible at the TCB interface." [Design
Specification and Verification requirement]

The above requirements suggest that all processor protection mechanisms that are visible at the
TCB interface should be tested. The scope and coverage of the security-relevant testing and test
documentation should also be consistent with those of TCB security-relevant testing and test
documentation for this division.

3.6.3 Hardware Security Testing vs. System Integrity Testing

Hardware security testing and system integrity testing differ in at least three fundamental ways.
First, the scope of system integrity testing and that of hardware security testing is different. System
integrity testing refers to the functional testing of the hardware/firmware components of a system
including components that do not necessarily have a specific security function (i.e., do not include
any protection mechanisms). Such components include the memory boards, busses, displays,
adaptors for special devices, etc. Hardware security testing, in contrast, refers to hardware and
firmware components that include protection mechanisms (e.g., CPU's and I/O processors). Failures
of system components that do not include protection mechanisms may also affect system security
just as they would affect reliability and system performance. Failures of components that include
protection mechanisms can affect system security adversely. A direct consequence of the distinction
between the scope of system integrity and hardware security testing is that security testing
requirements vary with the security class of a system, whereas system integrity testing requirements
do not.

Second, the time and frequency of system integrity and security testing are different. System
integrity testing is performed periodically at the installation site of the equipment. System security
testing is performed in most cases at component design and integration time. Seldom are hardware
security test suites performed at the installation site.

Third, the responsibility for system integrity testing and hardware security testing is different.
System integrity testing is performed by site administrators and vendor customer or field engineers.
Hardware security testing is performed almost exclusively by manufacturers, vendors, and system
evaluators.

3.6.4 Goals, Philosophy, and Approaches to Hardware Security Testing

Hardware security testing has the same general goals and philosophy as those of general TCB
security testing. Hardware security testing should be performed for processors that operate in normal
mode (as opposed to maintenance or test mode). Special probes, instrumentation, and special
reserved op-codes in the instruction set should be unnecessary. Coverage analysis for each tested
instruction should be included in each test plan. Cyclic test dependencies should be minimized, and
testing should be repeatable and automated whenever possible.

In principle, all the approaches to security testing presented in Section 3.3 are applicable to
hardware security testing. In practice, however, all security testing approaches reported to date have
relied on the monolithic testing approach. This is the case because hardware security testing is
performed on an instruction basis (often only descriptions of the hardware/firmware-implemented,
but no internal hardware/firmware design details, are available to the test designers). The generation
of test conditions is, consequently, based on instruction and processor documentation (e.g., on
reference manuals). Models of the processor protection mechanisms and top-level specifications of
each processor instruction are seldom available despite their demonstrable usefulness [3 and 4] and
mandatory use [13, class A1] in security testing. Coverage analysis is restricted in practice to
boundary-value coverage for similar reasons.

3.6.5 Test Conditions, Data, and Coverage Analysis for Hardware Security Testing

Lack of DTLS and protection-model requirements for processors' hardware/firmware in the
TCSEC between classes C1 and B3 makes the generation of test conditions for processor security
testing a challenging task (i.e., class A1 requires that FTLS be produced for the user-visible hardware
functions and thus these FTLS represent a source of test conditions). The generation of test data is
somewhat less challenging because this activity is related to a specific coverage analysis method,
namely boundary-value coverage, which implies that the test designer should produce test data for
both positive and negative outcomes of any condition.

Lack of DTLS and of protection-model requirements for processors' hardware and firmware
makes it important to identify various classes of security test conditions for processors that illustrate
potential sources of test conditions. We partition these classes of test conditions into the following
categories: (l) processor tests that help detect violations of TCB isolation and noncircumventability,
(2) processor tests that help detect violations of policy, and (3) processor tests that help detect other
generic flaws (e.g., integrity and denial of service flaws).

3.6.5.1 Test Conditions for Isolation and Noncircumventability Testing

(1) There are tests which detect flaws in instructions that violate the separation of user and TCB
(privileged) domain:

Included in this class are tests that detect flaws in bounds checking CPU and I/O processors,
top- and bottom-of-the-stack frame checking, dangling references, etc. [4]. Tests within this class
should include the checking of all addressing modes of the hardware/firmware. This includes single
and multiple-level indirect addressing [3 and 4], and direct addressing with no operands (i.e., stack
addressing), with a single operand and with multiple operands. Tests which demonstrate that all the
TCB processor, memory, and I/O registers are inaccessible to users who execute nonprivileged
instructions should also be included here.

This class also includes tests that detect instructions that do not perform or perform improper
access privilege checks. An example of this is the lack of improper access privilege checking during
multilevel indirections through memory by a single instruction. Proper page-or segment-presence
bit checks as well as the proper invalidation of descriptors within caches during process switches
should also be tested. All tests should ensure that all privilege checking is performed in all addressing
modes. Tests which check whether a user can execute privileged instructions are also included here.
Examples of such tests (and lack thereof) can be found in [3, 4, 22, and 23].

(2) There are tests that detect flaws in instructions that violate the control of transfer between
domains:

Included in this class are tests that detect flaws that allow anarchic entries to the TCB domain
(i.e., transfers to TCB arbitrary entry points and at arbitrary times), modification and/or
circumvention of entry points, and returns to the TCB which do not result from TCB calls. Tests
show that the local address space of a domain or ring is switched properly upon domain entry or
return (e.g., in a ring-based system, such as SCOMP, Intel 80286-80386, each ring stack segment
is selected properly upon a ring crossing).

(3) There are tests that detect flaws in instructions that perform parameters validation checks:

Included in this class are tests that detect improper checks of descriptor privileges, descriptor
length, or domain/ring of a descriptor (e.g., Verify Read (VERR), Verify Write (VERW), Adjust
Requested Privilege Level (ARPL), Load Access Rights (LAR), Load Segment Length (LSL) in
the Intel 80286-80386 architecture [24], Argument Addressing Mode (AAM) in Honeywell
SCOMP, [22 and 23], etc.).

3.6.5.2 Text Conditions for Policy-Relevant Processor Instructions

Included in this class are tests that detect flaws that allow user-visible processor instructions to
allocate/deallocate objects in memory containing residual parts of previous objects and tests that
detect flaws that would allow user-visible instructions to transfer access privileges in a way that is
inconsistent with the security policy (e.g., capability copying that would bypass copying restriction,
etc.).

This class also includes tests that detect flaws that allow a user to execute nonprivileged
instructions that circumvent the audit mechanism by resetting system clocks and by disabling system
interrupts which record auditable events. (Note that the flaws that would allow users to access audit
data in an unauthorized way are already included in Section 3.6.5.1 because audit data is part of
the TCB.)

3.6.5.3 Tests Conditions for Generic-Security Flaws

Included in this class are tests that detect flaws in reporting protection violations during the
execution of an instruction. For example, the raising of the wrong interrupt (trap) flag during a
(properly) detected access privilege violation may lead to the interrupt (trap) handling routine to
violate (unknowingly) the security policy. Insufficient interrupt/trap data left for interrupt/trap
handling may similarly lead to induced violations of security policy by user domains.

Also included in this class are tests that detect flaws in hardware/firmware which appear only
during the concurrent activity of several hardware components. For example, systems which use
paged segments may allow concurrent access to different pages of the same segment both by I/O
and CPU processors. The concurrent checking of segment privileges by the CPU and I/O processors
should be tested in this case (in addition to individual CPU and I/O tests for correct privilege checks
[3]).

The TCSEC requirements in the area of security testing state that security testing and analysis
must discover all flaws that would permit a user external to the TCB to cause the TCB to enter a
state such that it is unable to respond to communications initiated by other users. At the hardware/
firmware level there are several classes of flaws (and corresponding tests) that could cause (detect)
violations of this TCSEC requirement. The following classes of flaws are examples in this area.
(Other examples of such classes may be found in future architectures due to the possible migration
of operating system functions to hardware/firmware.)

(1) There are tests that detect addressing flaws that place the processors in an "infinite loop" upon
executing a single instruction:

Included in these flaws are those that appear in processors that allow multilevel indirect
addressing by one instruction. For example, a user can create a self-referential chain of indirect
addresses and then execute a single instruction that performs multilevel indirections using that chain.
Inadequate checking mechanisms may cause the processor to enter an "infinite loop" that cannot
be stopped by operating system checks. Lack of tests and adequate specifications in this area are
also explained in [3].

(2) There are tests that detect flaws in resource quota mechanisms:

Included in these flaws are those that occur due to insufficient checking in hardware/firmware
instructions that allocate/deallocate objects in system memory. Examples of such flaws include
those that allow user-visible instructions to allocate/deal locate objects in TCB space. Although no
unauthorized access to TCB information takes place, TCB space may be exhausted rapidly.
Therefore, instructions which allow users to circumvent or modify resource quotas (if any) placed
by the operating system must be discovered by careful testing.

(3) There are tests that detect flaws in the control of object deallocation:

Included in these flaws are those that enable a user to execute instructions that deallocate
objects in different user or TCB domains in an authorized way.

Although such flaws may not cause unauthorized discovery/modification of information, they
may result in denial of user communication.

3.6.6 Relationship Between Hardware/Firmware SecurIty Testing and the TCSEC
Requirements

In this section we review test condition and coverage analysis approaches for hardware/firmware
testing. The security testing requirements for hardware/firmware are partitioned into three groups:
(1) requirements for classes C1 and C2, (2) requirements for classes B1 to B3, and (3) requirements
for class A1. For hardware/firmware security testing, the TCSEC does not allow the derivation of
specific test-condition and coverage-analysis requirements for individual classes below class A1.
The dearth of explicit general hardware/firmware requirements in the TCSEC rules out class-specific
interpretation of hardware/firmware security testing requirements below class A1.

Security Classes C1 and C2

Test Conditions

For security classes C1 and C2, test conditions are generated from manual page descriptions of
each processor instruction, and from the description of the protection mechanisms found in the
processor's reference manuals. The test conditions generated for these classes include those which
help establish the noncircumventability and the isolation (i.e., tamperproofness) of the TCB. These
test conditions refer to the following processor-supported protection mechanisms:

(1) Access authorization mechanisms for memory-bound checking, stack-bound checking, and
access-privilege checking during direct or indirect addressing; and checking the user's inability to
execute processor-privileged instructions and access processor-privileged registers from
unprivileged states of the processor.

(2) Mechanisms for authorized transfer of control to the TCB domain, including those checking
the user's inability to transfer control to arbitrary entry points, and those checking the correct change
of local address spaces (e.g., stack frames), etc.

(3) Mechanisms and instructions for parameter validation (if any).

Other test condition areas, which should be considered for testing the processor support of TCB
noncircumventability and isolation, may be relevant for specific processors.

Test Coverage

The security testing guidelines of the TCSEC require that at least five specific tests be conducted
for class C systems in an attempt to circumvent the security mechanisms of the system. This suggests
that at least five test conditions should be included for each of the three test areas defined above.
Each test condition should be covered by boundary-value coverage to test all positive and negative
outcomes of each condition.

Security Classes B1 to B3

Test Conditions

For security classes B1 to B3, the test conditions for hardware/firmware security testing are
generated using the same processor documentation as that for classes C1 and C2. Additional class-
specific documentation is not required (e.g., DTLS is not required to include hardware/firmware
TCB components that are visible at the TCB interface-unlike class A1).

The test conditions generated for classes B1 to B3 include all those that are generated for classes
C1 and C2. In addition, new test conditions should be generated for the following:

(1) Processor instructions that can affect security policy (e.g., instructions that can allocate/
deallocate memory-if any), and instructions that allow users to transfer privileges between
different protection domains, etc.

(2) Generic security-relevant mechanisms (e.g., mechanisms for reporting protection violations
correctly) and mechanisms that do not invalidate address-translation buffers correctly during process
switches, etc.

(3) Mechanisms that control the deallocation of various processor-supported objects, and those
that control the setting of resource quotas (if any), etc.

The only test conditions that are specific to the B1 to B3 security classes are those for hardware/
firmware mechanisms, the malfunctions of which may allow a user to place the TCB in a state in
which it is unable to respond to communication initiated by users.

Test Coverage

The security testing guidelines of the TCSEC require that at least fifteen specific tests should be
conducted for class B systems in an attempt to circumvent the security mechanisms of the system.
This suggests that at least fifteen test conditions should be included for each of the three test areas
defined above (and for the three areas included in classes C1 through C2). Each test condition should
be covered by boundary-value coverage to test all the positive and negative outcomes of each
condition.

Security Class A1

The only difference between the hardware/firmware test requirements of classes B1 to B3 and
those of class A1 are (1) the processor test conditions derived for classes B1 to B3 (which should
also be included here) are augmented by the test conditions derived from DTLS and FTLS, and (2)
the test coverage should include at least twenty-five test conditions for each test area (included in
classes B1 to B3).

3.7 TEST PLAN EXAMPLES

In this section we present five test plan examples that have been used in security testing. An
additional example is provided to illustrate the notion of cyclic test dependencies and suggest means
for their removal. The first example contains a subset of the test plans for the access kernel primitive
in Secure XenixTM. Here we explain the format of test conditions and of test data necessary for test
conditions and focus on the notion of data flow analysis that might be presented in the coverage
section of test plans.

The second example contains a subset of the test plans for the open kernel primitive of Secure
XenixTM. Here we explain the use of the access-check generated or TCB kernels to eliminate
redundant tests without loss of coverage. In particular, we discuss the impact on test coverage of
the dependency of the open kernel primitive on the access kernel primitive. For example, during
the testing of the access primitive, the subpath starting at the "obj__access" check that includes
the "mand__access" and "discr__access" functions is tested (Figure 2, page 24). Then the open
primitive, which shares that subpath of the graph with the access primitive, need only be tested (1)
for the access check that is not shared with the access primitive, and (2) to demonstrate that the
data flow of open joins that of access. This can be done with only a few test conditions, thereby
reducing the test obligation without loss of coverage.

The third example of this section explains a security test plan for the read kernel primitive of
Secure XenixTM systems. The specifications of the read primitive are access-check dependent on
those of open. This means that a nonempty subset of the access checks necessary for read is done
in open and, therefore, need not be tested again for read. To obtain the same coverage for read as
that for open, or for access, one only needs to test (1) the existence of the access-check dependency
of read on open, and (2) the remaining access checks specific to read (not performed in open).
Since the testing of the access-check dependency requires only a few test conditions, the number
of test conditions for read is reduced significantly without loss of coverage. The subset of the test
plans required for read that are illustrated here is a subset of the test plans required for dependency
verification. in contrast with the first two examples, which contain only mandatory access control
(MAC) test conditions, this example includes some DAC test conditions.

The fourth example presents a subset of the test plans for the kernel and TCB isolation properties
of Secure XenixTM. These test plans are derived from a set of kernel isolation and
noncircumventability requirements for Secure XenixTM and are important for at least three reasons.
First, no formal model exists for these requirements for any system to date. This is true despite the
fundamental and obvious importance of isolation and noncircumventability requirements in
demonstrating the soundness of the reference monitor implementation in a secure system (at any
and all security classes above B2). Second, test plans for these requirements cannot be generated
from DTLS or FTLS of a secure system at any level (i.e., B2 to A1). This is because these
requirements are not necessarily specified in top-level specifications. This is true because isolation
and noncircumventability properties include low-level dependencies on processor architecture
details that do not correspond to the level of abstraction of TCB top-level specifications. Isolation
and noncircumventability properties also cannot be verified formally using the current
methodologies assumed by the tools sanctioned by the NCSC at this time (see the Appendix for the
justification of unmapped kernel isolation code in the SCOMP specification-to-code correspondence
example). Third, the kernel isolation and noncircumventability properties of a system depend to a
large degree on the underlying processor architecture and on the support the architecture provides
for kernel implementation. These test plan examples would therefore necessarily assume a given
processor architecture. (An example of test or verification conditions for such processor mechanisms
is provided by Millen in reference. [19])

In spite of the inherent architectural dependency of kernel isolation properties, we have selected
a few examples of test plans that assume a very simple architecture and therefore can be generalized
to other secure systems. The processor architecture, which is assumed by most implementations of
the machine-dependent code of UnixTM systems, includes only the following: (1) a two-state
processor (i.e., distinguished privileged mode versus unprivileged mode), (2) the ability to separate
kernel address space (e.g., registers and memory) from user space within the same process, which
could ensure that the kernel space cannot be read or written from user-space code, and (3) the ability
to restrict transfers of control to specific entry points into the kernel. Other facilities are not assumed,
such as special instructions that help the kernel and TCB primitive validate parameters and special
gate mechanisms that help distinguish between privileged and nonprivileged kernel invocations.

Of necessity, the test plan examples for the above-mentioned kernel primitives and isolation
examples are incomplete because it would be impractical to include complete test plans for these
kernel primitives here.

The fifth example presents two test plans used for the processors of the Honeywell SCOMP
system [22 and 23]. The first plan includes three test conditions for the ring crossing mechanism
of the SCOMP processor and their associated test data and coverage analysis. The second plan
presents a test condition for which test programs cannot be generated in the normal mode of
processor operation, illustrating the need for design analysis.

The last example of this section illustrates the notion of cyclic test dependencies that appear
among test programs. It also shows how the use of the access-graph and access-check dependencies
in defining test plans (especially coverage analysis) helps eliminate cyclic test dependencies.

3.7.1 Example of a Test Plan for "Access"

The access kernel primitive of Secure XenixTM has two arguments, path and amode. The first
argument represents the name of an object whose access privileges are checked by the primitive,
whereas the second argument represents the privileges to be checked. The following types of objects
can be referenced by the path names provided to access:

Files, Directories, Devices, Named Pipes, Xenix Semaphores, Xenix Shared Data Segments.

The following types of privileges and combinations thereof are checked by access:

Read, Write, Execute (and object's existence).

3.7.1.1 Test Conditions for Mandatory Access Control of "Access"

The following test conditions are derived from the interpretation of the mandatory access control
model [12].

(1) Whenever the type of object named by path is one of the set {File, Directory, Device}, then
the access call succeeds when executed by a process that wants to check the existence of amode =
Read, Execute privileges, or the existence of the object, if process clearance dominates the object
classification; otherwise, the access call fails.

(2) Whenever the type of object named by path is one of the set {Named Pipe, Xenix
Semaphore, Xenix Shared Data Segment}, then the access call succeeds when executed by a process
that wants to check the existence of amode = Read privilege, or the existence of the object, if process
clearance equals the object classification; otherwise, the access call fails.

(3) Whenever the type of object named by path is one of the set {File, Directory, Device,
Named Pipe, Xenix Semaphore, Xenix Shared Data Segment}, then the access call succeeds when
executed by a process that wants to check the existence of amode = Write privilege, or the existence
of the object, if process clearance equals the object classification; otherwise, the access call fails.

3.7.1.2 Test Data for MAC Tests

Environment Initialization

A subset of all clearances and category sets supported in the system is defined in such a way as
to allow all relationships between security levels to be tested (e.g., level dominance, incompatibility,
equality). For example, the chosen levels are UNCLASSIFIED/Null/, UNCLASSIFIED/B/,
CONFIDENTIAL/A, B/, SECRET/Null/, SECRET/All/, TOP SECRET/A/, and TOP SECRET/A,
B/. The security profile of the test operator is defined to allow the test operator to login at all of the
above levels.

A subset of all directory hierarchies supported in the system is defined in such a way as to allow
all relationships between objects of the hierarchy to be tested (e.g., child and parent directories-
see the discussion in Section 3.5.2.2). Three directories, denoted as directory 1, 3, and 6, are created
from the "home" directory at levels UNCLASSIFIED/A, B/, SECRET/A/, and TOP SECRET/A,
B/. Two directories, denoted as directory 4 and 5, are created from the SECRET/A/ directory 3 at
levels SECRET /All/ and TOP SECRET /A/. An additional CONFIDENTIAL /A, B/ directory,
denoted as directory 2, is created from the UNCLASSIFIED/A, B/ directory.

The test operator logs in at each of the above security levels and creates a file in each directory.
The discretionary access privileges are set on every file and directory to allow all discretionary
checks performed by the TCB to succeed. The directory hierarchy is thus created, and the definitions
of the security levels for each file and directory is shown in Figure 4.

Test Documents

The test operator logs in at each of the above security levels and invokes the access call with the
following parameters:

path: Every file pathname defined in the hierarchy.

amode: All access privileges individually and in combination.

Outcomes

Tables 1 and 2 show the expected outcomes of access indicating the correct implementation of
mandatory access checks. Note that in Tables 1 and 2 "Fail l" errors should provide no information
about the nature of the failure. This is the case because these failures are returned whenever the
invoker of access is at a lower level than that of the object being accessed. In particular, "Fail 1"
should not indicate the existence or nonexistence of files at levels that are higher than, or
incompatible with, the login level. Discovery of an object's existence or nonexistence at a higher
level than that of the accessor's would provide a covert storage channel. In contrast, "Fail 2" errors
allow the invoker of access to discover the existence of the file, because his security level dominates
that of the file. No covert channel provided by the object's existence is possible in this case.

3.7.1.3 Coverage Analysis

Model-Dependent MAC Coverage

The test conditions provided above cover all MAC checks for the access primitive. The test data
of this plan, however, cover the test conditions only partially. For example, condition (2) is not
covered at all because the object types included in the test data are only files. Conditions (1) and
(3) are only partially covered for the same reason. Environment reinitialization is necessary to allow
the testing of access with all other types of objects.

The above test data also provide partial coverage because they do not include the hierarchies
shown in Figures 3a and 3b, page 35. Re-execution of the above tests with the hierarchies shown
in Figures 3a and 3b would guarantee sufficient coverage of the MAC model hierarchy. The test
parameters and outcomes will generally differ if the additional tests suggested here are carried out.

Call-Specific MAC Coverage

Let us consider the coverage of individual arc paths and of combinations of arc paths as required
by data flow coverage of the access primitive. The question as to whether the graph of the access
primitive shown in Figure 2, page 24, is covered adequately or redundantly by the above test
conditions and data arises naturally. The following three cases of coverage illustrate primitive-
specific coverage analysis for test condition (1) and test data of Figure 4, page 55, and Table 1, page
57.

Case 1. A Single Arc Path

The test operator logs in at level UNCLASSIFIED/A, l3/ and invokes access with read as a
parameter on the file/home/directoryl/directory2/file2 at level CONFIDENTIAL/A, B/. As shown
in Figure 4 and Table 1, access is at the level of the invoking program (i.e., UNCLASSIFIED/A,
B/) and, therefore, the call to it will fail.

This test provides a single arc path coverage, namely that of arc path access - > "namei" - >
"obj__access" - > "mand__access," shown in Figure 2. Here "mand__access" returns "failure"
when it tries to resolve the file path name. Note that the file path name component "file2" cannot
be read from directory "directory\2" because the mandatory check fails. Note that mandatory checks
on the level of the file itself are also not performed here. The mandatory check failure is caused
earlier by path name resolution and returned to "namei."

Case 2. A Combination of Arc Paths

The test program logs in at level TOP SECRET/A/ and invokes access with Read as a parameter
on the file /home/directory3/file3 at level SECRET/A/. Access is at the level TOP SECRET/A/
(Figure 4 and Table 1), therefore, the call to it will succeed.

This test provides multiple arc path coverage. The first arc path is the same as in Case 1 above.
The "mand__access" check passes, however, and control is returned all the way up to access; see
Figure 2. The second arc path is "access" - > "obj__access" - > "mand access." The mandatory
check in "mand__access" is performed directly on the file and not on its parent directory as in Case
1. This check succeeds. The success result returns to "obj__access" which initiates a third arc path
traversal to "discr__access." The discretionary check passes (as set up in the environment definition)
and success is returned to "obj__access" and access.

Case 3. A Different Combination of Arc Paths

The test program logs in at level SECRET/A/ and invokes access with Read as a parameter on
the directory /home/directory3/directory5 at level TOP SECRET/A/. As shown in Figure 4 and
Table 1, access is at the level SECRET/A/. The call to it will fail.

Although this test appears to provide the same coverage as that of Case 1, in fact it does not. The
first arc path is the same as that in Case 1 above, except that the "mand__access" check on the path
name of the target object (which terminates with name "directory5" in directory/home/directory3)
succeeds and control is returned all the way up to access (see Figure 2, page 24). The second arc
path is then "access" -> "obj__access" -> "mand__access." The check in cmand__access" is
performed directly on the directory /home/directory3/directory5 and, unlike the check in Case 2, it
fails. This "failure" is returned to "obj__access" which reports it to access. Coverage analysis based
on a specific model interpretation in a given TCB primitive would require that the Case 1 test be
repeated with a directory replacing the file "file2." However, this new Case 1 test would become
indistinguishable from that of Case 3 in coverage analysis based on abstract models, and thus Case
3 would not necessarily be tested.

3.7.2 Example of a Test Plan for "Open"

The kernel primitive open has as arguments a path, oflag, and mode. The only relevant object
types named by path for open are the following:

Files, Directories, Devices, and Named Pipes.

The oflag parameter denotes different access privileges and combinations thereof. It also contains
other flags such as "o__excl," "o__creat," and "o__trunc" that specify object locking, default
creation, or truncation conditions. The mode argument of the open primitive is relevant only when
the object does not exist and the "o__creat" flag is used.

3.7.2.1 Test Conditions for "Open"

Test Condition for Access-Graph Dependency

Verify that the open kernel primitive shares the access primitive subgraph that includes the
object__access checks.

Examples of Test Conditions Specific to "Open"

(1) Verify that if the object specified by the path argument does not exist, the object is created with
the access privileges specified by the mode argument whenever the "o__creat" flag is on, with the
owner's user ID and the specified group ID, and with the invoker process' security level.

(2) Verify that if the object specified by the path argument exists, the open kernel primitive succeeds
whenever the requested privilege specified by the "o__flags" is granted to the person with access.
Verify that, in this case, the mode argument also has no effect on the existing privileges of the object.

(3) Verify that the open kernel primitive always fails when it is invoked:

· With the "write" access privilege for a Directory.

· On a nonexisting device.

· On Xenix Semaphores and Xenix Shared Data Segments.

3.7.2.2 Test Data for the Access-Graph Dependency Condition

Environment Initialization Parameters

A subset of all clearances and category sets defined in the tests of access is chosen in such a way
as to allow all relationships between security levels to be tested (e.g., level dominance,
incompatibility, equality). The chosen levels are UNCLASSIFIED/Null/, SECRET/All/, and TOP
SECRET/A/. The subset of the directory hierarchy defined in the tests of access that contains the
objects at the above chosen levels is selected for this test. The definition of the security levels and
discretionary access privileges and the creation and initialization of the object hierarchy are
performed in a similar way to that used in the test of access. The security profile of the test operator
is defined to allow him to login at all of the above levels.

The test operator logs in at each of the chosen security levels and invokes the open primitive with
the following parameters:

path: path names of the three files defined in the hierarchy.

o__flags: o__read, o__write, individually, and in the following combinations:

o__read| o|__excl, o__read|o__trunc, o__read|o__excl|o__trunc,

o__write| o__excl, o__write|o__trunc, o__write|o__excl|o__trunc,

For example, the test operator will use the following login, security level, files, and "o__flag"
parameters:

Case 1

The test program logs in at level SECRET/All/ and invokes open on file /home/directory3/
directory5/file5 at level TOP SECRET/A/ with o__read__only as the call option. Open is at the
level of the invoking program as shown in Table 3, therefore, the call fails.

Case 2

The test program logs in at level SECRET/All/ and invokes open on file /home/file0 at level
UNCLASSIFIED/Null/ with o__read__only as the call option. Open is at the level of the test
program as shown in Table 3, therefore, the call succeeds.

(Note that the above failure and success of the open invocations occur for the same reasons as
those explained in Cases 1 and 2 of the Coverage Analysis area of the access test plan in Section
3.2.4.1.)

Case 3

The test program logs in at level SECRET/All/ and invokes open on file /home/directory3/
directory4/file4 at level SECRET/All/ with "o__read__only" as the call option. Open is at the level
of the test program as shown in Table 4, therefore, the call succeeds.

Tables 3 and 4 show the expected outcome of open indicating the consistency of these outcomes
with those of access. Note that, as in the outcomes of access, "Fail 1,' errors in Tables 3 and 4
provide no information about the nature of the failure, which otherwise might indicate the existence
or nonexistence of files at levels that are higher than, or incompatible with, the login level. In contrast,
"Fail 2" errors allow the invoker of open to discover the existence of the file, because the user
security level dominates that of the file.

3.7.2.3 Coverage Analysis

Model-Dependent MAC Coverage

The testing of the access graph dependency of open on access provides the same model-dependent
MAC coverage for open as that provided for access. That is, after access is fully tested using data
flow coverage, the same coverage is obtained for open. Access-graph dependency testing confirms
that the access subgraph shared by the two primitives, which includes the "obj__access" function,
enforces the MAC policy. Since all object types relevant to open are included among those of access,
and since all access modes of open are included among those of access (the "exclusive" and
"truncation" modes introducing no additional modes independent of read and write), the only
additional model dependent MAC tests necessary for open are those which confirm the access-
graph dependency of open on access for the remaining types of objects (i.e., Directories, Devices,
and Named Pipes).

Call-Specific MAC Coverage

Additional primitive-specific test data are necessary to demonstrate that MAC policy is
discovered by the test plans. Test data, for example, must be provided for test conditions (2) and
(3) above.

3.7.3 Examples of a Test Plan for "Read"

The kernel primitive read used the file descriptor fildes to identify the target object of the read
action. The file descriptor is obtained from the kernel primitives open, creat, dup, fcntl, and pipe.
Since the primitives dup and pipe are not access-control-relevant, and since fcntl is tested
elsewhere, the only primitives and object types relevant to read are the following:

open, creat Files, Directories, Devices, and Named Pipes.

The read primitive uses fildes as one of its parameters, which is obtained from either open or
creat. Thus, read can be called only after either of these two calls have been performed successfully.
This establishes the access-check dependency condition-the only test condition that will be included
in the test plan.

3.7.3.1 Test Conditions for "Read"

For each object type in the set {Files, Directories, Devices, Named Pipes}, verify the following:

(1) That read fails, if neither open nor creat call has been performed before the read call.

(2) That read fails, if neither open nor creat call returned "success" before the read call.

(3) That read succeeds whenever an open call including the read privilege has been
successfully performed (creat has no read option).

3.7.3.2 Test Data for the Access-Check Dependency Condition

Environment lnitialization for Condition (1)

The test operator logs in as uid1.gid1 at a given security level, such as UNCLASSlFlED/Null/,
and attempts to read a file without calling open or creat first. An account for the test operator must
exist. Note that the initialization of the discretionary privileges is irrelevant only for the first
condition. Figure 5a illustrates the initialized environment.

Test Parameters for Condition (1)

After environment initialization, or program calls read with the parameters, illdes = 3, . . . 20.
Note that descriptors 0, 1, 2 are already opened for the standard input, output, and error files and
therefore cannot be used here. Note that fildes 20 is also an invalid file descriptor but is included
here to test that the read call fails when the file descriptor is out of range (i.e., a read specific test).

Outcomes for Condition (1) Tests

Table 5a shows the expected outcomes of the condition (1) test. "S"/ "F" denotes a success/failure
result.

Environment lnitialization for Condition (2)

A file (denoted as file2) is created in the test operator's "home" directory with "read-only"
discretionary access privilege initialized for the test operator. Testing consists of a logon as uid1.gid1
at security level UNCLASSIFIED/NULL/ followed by two attempts to call open and creat in such
a way that these calls fail. These calls will be followed by two attempts to call read with the fildes
presumed to be returned by the calls to open and creat. Figure 5b summarizes the initialization
needed for the required test.

Parameters for Condition (2) Tests

After environment initialization, the test operator or program performs the following actions
using the underlined parameters:

· Open a nonexisting file (file1) using the open call with "o__read__only" flag (open fails
because file1 does not exist). Then call read with the fildes returned by the open call.

· Create a file (denoted as file2) with any arbitrary mode using the creat call (creat fails
because file2 already exists and it has "read-only" privileges for the test operator). Then
the test operator or program calls read with the fildes returned by the creat call.

Outcomes for Condition (2) Tests

Testing demonstrates that read will fail in both cases because both open and creat returned
"failure" earlier. Table 5b shows the expected outcomes.

Environment lnitialization for Condition (3)

Two files (denoted as file1 and file2) are created in the user's "home" directory with "read-only"
and "write-only" discretionary access privileges respectively, defined for the test operator. The file
security levels are defined in such a way that all mandatory access checks succeed on both files.
Testing requires a logon as uid1.gid1 at the security level UNCLASSIFlED/NULL/ followed by
two calls to open and one to creat. Figure 5c describes the data needed for the required tests.

Test Parameters for Condition (3) Test

After environment initialization, the test operator or program performs the following actions with
the following parameters:

· Open file 1 using the open call with "o__ read__only" flag (open succeeds and returns
a valid fildes), then call read with the fildes returned by the open call.

· Open file2 using the open call with "o__read__only" flag (open succeeds and returns
a valid fildes), then call read with the fildes returned by the open call.

· Create a file (denoted as file3) in the test operator's "home" directory with all the
discretionary access modes permitted using the creat call (creat succeeds and returns
a valid fildes), then call read with the fildes returned by the creat call.

Outcomes for Condition (3) Tests

Testing demonstrates that read succeeds when the file descriptors from creat and open include
the read privilege; otherwise, read fails. Since open succeeds before calling the read kernel
primitive, read also succeeds only when an open call was performed with the read option flag;
otherwise, read fails. Although creat succeeds, read still fails because creat always opens an object
for write only, and thus read actions are not permitted. Table 5c shows the expected outcomes.

3.7.3.3 Coverage Analysis

Model-Dependent Coverage

The testing of the access-check dependency of read on open and creat provides the same model-
dependent coverage for read as that provided for open and creat. (Only a subset of this coverage
is explained in Section 3.7.2.3.) The testing of the access-check dependency confirms that the read
primitive cannot succeed unless the access checks that it requires have already been done in open
and creat. Since all object types relevant to read are included among those of open, and since the
read access privilege is covered in open, the only additional model-dependent tests necessary for
read are those performed by hardware (e.g., read authorization checks) and those that confirm the
access-check dependency of read on open and creat for the remaining types of objects (i.e.,
Directories, Devices, Named Pipes).

Call-Specific Coverage

Additional primitive-specific tests may be necessary for read depending upon its implementation.
For example, if the object-limit check performed by read is in any way different from those of other
primitives, it would need to be tested separately. Other conditions referring to object locking may
also be included in these primitive-specific tests.

3.7.4 Examples of Kernel Isolation Test Plans

The test conditions presented in the following example refer to the transfer of control from a
user-level program to the kernel of Secure XenixTM. Such transfers should take place only to entry
points determined by the system design.

The kernel code and data segments of Secure XenixTM are placed in ring (i.e., privilege level 0)
whereas user-level code and data segments are placed in ring 3. User-level programs, therefore,
cannot access kernel programs and data directly without transferring control to kernel programs
first (this property is assured by the processor security testing). The transfer of control from user-
level programs to the kernel can only take place in the following three ways:

· Through calls to a gate in the Global Descriptor Table (GDT), which is located in kernel
address space, via segment selector number 144.

· Through software interrupts controlled by gate as in the Interrupt Descriptor Table (IDT)
located in kernel address space.

· Through traps, which occur as the result of exceptional conditions and which either
cause the kernel to terminate the user process execution immediately or cause the kernel
to receive signals which eventually terminate the user process execution.

The test plans shown below illustrate that the above cases of transfer of control are the only ways
to transfer control to the kernel.

3.7.4.1 Test Conditions

(1) Call Gate. This tests that application programs cannot successfully access any GDT descriptor
except that provided by the segment selector number 144.

(2) Software interrupts. These test that whenever an application program uses the interrupt-
generating instructions "INT n" and "INTO," with n = / = 3,FO-FB, a general protection trap will
occur. For n = 3, the calling process will receive a SIGTRAP signal (a trap signal) and for N = FO-
FB, will cause the instructions following "INT n" to be interpreted as 80287 instructions. (Note:
The 80287 is the arithmetic co-processor.)

(3) Traps. These verify that the occurrence of traps will only affect the trap-generating process.
(Note: This condition cannot be tested by user-level test programs because the traps cause the
termination of the process running the test program. This condition can therefore only be verified
by review of the source code files containing machine dependent code, i.e., mdep/trap.c and mdep/
machdep.asm in Secure XenixTM.)

(4) Call validity. This tests that whenever a user-level program invokes the kernel with an invalid
kernel number, the call will fail.

3.7.4.2 Test Data

Environment Initialization for Conditions (1), (2) and (4)

Compile the test program using "cc -M2es -i" flags. These compilation flags refer to the small
memory model for C programs, with far pointers allowed and with separate instructions and data
segments. The code segment selector number for the program code will be Ox3F. The data segment
selector number will be 0x47. For each test condition, the program forks a child process to perform
each test as described below.

Test Parameters

The following sequences describe the steps of the test programs and the test parameters for each
condition:

(1) Loop with an index value from 0x8 to 0x190 incrementing the index value by 8. Access a
memory location whose segment selector number is provided by the index value.

(2) Loop with an index value from 0 to OxEF incrementing the index value by I. Execute
instruction "INT n" in the loop, where the value of n is provided by the index value.

(3) No test condition or parameters are necessary (namely, the Note of test condition (3) above).

(4) Invoke the kernel gate with the following INVALID kernel call numbers:

· 0x41 (outside of "sysent," the main kernel call table).

· 0x2228 (outside of "cxenix," the XenixTM system call table).

· 0x1740 (outside of "csecurity," the security system call table).

Then invoke the kernel with VALID kernel call numbers representing a user-visible kernel call
and a privileged kernel call.

Outcomes for the Test Conditions (1), (2), and (4) Above

(1) The process running the test program will receive a SIGSEGV signal (a segment violation signal)
for each access call except when the gate selector number is 0x90.

(2) The following is true for the process running the test for the software interrupts:

· Will receive a SIGSEGV signal for each index value except for n = 3.

· Will receive a SIGTRAP signal when n = 3.

· Will not receive any signal when n = 0xF0 - 0xFB, because these index values represent
valid entries for the 80287 arithmetic co-processor.

(3) No outcomes, since no tests are performed.

(4) Error EINVAL (i.e., invalid entry) will be received for all INVALID kernel call numbers (i.e.,
numbers outside the entry ranges of the main kernel call table, the XenixTM system call table, and
the security system call table). No error will be received for the kernel call using the valid kernel
call number (i.e., for any number within the table entry range). An error will be received for the
invocation of any privileged kernel call (primitive).

3.7.4.3 Coverage Analysis

The coverage of the above test conditions is based on boundary-value analysis. The test data
place each test program above (i.e., successful outcome) and below (i.e., unsuccessful outcome)
each boundary condition. The test data and outcomes represent the following degrees of condition
coverage:

(1) All kernel-call gate selection cases are covered.

(2) All software interrupt selection cases are covered.

(3) Not applicable (namely, the Note of condition (3) above).

(4) All the boundary conditions are covered. For complete coverage of each boundary
condition, all privileged kernel calls should be invoked, and all relevant out-of-range call
numbers should be tested. (Such tests are unnecessary because the range tests in kernel code
use the table ranges as defined constants.)

3.7.5 Examples of Reduction of Cyclic Test Dependencies

Consider the structure of typical test programs such as those for the open, read, write, close,
and fcntl TCB primitive of UnixTM to illustrate cyclic test dependencies and their removal. The test
program for each TCB primitive is denoted by tn where n is the first character of the function name.

The test program for open, to, opens an object, for instance a file, writes on it a predetermined
string of characters and closes the file. Then, it opens the file again and reads the file contents to
ensure that the correct file has been opened. Thus, to must invoke the TCB primitive write, close,
read in addition to open. The same sequence of TCB primitives is used for tr and tw to confirm that
the read and write TCB primitives use the correct file. Note that, even if a single file is created in
the test environment, the file system contains other system files that may be inadvertently read by
the kernel. Thus, the use of a predetermined string of characters is still necessary to identify the file
being written or read.

The test program for close, tc, has a similar structure to that of to. After tc opens an object (for
instance a file) and writes on it a predetermined string of characters, tc reads the string and closes
the file. Then tc opens the file again and reads the string. Tc must read the predetermined string of
characters both before closing the file and after reopening the file to ensure that the correct file was
closed. Even though close is a security-model-irrelevant TCB primitive, it must still be tested here
since the test programs to, tr,, and tw rely on it.

The TCB primitives open, read, and write are among the first to be tested because most other
test programs depend on them. If no access-graph or access-check dependencies are used, to, tr , tw
and to depend on each other as shown in Figure 6a. Note that the fcntl TCB primitive could have
been used instead of read in tc. However, this would not have decreased the total number of cyclic
test dependencies because the removed cyclic dependency between tc and tr would have to be
replaced by the cyclic dependency between tc and tfctl.

The structure of the above test programs is not unique. Some of the cyclic test dependencies
presented above, therefore, may not appear in other test programs. Other cyclic test dependencies
similar to the ones shown above, however, will still appear. The reason for this is that kernel isolation
and noncircumventability cause a test program for some TCB primitives to rely on other TCB
primitives, and vice versa, whenever the TCB primitives are tested monolithically.

The use of the access-check graph for testing open eliminates the need to invoke the TCB
primitives read, write, and close in to, and makes to dependent only on ta, the test program for
access. For example, since the access-check graph shows that both open and access use the same
function for the resolution file names [i.e., namei()], the read and write primitives are unnecessary
for file identification because the file name resolution has already been tested by ta. Figure 6b shows
the remaining cyclic dependencies between the test programs after all cyclic dependencies of to are
removed.

The use of the access-check dependencies between TCB primitives also helps remove cyclic
dependencies between test programs. For example, in tests for read and write, only the open TCB
primitive needs to be used to test the existence of the dependency in addition to those necessary to
set up the test environment (e.g., creat). Figure 6c shows the remaining dependencies between the
test programs ta, tr , tw, to, and tfcntl. Note that since test programs to, tr , and tw do not use the TCB
primitive close, and since close is security-model-irrelevant, the testing of close need not be
performed at all. Note that the remaining test dependencies are generally not always identical to
the ones shown in Figure 1, page 23. More dependencies than those shown in Figure 1 will remain
after the new test method is applied.

The example of the remaining test dependencies shown in Figure 6c does not imply that the test
program for access, ta, invokes only access. Ta must also invoke TCB primitives needed to set up
the test environment; therefore, it depends on the test programs for creat (tcr), ACL__control, and
on those of trusted processes login and mkdir. Also, tcr depends on the primitive access because
primitive creat shares a subgraph with access, Figure 2, page 24; therefore, the test program for
creat (tcr) depends on the test program for access. A cyclic test dependency therefore exists between
ta and tcr (not shown in Figure 6c).

To eliminate all such cyclic test dependencies, a small routine with limited functionality, which
is verified independently, could be added to the kernel to read out all the created test environments.
The actions of the test programs that set up the test environments could then be verified
independently. Judicious use of such a limited function read routine and of the new test method
could lead to the elimination of all cyclic test dependencies. The addition of such a routine to the
TCB, which could be done only in maintenance mode, would defeat our goal of test repeatability.

3.7.6 Example of Test Plans for Hardware/Firmware Security Testing

In the SCOMP documentation of processor security testing, the test conditions are identified by
the "verify" clauses. The test data are identified by the "verification" and the associated "test and
verification" (T&V) software description. The coverage analysis is identified by the "conclusion"
associated with each test and source of the "notes" of the T&V software description.

The complete understanding of the test plans presented below requires some knowledge of ring
mechanisms. A good description of the SCOMP ring mechanism can be found in (24]. In the example
presented below, the following abbreviations have been used:

· Ring numbers are represented by the contents of registers Ro-R3 such that R0 < = R1 <
= R2 , = R3, and 0 < Ri < = 3.

· Reff = max(Rcurr, Rcaller) is the effective ring of the accessor, where Rcurr is the current
ring of execution and Rcaller is the ring of the caller program.

· The offset is the entry point into the segment f the called program. (offset = O)

· Rfrom.,(Rto) is the ring from (to) which control is transferred with Rto = < Rfrom calls and
Rto > = Rfrom. for returns.

· The T register contains the segment number of the stack segment associated with the
current ring.

3.7.6.1 Test Conditions for the Ring Crossing Mechanism

(1) Test that the ring-call mechanism changes the ring numbers such that R to < = Rfrom) transfers
to entry point zero (offset = O) of the called-program segment and requires that the execute bit is
turned on in the descriptor for the called-program segment.

(2) Test that the ring-return mechanism changes the ring numbers such that Rto > = Rfrom.

(3) Test that each ring is associated with a different per-ring stack segment after the ring call/return
is made.

3.7.6.2 Test Date

(1) Environment Initialization

The following sections of the T&V software descriptions T200 and T1100 contain the
environment initialization used by the test programs which invoke the SCOMP call (LNJR) and
return (RETN) instructions.

Test and Verification Software Description

T200 TCALL (Test CAll and Return Instructions):

A. Execute return and call instructions between two rings to test a single ring change.

B. Test multilevel ring change. Change rings from ring 0 to ring 3 (one ring at a time), return to
ring 0 in reverse order.

C. Test transfer of T register data on ring changes.

ALGORITHM: TEST NUMBER:

Put known values in T registers

Return to ring 3 (TCALL3) 200

Call to ring 0 (TCALL) 201

Return to ring 1 (TCALL1) 202

Return to ring 2 (TCALL2) 203

Return to ring 3 (TCALL3A) 204

Check T register 205

Call to ring 2 (TCALL2) 206

Check T register 207

Call to ring 1 (TCALL1) 208

Check T register 209

Call to ring 1 (TCALL) 20A

Check T register 20B

Notes:

1. Halt on failures, identify the test failed.

2. For call, set R <= Reff < = R3, offset = 0, and execute permission is "on."

3. Negative tests are not required, these are tested under Trap Handling (namely, T1100 below).

4. Identify controlling descriptors for each test.

5. Inputs identify supporting code and data locations and controlling descriptors.

6. Separate blocks are shown in the structure chart since the process is spread across three rings.

Outside Services Required: None.

Test and Verification Software Description

T1100 TRING (Test Ring Traps)

Execute call using descriptors with the following trap conditions:

A. Reff > R3, segment offset 0 (use page offset 0), execute permission off.

B. Execute return with Rto < Reff.

ALGORlTHM: TEST NUMBER:

Change to ring 1

Call - Eoff 1100

Call - Offset NEO 1101

Call - Reff GT R3(0) 1102

Return - Reff GT Rto 1103

ALGORlTHM: TEST NUMBER:

Change to ring 2

Call - Reff GT R3(0) 1104

Call - Reff GT R3(1) 1105

Return - R,eff GT Rto(0) 1106

Return - R,eff GT Rto, (1) 1107

Change to ring 3

Call - Reff GT R3(0) 1108

Call - Reff GT R3(1) 1109

Call - Reff GT R3(2) 110A

Return - Reff GT Tto (1) 110B

Return - Reff GT Tto,(I) 110C

Return - Reff GT Tto (2) 110D

Change to ring 0 via seg 4

Notes:

1. Halt on failure; identify test failed.

2. Identify controlling descriptors for each test.

3. Provide trap handler that verifies 5PM hardware trap functions and recovers from the trap.
Correct functioning should render the expected trap transparent.

4. All of the tests are executed in ring 3.

Outside Services Required: S1 - Trap Handler (TH14)

(2) Test Parameters

The test programs require no input for these test conditions. The outputs of the test program
represent the test outcomes defined below.

(3) Test Outcomes

Outcomes for Test Condition (1)

· Success: R1 < = Reff <=R3 and offset =0 and E privilege = OFF; (namely, test numbers
201, 206, 208, and 20A).

· Failure: R3> = Reff or offset 0, or E privilege = OFF; (namely, test numbers 1102,
1104-1105, 1108-1110A, or 1101, or 1100).

Outcomes for Test Condition (2)

· Success: R1 Reff (namely, test numbers 200, 202, 203, and 204).

· Failure: Rto < Reff (namely, test numbers 1103,1106,1107,110B-110D).

Outcomes for Test Condition (3)

· Success: T registers contain the stack segment number placed in there in T200. This
outcome is obtained for test numbers 205, 207, 209, and 20B.

· Failure: This outcome is not expected.

3.7.6.3 Coverage Analysis

The test conditions (1)-(3) above have been derived from descriptions of the SCOMP processor
and of the Security Protection Module (SPM). The SCOMP FTLS of the user-visible hardware
functions were either incomplete or unavailable at the time of processor security testing and,
therefore, could not be fully used for the generation of test conditions [3]. Since a formal model of
the protection mechanisms of the SCOMP processor was unavailable, the documentation of the
SCOMP processor and SPM were the only available sources of test conditions.

The test coverage analysis for the conditions (1)-(3) above is based on boundary value coverage.
Note that test condition (1) includes three related subconditions, namely (offset = 0) and (E privilege
= ON). Furthermore, subcondition (Ro <=Reff<) requires at least three calls (i.e., R3 to R1, m R3 to
R1, R3 to R2) be made and that each be combined with subconditions (offset = 0) and E privilege =
ON). Though subcondition R3 > Reff requires that six calls be made (i.e., for Reff > R3 = 0, Reff > R3
= ), 1, Reff> R3 = 0, 1, 2), these subconditions cannot be combined with subconditions (offset 0)
and E privilege = OFF) because all these related subconditions return failure. Test condition (2)
similarly requires that multiple calls be made. It should be noted that for test condition (3) the
boundary-value coverage can only cover the success subcondition in normal mode. The lack of a
current stack segment number in the T register after a call or a return could only happen due to
processor or SPM failures.

Several test conditions may be desired for processor security testing for which test programs
cannot be built in the normal mode of operation. The example of this is provided by the invalidation
of current process descriptions in the processor cache before dispatching the next process. (A
complete test of the invalidation function for descriptions in the cache can be performed in privileged
mode or in ring 0 as outlined in the conclusions below.)

Test Conditions for Descriptor Invalidation

Test that the descriptors contained in the cache are invalidated prior to the dispatch function.

Test Data

A test to insure invalidation of SPM descriptors after dispatch is not in the test software.

Verification by Analysis

The invalidation function of dispatch involves resetting of all SPM cache validity bits for direct
memory descriptors used by the CPU. This requires invalidation of 256 and 64 cache locations in
the Virtual Memory Interface Unit (VMIU) and Descriptor Store boards, respectively. Analysis has
confirmed the proper implementation of this function.

Conclusions

A test to verify the dispatch invalidation function could be implemented by using two descriptor
structures, each mapping CPU memory references to different areas of memory. By checking usage
(U and M bits) of each direct memory descriptor and the actual access to different locations in
memory, the invalidation of previously stored descriptors in the SPM cache could be determined.
The VMIU portion of the test would use a page descriptor structure (256 page located within 16
contiguous segments) with checks provided for each page of memory. The descriptor store board
portion of the test would be constructed in a similar manner except 64 direct segment descriptors
would be used.

3.7.7 Relationship with the TCSEC Requirements

In this section we present the documentation requirements for test plan and procedures stated by
the TCSEC and additional recommendations derived from those requirements. Responsibility for
documenting test plans and procedures belongs both to evaluators and to vendors because security
testing evidence is produced by both for different purposes. It should be noted that the evaluator's
test documentation and programs will not fulfill the vendor responsibility for providing test
documentation and test programs. Wherever appropriate, this section differentiates exclusive
evaluator responsibility from that of the vendors. Citations of specific evaluator responsibility
provided by the TCSEC are omitted here because they are explained in detail in Section 10 of
reference [13].

The introductory section of the test documentation should generally identify the product to be
evaluated and give a high-level description of the system being tested, the types of applications for
which it is designed, and the evaluated product class for which it is being tested.

PURPOSE AND SCOPE OF SECURITY TESTING BY EVALUATORS

A section should state the objectives of security testing conducted by the vendor and describe
the role that this testing plays in supporting the Technical Evaluation Phase by the NCSC. It should
state the purpose of the test plan and how it will be used. It should also define the intended scope
of the effort in terms of both hours of effort and elapsed time, as well as the technical scope of the
effort.

ROLES AND RESPONSIBILITIES OF SYSTEM EVALUATORS

A section should describe how the test team is organized and identify the team leader and all
members by name and organization, qualifications, and experience. Its purpose is two fold. First,
it should clearly delineate team members' responsibilities and relationships. Second, it should
provide sufficient background information on the team's prior functional testing experience to
substantiate that the team is qualified to conduct the tests. It should describe the level of previous
experience that each team member has with the system being evaluated, whether all team members
have completed an internals course for the system, how well the team understands the flaw
hypothesis penetration testing methodology and vulnerability reporting process, and other relevant
qualifications. This section should specifically address test team education, skill, and experience.

A section should also identify any responsibilities for coordination, review and approval of the
test plan, and procedures and reports that lie with personnel outside the test team.

SYSTEM CONFIGURATION

A section should specify the hardware and software configuration used for testing to include the
location of the test site. This configuration should be within the configuration range recommended
by the vendor for approval by the NCSC during the Vendor Assistance Phase. The vendor will be
required to identify and recommend a test configuration to the NCSC early in the evaluation process.
The vendor's recommendation will be considered by the NCSC test team in selecting the "official"
test configuration.

Hardware Configuration

A subsection should identify the CPU, amount of random access memory (RAM), I/O controllers,
disk drives, communications processors, terminals, printers, and any other hardware devices
included in the test configuration by specifying the vendor's model number and quantity of each
configuration item. Each peripheral should be given a unique identifier that associates it with a
specific controller port. Communications parameter settings should be specified where appropriate.
It should be possible to duplicate the test configuration exactly from the information provided.

Software Configuration

A subsection should identify the version of the vendor's operating system included in the test
configuration, as well as each specific TCB software component that is not part of the operating
system. It should include sufficient information to generate the system from the TCB test software
library along with the vendor's distribution tapes. It is very useful to include a summary of device
driver file names and the file system directory structures along with a description of their general
contents.

SECURITY TEST PROCEDURES (TO BE FOLLOWED BY BOTH VENDORS AND
EVALUATORS)

The TCSEC states the following test documentation requirement:

Class C1 to A1. "The system developers shall provide to the evaluators a document that
describes the test plan, test procedures that show how the security mechanisms were tested
and the results of the security mechanisms' functional testing."

A section should provide both an overview of the security testing effort and detailed procedures
for each of the security test plans. Security testing will include detailed procedures for executing
any test plan that is needed to provide significant coverage of all protection mechanisms. This
portion of the test plan must be detailed; it will require the test team to generate the test plans for
each TCB primitive.

Review and Evaluation of Test Documentation for Each TCB Primitive

A subsection will present an evaluation of the method of TCB primitive testing used by the
vendor's development team, the completeness of the coverage provided by the vendor's tests for
the TCB primitive, and any shortfalls that will need to be covered by the security testing team. This
evaluation should include a discussion of the extent to which the vendor's tests used black-box
(which does not necessarily assume any knowledge of system code or other internals) or gray-box
coverage (which assume knowledge of system code and other internals). Black-box test coverage
is best suited for C1 to B1 class systems. Gray-box coverage of a system's security protection with
respect to the requirements in the TCSEC is best suited for B2 to A1 class systems. In terms of
TCB-primitive coverage, this subsection should identify any relevant interfaces or mechanisms that
the vendor has previously failed to test, as well as how thoroughly the vendor has previously tested
each interface or mechanism with respect to each TCSEC requirement.

Test Plans

Test Conditions

This section identifies the test conditions to be covered. It should include explanation of the
rationale behind the selection and the order in which the tests are executed. It is recommended that
the detailed test procedures for each test condition be compiled in annexes in a format that enables
the test personnel to mark steps completed to ensure that procedures are performed correctly.

These test conditions should be derived from interpretations of the following:

· Protection philosophy and resource isolation constraints (for classes C1 and C2).

· Informal security models (class B1).

· DTLS and formal security models (classes B2 and B3), FTLS (class A1).

· Accountability requirements (all classes).

Test Data

The test data should include the definition of the following:

· Environment initialization.

· Test parameters.

· Test outcomes.

Coverage Analysis

The coverage analysis section of a test plan should justify the developer's choice of test conditions
and data, and should delimit the usefulness of the test with respect to security of the system.

Test Procedure Format

Whenever security testing is not automated extensively, the developer's test documentation
should include test scripts. These should contain:

· A description of the environment initialization procedure.

· A description of the execution test procedure.

· A description of the result identification procedure.

Procedure for Correcting Flaws Uncovered During Testing

A subsection should describe the procedure for handling the identification and correction of flaws
uncovered during the course of functional testing. It should specify how this information was
provided to the vendor's test team, how much time was allocated to correct the flaw, and how testing
again was conducted to verify that flaws have been corrected.

An Example Test Report Format

The TCSEC includes the following requirements for reporting the test results:

Classes C1 to A1. "The system developer shall provide to the evaluators a document that
describes [the] results of the security mechanisms' functional testing."

A section should identify the vendor of the evaluated product and give a high-level description
of the system that was tested, the types of applications for which it is designed, and the class for
which it is being evaluated.

Test System Configuration

A section should provide a general description of the test system configuration. It need not be as
detailed as the test plan, but should give enough detail to identify the hardware and software context
for the test results.

Test Chronology

A section should provide a brief chronology of the security testing effort. It should indicate when
testing began, where it was conducted, when each milestone was completed, and when testing was
completed.

Results of Security Testing

A section should discuss each flaw uncovered in the system during security testing. It should
describe any action taken to correct the flaw as well as the results of retesting. It may be useful to
define a "level of criticality" for classifying the flaws in order to distinguish major problems that
might impact the final rating from minor discrepancies or those for which a work-around was found.

List of Uncorrected Flaws

A section should identify any problems that were uncovered during testing that were not corrected
to the test team's satisfaction.

4. COVERT CHANNEL TESTING

Covert channel testing is required in order to demonstrate that the covert channel handling method
chosen by system designers is implemented as intended. These methods include prevention and
bandwidth limitation. Testing is also useful to confirm that the potential covert channels discovered
in the system are in fact real channels. Testing is also useful when the handling method for covert
channels uses variable bandwidth-reduction parameters (e.g., delays) that can be set by system
administrators (e.g., by auditors). Testing can ensure that these mechanisms reduce the channel
bandwidths to the correct limits intended by system administrators.

Bandwidth estimation methods that are necessary for the handling of covert channels may be
based on engineering estimation rather than on actual measurements. Bandwidth estimations
provide upper bounds for covert channels before any handling methods are employed. In contrast,
covert channel testing always requires that actual measurements be performed to determine the
covert channels' bandwidths after the chosen handling method. Similarly, whenever covert channels
are prevented (i.e., eliminated), testing of actual code of the implemented system is required.

4.1 COVERT CHANNEL TEST PLANS

The test plans used for covert channel testing have the same structure as those used for security
functional testing. That is, for testing each covert channel a test condition and the test data should
be written, and coverage analysis should be performed for that channel.

The test conditions for channels differ depending on the choice of the covert channel handling
method. Test conditions would state that no information leakage is possible through the previously
extant channel for covert channels that are eliminated. For covert channels handled by bandwidth
limitation, the condition would state that the measured bandwidth of the respective channel is below
the predicted limit chosen for the channel. If the test is used to measure the bandwidth after nonzero
delay values are used, the predicted bandwidth limit is the target bandwidth chosen or provided by
the default values of the added delays. If the test is used to measure the bandwidth before nonzero
delays are used, the predicted bandwidth is the estimated maximum bandwidth of each channel.

The test data for each channel consists of the test environment definition, the test parameters
used, and the outcomes of the test. The test environment definition consists of a description of the
actual covert channel scenario defining how the sender and recipient processes leak information.
This definition may include a description of the synchronization methods used by the sender and
receiver, the creation and the initialization of the objects (if any) used by the sender/receiver to leak
information, the initialization and resetting of the covert channel variable, etc. If channels are
aggregated serially or in parallel, and if specific encodings are used, the aggregation and encoding
methods should be defined. (Note that neither channel aggregation nor special bit encodings need
to be used in testing as these are neither required nor recommended by either [13] or its covert
channel guidelines.)

It should be noted that in many cases of resource exhaustion channels, the test program need not
actually leak any string of bits. This is acceptable only in cases when the exhaustion of one of these
resources deteriorates system performance to such an extent that no information could possibly be
transmitted within 1 second. In such cases, it is sufficient to measure the elapsed time from the
beginning of the covert channel primitive invocation until the resource exhaustion error is returned
to test the upper bound of the achievable bandwidth.

The test parameters consist of the values set to run the measurements and include the number of
bits leaked for each channel, the distribution of 0's and 1's chosen for the test, the representation
of 0's and 1's (e.g., using the states of the covert channel variable or system objects), the delay
values chosen for testing, the number of objects used and their types and privileges, etc.

The test outcomes specify the expected results of the test. As with test conditions, the test
outcomes are similar for all channels. For each channel, they define the target limit of the actual,
measured channel bandwidth.

Coverage analysis for covert channel testing requires the demonstration that the placement of
delays and randomization points in code covers all information flow paths. Credible assurance of
correct handling of covert channels cannot be provided without such analysis.

To understand the complexity of covert channel testing and the need for covering all information
flow paths, consider a generic example of covert channels provided by a single variable. Assume
that the variable can be viewed (altered) by V (A) primitive calls of the TCB. These primitives can
create up to V x A covert channels. Testing would have to ensure that, if the covert channel handling
method is based on placement of bandwidth reduction delays, the placement of those delays limits
bandwidth of all these covert channels to a specified value. In a system that has N variables (or
attributes) that create covert storage channels, sum__{i = ... .N} (VixAi) test programs would have
to be generated to assure correct placement of delays. For a UnixTM-like system, this would require
approximately 3,000 test plans and programs for covert channel testing, many of which would be
redundant. This would clearly be impractical.

The assurance that covert channel tests cover all possible flows in the TCB can be provided by
(1) a test of a single instance of covert channel usage, which would test that the channel is eliminated
or delayed, and by (2) an analysis that shows that all other instances of the same channel are either
eliminated or have their bandwidth reduced by correct placement of delays. This assurance allows
the elimination of all redundant covert channel tests without loss of coverage.

4.2 AN EXAMPLE OF A COVERT CHANNEL TEST PLAN

In this section we present an example of a test plan for a covert storage channel. This channel,
called the Upgraded Directory Channel, has been described in references [15] and [16]; therefore,
it will not be described here in detail. Measurements and engineering estimations, which predict
the bandwidth of this channel in Secure XenixTM running on a 6 megahertz personal computer AT,
have been reported in reference [16]. Other types of engineering estimations which can determine
the maximum bandwidths of noise less covert channels have been presented in reference [21].

See Section 4.3, "Relationship with the TCSEC Requirements," which contains an example of a
covert channel test plan.

4.2.1 Test Plan for the Upgraded Directory Channel

System Version: PS/2 Model 80.

Covert Channel Type: MAC conflict channel [19].

Variable Name: direct - > d__ino.

4.2.1.1 Test Condition

The test condition for the "upgraded directory channel" is:

The measured bandwidth is lower than the predicted bandwidth after delay is added.

4.2.1.2 Test Data

Environment Initialization

The test operator logs in at a security level called "Low" and initializes a receiver process. Then
the operator logs in at a level called "High" and initializes a sender process. Level High must
dominate level Low. The receiver process creates an upgraded directory at level High and the sender
process, which is at the same level as that of that directory, signals to the receiver process a 1 or a
0 by either creating or not creating an object in that directory. The receiver process may detect 0s
and 1s by trying to remove the upgraded directory. Success or failure of the removal operation
signals 0s or 1s because a directory can only be removed when there is no object in that directory [15].

Note: Both the sender and the receiver use four directories to amortize the synchronization and
environment set up delay for every bit over four bits (i.e., four-bit serial aggregation). This covert
channel scenario is shown in Figure 7.

Parameters

Number of Bits leaked: 8.

Distribution of information used by the test program: 01100011, which represents character "c."

Object Type: directory.

Number of objects used: 4 directories (serial four-bit aggregation).

Measurements: The rmdir (nonempty directory) elapsed time is 3020 ms. (delayed). The rmdlr
(empty directory) elapsed time is 180 ms. The rmdir (average) elapsed time is 1600 ms.

Outcome: The measured bandwidth is less than the predicted bandwidth of 0.566 bit/sec (with
delay). If delay is removed, the predicted bandwidth is 2.8 bits/sec.

4.2.1.3 Coverage Analysis

The trusted process rmdir is the only primitive that reads variables in this covert channel.

4.2.2 Test Programs

The test programs are included in files up/dirs.c and dirr.c (not shown here).

4.2.3 Test Results

The measured bandwidth is 0.5 bit/sec. The reason the test results for the "no delay" case are not
included here is that this delay is built into the system configuration. The auditor cannot turn off or
set the delay.

4.3 RELATIONSHIP WITH THE TCSEC REQUIREMENTS

The TCSEC states the following requirement for the documentation of covert channel testing:

Classes B2 to A1. "The system developers shall provide to the evaluators a document
that ... shall include the results of testing the effectiveness of the methods used to reduce
covert channel bandwidths."

To satisfy this requirement the testing of the covert channel bandwidth must be performed. The
following format is recommended for the documentation of covert channel test plans.

(1) Test Conditions

These conditions should be derived from covert channel handling methods and should include:

· Elimination of covert channel conditions (whenever appropriate).

· Bandwidth limitation conditions based on measurements or engineering estimations.

· Covert channel audit conditions (where appropriate).

(2) Test Data

Test data should include:

· Environment initialization data and/or a brief scenario of covert channel use;

· Test parameter definition.

· Test outcome (a blocked channel, an eliminated channel, or measured bandwidth below
the predicted value).

(3) Coverage Analysis

This analysis should contain an explanation of why the test covers all modes of covert information
leakage through an individual channel, through a channel variable, or through a class of channels.

5. DOCUMENTATION OF SPECIFICATION-TO-CODE
CORRESPONDENCE

The correspondence of the formal specification and source code is also a test documentation
requirement of the TCSEC. The test documentation requirements of the TCSEC state:

Class A1. "The results of the mapping between the formal top-level specification and the
TCB source code shall be given."

This A1-exclusive requirement is only peripherally related to security testing. We have only
included it as an appendix for the interested reader. The detailed set of FTLS-to-code correspondence
requirements is provided by A Guideline to Formal Verification Systems (NCSC-TG-014).

APPENDIX

Specification-to-Code Correspondence

1. Overview

The requirements of the FTLS-to-code correspondence stated in Section 5 define the scope of
the mapping process. The mapping may be informal but it must:

· Show the consistency of the TCB implementation code and FTLS.

· Include all elements of the FTLS in the correspondence.

· Describe code not specified in the FTLS (if any) excluded from the mapping.

Although the mapping may be informal, it is instructive to review its theoretical underpinnings.
These underpinnings are summarized in Sections 1-4 of reference [25] and in Sections III and IV
of reference [26].

Consider two specifications of a finite state machine M denoted by Mf and Mc. The specification
Mf is the FTLS of M, and Mc is the implementation specification (i.e., code) of M. A common thread
of all formal verification methods is the requirement to demonstrate that any state of the machine
specification Mc represents a state of another, more abstract, machine specification Mf . Alternate
methods exist that attempt to formally establish this representation.

The first method is based on defining a function phi with an application to a state Sc of Mc that
yields the "corresponding" state of Mf. The function F defines the mapping between the two machine
specifications. This mapping expresses properties of the correspondence between Mc and Mf. For
example, if the property of Mc is to mimic Mf step by step, the mapping function F should be defined
in such a way that the i-th state of Mf corresponds to the i-th step of Mc. If the notions of a secure
state and state transition are defined in Mf, and if all state transitions of Mf leave it in a secure state,
the property of the function F is defined in such a way that all mapped states of Mc are secure and
all state transitions of Mc leave it in a secure state. In general, the mapping function F should capture
the specific property, or properties, desired for the mapping.

The first mapping method, called "refinement mapping" in reference [25], is applicable to large
classes of problems of establishing code-to-specification correspondence, including
correspondence of concurrent program code. In many cases, however, the refinement mapping
cannot be found. Reference [25] shows that in a very large class of mapping cases it is possible to
augment the implementation specification of Mc (i.e., the code) with extra state components (called
the "history" or "prophecy" variables) in a way that makes it possible to produce refinement
mapping.

The second method is based on defining a function G whose application to an assertion Af defined
on a state of Mf yields an assertion Ac about the state of Mc. This alternate mapping should also
capture similar properties of Mf and Mc as those defined above. These two notions of mapping
defined by F and G are inverses of each other, as argued in reference (16], in the sense that:

For any assertion Af about the states of Mf and any state Sc of Mc, Af is true of state F(Sc) of
Mf if and only if assertion G(Af) is true of state Sc.

Examples of how the two mapping definitions are applied to system specifications and design
are provided in [26]. A further example, which uses similar methods for the generation of correct
implementation code from abstract specifications, is given in [27]. In both references, the mappings
are defined on types, variables, constants, operators (e.g., logic operators), and state transformations.
The common characteristics of all formal mappings are (1) the mapping definition, (2) the
identification and justification of unmapped specifications (if any), (3) the specification of the
properties that should be preserved by the mappings, and (4) the proofs that these properties are
preserved by the defined mappings.

2. Informal Methods for Specification-to-Code Correspondence

Informal methods for FTLS-to-code correspondence attempt, to a significant degree, to follow
the steps prescribed by formal methods. Two informal exercises of FTLS-to-code correspondence
are presented briefly in references [28 and 29], one based on FTLS written in SPECIAL and the
other in Ina Jo. Analysis of both exercises, one of which was carried out on the SCOMP kernel [28],
reveals the following common characteristics.

2.1 Mapping Definition

The mapping units of both FTLS and code are identified and labeled explicitly. For example,
each "processing module" is identified both in FTLS and code. This identification is aided by:

· Intermediate English language specification or program design language specifications,
and/or

· Naming conventions common to FTLS and code (if common conventions are used).
Processing modules are represented by the "transform" sections of Ina Jo and by the
module V, O, and OV functions of SPECIAL.

Alternatively, the mapping units may consist of individual statements of FTLS and
implementation code.

Correspondences are established between similarly labeled mapping units. This is particularly
important for the units that have user visible interfaces. Correspondences include:

· Data structures used by processing modules (namely variables, constants, and types of
the FTLS) are mapped in their correspondent structures of code.

· Effects of processing modules and operators (e.g., logic operators) that are mapped to
the corresponding code functions, procedures, statements and operators.

In addition, whenever the effects sections of a processing module identify exceptions separately,
the correspondence of FTLS exceptions and code exceptions should also be included explicitly.

2.2 Unmapped Specifications

The process of establishing the FTLS-to-code correspondence may reveal that the FTLS has no
corresponding code or has incomplete code specifications. This situation is explicitly excluded by
the TCSEC requirements, because all elements of the FTLS must have corresponding code. More
often, significant portions of the implementation specifications (i.e., code) remain unmapped.
Mismatches between FTLS and implementation specification may occur for many different reasons,
which include:

· FTLS and code are written in languages with very different semantics. This is the case
whenever FTLS are written in nonprocedural languages and code is written in a
procedural language. In this case, the correspondence between the assertions of the
nonprocedural language and the functions, procedures, and control statements of the
procedural language are difficult to establish. Some unmapped implementation code
may represent implementation language detail which is not mapped explicitly to FTLS
and which does not affect adversely the properties preserved by the mapping (discussed
below).

· The domain or range of an FTLS function may be incorrectly identified during code
development. In this case the mapping process should be able to identify the cause of
the FTLS and implementation code mismatch.

· A significant part of the TCB code is not visible at the user interface and thus, has no
correspondent FTLS. This code, which includes internal daemons, such as daemons for
page/segment replacement, daemons that take system snapshots, and so on, is
nevertheless important from a security point of view because it may introduce
information flows between TCB primitives in unexpected ways. The effect of such code
on the mapping, or lack thereof, should be carefully analyzed and documented.

· Unmapped TCB implementation code includes code which ensures the
noncircumventability and isolation of the reference monitor and has no specific
relevance to the security (i.e., secrecy) policy supported. For example, TCB
implementation code which validates the parameters passed by reference to the TCB is
policy independent and may cause no covert channel flows (because all relevant flows
caused by these checks are internal to the process invoking the TCB).

· Unmapped TCB implementation code may include accountability relevant code (e.g.,
audit code), debugging aids, and performance monitoring code, as well as other code
which is irrelevant to the security supported. The presence of such code within the TCB
may introduce information flows within the TCB primitives and may introduce
additional covert channels. The effect of such unmapped code on the mapping should
be analyzed and documented.

· The TCB may contain implementation code that is relevant to the security policy supported
by the system but irrelevant to the properties that could be verified using the FTLS. For
example, the correctness of some of the discretionary access control policies may not
be easily verified with the currently available tools. Therefore, the complete mapping
of code implementing such policies to the corresponding FTLS may have limited value.
However, the information flows generated by such code should be analyzed and
documented.

2.3 Properties Preserved by the Mapping

A key characteristic of any FTLS-to-code mapping is the specification of the security property
of the FTLS that should be included in implementation code. Such security properties include
specifications of MAC and DAC policy components, object reuse components, and accountability
components, all of which are user visible at the TCB interface. These properties should also include
specifications of equivalence between information flows created by FTLS and those created by
implementation functions, procedures, variables and mapped code. Other safety properties and
liveness properties may also be included. For each mapped module, the properties preserved by that
module should be documented.

It must be noted that current emphasis of practical work on FTLS-to-code mapping is exclusively
focused on the maintenance of (1) mandatory access control properties of FTLS in implementation
code, and (2) the equivalence between covert channel flows of the FTLS and those of the
implementation code.

2.4 Correlation Arguments

The documentation of each correspondence between mapping units should include a convincing
argument that the desired properties are implemented by code despite unmapped specifications or
code (if any). Lack of such documentation would cast doubts on the validity of the mapping and
on the usefulness of demonstrating formally such properties of FTLS. For example, little use is
made of the soundness of information flows of FTLS whenever flow equivalence between FTLS
primitives and variables and those of implementation code is not established.

3. An Example of Specification-to-Code Correspondence

The module whose FTLS mapping to implementation code is illustrated in this section is
"get__segment__access" system call of the Honeywell's Secure Communication Processor
(SCOMP). The FTLS is written in SPECIAL, the language supported by the Hierarchical
Development Methodology (HDM) developed at SRI International, and the implementation code
is written in UCLA Pascal. The system call "get__segment__access" returns the access privileges
of the invoking process (e.g., user process) for a uniquely identified segment. The effect of this call
is similar to that of the access system call of UnixTM when applied to files, namely Tables 1 and 2,
page 58. Figure 8 below shows the FTLS of "get__segment__access" and Figures 9a; 9b, parts 1
and 2; and 9c show its implementation code. Note that Figure 9a identifies the "def.h" file, namely
the file of included header definitions of the module, Figure 9b, parts 1 and 2, contain the actual
code of the module (i.e., in the ".p" file), and Figure 9c contains the code of implementation function
"get__segment__info," which is invoked by the code of "get__seg__access."

3.1 Mapping Definition

Mapping Units

The mapping units for both the FTLS and the implementation code of SCOMP are the individual
language statements. To establish the mapping each statement of the implementation code is labeled
unambiguously (i.e., using the code or data file name and the statement line number). Statement
level labeling of data definitions (i.e., "def.h" files) and code (i.e., ".p" files) is shown in Figures
9a; 9b, parts 1 and 2; and 9c.

Correspondence of Labeled Units

The statement level mapping of FTLS to code is established in SCOMP by adding to each
SPECIAL statement of the "get__segment__access" module the corresponding individual (or group
of) UCLA Pascal statement(s). Figure 8 shows this.

LEGEND

^ = pointer to

!= = not equal

~ = negation

User Visible Effects, Exceptions, and Data Structures

The only user visible effect of this VFUN (i.e., state returning function) is mapped to the language
statements "segment.p 931-939" as part of the nondiscretionary access check performed to
determine whether the calling process has MAC access to the segment passed as a parameter.
Whenever this check is passed (in "segment.p 931"), the accesses of the caller process to the segment
are returned through "seg__access__p" parameter. Note that the UCLA Pascal function
"non__discretionary__access__allowed(...)" is mapped to the SPECIAL function
"valid__flow(...)." The former calls the UCLA Pascal version of the latter (neither shown here). In
addition, the function "non__discretionary__access__allowed(...)" also performs checks to
determine whether the invoking process has special system privileges that would allow it to bypass
the MAC checks of "valid__flow(...)." Since the properties of interest to the FTLS verification do
not include the effects of the system privileges, only the SPECIAL function corresponding to
"valid__flow(...)" is used in the VFUN "get__segment__access" (namely, comment in the FTLS).
Note that the derived SPECIAL function "get__object__access" corresponds to the UCLA Pascal
function "get__segment__info," defined in "segment.p 945-978" and invoked in "segment.p 914-
917," and that both are invisible at the TCB interface when used in the corresponding modules
"get__segment__access" and "get__seg__access."

The two visible exceptions of the VFUN, namely "invalid__segment__name" and
"segment__does__not__exist," are mapped to the exceptions with the same name of the UCLA
Pascal code found in statements "segment.p 908, 911, 922, and 938."

The only visible data structures are the parameters exchanged by the caller process and the VFUN
module. The mapping of these parameters is shown in the header file at lines "def.h 370 and 372."

3.2 Unmapped Implementation Code

The following lines of UCLA Pascal code have no correspondent SPECIAL code:

· segment.p 888-898-Implementation language detail (i.e., declarations of function
parameters and internal system data structures).

· segment.p 901-904 (and 954-957)-Conditional compilation of debugging code}.

· segment.p 905-906 and 940-942-Implementation code of the reference monitor
mechanism (i.e., code that validates parameters passed by reference that helps maintain
noncircumventability and isolation properties).

· segment.p 912-913, 920-921, 924-930-Implementation details referring to internal
data structures that remain invisible to the user interface. Note the use of locking code,
which ensures that internal sequences of kernel actions cannot be interrupted.

· segment.p 900, 943-Implementation language detail (i.e., control statements).

3.3 List of Properties Preserved by the Mapping

The properties preserved by the mapping are:

· Mandatory Access Control to objects of type p segment.

· Equivalence of information flows visible at the TCB interface.

3.4 Justification for the Maintained Properties and for Unmapped Code

MAC Properties of Segments

In both the SPECIAL FTLS and UCLA Pascal code versions of "get__segment__access," control
returns suCcessfully to the invoking process only if the "valid__flow" and the "non
disCretionary__access__allowed" checks pass. As explained above, these checks are equivalent
from an unprivileged user's point of view. Furthermore, in both the FTLS and code versions, the
unsuccessful returns are caused by the same sets of exception checks, namely (1) wrong object
type, segment is not in the required file system partition (consistency checks); and (2) unmounted
segment, failed MAC check, and inexistent segment (MAC relevant checks). The two additional
exception checks present in the implementation code are not MAC specific checks. Instead, they
are checks of the reference monitor mechanism (e.g., parameter validation), and thus irrelevant for
MAC property verification.

Equivalence of Information Flows

The only visible flows of information through the interface of the "get__segment__access"
module are those provided by the successful and the unsuccessful returns. These returns take place
in identical FTLS and code conditions (namely, the mapping definition documented above and
Figures 8; 9a; 9b, parts 1 and 2; and 9c). The additional exception returns of the implementation
code to the invoker (i.e., the parameter validation exceptions) cannot introduce flows between
different processes. Therefore, the equivalence of the FTLS (SPECIAL) flows and the
implementation code (UCLA Pascal) flows is preserved.

Justification for Unmapped Code

The unmapped code cannot affect the mapping properties that must be preserved for the following
reasons:

· The syntax of the parameter declarations and of the control statements are property
irrelevant language details.

· The debugging code is not compiled in the TCB in the normal mode of operation.

· The code implementing the reference monitor checks is not specific to either of the above
properties (although the functional correctness of these checks is required for secure
system operation, such proof of correctness is not required for A1 systems currently).

· The code which implements internal kernel actions in a manner that cannot be interrupted
is not visible at the TCB interface (although its functional correctness is required in
secure systems, it is not always demonstrable using currently approved tools for A1
systems).

GLOSSARY

Access

A specific type of interaction between a subject and an object that results in the flow of information
from one to the other.

Administrative User

A user assigned to supervise all or a portion of an ADP system.

Audit

To conduct the independent review and examination of system records and activities.

Audit Trail

A set of records that collectively provides documentary evidence of processing used to aid in
tracing from original transactions forward to related records and reports and/or backwards from
records and reports to their component source transactions.

Auditor

An authorized individual, or role, with administrative duties, which include selecting the events
to be audited on the system, setting up the audit flags which enable the recording of those events,
and analyzing the trail of audit events.

Authenticate

To establish the validity of a claimed identity.

Authenticated User

A user who has accessed an ADP system with a valid identifier and authentication combination.

Bandwidth

A characteristic of a communication channel that is the amount of information that can be passed
through it in a given amount of time, usually expressed in bits per second.

Bell-LaPadula Model

A formal state transition model of computer security rules. In this formal model, the entities in
a computer system are divided into abstract sets of subjects and objects. The notion of a secure state
is defined and it is proven that each state transition preserves by moving from secure state to secure
state, thus inductively proving that the system is secure. A system state is defined to be "secure" if
the only permitted access modes of subjects to objects are in accordance with a specific security
policy. In order to determine whether or not a specific access mode is allowed, the clearance of a
subject is compared to the classification of the object and a determination is made as to whether
the subject is authorized for the specific access mode. The clearance/classification scheme is
expressed in terms of a lattice. (Also see Lattice).

Channel

An information transfer path within a system. May also refer to the mechanism by which the
path is effected.

Covert Channel

A communication channel that allows a process to transfer information in a manner that violates
the system's security policy. (Also see Covert Storage Channel and Covert Timing Channel.)

Covert Storage Channel

A covert channel that involves the direct or indirect writing of a storage location by one process
and the direct or indirect reading of the storage location by another process. Covert storage channels
typically involve a finite resource (e.g., sectors on a disk) that is shared by two subjects at different
security levels.

Covert Timing Channel

A covert channel in which one process signals information to another by modulating its own use
of system resources (e.g., CPU time) in such a way that this manipulation affects the real response
time observed by the second process.

Coverage Analysis

Qualitative or quantitative assessment of the extent to which the test conditions and data show
compliance with required properties, e.g., security model and TCB primitive properties, etc. (Also
see Test Condition and Test Data.)

Data integrity

The state that exists when computerized data are the same as those that are in the source documents
and have not been exposed to accidental or malicious alteration or destruction.

Descriptive Top-Level Specification (DTLS)

A top level specification that is written in a natural language (e.g., English), an informal program
design notation, or a combination of the two.

Discretionary Access Control (DAC)

A means of restricting access to objects based on the identity of subjects and/or groups to which
they belong or on the possession of a ticket authorizing access to those objects. The controls are
discretionary in the sense that a subject with a certain access permission is capable of passing that
permission (perhaps indirectly) onto any other subject.

Dominate

Security level S1 is said to be the dominate security level if the hierarchical classification of S1
is greater than or equal to that of S2 and the nonhierarchical categories of S1 include all those of
S2 as a subset.

Exploitable Channel

Any channel that is usable or detectable by subjects external to the Trusted Computing Base.

Flaw

An error of commission, omission, or oversight in a system that allows protection mechanisms
to be bypassed.

Flaw Hypothesis Methodology

A system analysis and penetration technique where specifications and documentation for the
system are analyzed and then flaws in the system are hypothesized. The list of hypothesized flaws
is prioritized on the basis of the estimated probability that a flaw actually exists and, assuming a
flaw does exist, on the ease of exploiting it and on the extent of control or compromise it would
provide. The prioritized list is used to direct the actual testing of the system.

Formal Proof

A complete and convincing mathematical argument, presenting the full logical justification for
each proof step and for the truth of a theorem or set of theorems. The formal verification process
uses formal proofs to show the truth of certain properties of formal specification and for showing
that computer programs satisfy their specifications.

Formal Security Policy Model

A mathematically precise statement of a security policy. To be adequately precise, such a model
must represent the initial state of a system, the way in which the system progresses from one state
to another, and a definition of a "secure" state of the system. To be acceptable as a basis for a TCB,
the model must be supported by a formal proof that if the initial state of the system satisfies the
definition of a "secure" state and if all assumptions required by the model hold, then all future states
of the system will be secure. Some formal modeling techniques include state transition models,
temporal logic models, denotational semantics models, and algebraic specification models.

Formal Top-Level Specification (FTLS)

A Top Level Specification that is written in a formal mathematical language to allow theorems
showing the correspondence of the system specification to its formal requirements to be
hypothesized and formally proven.

Formal Verification

The process of using formal proofs to demonstrate the consistency (design verification) between
a formal specification of a system and a formal security policy model or (implementation
verification) between the formal specification and its program implementation.

Functional Testing

The portion of security testing in which the advertised features of a system are tested for correct
operation.

Lattice

A partially ordered set for which every pair of elements has a greatest lower bound and a least
upper bound.

Least Privilege

This principle requires that each subject in a system be granted the most restrictive set of privileges
(or lowest clearance) needed for the performance of authorized tasks. The application of this
principle limits the damage that can result from accident, error, or unauthorized use.

Mandatory Access Control (MAC)

A means of restricting access to objects based on the sensitivity (as represented by a label) of the
information contained in the objects and the formal authorization (i.e., clearance) of subjects to
access information of such sensitivity.

Multilevel Device

A device that is used in a manner that permits it to simultaneously process data of two or more
security levels without risk of compromise. To accomplish this, sensitivity labels are normally stored
on the same physical medium and in the same form readable by machines or humans as the data
being processed.

Object

A passive entity that contains or receives information. Access to an object potentially implies
access to the information it contains. Examples of objects are records, blocks, pages, segments,
files, directories, directory trees, and programs, as well as bits, bytes, words, fields, processors,
video displays, keyboards, clocks, printers, and network nodes, etc.

Process

A program in execution. It is completely characterized by a single current execution point
(represented by the machine state) and address space.

Protection Critical Portions of the TCB

Those portions of the TCB, the normal function of which is to deal with the control of access
between subjects and objects.

Read

A fundamental operation that results only in the flow of information from an object to a subject.
Read Access (Privilege) Permission to read information.

Security Level

The combination of a hierarchical classification and a set of nonhierarchical categories that
represents the sensitivity of information.

Security Policy

The set of laws, rules, and practices that regulate how an organization manages, protects, and
distributes sensitive information.

Security Policy Model

An informal presentation of a formal security policy model.

Security Relevant Event

Any event that attempts to change the security state of the system, e.g., change discretionary
access controls, change the security level of the subject, or change a user's password, etc. Also, any
event that attempts to violate the security policy of the system, e.g., too many attempts to login,
attempts to violate the mandatory access control limits of a device, or attempts to downgrade a file,
etc.

Security Testing

A process used to determine that the security features of a system are implemented as designed
and that they are adequate for a proposed application environment.

Single Level Device

A device that is used to process data of a single security level at any one time. Since the device
need not be trusted to separate data of different security levels, sensitivity labels do not have to be
stored with the data being processed.

Subject

An active entity, generally in the form of a person, process, or device that causes information to
flow among objects or changes the system state. Technically, a process/domain pair.

Subject Security Level

A subject's security level is equal to the security level of the objects to which it has both read
and write access. A subject's security level must always be dominated by the clearance of the user
the subject is associated with.

TCB-primitive

An operation implemented by the TCB whose interface specifications (i.e., names, parameters,
effects, exceptions, access control checks, errors, and calling conventions) are provided by system
reference manuals or DTLS/FTLS as required.

Test Condition

A statement defining a constraint that must be satisfied by the program under test.

Test Data

The set of specific objects and variables that must be used to demonstrate that a program produces
a set of given outcomes.

Test Plan

A document or a section of a document which describes the test conditions, data, and coverage
of a particular test or group of tests. (Also see Test Condition, Test Data, and Coverage Analysis.)

Test Procedure (Script)

A set of steps necessary to carry out one or a group of tests. These include steps for test
environment initialization, test execution, and result analysis. The test procedures are carried out
by test operators.

Test Program

A program which implements the test conditions when initialized with the test data and which
collects the results produced by the program being tested. Top Level Specification (TLS) is a
nonprocedural description of system behavior at the most abstract level. Typically a functional
specification that omits all implementation details.

Trusted Computer System

A system that employs sufficient hardware and software integrity measures to allow its use for
simultaneously processing a range of sensitive or classified information.

Trusted Computing Base (TCB)

The totality of protection mechanisms within a computer system-including hardware, firmware,
and software-the combination of which is responsible for enforcing a security policy. It creates a
basic protection environment and provides additional user services required for a trusted computer
system. The ability of a trusted computing base to correctly enforce a security policy depends solely
on the mechanisms within the TCB and on the correct input by system administrative personnel of
parameters (e.g., a user's clearance) related to the security policy.

Trusted Path

A mechanism by which a person at a terminal can communicate directly with the Trusted
Computing Base. This mechanism can only be activated by the person or the Trusted Computing
Base and cannot be imitated by those untrusted. Any person who interacts directly with a computer
system.

Verification

The process of comparing two levels of system specification for proper correspondence (e.g.,
security policy model with top level specification, TLS with source code, or source code with object
code). This process may or may not be automated.

Write

A fundamental operation that results only in the flow of information from a subject to an object.

Write Access (Privilege)

Permission to write to an object.

REFERENCES

1. Howden, W.E., "The Theory and Practice of Functional Testing," IEEE Software, September
1985, pp. 18-23.

2. Haley, C.J. and Mayer, F.L., "Issues on the Development of Security Related Functional Tests,"
Proceedings of the Eighth National Computer Security Conference, National Bureau of
Standards, Gaithersburg, Maryland, September 1985.

3. Gligor, V.D., "An Analysis of the Hardware Verification of the Honeywell SCOMP," Proceedings
of the IEEE Symposium on Security and Privacy, Oakland, California, April 1985.

4. Gligor, V.D., "The Verification of the Protection Mechanisms of High Level Language
Machines," International Journal of Computer and Information Sciences, Vol. 12, No. 4, August
1983, pp. 211-246.

5. Petschenik, N., "Practical Priorities in System Testing," IEEE Software, September 1985, pp.
1-23.

6. Myers, G.J., The Art of Software Testing, John Wiley and Sons, New York, 1979.

7. Howden, W.E., "Functional Program Testing," IEEE Transactions on Software Engineering,
Vol. SE-6, No. 3, May 1980, pp. 162-169.

8. Clark, D., "Ancillary Reports: Kernel Design Project," M.I.T. Laboratory Computer Science,
Cambridge, Massachusetts, Technical Memo 87, June 1977.

9. Gligor, V.D.; Chandersekaran, C.; Jiang, W.D.; Johri, A.; Luckenbaugh, G.L. and Reich, L.E.,
"A New Security Testing Method and Its Application to the Secure Xenix Kernel," IEEE
Transactions on Software Engineering, Vol. SE13, No. 2, February 1987, pp. 169-183.

10. Laski, J.W. and Korel, B., "A Data Flow Oriented Program Testing Strategy," IEEE Transactions
on Software Engineering, Vol. SE-9, No. 3, May 1983, pp. 347-354.

11. Rapps, S. and Weyuker, E.J., "Data Flow Analysis Techniques for Test Data Selection,"
Proceedings of the Sixth International Conference on Software Engineering, 1982, pp. 272-278.

12. Luckenbaugh, G.L.; Gligor, V.D.; Dotterer, L.J.; Chandersekaran, C.S. and Vasudevan, N.,
"Interpretation of the Bell-LaPadula Model in Secure Xenix," Proceedings of the Ninth National
Computer Security Conference, Gaithersburg, Maryland, September 1986.

13. Department of Defense Trusted Computer System Evaluation Criteria, DoD 5200.28-STD,
December 1985 (supersedes CSC-STD-001 -83, dtd 15 Aug 83), Library No. 5225,711.

14. Department of Defense-ADP Security Manual-Techniques and Procedures for
Implementing, Deactivating, Testing, and Evaluating Secure Resource Sharing ADP Systems,
DoD 5200.28-M, revised June 1979.

15. Gligor, V.D.; Chandersekasan, C.S.; Jiang, W.D.; Johri, A.; Luckenbaugh, G.L. and Vasuderan,
N., "Design and Implementation of Secure Xenix," IEEE Transactions on Software Engineering,
Vol. SE-13, No. 2, February 1987, pp. 208-221.

16. Tsai, C.R. and Gligor, V.D., "A Bandwidth Computation Model for Covert Storage Channels
and its Applications," Proceedings of the IEEE Symposium on Security and Privacy, Oakland,
California, April 1988.

17. Aerospace Report No. TOR-0086 (6777-25)1, "Trusted Computer System Evaluation
Management Plan," 1 October 1985.

18. National Computer Security Center, Trusted Product Evaluations-A Guide For Vendors,
NCSC-TG-002, Version-1, 22 June 1990.

19. Millen, J.K., "Kernel Isolation for the PDP-11/70," Proceedings of the IEEE Symposium on
Security and Privacy, Oakland, California, 1982.

20. Cugini, J.A.; Lo, S.P.; Hedit, M.S.; Tsai, C.R.; Gligor, V.D.; Auditham, R. and Wei, F.J.,
"Security Testing of AIX System Calls using Prolog," Proceedings of the Usenix Conference,
Baltimore, Maryland, June 1989.

21. Millen, J.K., "Finite-State Noiseless Covert Channels," Proceedings of the Computer Security
Foundation Workshop II, Franconia, New Hampshire, June 1989. (IEEE Catalog Number
89TH02550)

22. Carnall, J.J. and Wright, A. F., "Secure Communication Processor Hardware Verification
Report," Technical Report, Honeywell Inc., Program Code No. 7P10, prepared for Contract No.
HAVELEX N00039-77-C-0245.

23. Honeywell, Inc., "Secure Communication Processor-Test and Verification Software
Description," Technical Report, Rev. 3, April 1980, Program Code No. 7P10, prepared for Contract
No. NAVELEX N00039-77-C-0245.

24. Vickers-Benzel, T., "Overview of the SCOMP Architecture and Security Mechanisms," The
MITRE Corporation, Technical Report, MTR-9071, September 1983.

25. Abadi, M. and Lamport, L., "The Existence of Refinement Mappings," Research Report 29,
Systems Research Center, Digital Equipment Corporation, August 1988.

26. Berry, D.M., "Towards a Formal Basis for the Formal Development Method and the Ina Jo
Specification Language," IEEE Transactions on Software Engineering, VoL. SE-13, No. 2,
February 1987, pp. 184-201.

27. Yu, C.F. and Gligor, V.D., "A Formal Specification and Verification Method for the Prevention
of Denial of Service in ADATM Services," Institute for Defense Analyses, Paper No. P-2120, July
1988.

28. Vickers-Benzel, T., "Analysis of a Kernel Verification," Proceedings of the IEEE Symposium
on Security and Privacy, Oakland, California, April 1984.

29. Solomon, J., "Specification-to-Code Correlation," Proceedings of the lEEE Symposium on
Security and Privacy, Oakland, California, April 1982.