1 Test Methodology

Simple Web Server Test Plan

Objective

To specify a range of test, and the success metrics, required to qualify the release of the functionality provided by the Simple Web Server Tool ( https://bugs.openjdk.java.net/browse/JDK-8260510)

The range of testing will cover functional tests, reliability tests, stress (stability), scale, and security tests. Additionally, elementary performance analysis and code coverage analysis will be carried.

Testing Scope

What's in scope

• Functional Tests, reliability tests, stress and stability tests, scale tests, interoperability tests and security test. The latter taking guidance from the OWASP framework - Top ten web security issues https://owasp.org/www-project-top-ten/ and test guide https://owasp.org/www-project-web-security-testing-guide/

  • API test. The SWS exposes internal APIs, as such a full range of API tests will be developed similar to JCK level testing

What's out of scope

• Formal JCK testing. However, there will be equivalent API level tests.

2 Test Inventory

Regression Tests Inventory

1. Functional tests

Component Functional test cases

tests	Count
????
????

Command line test cases

tests	Count
????
????

2. API tests

The SWST has an API, and this should be extensively tested for functional correctness and operational rubustness.

JCK equivalent tests

tests	Count
????	??
????	??

3. SPI tests

SPI tests

tests	Count
????	??
????	??

4. HTTP Response Conformance tests

Establish that the server returns the correct HTTP responses for relevant

HTTP Resposne tests

tests	Count
????	??
????	??

5. HTML Response Conformance tests

Validate that any HTML response bodies are well formed and correct HTML validation

HTML Response Validation tests

tests	Count
????	??
????	??

6. HTML Response WCAG Conformance tests

Validate that any HTML response bodies are valid as per WCAG (Web content accessibility guidelines - e.g. screenreaders)

HTML Response Validation tests

tests	Count
????	??
????	??

Other Test Inventory

Stability, Reliability and Scale Tests

Reliability is defined as the probability that a product, system, or service will perform its intended function adequately for a specified period of time, or will operate in a defined environment without failure.

Stability is a complex term but for the purposes of this use case can be defined as reliability at a given scale

The main objective is to establish that the SWST performs reliably and stable under known conditions. Also, that a know set of metrics are establish in terms of scale, performance and throughput. In performing reliably it should be able to provide continuous service of a defined number of days, and should not exhibit any appreciable or excessive resource retention, in particular excessive memory growth. Tests to establish the number of connections and number of requests per second the SWST will handle

Reliability tests scenarios

tests	Count
???	???

Stability tests scenarios

tests	Count
???	???

Scale Metrics tests

tests	Count
???	???

Performance Metrics Tests

Stress tests

tests	Count
???	???

Request Performance Metrics tests

tests	Count
???	???

Microbenchmark Tests

TBD

tests	Count
???	???

Interoperability Tests

TBD

• Java HttpClient
• Apache HttpClient
• curl
• Basic Common Browser testing

4. Others

investigate basic security and vulnerability tests - OWASP Zed Attack Proxy tests

OWASP ZAP Tests

tests	Count
???	???

Success criteria

Regression Tests

Criteria	Planned	Status	Result	Comments
Public API coverage	100%	100%
Block coverage of new code	??%	??%
Test stability	SBR runs and ATR			make sure there are no intermittent failures. All good in ATR and SBR runs
Open test issues				makes sure all the tests are ready
Pass Rate	100%

Test Configurations

N/A