-
Task
-
Resolution: Unresolved
-
P3
-
jfx14
We have many tests that are currently ignored, especially in the "controls", "graphics", and "systemTests" projects. The task tracks the effort of evaluating why the tests are ignored, fixing them if possible, and filing a bug to track it if not.
We should start by doing the following for each area that we will evaluate (I will file a sub-task for each of controls, graphics, and maybe web to record the results of each area. This will include the system tests related to that area.
The main task will be:
1. Remove the `@Ingore` tags from all tests in an area, do a test run
2. Record the results.
3. Analyze the results and file bugs
After running an analyzing the tests, please record your results in the appropriate sub-task. Then you can file new bugs relating to the test failures. We don't need a separate bug for each test method, but instead a single bug for a related group of tests (use your judgment as to the granularity of number of tests per bug). If there is already a bug filed for a test failure, then note that in your report and don't file a new bug (but do link it to the appropriate sub-task).
The next step will be fixing the bugs. Any commit that is pushed to the repo needs to be done using a different bug ID and *not* using this Task ID or any of the Sub-task IDs. This task, and its sub-tasks, will be used to track the failures. Each new bug that is filed should be linked to the component-specific sub-task in question.
One of the following things might be done for each of the failing tests.
A. If a test passes on all platforms, and is stable, then the @Ignore can be removed.
B. If the test can be made to run on all platforms by just changing the test itself, then that change can be done along with removing the @Ignore.
C. If the test can be made to run on some platforms, then the failing platform(s) can be skipped with `assumeTrue`
D. If the test is inherently unstable, or if it is failing as a result of a product bug, then the bug ID of the bug (test or product) that is causing the failure can be added as a string argument to the `@Ignore` (if it isn't already there).
We should start by doing the following for each area that we will evaluate (I will file a sub-task for each of controls, graphics, and maybe web to record the results of each area. This will include the system tests related to that area.
The main task will be:
1. Remove the `@Ingore` tags from all tests in an area, do a test run
2. Record the results.
3. Analyze the results and file bugs
After running an analyzing the tests, please record your results in the appropriate sub-task. Then you can file new bugs relating to the test failures. We don't need a separate bug for each test method, but instead a single bug for a related group of tests (use your judgment as to the granularity of number of tests per bug). If there is already a bug filed for a test failure, then note that in your report and don't file a new bug (but do link it to the appropriate sub-task).
The next step will be fixing the bugs. Any commit that is pushed to the repo needs to be done using a different bug ID and *not* using this Task ID or any of the Sub-task IDs. This task, and its sub-tasks, will be used to track the failures. Each new bug that is filed should be linked to the component-specific sub-task in question.
One of the following things might be done for each of the failing tests.
A. If a test passes on all platforms, and is stable, then the @Ignore can be removed.
B. If the test can be made to run on all platforms by just changing the test itself, then that change can be done along with removing the @Ignore.
C. If the test can be made to run on some platforms, then the failing platform(s) can be skipped with `assumeTrue`
D. If the test is inherently unstable, or if it is failing as a result of a product bug, then the bug ID of the bug (test or product) that is causing the failure can be added as a string argument to the `@Ignore` (if it isn't already there).
1.
|
Evaluate ignored controls tests |
|
Resolved | Ajit Ghaisas | |
2.
|
Evaluate ignored graphics tests |
|
Resolved | Ambarish Rapte | |
3.
|
Evaluate ignored web tests |
|
Resolved | Arun Joseph (Inactive) |