2018-04-19 Android CTS测试扫盲

CTS是Compatibility Test Suite（即兼容性测试包）的缩写，这是Android官方提供的一款自动化测试工具。
凡是做安卓手机，或者别的什么搭载安卓系统的设备，都绕不开这个测试。

该测试准备工作主要分为两个部分，一是测试环境的搭建，二是手机端的设置。

CTS的测试环境一般是搭建的Ubuntu系统上，需要安装jdk, android sdk等，并配置相应的环境变量/etc/profile。这些安装配置完成后，就可以Android官网下载CTS packages了，下载之后解压。

cts packages: https://source.android.com/compatibility/cts/ (需科学上网)

手机端的设置注意事项如下：

1. 将被测设备恢复出厂设置: Settings > Backup & reset > Factory data reset

2. 手机语言设置为美国英语: Settings > Language & input > Language

3. 如果被测设备支持GPS, Wi-Fi或移动网络，打开位置服务: Settings > Location > On

4. 链接一个接入互联网的支持IPv6的Wi-Fi: Settings > Wi-Fi

5. 确保被测设备没有设置锁屏密码或图案: Settings > Security > Screen lock > None

6. 打开USB调试: Settings > Developer options > USB debugging.

   注意: Android 4.2及之后的版本, Developer options 是默认隐藏的。进到 Settings > About phone 并且点按 Build number 7次，然后回到上一界面，该选项才会出现。 (http://developer.android.com/studio/run/device.html#developer-device-options) 供参考

7. 确保时间被设为12小时格式: Settings > Date & time > Use 24-hour format > Off

8. 开启该选项: Settings > Developer options > Stay Awake > On

9. 开启该选项: Settings > Developer options > Allow mock locations > On

   注意: 只有 Android 5.x 和 4.4.x版本支持该选项。

10. 关闭该选项: Settings > Developer options > Verify apps over USB > Off

**注意:** Android 4.2. 之后才有此项要求。

11. 启动浏览器，关掉所有弹框。

12. 手机通过USB连接电脑。

    Note: When you connect a device running Android 4.2.2 or later to your computer, the system shows a dialog asking whether to accept an RSA key that allows debugging through this computer. Select Allow USB debugging.

13. Install and configure helper apps on the device.

**Note:** For CTS versions 2.1 R2 through 4.2 R4, set up your device (or emulator) to run the accessibility tests with:
`adb install -r android-cts/repository/testcases/CtsDelegatingAccessibilityService.apk`
On the device, enable: **Settings > Accessibility > Accessibility > Delegating Accessibility Service**

**Note:** For CTS versions prior to 7.0, on devices that declare `android.software.device_admin`, set up your device to run the device administration test using:
`adb install -r android-cts/repository/testcases/CtsDeviceAdmin.apk`

In Settings > Security > Select device administrators, enable the two`android.deviceadmin.cts.CtsDeviceAdminReceiver*` device administrators. Ensure the`android.deviceadmin.cts.CtsDeviceAdminDeactivatedReceiver` and any other preloaded device administrators remain disabled.

14. Copy the CTS media files to the device as follows:

**Note:** For CTS 2.3 R12 and later, if the device supports video codecs, the CTS media files must be copied to the device.

*   Navigate (cd) to the path the media files are downloaded and unzipped to.
*   Change the file permissions: `chmod u+x copy_media.sh`
*   Run `copy_media.sh`:
    *   To copy clips up to a resolution of 720x480, run: `./copy_media.sh 720x480`
    *   If you are not sure about the maximum resolution, try `./copy_media.sh all` so that all files are copied.
    *   If there are multiple devices under adb, add the -s (serial) option to the end. For example, to copy up to 720x480 to the device with serial 1234567, run: `./copy_media.sh 720x480 -s 1234567`

配置完成后，将手机通过数据线连接到PC, 然后从命令终端（terminal）进入到相应路径，然后尝试执行run cts --plan命令；

To run a test plan:

Connect at least one device.
Press the home button to set the device to the home screen at the start of CTS.
While a device is running tests, it must not be used for any other tasks and must be kept in a stationary position (to avoid triggering sensor activity) with the cameras pointing at an object that could be focused.
Do not press any keys on the device while the CTS is running. Pressing keys or touching the screen of a test device will interfere with the running tests and may lead to test failures.
Launch the CTS console by running the cts-tradefed script from the folder where the CTS package has been unzipped, e.g. $ ./android-cts/tools/cts-tradefed
Start the default test plan (contains all test packages) by appending: run cts --plan CTS . This kicks off all CTS tests required for compatibility.
For CTS v1 (Android 6.0 and earlier), enter list plans to view a list of test plans in the repository or list packages to view a list of test packages in the repository.
For CTS v2 (Android 7.0 and later), enter list modules to see a list of test modules.
Alternately, run the CTS plan of your choosing from the command line using: cts-tradefed run cts --plan
Note: When running Android 6.0 (Marshmallow) CTS only, we recommend you use the --skip-preconditions option to skip the experimental pre-conditions feature that may cause issues for when executing CTS tests.

View test progress and results reported on the console.
If your device is Android 5.0 or later and declares support for an ARM and a x86 ABI, you should run both the ARM and x86 CTS packages.
Using the CTS v1 console
For Android 6.0 or earlier, you'll use CTS v1.

Selecting plans
The following test plans are available:

CTS—all tests required for compatibility.
Signature—the signature verification of all public APIs
Android—tests for the Android APIs
Java—tests for the Java core library
VM—tests for ART or Dalvik
Performance—performance tests for your implementation
These can be executed with the run cts command.

CTS v1 console command reference
Table 1. This table summarizes the CTS v1 console commands for various uses.

Host Description
help Display a summary of the most commonly used commands
help all Display the complete list of available commands
exit Gracefully exit the CTS console. Console will close when all currently running tests are finished
Run Description
run cts Run the specified tests and displays progress information. One of --plan, --package, --class or --continue-session needs to be specified
The CTS console can accept other commands while tests are in progress

If no devices are connected, the CTS desktop machine (or host) will wait for a device to be connected before starting tests

If more than one device is connected, the CTS host will choose a device automatically

--plan <test_plan_name> Run the specified test plan
--package/-p <test_package_name> [--package/-p <test_package2>...] Run the specified test packages
--class/-c <class_name> [--method/-m <test_method_name> Run the specified test class and/or method
--continue-session Run all not executed tests from previous CTS session; the sessions testResult.xml will be updated with the new results
--shards <number_of_shards> Shard a CTS run into given number of independent chunks, to run on multiple devices in parallel
--serial/-s <deviceID> Run CTS on the specific device
-t <class_name>#<test_method_name> Run a specific test method
--force-abi 32|64 On 64-bit devices, run the test against only the 32-bit or 64-bit ABI
List Description
list packages List all available test packages in the repository
list plans List all available test plans in the repository
list invocations List 'run' commands currently being executed on devices
list commands List all 'run' commands currently in the queue waiting to be assigned to devices
list results List CTS results currently stored in repository
list devices List currently connected devices and their state
'Available' devices are functioning, idle devices, available for running tests

'Unavailable' devices are devices visible via adb, but are not responding to adb commands and won't be allocated for tests

'Allocated' devices are devices currently running tests

Add Description
add derivedplan --plan <plan_name>
--result/-r
[pass | fail | timeout | notExecuted]
[--session/-s <session_id>] Create a plan derived from given result session; use this option to rerun reports and validate test issues
Using the CTS v2 console
For Android 7.0 or later, you'll use CTS v2.

Selecting plans
Available test plans include the following:

cts—Runs CTS from an pre-existing CTS installation.
cts-camera— Runs CTS-camera from a pre-existing CTS installation.
cts-java— Runs Core Java Tests from a pre-existing CTS installation.
cts-pdk— Runs Tests useful on validating a PDK fusion build.
everything— Common config for Compatibility suites.
Other available configurations include the following:

basic-reporters— Configuration with basic CTS reporters.
collect-tests-only—Runs CTS from a pre-existing CTS installation.
common-compatibility-config— Common config for Compatibility suites.
cts-filtered-sample— Common config for Compatibility suites.
cts-known-failures— Configuration with CTS known failures.
cts-preconditions— CTS precondition configs.
host— Runs a single host-based test on an existing device.
instrument— Runs a single Android instrumentation test on an existing device.
native-benchmark— Runs a native stress test on an existing device.
native-stress— Runs a native stress test on an existing device.
recharge— A fake test that waits for nearly-discharged devices and holds them for charging.
testdef— Runs tests contained in test_def.xml files on an existing device.
util/wifi— Utility config to configure Wi-Fi on device.
util/wipe— Wipes user data on device.
All of these plans and configs can be executed with the run cts command.

CTS v2 console command reference
Table 1. This table summarizes the CTS V2 console commands for various uses.

Host Description
help Display a summary of the most commonly used commands
help all Display the complete list of available commands
version Show the version.
exit Gracefully exit the CTS console. Console will close when all currently running tests are finished.
Run Description
run cts
Run the default CTS plan (that is, the full CTS invocation).

The CTS console can accept other commands while tests are in progress.

If no devices are connected, the CTS desktop machine (or host) will wait for a device to be connected before starting tests.

If more than one device is connected, the CTS host will choose a device automatically.

--plan <test_plan_name> Run the specified test plan.
--module/-m <test_module_name> [--module/-m <test_module2>...] Run the specified test module or modules. For example, run cts --module CtsGestureTestCases executes the gesture test module (this can be shortened to run cts -m Gesture).
run cts -m Gesture --test android.gesture.cts.GestureTest#testGetStrokes runs the specific package, class, or test.
--subplan <subplan_name> Run the specified subplan.
-- module/-m <test_module_name> -- test <test_name> Run the specified module and test. For example, run cts -m Gesture --test android.gesture.cts.GestureTest#testGetStrokes runs the specific package, class, or test.
--retry Retry all tests that failed or were not executed from the previous sessions. Use list results to get the session id.
--shards <number_of_shards> Shard a CTS run into given number of independent chunks, to run on multiple devices in parallel.
--serial/-s <deviceID> Run CTS on the specific device.
--include-filter <module_name> [--include-filter <module2>...] Run only with the specified modules.
--exclude-filter <module_name> [--exclude-filter <module2>...] Exclude the specified modules from the run.
--log-level-display/-l <log_level> Run with the minimum specified log level displayed to STDOUT. Valid values: [VERBOSE, DEBUG, INFO, WARN, ERROR, ASSERT].
--abi <abi_name> Force the test to run on the given ABI, 32 or 64. By default CTS runs a test once for each ABI the device supports.
--logcat, --bugreport, and --screenshoot-on-failure Give more visibility into failures and can help with diagnostics.
--device-token Specifies a given device has the given token eg. --device-token 1a2b3c4d:sim-card..
--skip-device-info Skips collection of information about the device. Note: do not use this option when running CTS for approval.
--skip-preconditions Bypasses verification and setup of the device's configuration, such as pushing media files or checking for Wi-Fi connection.
List Description
list modules List all available test modules in the repository.
list plans or list configs List all available test plans (configs) in the repository.
list subplans List all available subplans in the repository.
list invocations List 'run' commands currently being executed on devices.
list commands List all 'run' commands currently in the queue waiting to be assigned to devices.
list results List CTS results currently stored in repository.
list devices List currently connected devices and their state.
'Available' devices are functioning, idle devices, available for running tests.

'Unavailable' devices are devices visible via adb, but are not responding to adb commands and won't be allocated for tests.

'Allocated' devices are devices currently running tests.

Dump Description
dump logs Dump the tradefed logs for all running invocations.
Add Description
add subplan --name/-n <subplan_name>
--result-type
[pass | fail | timeout | notExecuted]
[--session/-s <session_id>] Create a subplan derived from previous session; this option generates a subplan that can be used to run a subset of tests.

The only required option is --session. Others are optional but, when included, must be followed by a value. The --result-type option is repeatable; for example add subplan --session 0 --result-type passed --result-type failed is valid.

参考内容：Android官网