<br />

There are multiple ways to use Firebase Test Lab to run tests on
your Android app, including the [command line interface](https://firebase.google.com/docs/test-lab/android/command-line),
[Android Studio](https://firebase.google.com/docs/test-lab/android/android-studio), the [Test Lab UI](https://firebase.google.com/docs/test-lab/android/firebase-console) in the
Firebase console, and the [Testing API](https://firebase.google.com/docs/test-lab/reference/testing/rest).
However you choose to start your tests, the results are
stored in the Firebase project that you specify. You can explore the results using
[ToolResults API](https://developers.google.com/apis-explorer/#search/toolresults/toolresults/v1beta3/)
in addition to any of the tools above. This page describes how to
review and analyze these test results.

## Key concepts

To see the results from all your previous test runs, select **Test Lab** in the
left navigation panel of your project in the
Firebase console. This page displays all the test runs from the
apps that you have tested with your project using Test Lab.

To review test results, you first need to understand three concepts:

When you run a test or a set of test cases against devices and configurations
you've selected, Test Lab runs the test against your app in a batch, then
displays the results as a **test matrix**.

**Devices × Test Executions = Test Matrix**

Device
:   A physical or virtual device (Android only) you run a test on, such as a
    phone, tablet, or wearable device. Devices in a test matrix are identified by
    device model, OS version, screen orientation, and locale (also known as
    geography and language settings).

Test, test execution
:   A test (or a set of test cases) to be run on a device. You can run one test
    per device, or optionally shard the test and run its test cases on
    different devices.

Test matrix
:   Contains the statuses and test results for your test executions. If any
    test execution in a matrix fails, the whole matrix fails.

The following sections explain how to navigate test results.

## Interpret test history results

When you navigate to your test results by selecting **Test Lab**, you see
the results of tests you have run so far.

Testing history is grouped by app. Only the most recent five test matrices are
shown for each app; if more are available, you can click the **All Matrices**
link at the bottom of the app test list to see the complete list for that app.

## Interpret test matrix results

When starting a test through the [Test Lab UI](https://firebase.google.com/docs/test-lab/android/firebase-console), you are
redirected to a page where you can see your test matrix and click a specific
test execution to view test results. [Android Studio](https://firebase.google.com/docs/test-lab/android/android-studio)
and [the gcloud command](https://firebase.google.com/docs/test-lab/android/command-line) provide a URL for the test matrix
results page as well.

In a typical test matrix, you might run a test across a dozen or so different
devices. Each test execution can have a different outcome. The possible outcomes
for any test execution in a test matrix include the following:

- Passed ![Test Passed](https://firebase.google.com/static/docs/test-lab/images/test-passed.png): No failures were encountered.
- Failed ![Test Failed](https://firebase.google.com/static/docs/test-lab/images/test-failed.png): At least one failure was encountered.
- Inconclusive ![Test Passed](https://firebase.google.com/static/docs/test-lab/images/test-inconclusive.png): Test results were inconclusive, possibly due to a Test Lab error.
- Skipped ![Test Passed](https://firebase.google.com/static/docs/test-lab/images/test-skipped.png): The selected dimension values for some test executions in the matrix were incompatible. This occurs when devices that you selected are incompatible with one or more of the Android API levels that you selected.

To review aggregated test results for all test matrices for a given app in your
Firebase project, click the name of the app, as shown in the following example:

*Example test matrix results page with only four test executions*
![Test Matrix Results](https://firebase.google.com/static/docs/test-lab/images/test-matrix-results.png)

This takes you to the test matrix list for your app, where you can click the
name of any test matrix to see the test matrix results, and where you can click
the name of the app (shown in the red box below) to view the test matrix list
for other apps associated with your Firebase project.

*Example test matrix list page*
![Test Matrix List](https://firebase.google.com/static/docs/test-lab/images/test-matrix-list.png)

A test matrix can pass, fail, or be inconclusive. A test matrix is shown
as failed or inconclusive if any test executions in that matrix fail or are
inconclusive.
| **Note:** **Device issues sometimes increase test execution time.** If you notice that a test is taking longer than usual, it could be a sign of an issue in Test Lab. Don't worry, though, Test Lab only charges you for the time your test is actually running.
| **Note:** **Crashes can result in skipped tests.** If some of your test cases don't appear in your results, it might be due to a crash that terminated the Test Lab test runner. To prevent a crash in one test from impacting other tests, run each test case in isolation with the [Android Test Orchestrator](https://firebase.google.com/docs/test-lab/android/firebase-console#new_instrumentation_tests_with_orchestrator).

## Interpret Robo test results

If you ran your tests with Robo, your results include videos and screenshots of
Robo crawling your UI, in addition to the usual test metrics. Those video and
screenshots include visual indications of the actions Robo took during the crawl,
similar to the 'Show touches' feature in Android. You can use the indications to
help you follow along with Robo's progress, and reproduce any bugs it might
uncover.

*Example Robo test results video*

![Robo test results](https://firebase.google.com/static/docs/test-lab/images/robo_touchpoints.gif)

## Interpret results from a single test execution

From the test matrix results page, click on one of the test executions to see
the result of that specific test execution.

*Example test execution results page*
![Test execution results](https://firebase.google.com/static/docs/test-lab/images/test-execution-results.png)

On this page, you can see the time required for each test execution. You can
also see the results for specific test cases that correspond to methods in your
test APK (for instrumentation tests) and detailed test results, including test
logs, screenshots, and videos. For [Robo test](https://firebase.google.com/docs/test-lab/android/robo-ux-test), detailed test
results also include an activity map that graphically shows the UI paths that
were visited by Robo test.

### Partitioned instrumentation test results

To help you interpret instrumented test results, Test Lab separates each test
into its own detailed report page, complete with stack traces, logs, and videos.
This feature works whether or not you are using Android Orchestrator.

*Example testcase results page*
![Testcase results](https://firebase.google.com/static/docs/test-lab/images/testcase-results.png)

## Interpret accessibility results

Robo tests use Android Accessibility Scanner to detect accessibility issues
in your app (note that you can also run a scan locally on your device).
For instructions on how to review and interpret the
accessibility results of your Robo test, visit
[Get started with Accessibility Scanner](https://support.google.com/accessibility/android/faq/6376582?hl=en&visit_id=637237771125039843-484192898&rd=1).

For general information on how to improve the accessibility of your app, visit
the [Android Developer Accessibility documentation](https://developer.android.com/guide/topics/ui/accessibility/index.html).

## Performance metrics

Tests run on physical devices also return performance metrics:

| Metric | Required device configuration |
|---|---|
| App startup time | API 19+ |
| CPU usage | API 21+ |
| Memory usage | API 21+ |
| Network activity | API 21+ |
| Frames per second | API 21+ and includes a [`SurfaceView`](https://developer.android.com/reference/android/view/SurfaceView) |

## Detailed test results

Detailed test results are available for 90 days after you run a test and are
stored in a Google Cloud Storage bucket (but are also visible in the
Firebase console). You can view detailed test results in the Cloud Storage
bucket when you click **Test results** on the test execution results page.
When detailed test results are no longer available, you can still see which
tests passed or failed.

If you want to retain detailed test results for longer than 90 days, you can
send these test results to a Cloud Storage bucket that you own using the
[**--results-bucket** gcloud command-line option](https://cloud.google.com/sdk/gcloud/reference/firebase/test/android/run).
You can then set the **Age** setting to determine how long results are stored in
your Cloud Storage bucket. See
[Lifecycle conditions](https://cloud.google.com/storage/docs/lifecycle#conditions)
for information about how to change the **Age** setting.
| **Note:** The project that owns this Cloud Storage bucket must have billing enabled to retain detailed test results for longer than 90 days.

When running a test, if you receive the error, `does not have storage.objects.create access to the Google Cloud Storage object. Permission 'storage.objects.create' denied on resource (or it may not exist)..`, the principal executing the test may not have the correct permission on the project. By default, Firebase creates a bucket for you in Google Cloud Storage, however, it requires the principal executing the test to have "roles/editor", which is a permissive role. If you cannot grant that role to the principal, you can use the [**--results-bucket** gcloud command-line option](https://cloud.google.com/sdk/gcloud/reference/firebase/test/android/run).