How I Reduced Flutter Integration Suite Result Time by Almost 50% | by Akanksha | Dec, 2022

Using Turnaround Time Improvements github job strategy

photo by guillaume jellette Feather unsplash

If you are new to Flutter integration testing, I suggest to search integration testing document as a condition. Flutter supports smooth integration with Firebase Test Lab. To learn about more integrations, please visit Integration Suite with Firebase Test Lab,

Firebase Test Lab is a cloud-based app testing infrastructure that lets you test your app on a wide variety of devices and configurations, so you can get a better idea of ​​how it will perform in the hands of live users.

Let’s say we have 30 E2E flows written for Flutter integration tests. The total execution time to get the result takes on average about 30 minutes (one minute per e2e flow). I assumed we were fine for 30 minutes to get automation test feedback.

But if our E2E throughput increases from 30 to 300, the average response time is five hours. That’s too much waiting time.

One solution that would come to our mind would be to run the tests in parallel mode.

But we have an open issueFirebase Test Lab does not support parallel execution (sharding of tests) for Flutter.

I reduced my automation turnaround time by using github job strategy, I will discuss my approach step by step.

Step 1: Let’s say we have four test files: feature1_tests.dart, feature2_tests.dart, feature3_tests.dartAnd feature4_tests.dart, Each file will contain test widget cases. Each feature test file would look like the following (main method with multiple test cases):

import 'package:flutter_test/flutter_test.dart';
import 'package:integration_test/integration_test.dart';

void main()
final binding = IntegrationTestWidgetsFlutterBinding.ensureInitialized()
as IntegrationTestWidgetsFlutterBinding;
binding.framePolicy = LiveTestWidgetsFlutterBindingFramePolicy.fullyLive;

group('Feature 1', ()
testWidgets('Scenario 1', (WidgetTester tester) async );

testWidgets('Scenario 2', (WidgetTester tester) async );
);

Step 2: We will define our test suite which will have a main method calling several features. Say, Suite1 And Suite2,


// Suite1.dart File , with feature 1 and Feature 2
import 'tests/feature1_tests.dart' as feature1_tests;
import 'tests/feature2_tests.dart' as feature2_tests;

void main()
feature1_tests.main();
feature2_tests.main();


// Suite2.dart File , with feature 3and Feature 4

import 'tests/feature3_tests.dart' as feature3_tests;
import 'tests/feature4_tests.dart' as feature4_tests;

void main()
feature3_tests.main();
feature4_tests.main();

Comment: Based on your project and features you can build your suite accordingly.

Step 3: Build Your GitHub Workflow to Run Two Suites Using Parallels github job strategy,

name: Parallel Test Execution On Firebase Test Lab

on:
workflow_dispatch:

env:
ANDROID_HOME: /Users/runner/Library/Android/sdk
JAVA_HOME: /Users/runner/hostedtoolcache/Java_Microsoft_jdk/11.0.13/x64/Contents/Home
flutter_version: "3.3.9"

jobs:
integration-tests:
runs-on: macos-11
timeout-minutes: 60
continue-on-error: true
strategy: # GitHub Strategy
fail-fast: false
max-parallel: 2
matrix:
tests: [SUITE1,SUITE2]

steps:
- name: Checkout Repository And Submodules
uses: actions/checkout@v2

- id: 'auth'
uses: 'google-github-actions/auth@v0'
with:
service_account: 'Your Service Account Link'
credentials_json: $

- name: 'Set up Cloud SDK'
uses: 'google-github-actions/setup-gcloud@v0'

- name: 'Use gcloud CLI'
run: 'gcloud info'

- name: Java Setup
uses: actions/setup-java@v3
with:
distribution: 'temurin'
java-version: '11'

# Caching Flutter SDK download based on version
- name: Flutter SDK Download
uses: subosito/flutter-action@v2
with:
flutter-version: $ env.flutter_version
channel: 'stable'
cache: true
cache-key: flutterA-$ env.flutter_version
cache-path: $ runner.tool_cache /flutterA-$ env.flutter_version

- name: Download Flutter Dependencies
run: |
flutter pub get

# Assuming shell script to Run on Firebase is Placed at file location - integration_test/firebase_parallel_test_runner.sh
- name: Assign Permission to shell file
run: |
chmod 777 ./integration_test/firebase_parallel_test_runner.sh

# Passing $matrix.tests as positional parameter to shell file
# If $matrix.tests = SUITE1 , we will upload Suite1.dart based test apk and same as SUITE2

- name: Run Suite
run: |
sh .integration_test/firebase_parallel_test_runner.sh $matrix.tests

Step 4: Create our bash shell script to run the suite on Firebase Test Lab. We’ll take a param from the command line or a GitHub variable and use that to build our test APK specific to that suite.

# Firebase Test Lab required Gcloud setup on Your Machine

DEBUG_APK_PATH=build/app/outputs/flutter-apk/app-debug.apk # File path of debug APk
TEST_APK_PATH=build/app/outputs/apk/androidTest/staging/debug/app-debug-androidTest.apk # File Path of Test Apk
SUITE=$1 # Positional Argument which will give Suite Name i.e SUITE1 and SUITE2 in our case
flutter build apk --debug # Build the Debug APk

pushd android
./gradlew app:assembleAndroidTest
popd

if [ "$SUITE" == "SUITE1" ]; then
pushd android
./gradlew app:assembleDebug -Ptarget=integration_test/suites/suite1.dart
popd

if gcloud firebase test android run --type instrumentation --app $DEBUG_APK_PATH --test $TEST_APK_PATH --timeout 45m
then
exit 0 # Exit with 0 if all test passed else exit with 1
else
exit 1
fi
fi

if [ "$SUITE" == "SUITE2" ]; then
pushd android
./gradlew app:assembleDebug -Ptarget=integration_test/suites/suite2.dart
popd

if gcloud firebase test android run --type instrumentation --app $DEBUG_APK_PATH --test $TEST_APK_PATH --timeout 45m
then
exit 0
else
exit 1
fi
fi

Yes! you are all set. Now both your suites will run in parallel. This will reduce your turnaround time.

source code is available in GitHub repository,

Leave a Reply