> For the complete documentation index, see [llms.txt](https://wong-coupon.gitbook.io/flutter/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://wong-coupon.gitbook.io/flutter/my-flutter/quality-delivery/widget-test-golden-regression.md). # Widget Tests and Golden Regression How I combine widget tests with golden PNGs, stabilize rendering, and review visual regressions without hiding failures behind update-goldens ## Result In my project, widget tests already verify text, state, and user actions. However, a test can still pass when padding shifts, a color changes, or an icon renders incorrectly. I add golden tests for UI states with high review value. Widget tests protect the behavioral contract, while golden PNGs protect the visual contract. ``` Fixture + mock dependency │ ▼ Deterministic widget shell viewport · DPR · safe area · theme · locale · fonts │ ├──► Semantic assertions │ text · state · interaction │ └──► Golden comparator ── match ──► pass │ mismatch ▼ master · test · diff images │ ┌──────────┴──────────┐ ▼ ▼ UI regression Intended change fix implementation update + review baseline ``` Both test branches use the same fixture but answer different questions. Semantic assertions explain which behavior broke. A golden diff makes a layout change visible during review. In the source I reviewed, the golden harness fixes these inputs: | Input | Value | | ------------------ | ------------------------------------ | | Logical viewport | `375 × 812` | | Device pixel ratio | `3.0` | | Physical view | `1125 × 2436` | | Safe area | top `44`, bottom `34` logical pixels | | Text scaling | `TextScaler.noScaling` | | Theme | Light | | Fuzzy tolerance | `0.01%` different pixels | The current source has only two `375 × 812` baseline PNGs for two states of the same page. Goldens run indirectly in the root suite through `make test`. I treat this as an initial adoption step, not evidence that the entire UI has visual regression coverage. ## Problem ### Widget tests can pass while the UI is still wrong A semantic widget test usually checks assertions like these: ```dart expect(find.text('Notification settings'), findsOneWidget); expect(find.byIcon(Icons.check), findsOneWidget); ``` These two assertions do not detect changes such as: * A card losing its padding. * A selected border changing color. * Text wrapping onto a new line. * An icon shifting away from its baseline. * A button still existing while its height or spacing changes. A golden test captures the widget area that needs protection and compares the rendering with an approved baseline. It complements widget tests; it does not replace them. | Test type | Main question | Limitation | | ----------- | ---------------------------------------------------------- | ---------------------------------------------------------------------- | | Widget test | What does the user see, and which state follows an action? | Does not fully protect spacing, colors, typography, or layout. | | Golden test | Does this state render like the baseline? | Does not prove that callbacks, APIs, or business behavior are correct. | ### Screenshots become flaky when render inputs are unstable A single `matchesGoldenFile` line does not make a test reliable. The image can change when: * Flutter or the engine is upgraded. * The baseline is generated on a different OS. * The viewport, DPR, or safe area changes. * A font or icon font has not been loaded. * Locale, theme, or text scale comes from the external environment. * The fixture contains a timestamp or random data. * A network image has not been precached. * An animation or stream continues producing frames. Until these inputs are controlled, increasing the tolerance only turns the test green by hiding the real cause. ### `--update-goldens` is not a fix for a red test `--update-goldens` records the expected output again. If I run this command as soon as a test fails, I can turn a regression into the new baseline without noticing. I separate the two operations: ``` Verify ─► compare with baseline ─► fail when the difference exceeds tolerance Update ─► write a new baseline ─► only after the UI change is approved ``` Baseline PNGs are part of code review. Reviewers need to know which images changed, why they changed, and whether an image contains data that should not enter the repository. ## Solution ### Choose states worth protecting with goldens I do not snapshot every widget. I prioritize states where a small change can affect many users or many screens: * Empty, loading, success, and error states with different layouts. * Selected and unselected states that change borders, icons, or hierarchy. * Design-system components that are widely reused. * Layouts at narrow widths, with long text, or at accessibility scale. * Pages with many aligned elements that can regress during refactoring. I do not start with UI containing random timestamps, infinite animations, or uncontrolled network images. Those inputs must first be replaced with a deterministic clock, fixture, or image provider. Baseline names should describe the state instead of using sequence numbers: ``` test/ ├── goldens/ │ ├── notification_card_light_empty.png │ └── notification_card_light_selected.png └── widgets/ └── notification_card_test.dart ``` ### Keep test packages in the right scope For the toolchain in this article, the main package is locked at `3.3.0`: ```yaml dev_dependencies: flutter_test: sdk: flutter golden_screenshot: 3.3.0 ``` I place packages used only by tests in `dev_dependencies`. If a project already lists the dependency under `dependencies`, search for all runtime imports before moving it, then run `pub get`, analyze, and the focused test again. Do not copy the latest version automatically. Golden output depends on the Flutter engine and the comparator package, so each upgrade may require another baseline review. ### Fix the viewport, DPR, and MediaQuery This helper is shortened from the pattern I use: ```dart import 'dart:ui' as ui; import 'package:flutter/material.dart'; import 'package:flutter_test/flutter_test.dart'; import 'package:golden_screenshot/golden_screenshot.dart'; Future pumpGoldenWidget( WidgetTester tester, { required Widget child, ui.Size logicalSize = const ui.Size(375, 812), double devicePixelRatio = 3, Locale locale = const Locale('en'), EdgeInsets safePadding = const EdgeInsets.only( top: 44, bottom: 34, ), double allowedDiffPercent = 0.01, }) async { tester.view.physicalSize = ui.Size( logicalSize.width * devicePixelRatio, logicalSize.height * devicePixelRatio, ); tester.view.devicePixelRatio = devicePixelRatio; addTearDown(tester.view.resetPhysicalSize); addTearDown(tester.view.resetDevicePixelRatio); await tester.pumpWidget( MaterialApp( debugShowCheckedModeBanner: false, locale: locale, theme: ThemeData.light(), home: MediaQuery( data: MediaQueryData( size: logicalSize, padding: safePadding, devicePixelRatio: devicePixelRatio, textScaler: TextScaler.noScaling, ), child: Material(child: child), ), ), ); await tester.pumpAndSettle(); tester.useFuzzyComparator( allowedDiffPercent: allowedDiffPercent, ); } ``` `physicalSize` is calculated by multiplying the logical size by the DPR. With this configuration, the test view has a physical size of `1125 × 2436`, while the current baseline is stored as a `375 × 812` PNG. The two `addTearDown` calls are important. Without resetting the view, one golden test can alter the environment of the next test. This example fixes the light theme and disables text scaling to create a stable baseline. It does not prove dark mode, responsive layout, or accessibility scale. When those variants are part of the contract, I create separate tests and baselines instead of mixing them into one image. ### Pin locale, theme, and dependencies The golden helper should accept or explicitly set every input the widget reads at runtime: ``` Golden test shell ├── ThemeData / brightness ├── Locale + localization delegates ├── MediaQuery size / padding / textScaler ├── defaultTargetPlatform when the UI has platform branches ├── Clock or fixture date ├── Feature flags ├── Fake service / repository response └── Font, icon font, and image assets ``` In this public example, I do not call a real API. The selected state comes from a deterministic fixture: ```dart testWidgets('selected state keeps behavior and visual contract', ( tester, ) async { await pumpGoldenWidget( tester, child: const ListTile( key: ValueKey('notification-item'), title: Text('Notification settings'), trailing: Icon(Icons.check), selected: true, ), ); expect(find.text('Notification settings'), findsOneWidget); expect(find.byIcon(Icons.check), findsOneWidget); await expectLater( find.byKey(const ValueKey('notification-item')), matchesGoldenFile( '../goldens/notification_card_light_selected.png', ), ); }); ``` The finder passed to `matchesGoldenFile` must match exactly one widget. Flutter captures the first `RepaintBoundary` ancestor of that widget. When I need a more explicit image boundary, I wrap the component in its own `RepaintBoundary`. I keep semantic assertions beside the golden assertion. If an important icon disappears but the pixel difference remains below the tolerance, the semantic test must still fail. ### Load fonts before capturing Flutter tests use Ahem as the default font. If the app uses a custom font or icon font but the test does not load it, text and glyphs can differ from production. For a font shared by many tests, I load it in `test/flutter_test_config.dart`: ```dart import 'dart:async'; import 'package:flutter/services.dart'; import 'package:flutter_test/flutter_test.dart'; Future _loadFont(String family, String assetPath) async { final data = await rootBundle.load(assetPath); final loader = FontLoader(family) ..addFont(Future.value(data)); await loader.load(); } Future testExecutable( FutureOr Function() testMain, ) async { setUpAll(() async { await _loadFont('AppIcons', 'assets/fonts/app-icons.ttf'); }); await testMain(); } ``` The asset path and family in this example are placeholders. They must match the real project's `pubspec.yaml`. If only one file needs a special font, I load it in that file's `setUpAll`. I do not depend on fonts installed on a developer machine. ### Interpret fuzzy tolerance correctly With `golden_screenshot` 3.3.0, `allowedDiffPercent` uses percentage units. The comparator source explicitly states that `0.1` means `0.1%`. Therefore: ```dart tester.useFuzzyComparator(allowedDiffPercent: 0.01); ``` allows **0.01%** different pixels, not 1%. The comparator passes when the images match exactly or when `diffPercent <= allowedDiffPercent`. If the difference exceeds the threshold, it creates failure output and fails the test. I set the threshold in a shared helper and do not let each test choose an arbitrary value. `0.01%` is the value I verified in the current source, not a best practice for every app. If you want an exact match, keep Flutter's default comparator and omit `useFuzzyComparator`. If you want a looser tolerance, inspect the masked diff first and document which rendering noise is being accepted. ### Choose one shadow policy `golden_screenshot` 3.3.0 supports two shadow-related approaches: * `testWidgets` uses the default Flutter test behavior; the comparator can be replaced separately with `useFuzzyComparator`. * `testGoldens` is a wrapper that enables shadows during the test and restores them afterward. The source I reviewed uses `testWidgets` with the fuzzy comparator. I retain that approach here so the baseline does not change outside the article's scope. If you switch to `testGoldens`, treat it as a visual-policy change. Run the focused test again, review every changed image, and do not mix both policies in the same golden suite without a clear reason. ### Run the focused test first I run the file I am editing first: ```bash fvm flutter test \ test/widgets/notification_card_test.dart \ --reporter compact ``` Expected results: * Semantic assertions pass. * The golden comparator reads the correct baseline relative to the test file's directory. * No new file appears under `failures/`. * The command returns `0`. After the UI change is confirmed, I update only that test file: ```bash fvm flutter test \ test/widgets/notification_card_test.dart \ --update-goldens ``` After this command, I always inspect the PNG `git diff`. If multiple baselines change unexpectedly, I stop and find the render input that has not been pinned. The current source also has a target that updates every root golden: ```bash make update-snapshot ``` This target calls `flutter test --update-goldens` without limiting the file. I use it only when I really intend to check the full root suite, such as after upgrading Flutter or changing a shared font. ### Preserve the real suite exit status My root suite pipes `flutter test` through `tee` so it can display and save the log at the same time. The Makefile must return the test command's exit status, not the exit status from `tee`: ```makefile SHELL := /bin/bash test: @mkdir -p reports @status=0; \ fvm flutter test --coverage --reporter compact 2>&1 \ | tee reports/test-output.log; \ test_status=$${PIPESTATUS[0]}; \ if [ $$test_status -ne 0 ]; then status=$$test_status; fi; \ exit $$status ``` In the current source, `make test` continues by running package tests and aggregating the JSON report. A golden mismatch at the root still remains in the final status instead of being hidden as a pass by `tee`. The pipeline currently calls: ``` analyze │ ▼ separate widget test │ ▼ make test ├── root tests + golden └── package tests │ ▼ coverage report ``` Goldens are already part of the automatic root gate. However, the current CI source does not save diff images as artifacts. ### Keep diff images when CI fails When `LocalFileComparator` or the fuzzy comparator detects a mismatch, Flutter creates diagnostic images: * master image; * test image; * isolated diff; * masked diff. CI needs to upload the `failures/` directory even when the job fails. The proposed configuration is: ```yaml artifacts: when: always paths: - test/**/failures/ ``` This improvement is not present in the source pipeline I reviewed. Before applying it, create an intentional mismatch on a test branch, verify that the glob points to the right directory, and check that the job produces both outcomes: 1. The job enters the failed state. 2. All four diff images remain downloadable. The suite should also remove old failure artifacts before running. Otherwise, a passing run can still upload images left by an earlier failure. ### Do not treat `dart_test.yaml` as a fake path source The golden key is resolved relative to the test file: ```dart matchesGoldenFile('../goldens/notification_card_light_selected.png') ``` In the source I reviewed, the actual baseline path comes from this argument. Neither the `Makefile` nor the test reads custom keys such as `golden_dir` or `failure_dir` from `dart_test.yaml`. I do not describe those keys as active configuration. To change path resolution across the project, install a real `GoldenFileComparator` in `flutter_test_config.dart` and write tests for that comparator. ### Review baselines like code This is the checklist I use before accepting a new PNG: 1. It belongs to the intended test and state. 2. Text, spacing, icons, colors, and hierarchy match the acceptance criteria. 3. It contains no timestamp, animation frame, or random data. 4. It contains no person name, account, endpoint, or internal data. 5. The baseline was generated with the project's approved Flutter version and OS policy. 6. Semantic assertions still pass. 7. The threshold was not increased only to turn the test green. I choose one standard OS/runner for generating baselines. CI verifies them; it does not run `--update-goldens` automatically and commit the output. When Flutter is upgraded or a shared font changes, I separate the baseline update into its own change so reviewers can see the scope of churn. ### Common failures | Symptom | Common cause | How to diagnose and fix it | | ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------ | | `Could not be compared against non-existent file` | The baseline is missing or the relative path is wrong. | Check the path from the test file's directory; use `--update-goldens` only when intentionally creating a baseline. | | Text becomes boxes or its layout changes | A custom font or icon font was not loaded. | Load it in `flutter_test_config.dart` or `setUpAll`; do not depend on fonts installed on the host. | | Golden fails after a Flutter upgrade although app code is unchanged | Engine or font rendering changed. | Generate the baseline on the standard runner and review the migration diff in a separate change. | | `pumpAndSettle` times out | An animation or stream keeps producing frames. | Pump a deterministic number of frames or wait for a finder/state with a timeout. | | Golden passes although a small icon is wrong | The change is below the fuzzy tolerance. | Keep a semantic assertion for important details or lower the threshold. | | CI reports a mismatch but reviewers cannot see images | `failures/` is not uploaded when the job fails. | Add an artifact with `when: always`, create a test mismatch, and verify the path. | | Many PNGs change after an update | Locale, theme, font, clock, or viewport is not pinned; or the update command ran the full suite. | Run the focused file, inspect each diff, and stop before committing out-of-scope baselines. | ### Trade-offs * Goldens catch visual regressions well but increase the cost of storing and reviewing binary files. * A snapshot that is too large makes diffs hard to read; one that is too small misses layout relationships. * One viewport does not prove responsive behavior. * Disabling text scale stabilizes the baseline but does not test accessibility. * Fuzzy tolerance reduces rendering noise but can hide small changes. * Mocked dependencies stabilize the image but do not prove that production APIs still satisfy their contracts. * Running goldens in the root suite creates an immediate gate but gives slower local feedback than a dedicated golden target. * Artifacts help diagnosis but do not replace human review. ### Verify the result A complete golden state should pass this matrix: | Case | Semantic assertion | Golden comparison | Expected result | | ------------------------------------------------- | ------------------ | -------------------------------------- | ----------------------------------------------------------- | | Unchanged standard state | Pass | Pass | No new diff is created. | | Important text or icon disappears | Fail | May fail or remain below the threshold | The error identifies the semantic contract. | | Padding, color, or layout changes unintentionally | May pass | Fail | Master, test, isolated diff, and masked diff are available. | | UI change is approved | Pass | Fail before update; pass after update | The PNG diff is reviewed. | | Baseline is deleted | May pass | Fail | Verification does not create the baseline automatically. | | Animation does not stop | May not run | Does not reach comparison | A clear timeout occurs; no image is updated. | My local verification on 2026-09-02 did not reach the golden assertion. `flutter test` stopped during dependency resolution because the environment lacked access to a private Git dependency. Retrying with `--no-pub` was also invalid because the package cache lacked many hosted packages. The current evidence is therefore separated explicitly: * Test structure, comparator, Make target, and CI wiring: verified from source. * The two baseline PNGs and their dimensions: inspected from tracked files. * Focused golden pass on Flutter 3.41.2: not confirmed in the research environment. * CI upload of failure images: not implemented. I do not use this dependency failure to claim that the golden passes or fails. Runtime evidence is valid only after dependencies resolve and the assertion actually runs. ### Related articles Line coverage measures which code executed; it does not measure whether the UI matches a baseline. I keep these two results separate. {% content-ref url="/pages/GYRjxkG3zMjUYpCTRcv4" %} [Coverage](/flutter/my-flutter/foundations/coverage-codecov.md) {% endcontent-ref %} When a component has a contract by width or theme, create a baseline for each variant. The next two articles own responsive UI organization and dark mode; this article covers only visual verification. {% content-ref url="/pages/CTKWji67N7nUi8hjGxHc" %} [Responsive](/flutter/my-flutter/ui-media/responsive.md) {% endcontent-ref %} {% content-ref url="/pages/DDwBamhA1zlGZaY4h0sH" %} [Dark Mode](/flutter/my-flutter/ui-media/dark-mode.md) {% endcontent-ref %} Widget and golden tests suit an isolated component or page. When I need to test a complete flow across multiple screens and dependencies, I move up to integration/BDD tests. {% content-ref url="/pages/5gKOCHnCinMIWgGm3uxB" %} [BDD with Flutter Gherkin](/flutter/my-flutter/quality-delivery/bdd-flutter-gherkin.md) {% endcontent-ref %} ### Versions reviewed * Flutter: `3.41.2`. * Dart: `3.11.0`. * `golden_screenshot`: `3.3.0`. * Current baseline: `375 × 812` RGBA PNG. * Platform scope: Shared, run with `flutter test` on the host. ### References * [Flutter API — `matchesGoldenFile`](https://api.flutter.dev/flutter/flutter_test/matchesGoldenFile.html) * [Flutter API — `LocalFileComparator`](https://api.flutter.dev/flutter/flutter_test/LocalFileComparator-class.html) * [Flutter API — `goldenFileComparator`](https://api.flutter.dev/flutter/flutter_test/goldenFileComparator.html) * [`golden_screenshot` changelog](https://pub.dev/packages/golden_screenshot/changelog) * [GitLab — Job artifacts](https://docs.gitlab.com/ci/jobs/job_artifacts/) ## Conclusion Widget tests tell me what the interface must do. Golden tests tell me what that state must look like. I use both layers together because a pixel diff cannot explain behavior, while a semantic assertion cannot see the entire visual regression. The most important part is not `matchesGoldenFile`. I need to stabilize render inputs, interpret tolerance correctly, preserve the real exit status, and review PNGs like code. `--update-goldens` is valid only after a UI change has been confirmed; it is not a button for bypassing a red test. Goldens fit components or pages with a stable visual contract. For flows spanning multiple screens, native UI, or real dependencies, I use integration tests instead of stretching goldens beyond their scope. [Buy Me a Coffee](https://buymeacoffee.com/ducmng12g) | [Support Me on Ko-fi](https://ko-fi.com/I2I81AEJG8)