feat(test_runner): support live Firebase and HTTP under flutter test#2321
Open
sharjeelyunus wants to merge 27 commits into
Open
feat(test_runner): support live Firebase and HTTP under flutter test#2321sharjeelyunus wants to merge 27 commits into
sharjeelyunus wants to merge 27 commits into
Conversation
Bridge Cloud Functions, Auth, and Firestore pigeon calls to HTTPS so YAML app tests can exercise real backends without native plugins. Preserve route settings for navigation waits, register startTimer delay timers for lifecycle cleanup, and keep unit tests free of real credentials.
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
…bility Removed unused HTTP override logic from various files and simplified the implementation of HTTP client creation. Enhanced code formatting for better readability and maintainability.
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
Replaced direct construction of HttpResponse with a dedicated method to convert MockAPIResponse to a runtime-compatible format. This change enhances code clarity and maintainability by centralizing response formatting logic.
…der overlay Updated the ensemble test runner to utilize a new TestApiProviderOverlay, enhancing the handling of API mocks and improving the clarity of API interactions. This change includes renaming and refactoring related components to streamline the testing process and ensure better maintainability.
…test vocabulary Eliminated unused API mock steps such as `mockNetworkOffline`, `mockNetworkOnline`, `expectApiRequest`, `expectApiRequestContains`, and `expectApiHeader` from the test vocabulary and related files. This refactor enhances clarity and maintainability of the test runner by focusing on essential functionalities.
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
…oved clarity Modified the logStorage functionality to log all public storage values when no key is provided, enhancing its usability. Updated the schema and examples in the ensemble_tests_schema.json to reflect changes in fixture paths and descriptions, ensuring consistency across the test runner. Introduced an optionalStorageKey argument kind to streamline the handling of storage keys.
… vocabulary Eliminated deprecated state handling steps such as `setState`, `expectState`, and related assertions from the test runner. This refactor enhances clarity and maintainability by focusing on essential functionalities, while also updating the associated documentation and schema to reflect these changes.
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
…larity Updated the 'wait' step to utilize real-time delays with runAsync, improving its functionality. Modified the associated schema to include a new 'keychain' property, ensuring better organization and clarity in the test definitions. Adjusted related documentation and tests to reflect these changes.
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
Updated the ExtendedStepHandlers to log storage and screenshot actions by writing output to files, improving traceability of test results. Introduced a new method for capturing the widget tree and logging API call counts, ensuring better organization of test artifacts. Adjusted related tests to verify the existence and content of generated log files.
… are available Implemented a new method to load application fonts from the FontManifest.json, enhancing the test harness's ability to handle font dependencies. Added a test to verify that font loading is safe even when the manifest is unavailable, ensuring robustness in the testing environment.
… support Added support for capturing screenshots with device frames in the test runner. Updated the schema to include new properties for device frame, platform, and model in screenshot steps. Introduced a new utility for resolving screenshot devices based on platform and model, and updated related tests to verify the new functionality. This enhancement improves the visual fidelity of screenshots captured during tests.
…lated logic Eliminated the deviceFrame property from the ensemble tests schema and refactored related logic in the screenshot handling to streamline the screenshot functionality. Updated tests to reflect these changes, ensuring that the default device model is now iPhone 15 Pro. This refactor enhances clarity and simplifies the screenshot capturing process.
Refactored the logging functionality in ExtendedStepHandlers and TestStepExecutor to write structured JSON outputs for API calls and storage logs, enhancing readability and traceability. Updated the file extension handling in TestLogger to support dynamic extensions, ensuring logs are saved in appropriate formats. Adjusted related tests to verify the new JSON structure and file extensions, improving the overall logging mechanism in the test runner.
Added a new options section to the ensemble tests schema, allowing configuration of screenshot settings such as enabling/disabling screenshots, platform, model, and step inclusion/exclusion. Enhanced the screenshot capturing functionality to utilize these options, ensuring automatic screenshots are taken based on specified criteria. Updated related classes and tests to support the new options structure, improving the flexibility and usability of the screenshot feature in the test runner.
…ests Introduced a new `logPerformance` step to capture and log Flutter app frame timing metrics as JSON artifacts. Updated the ensemble tests schema to include performance options, allowing users to enable or disable performance logging. Enhanced the test runner to automatically write performance logs at the end of test executions when enabled. Updated related classes, tests, and documentation to support this new feature, improving the overall performance monitoring capabilities of the test runner.
Added a new `config.yaml` file to define suite-wide settings for screenshots, performance logging, and other test artifacts. Updated the ensemble tests schema to support this configuration, allowing users to enable or disable features globally. Refactored related classes and methods to utilize the new configuration structure, enhancing the flexibility and maintainability of the test runner. Updated documentation to reflect these changes and provide examples of the new configuration options.
…and correlation Expanded the performance logging capabilities in the ensemble test runner by introducing new metrics and correlations. Added attributes for tracking performance markers, including worst steps and screens, jank clusters, and API call correlations. Updated the `writePerformanceLog` function to include these new metrics in the output JSON. Enhanced the `EnsembleTestRunner` to record performance markers during test execution, providing a comprehensive overview of performance data. Updated tests to validate the new logging features and ensure accurate performance reporting.
Enhanced the ViewUtil class to include a new property, testId, in the model construction process. This allows for the preservation of explicit testId values without requiring an id. Additionally, added a new test case to verify this functionality, ensuring that the testId is correctly assigned and that the id property is not included when not specified.
Introduced repeatable `--input key=value` flags for passing test inputs via the command line. Updated the test runner to reference these inputs in the test YAML using `${inputs.key}`. Enhanced the `EnsembleTestParser` and related classes to handle input values, ensuring they can be utilized in initial states, mocks, and steps. Added tests to validate the correct resolution of CLI inputs and error handling for missing inputs, improving the flexibility and usability of the test framework.
…mble tests Refactored the ensemble test runner to improve mock handling by introducing a new `mocks` section in the test schema, allowing for the use of `.mock.json` files for API responses. Updated the `EnsembleTestCase` model to include `mockFiles` and `scenarios`, enabling scenario-based testing with variable substitutions. Enhanced the `EnsembleTestParser` to parse scenarios and their variables, ensuring proper integration with the test execution flow. Updated documentation and tests to reflect these changes, improving the flexibility and usability of the test framework.
Enhanced the ensemble test framework by introducing a `delayMs` property for mocked API responses, allowing for simulated latency in tests. Updated the documentation to reflect this new feature and added tests to ensure that mocked APIs honor the specified delay before returning responses. This improvement aligns mock behavior with real API loading times, enhancing the realism of test scenarios.
Enhanced the ensemble test runner by introducing a new `record` configuration option, allowing users to enable session recordings during tests. Implemented functionality to capture and save recordings as GIFs along with a JSON manifest. Additionally, added a feature to generate a screenshot contact sheet, compiling automatic screenshots into a single image for easier review. Updated the schema, models, and documentation to reflect these new capabilities, improving the overall testing experience and artifact management.
… and layout adjustments Improved the screenshot capturing process in the ensemble test runner by adding functionality to highlight specific UI elements during screenshot capture. Implemented methods to ensure the target elements are visible before capturing, and adjusted the layout of the screenshot contact sheet to center tiles dynamically. These enhancements provide clearer visual feedback in test results and improve the overall usability of the screenshot features.
Removed the check for pending live calls during screenshot capture to streamline the process. Changed the screenshot contact sheet format from JPG to PNG for better quality and adjusted the layout by increasing tile dimensions and label sizes. Enhanced the label truncation logic for improved readability. These changes aim to enhance the overall screenshot functionality and visual presentation in the ensemble test runner.
Eliminated the recording functionality from the ensemble test runner, including the associated configuration and related code. Updated the schema and documentation to reflect the removal of the `record` option. Adjusted tests to ensure compatibility with the new configuration structure, enhancing the clarity and maintainability of the test framework.
Improved the EnsembleTestHarness by adding support for font family aliases and asset candidates, allowing for more flexible font loading. Implemented methods to resolve package information from app metadata files, with fallbacks for missing data. Added comprehensive tests to validate the new functionality, ensuring robust handling of font assets and package information in the test framework.
Introduced a new LiveFlutterTestOutputFilter to manage the output of running Flutter tests, allowing for selective emission of logs during test execution. Updated the extractSuiteReport function to conditionally include or exclude screen tracker lines based on parameters. Enhanced the extractKnownFailure function to capture actionable error messages from the Flutter test framework. Added tests to validate the new filtering behavior and report extraction logic, improving the clarity and usability of test outputs in the ensemble test runner.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bridge Cloud Functions, Auth, and Firestore pigeon calls to HTTPS so YAML app tests can exercise real backends without native plugins. Preserve route settings for navigation waits, register startTimer delay timers for lifecycle cleanup, and keep unit tests free of real credentials.