Rainforest QA — Element matching with DOM

Overview

About the product

• Rainforest QA is a no-code test automation platform
• Users can easily creating test steps with predefined actions such as clicking, scrolling, or typing
• To identify UI elements like a login button or password field, users can capture screenshots from a live preview on a virtual machine.
• During test execution, the agent scans the website to find elements that match the screenshots.

Problem with relying on image matching

• Flakiness: Image matching can be unreliable, leading to inconsistent test results.
• Platform compatibility: It often doesn't perform well across different platforms.
• Maintenance overhead: Frequent UI changes can result in significant ongoing maintenance.
• False negatives: Tests may fail when UI changes occur, even if the underlying functionality remains intact.

Goals

• Explore the potential of DOM matching in our QA product.
• Ultimately, reduce test flakiness by introducing DOM matching as a fallback when image matching failed.
• For the sake of simplicity, I’ll be focusing on the element modal in this case study.

Process overview

Challenges

• Scoping: With many possibilities for DOM matching, we need to decide what should be included in the MVP.
• User friendly: How can we prevent users from entering incorrect DOM snippets? Even technical users may find it challenging to identify the correct DOM snippet.
• Timeline: as an experimental project, we aim to validate the product quickly without investing excessive time and effort.

01 — Research & concept validation

Secondary research — gaining a deeper understanding of the problem space

• Learn about DOM and its potential: read DOM related articles and chat with PM and developers collecting concerns, feedbacks, and their unique perspective.
• Competitor research: understand how other products are currently using the DOM.

Design exploration

Based on our findings and research, I began exploring various design concepts. While these designs might appear high-fidelity, I used Rainforest’s internal design system to quickly produce a range of options.

Streamline for Speed: Simple & Fast Shipping

Ultimately, we decided to go with the simplest option;

• It’s easier and faster to built as we can utilize existing components.
• Given that the DOM is quite technical, we expect early adopters to be more technical users. As a result, we don’t need to simplify the DOM input and can instead use a straightforward text input field.

Results

• We successfully implemented the product and quickly delivered it to users for feedback.
• From their responses, we discovered that the DOM matching feature effectively resolved the flaky issue.
• As a result, we've decided to invest additional time and resources into developing the next version.

02 — Further improvement: Dynamic matching mode

After launching and gathering feedback, we're ready to move forward. I explored several ideas and created interactive prototypes for testing.

Context

With the old model, you selected one matching method, and if a match wasn't found, the test would fail immediately.

In the new model, If the agent fails to find an element that matches the first selected method, it'll try to find the element using the next enabled fallback method. This approach increases the likelihood of the test passing.

Answering the big questions: How should we present the DOM matching option in the element popup?

Each option has its pros and cons;

1. Fallback as a property of the main method: This approach requires the least implementation effort but is the least user-friendly, as it may not be immediately clear to users how to manage or configure fallback options.
2. List view: This option offers a good balance between ease of understanding and efficient use of existing components. It presents the options as a simple list so it’s not overwhelming users with too much information.
3. Full-screen modal: allow users to fully focus on the matching methods and and provides the most flexibility for future feature expansions. However, we might need to implement the modal from scratch, which could be resource-intensive.

03 — Final design

After a few iterations, we decided to go with option 2.

• Clean design: Minimal information to avoid overwhelming users
• Sequence indicators: arrows show the order of the three matching methods.
• Simultaneous viewing: Users can see the values of all three matching methods at once, without navigating through separate menus.
• Efficient Implementation: Utilizes existing components to reduce development effort.