Embracing Conditional Test Skipping in Playwright

If you're using Playwright for your automation testing, you've probably appreciated its straightforward and simple test writing capabilities. Today, we're focusing on a feature that's simple yet impactful: skipping tests when certain conditions are met. This functionality makes Playwright not just useful, but a smart choice for efficient testing.

In the dynamic landscape of web testing, efficiency is key. With Playwright’s test.skip(), we’re not just skipping tests; we’re strategically choosing where to focus our efforts for maximum impact. Whether it’s adapting to different environments, managing varying user scenarios, or handling unique application behaviours, test.skip() allows us to tailor our tests intelligently.

We’ll explore various use cases where conditional test skipping can streamline your testing process, share best practices, and delve into some examples that illustrate its practical benefits.

By harnessing the power of test.skip(), we can make our test suites not only more responsive but also more efficient and focused. Let’s dive in and discover how to make every test count by skipping the right ones.

test.skip()

The test.skip() function in Playwright is a pivotal feature for effective test management. This function allows you to conditionally skip tests based on specific criteria. The syntax is straightforward yet powerful:

test.skip(condition, description);

In this structure, condition is a Boolean expression that determines whether to skip the test, and description is an optional string that explains why the test was skipped.

Basic Usage

test('Feature only relevant for Chrome', async ({ browserName }) => {
  test.skip(browserName !== 'chromium', 'This feature is only relevant in Chrome');
  // Test code specific to Chrome goes here
});

test.skip() can also be unconditional which will immediately abort the the test when test.skip() is called.

Scenarios for Skipping a Test

Conditional test skipping is especially valuable in several key scenarios:

1. Browser-Specific Behaviour: Not all features need to be tested on every browser. For instance, if a feature is exclusive to Chrome, there’s no need to run those tests on Firefox or Safari.

test('Exclusive Feature for Firefox', async ({ browserName }) => {
  test.skip(browserName !== 'firefox', 'Only relevant for Firefox');
  // Test code for Firefox-specific feature
});

2. Environment-Dependent Features: Sometimes, features behave differently or are even irrelevant in certain environments. For example, testing a feature that's only enabled in production environments doesn't make sense in a staging setup.

test('Production Only Feature', async ({}) => {
  test.skip(process.env.NODE_ENV !== 'production', 'Runs only in production');
  // Test code for production-only feature
});

3. Feature Flags: If you’re using feature flags to toggle functionality, it’s efficient to skip tests for features that are turned off in the current testing environment.

test('Feature Behind Flag', async ({}) => {
  test.skip(!isFeatureFlagEnabled('new-feature'), 'Feature flag is off');
  // Test code for the feature behind the flag
});

4. Resource-Intensive Tests: In cases where certain tests are particularly heavy on resources, you might choose to skip them during routine checks and reserve them for specific testing sessions.

test('Resource-Intensive Test', async ({}) => {
  test.skip(process.env.ci, 'Skipped in CI due to resource intensity');
  // Test code for resource-intensive process
});

5. Known Issues: If there’s a known bug that’s yet to be fixed, you might temporarily skip tests related to that bug to keep your test suite green and focus on other areas.

test('Test for Feature with Known Bug', async ({}) => {
  test.skip(hasKnownBug('BUG-123'), 'Known issue, pending fix');
  // Test code for the buggy feature
});

By using test.skip() strategically, as demonstrated in these examples, you can tailor your test suite to be more efficient and contextually relevant.

Unpacking an Example

Let's take a closer look at a practical example to understand how test.skip() and conditional testing works in Playwright.

Consider the following test example:

test('Verify page Title', async ({ page, browserName }) => {
  test.skip(browserName === 'webkit', 'Skipping Safari due to BUG-123');

  await page.goto('https://playwright.dev/');
  const title = page.locator('.navbar__inner .navbar__title');

  if (browserName === 'firefox') {
    await expect(title).toHaveText('Playright');
  }
  await expect(title).toHaveText('Playwright');
});

In this setup, we're dealing with two fixtures provided by Playwright's test runner: page and browserName. page represents the browser page, and browserName is a string indicating the browser type (like 'chromium', 'firefox', or 'webkit').

Skipping Logic

The line test.skip(browserName === 'webkit' is our conditional skip. This statement tells Playwright to skip this test if the browser being used is WebKit. The reasoning behind this could be that the feature being tested does not apply to WebKit browsers or behaves differently in a way that doesn't require testing in this test case.

Conditional Testing for Firefox

Further in the code, there's an if-statement checking if browserName equals 'firefox'. If true, it asserts that the page title should be 'Playright'. This is an example of browser-specific testing where you might expect different behaviour or content in different browsers. For Firefox, we’re expecting a different title, possibly due to a feature or bug that's specific to Firefox.

By using a combination of skip and conditional testing we can be concise when writing our test cases and make them adapt to the scenarios in test regardless of peculiarities such as browser or environment.

Best Practices

• Placement of test.skip(): It's essential to place test.skip() at the beginning of your test. This ensures that the skipping logic is evaluated before any test steps are executed, making your tests cleaner and more efficient.
• Descriptive reasoning: The description 'Skipping Safari due to BUG-123' is self-descriptive, providing clear context about why the test is skipped for Webkit. Meaningful skip descriptions are crucial for easy understanding of test case skip conditions and reporting, especially as the number of conditions can grow in size and complexity.

Good to Know

Expanding our knowledge beyond the basic use of test.skip(), let's explore some tips and techniques that can further enhance your testing strategy in Playwright.

• test.only(): This method is the opposite of skipping. It tells Playwright to run only this particular test in the file. It's useful when you want to focus on a specific test without running the entire suite.

test.only('Focused Test', async ({ page }) => {
  // This test will run exclusively
});

• test.fixme(): This method is a special form of skip. It's used to mark tests that are known to be failing and need fixing. It's a way to acknowledge that something isn't working as expected, yet avoiding the failure noise in your test reports.

test.fixme('Broken Feature Test', async ({ page }) => {
  // This test is known to be failing and needs attention
});

• Reporting: When tests are skipped using test.skip(), they appear in test reports, clearly indicating that they were not executed. This feature is essential for maintaining transparency in your test coverage. It ensures that you have a complete picture of what was tested and what was intentionally omitted, aiding in accurate assessments of your test suite’s health and coverage.
• Can be used in Hooks: While test.skip() is primarily used within individual test definitions, it can also be employed inside hooks like beforeEach or afterEach. However, its utility in these contexts might be somewhat limited and not recommended. The skip condition in hooks will apply to all tests in the file, which might not always be desirable. It's more common and practical to use test.skip() within specific tests or even in a test.describe block, where the conditions for skipping are clearly defined and confined to that test and/or test describe block.

beforeEach(async ({ test }) => {
  test.skip(someCondition, 'Skipping all tests in this suite under certain condition');
});

• Hooks are still executed: Hooks such as beforeEach and afterEach are still executed if the conditional skip annotation was called inside the test body, because Playwright has to run the test function to get to the test.skip call. A workaround would be to use an anonymous describe block:

test.describe(() => {
  // Anonymous describe to scope the next test.skip() call that affects all tests in the describe block.
  test.skip(() => true, 'reason');
  test('test title', () => {});
});

Common Pitfalls

While using test.skip() and its variants can significantly enhance your testing strategy, there are common pitfalls that one should be aware of to avoid counterproductive outcomes.

Misplaced Skipping Logic

One of the most critical aspects of using test.skip() effectively is its placement within your test code. It's essential to place this function at the start of your test, before any other operations or assertions.

test('Should not run this for WebKit', async ({ page, browserName, testData }) => {
  await testData.seedUsers();
  await page.goto('https://example.com');
  test.skip(browserName === 'webkit', 'Not for WebKit');
  // Further test steps...
});

In this example, the test will seed a user and navigate to the URL even in WebKit browsers before it evaluates the skip condition. This is not only inefficient but could lead to partial test execution and leaving residual test data. Placing test.skip() too late in the test can also be an issue. This mistake can lead to a part of the test being executed when it should have been skipped entirely, potentially causing confusion when analysing test results.

Overusing Skip

While skipping tests can be a powerful tool, overusing it can lead to gaps in your test coverage. It's important to use test.skip() wisely. Skipping tests should be a deliberate decision, backed by a clear rationale, not a frequent go-to for every minor inconvenience or unoptimised test scenario.

Hope this was helpful and got you to think about the whys and whens before utilising test.skip in your testing. Remember, every test skipped for the right reason is a step towards a more streamlined, effective, and intelligent testing process (and hopefully a faster and passing pipeline).

Additional Resources

To continue your journey with Playwright and test skipping, please refer to the official Playwright docs on Test skipping; Playwright documentation for more detailed information and guidelines on using test.skip() and other test control methods.