Electron framework regression testing with spectron

When your development process comes to a point where multiple changes can impact the existing features of your application, it’s probably time to implement regression tests. 

With these tests, you are making sure that your product is stable and that any newly implemented features won’t crash it.

In this blog post, we’ll be covering our experience of writing regression tests in Spectron with Jest. This was done for an application developed in the Electron framework with React. 

Also, we will try to explain our dilemma on why we chose that test tech stack and why we thought it would be the best possible solution for our client.

This is not a complete “how to” blog post, but simply sharing experience and what we went through when writing the tests.

We will provide some pseudo code, but there will not be a full github code example. It is not important to know about the tech stack if you just want to understand what we’re talking about. 

The whole project has been finished recently and we decided to write down our experiences and thoughts to see what we’ve learned. We’ve been working on it through the last year, and the automation of regression tests was the last phase.

Why you Need to Automate Your Tests

So you can understand better how the situation was developing, we’ll get back to the start.  The client requested an Electron-based application and our job was to finish the application with React for the UI. 

Backend was served and maintained by the client. After a couple of months we’ve come to a point where most of the application was done, but the client was still adding backend features.

After each added feature they were doing manual testing to see if a regression happened. It didn’t happen every time, but they still had to check it manually, which was very time-consuming. That’s why we came to the idea of automating the process. 

If you encountered this article I guess you were searching for the right tool for automating tests in your Electron app. So were we.

Most results that we’ve encountered were about Selenium and Spectron, but we’ll go into detail about the two later. However, the main reason why we didn’t automate our tests with Selenium was that you would probably have to manually set up WebdriverIO to wrap around Selenium so you can control the Electron app.

On the other hand, if you choose Spectron, you get out-of-the-box functionalities that allow you to use WebdriverIO with Selenium to control Electron applications. 

Furthermore, Spectron is the official open source framework developed by the same guys that are working on Electron. 

In the official documentation of Electron, there is only one page of resources about Spectron. That was a little disappointing, but it checked all the boxes for the specific things we were looking for. Here’s the list:

• CI ready
• Out of the box API
• Multi Window
• Compatible with popular test frameworks

Everything seemed ideal, and then we’ve checked the official Spectron git project. There were issues opened, which is normal for every project, but we’ve also seen that those issues were being solved and maintained. 

At this point basically everything was leading us to choose the Spectron framework as we would get out of the box API with WebdriverIO. That meant we wouldn’t even need to touch Selenium. 

This way we would be faster with the initial project setup, as well with writing tests. 

Since the tech stack we were working with was new, we were pretty sure that everything was going to be maintained thoroughly. This made sure that dependency updates for the project would be easy to upgrade. 

Spectron vs Selenium Dilemma

As mentioned previously, we’ll try to demonstrate the pros and cons in the Spectron and Selenium comparison.

Why not Selenium

Selenium WebDriver allows you to write tests that can control the web browser. It provides an API that can reproduce user interaction with browsers.

It’s an open source tool which provides functionalities such as navigating through the web, application interaction, JavaScript execution, etc.

This means that you must write your own and custom API to control Selenium, and it would result in having to waste a lot of time on that. 

Why Spectron

On the other hand, with Spectron, you don’t need to even touch Selenium. WebdriverIO (WDIO) takes care of controlling Selenium WebDriver on our behalf. Spectron is then wrapping the WDIO and allowing us to test any Electron application. These have bindings for Selenium while using Javascript. 

Github

One thing that I want to mention here is actually a big deal. Since Spectron is the official testing framework for Electron (suggested by their team), you would guess that everything would be fine. But you’re wrong.

Since Spectron is dependent on WebdriverIO, every time WDIO would get a major update, the Spectron team would fall behind. 

Even when the version would get bumped, there would be a lot of issues and breaking changes that just takes up your time for maintenance. 

We decided to lock the most stable version for our tests, and then ignored the new major updates completely.

In the end, it didn’t seem like a big deal. Everything was working and the test runs were completely fine. 

Project setup 

Architecture

We used the Jest testing framework and decided to separate the test suites into a single folder. 

Inside it, we categorized and divided test actions by specific “domain.” For example, if you want to test if your app is starting up correctly you would create a startup.test.ts. 

All tests have a .test extension which distinguishes test suites from other parts of the code. 

The application was then diced up into specific pages and all the code was put into the “pages” folder. 

Page as Object Pattern

Instead of plain javascript, we decided to go with Typescript. Also, Typescript features gave us the ability to use the full potential of object-oriented programming. 

Next step was to implement the Page as Object pattern to abstract the code from the actual tests. If you want to find out more about it, there is a nice article with code examples on the official WebdriverIO documentation.

The main point of that principle is to think of application elements as “first-class citizens”. By using that pattern you get clean classes that provide nice features such as:

• Inheritance between similar pages
• Lazy loading of elements
• Encapsulation of methods

First, you start off by making the main page object. We called that a BasePage in which we extracted all the selectors and methods. Every following page object inherits these. 

For example, there you can put methods to check if a specific page is loaded or to verify the page integrity.

Also you can abstract actions that can happen on every page such as click, adding values to text fields, clearing values, checking if a specific element exists.

After that, a specific page, i.e. LoginPage, can inherit BasePage and use all of its methods. You can then define the important selectors and actions that are required.

Pattern code example

This part was made to give you insight into how we implemented the pattern and how it benefited us. This is just an example to demonstrate how we did it. I’m sure if you know how to organize your code better, you will do it. 

First we have a base page where we abstracted all of the methods:

export class BasePage {
    driver: Application;
    /**
     * Class Constructor.
     *
     * @constructor
     * @param driver - Electron Application driver.
     */
    constructor(driver: Application) {
        super(driver);
        this.driver = driver;
    }
    /**
     * Helper method for verifying that a given element is visible.
     * Waits a certain amount of milliseconds, based on `timeout`, for the element to be rendered.
     *
     * @async
     * @param element - The selector for the element we want to verify.
     * @param timeout - The amount of milliseconds to wait for the element to appear.
     * @returns a Promise for the visibility of the element.
     */
    protected async verifyOnPage(element: string, timeout: number = mediumWaitMs): Promise<boolean> {
        await this.driver.client.waitForVisible(element, timeout);
        return await this.driver.client.element(element).isVisible();
    }
    /**
     * Helper method for clicking a given element.
     * Waits a certain amount of milliseconds, based on `timeout`, for the element to be rendered.
     *
     * @async
     * @param element - The selector for the element we want to click.
     * @param timeout - The amount of milliseconds to wait for the element to appear.
     */
    protected async click(element: string, timeout: number = 1000): Promise<void> {
        await this.driver.client.pause(timeout);
        await this.driver.client.click(element);
        await this.driver.client.pause(timeout);
    }
    /**
     * Helper method for adding a given value to an element.
     * Waits a certain amount of milliseconds, based on `timeout`, for the element to be rendered.
     *
     * @async
     * @param element - The selector for the element we want to add a value to.
     * @param value - The value to add to the `element`.
     * @param timeout - The amount of milliseconds to wait for the element to appear.
     */
    protected async addValue(element: string, value: string, timeout: number = 1000): Promise<void> {
        await this.driver.client.pause(timeout);
        await this.driver.client.clearElement(element);
        await this.driver.client.element(element).addValue(value);
    }
    /**
     * Clears the value of input element
     *
     * @param element element id
     */
    protected async clearValue(element: string): Promise<void> {
        await this.driver.client.clearElement(element);
    }
    /**
     * Helper method for checking if an element exists on the page.
     * Waits a certain amount of milliseconds, based on `timeout`, for the element to be rendered.
     *
     * @async
     * @param element - The selector for the element we want to click.
     * @param timeout - The amount of milliseconds to wait for the element to appear.
     */
    protected async isElementExisting(element: string, timeout: number = 1000): Promise<boolean> {
        await this.driver.client.pause(timeout);
        return await this.driver.client.isExisting(element);
    }
    /**
     * Gets HTML element value
     * @param elementId HTML element id attribute
     */
    protected async getInputElementValue(elementId: string): Promise<string> {
        return await this.driver.client.element(elementId).getValue();
    }
    /**
     * Gets HTML element value
     * @param elementId HTML element id attribute
     */
    protected async getElementAttribute(elementId: string, attribute: string): Promise<string> {
        return this.driver.client.element(elementId).getAttribute(attribute);
    }
    /**
     * Helper method for getting text from a given element.
     * Waits a certain amount of milliseconds, based on timeout, for the element to be rendered.
     *
     * @async
     * @param element - The selector for the element we want to obtain text from.
     * @param timeout - The amount of milliseconds to wait for the element to appear.
     * @returns A Promise for the text contained in the `element`
     */
    protected async getText(element: string, timeout: number = 1000): Promise<string> {
        await this.driver.client.pause(timeout);
        return await this.driver.client.element(element).getText();
    }
    /**
     * Helper method for pressing enter key.
     * Waits a certain amount of milliseconds, based on `timeout`, for the element to be rendered.
     *
     * @async
     * @param element - The selector for the element we want to add a value to.
     */
    protected async pressEnter(element: string, ): Promise<void> {
        return this.driver.client.element(element).keys('Enter');
    }
}

Then, as we said we have a LoginPage where we’re inheriting the BasePage class

export default class LoginPage extends BasePage {
  usernameInput: string;
  passwordInput: string;
  loginButton: string;
  createAccount: string;
  changeUserButton: string;
  /**
   * Class Constructor.
   *
   * @constructor
   * @param driver - Electron Application driver.
   */
  constructor(driver: Application) {
    super(driver);
    this.usernameInput = '#usernameInput';
    this.passwordInput = '#passwordInput';
    this.loginButton = '#logInButton';
    this.createAccount = '#registerLink';
    this.changeUserButton = '#changeUserButton';
  }
  /**
   * Returns True if the page is done loading, based on an HTML identifier
   * unique to that page.
   *
   * @async
   * @returns a Promise for the true/false load status of the page.
   */
  public async loaded(): Promise < boolean > {
    return await this.verifyOnPage(this.usernameInput);
  }
  public async checkIfUserLoggedOut(): Promise < boolean > {
    return await this.isAwaitedElementVisible(this.changeUserButton, 60000);
  }
  /**
   * Clicks on register button
   */
  public async createAccountClick(): Promise < void > {
    await this.click(this.createAccount);
  }
  /**
   * Returns true as a string if
   * username is disabled
   */
  public async isUsernameDisabled(): Promise < string > {
    return await this.get$Element(this.usernameInput).getAttribute('disabled');
  }
  /**
   * Returns true as a string if
   * log in button is disabled
   */
  public async isLogInButtonDisabled(): Promise < string | null > {
    return await this.get$Element(this.loginButton).getAttribute('disabled');
  }
  public async fillCredentials(username: string, password: string): Promise < void > {
    await this.addValue(this.usernameInput, username);
    await this.addValue(this.passwordInput, password);
  }
  /**
   * Returns true if all element selectors are present on the page.
   *
   * @async
   * @returns A Promise for the true/false display status of all element selectors.
   */
  public async verifyPageIntegrity(): Promise < boolean > {
    return (
      (await this.verifyOnPage(this.usernameInput)) &&
      (await this.verifyOnPage(this.passwordInput)) &&
      (await this.verifyOnPage(this.loginButton))
    );
  }
}

And in the end, we have the result of the written tests in Jest testing framework.

 /**
 * Basic tests to make sure creating an account works.
 */
describe('setup test: ', () => {
  test('should launch app', async () => {
    expect(client.isLoaded()).toBeTruthy;
  });
  test('should title of app be correct', async () => {
    const title = await client.getAppTitle();
    expect(title).toBe(‘TestApp’);
  });
  test('should all element ids be correct', async () => {
    expect(await client.loginPage.verifyPageIntegrity()).toBeTruthy();
    await client.loginPage.createAccountClick();
    expect(await client.registerPage.verifyPageIntegrity()).toBeTruthy();
  });
  test("should click on 'already have an account'", async () => {
    await client.registerPage.returnToLoginClick();
    expect(await client.loginPage.verifyPageIntegrity()).toBeTruthy();
    await client.loginPage.createAccountClick();
    expect(await client.registerPage.verifyPageIntegrity()).toBeTruthy();
  });
});

Conclusion

The decision to automate the regression testing, or basically to automate anything, is always a smart decision.

With automated regression tests we saved a lot of time that was getting wasted on manual testing. Using Typescript helped us a lot to avoid type errors and easily use object oriented programming to abstract a large amount of code. 

The Page as Object pattern was also a good decision, which proved to have simple written tests that required less maintenance. 

Choosing Spectron as the Electron test framework is debatable if we consider all the hardships that we’ve encountered while using it, but it definitely helped us do our job faster than writing our own Selenium and WebdriverIO API. 

Also, being an officially supported library for testing didn’t mean a lot. You are reliant on the vision of the company, and the team responsible for maintaining the library which can be risky. If they fall behind, it might affect your project. Heavily.

I hope we managed to give useful insight on this matter. Automation of regression tests can maybe seem redundant to some developers, but the feeling of insurance that you get whenever a new feature is added is simply irreplaceable. If you add the amount of time it saves on top of that, there is really no denying that investing time in writing them is a smart idea.

If you have any questions, or want to work with an experienced team, feel free to contact us at business@decode.agency