A bit of context

Right from the start, the team goal was to provide the features required to have a Minimum Viable Product so that we could onboard clients to have feedbacks and help us to prioritize the next features. We had to find a suitable code quality and tools plus make compromises as most developers and companies does. We wanted to include some tests to help us to deliver a good enough quality for our clients, and we made some decisions over the tests subjects to make it count and make it smart.

This story aims to share what, I think, were the best decisions we made — and also the bad ones — to maybe help you as well to find the good balance regarding the tests on the Front-End side. I also share my thoughts and what we plan to do in the future to improve our tests and processes.

This is the first story covering mostly the decisions we took, and we will publish more stories covering specific in-depth topics related to how we write our tests (syntax, tips, matrices, SUT, etc) and intended mostly to the developers.

DIY

Right from the start, as developers we used Cypress to be able to create end-to-end tests in the CI and locally. Indeed, we were responsible to provide unit, integration and end-to-end tests. However, it became far too much after a few weeks. When working in the early days of the project, having more time dedicated to the tests than to the delivery of new features was not efficient. We would love as engineers to have this luxury, but it was clearly not compatible with the shareholder expectations. So, we had to take actions to increase the productivity, and we decided to recruit some QAs to delegate all the end-to-end tests and reduce as well our expectations of coverage to focus on the delivery.

QAs FTW

So, after three months, we had in each of our team a Quality Assurance engineer to help us ensure that we were not introducing issues. This is an important info for you to consider for the rest of this story because we made many decisions regarding the tests since we had a safety net provided through the QAs, so we could just test less and develop more. Our QAs were using Cypress to provide end-to-end tests, and so we stopped to use Cypress on our side.

QAs is good money

Unfortunately, even with QAs there was so much features to cover that we decided not to perform any kind of visual testing; no end-to-end snapshots (screenshots of the UI before and after some changes to know if something changed) for the QAs, no DOM testing for the developers. I think that it was a good fallback decision for a project  in early stage to avoid creating an armada of tests and also to keep up with the features. Furthermore, snapshots and DOM testing are time-consuming tests, so it was definitely not a good fit for us at this point.

Undoubtedly, investing into the QAs was good. No matter what, developers will introduce some bugs and users will use the features in a way that nobody expects, discovering new issues. Having a QA can reduce the time spent by the developers to test, and he can find more real-life bugs. So, I suggest you to really think about where is the value in the coverage and who is the best candidate in a team to bring that up. It's all about finding the right balance between a QA and a developer and the quality you wish to have.

End-to-End is a must-have at some point

Nonetheless, two years later we were more stable besides we reached some important milestones that allowed us to step back and take some time to improve the quality. We have now some technical subjects to work on. Some of them can alter every single page in the project, and we would love to be able to ensure that everything is still working as expected. But we have a lack of end-to-end testing — both at the local level and at the CI level — so we just can't do it at ease.

Since we don't have snapshots to review the UI changes, we have to review everything as developers, but also as Product Owners and obviously as well as QAs. It is time-consuming for everyone, and it became grunt works.

Thus, it is hard to tell at which point it becomes a good investment for the developers to have end-to-end tests with snapshots. For sure, it's a bit too much when you need to deliver a v1 quickly but becomes a must-have when you start working on UI overhauls or big-bang refactoring.

As a first step, having at least all the critical user workflows covered by some snapshots would help a lot because it's where your project value lives and where the features need to be bulletproof. It's also where the improvements/refactoring will reside, so having UI testing is very neat!

Testing — The developer's side

As I mentioned above, as developers we decided to provide only unit and integration tests. We said bye-bye to Cypress, and we stopped to test the DOM for the components and directives — for now —. We knew what to do, but we needed to define also how to do it. In a team of 15 developers, how do they review each other code and specs? And so, the question of “What to test?” needed an answer.  

Indeed, what to test?

Defining when we need to test something was — and is in fact — the most complicated part of our development process. We don't like to test just for the sake of the tests, nor aim to a certain amount of coverage just to show off. We like to make it count and invest time when we think it's useful — at least until we have enough features to aim at quality over delivery —. The drawback is that it's very difficult to define this usefulness because it is personal for every developer at QIMA. If he thinks that he needs to add some coverage, or not.

Yet, to draw the line somewhere, we defined some thresholds at 50% on the new code coverage in the CI with Sonar. Nevertheless, we did not consider these checks as mandatory to merge a PR. So for now, it's still mostly useless for us, but it's always nice to have this kind of information and one day, we will take it into consideration and make it a blocker.

Then, how do we define when we estimate that a feature needs coverage? If we could write a recipe, it would look like this:

  • If the features comes from our libraries/core, we need them robust and adding more coverage is a good way to ensure it
  • If the feature could be flagged as utile (e.g: a utility function), it's often a good candidate to test it in-depth — for example: a function that checks if a given unknown argument is a string
  • If the feature is related to mathematical operations, having a matrix of different results is useful especially for edge cases
  • If the feature involves a service, it means that it's reusable and like the library and utile code, it's a better candidate to add coverage
  • If the feature comes from a component/directive/pipe, it's less likely to be useful to cover it — it's closer to the UI so closer to have QAs tests, if well written, clean and typed, the main logic will not come from this code

But in the end, we don't have rules written in marble — only suggestions. We are autonomous and it's all about trust. Nonetheless, the tests are like the code: if you let the developers do what they want, it will be a considerable mess and the code base will no longer be representing a team work. So with time, we introduced new rules to ban some bad practices and enforce some good ones.

GWT pattern

Well, we just couldn't miss it. We also decided to use the famous GWT pattern to have some consistency across the developers when writing our tests. Based on the complexity of the tests, we may add some explicit comments to separate the three blocks, but it's totally optional.

Example:

// With explicit comments
it('should authenticate the user', async (): Promise<void> => {
  expect.assertions(1);
  
  // GIVEN
  const email = faker.internet.email();
  const password = faker.internet.password();

  // WHEN
  const result = await service.login(email, password);

  // THEN
  expect(result).toStrictEqual(true);
});
  
// Without comments
it('should authenticate the user', async (): Promise<void> => {
  expect.assertions(1);
  const email = faker.internet.email();
  const password = faker.internet.password();

  const result = await service.login(email, password);

  expect(result).toStrictEqual(true);
});

By the way, if you have some ideas to enforce this pattern — with a linter or any other tool —, I would love to hear them!

Side-note: quite similar — I fear to write identical — to AAA.

Dedicated mock generators

One of the issues we faced was to mock the classes that were not belonged to us. For example, mocking the Router of Angular. We end up with multiple ways to do it, and obviously, it was not efficient.

To improve this, we decided to create a new folder dedicated to every mock in the project so that each time we needed to mock something, we end up with the same mocks. And it became so useful that we decided to do that as well for every exported interface in the project.

One of the drawbacks though was to create them in a way to detect the flaky tests. Indeed, we updated all of our mocks to use Faker to add some randomness to the models.

At some point, we also included an argument to the generators to optionally give a partial model at creation level to simplify even more the boilerplate.

Example:

interface IUser {
  firstname: string;
  lastname: string;
  email: string;
  isAdmin: boolean;
  hobbies: string[];
}

export class UserGenerator {

  // Optionally override the data
  public static mockIUser(user?: Partial<IUser>): IUser {
    return {
      firstname: faker.name.firstName(),
      lastname: faker.name.lastName(),
      email: faker.internet.email(),
      isAdmin: faker.random.boolean(),
      hobbies: [faker.random.word()],
      ...user,
    };
  }
}

// Usage examples
const randomUser = UserGenerator.mockIUser();
const marcoPolo = UserGenerator.mockIUser({
  firstname: 'Marco',
  lastname: 'Polo',
  email: 'marco.polo@qima.com'
});

Testing modules

The more our code base grew, the more difficult it was to handle the breaking changes. For example, if a module A needs HttpClient, we need to import HttpClientTestingModule to test it. If a module B needs to import the module A then this module needs HttpClientTestingModule as well.

With the modularity in Angular, we already have a solution: import the module A into the module B. However, doing the same when using TestBed is not a good fit — and we already showed you a good example with the HttpClient above. So said OK, let's create some testing modules as well. So, we created for each module a dedicated testing module which includes the original module to have the declarations — because you cannot declare a component in two modules — and contains extra DI tokens to override the code for testing purpose.

Architecture example for a login component:

    ├── login
    │   ├── login.component.html
    │   ├── login.component.scss
    │   ├── login.component.ts
    │   ├── login.testing.module.ts
    │   └── login.module.ts

Well, it sounds good, right? We ended up to maintain only these testing modules when the related module changed and import them where need them and voilà. Everything was great, easy, readable, and neat.

Speed battle

Sadly, at some point we figured out that our CI Front-End test step became slower than the Back-End one. And it was a defeat for us, the absolute shame in fact — just kidding —. Thus, we dig into this nonsense right away, and we found out that everything was cooler with the testing modules except the performances. And we didn't see it at first sight because we made the migration to this “design pattern” progressively. So with a lot of pain and suffering, we killed this idea, and we decided to go back to the good old fashion way: make it simple and just don't try to create a factorized TestBed. And basically, we speed up our tests by 50% without even having testing modules everywhere yet. This is the way.

Like I mentioned, it's not all about the things that worked but also about our mistakes.

Side quest: Jasmine to Jest

The reason to have Jasmine at first

We started the project with Jasmine and Karma since it's the default testing frameworks provided by Angular. It was a logical choice to have good examples when following the documentation and also when using the Angular CLI. The tests structure was the same for every developer, and it helped the productivity, especially at the beginning of a project.

Time for a change

Nevertheless, at some point we decided to switch to Jest for performances reason.
Indeed, one of the main advantage is the cache which improve the speed locally and in the CI.  It has also 2 times more stars on GitHub (15k vs 35k at the time of writing this story) which makes it a better candidate on the long run due to the Open-Source contributions.

You can also compare in-depth the differences from the link below.

Jasmine vs Jest | What are the differences?
Jasmine - DOM-less simple JavaScript testing framework. Jest - Painless JavaScript Unit Testing.
StackShareCristian Giagante

A smooth migration

Jest has the advantage of shipping a copy of Jasmine which makes the migration way easier. We could keep what became our legacy tests and write the new ones with the proper syntax for Jest.

For example:

it('should do something', (): void => {
  // Jasmine syntax
  spyOn(service, 'foo').and.returnValue('bar');
    
  // Jest syntax
  jest.spyOn(service, 'foo').mockReturnValue('bar');
});

Both syntaxes are valid which minimize the cost of the migration. It's not the best to live with those two worlds, but it's nice to have the choice to either refactor everything or just live with it.

⚠️ It reduced the grunt task complexity related to the migration, but it was not trivial nonetheless. It may vary from a project to another depending on your test frameworks configurations and code base.

Basically, for a second run — when there is no cache yet — the tests became at least 50% faster than previously.

Matrices

Sometimes, we need to test the outputs of a function based on the given arguments. It may be difficult depending on the total number of outputs to have clean, understandable and unique tests. For example, if you need to write the tests for a function that calculates the total price of a service you provide which takes a service as parameter and can have multiple keys which alter the result — like the price, the vat, the currency, the discount, etc —. We finally found a way to write them that satisfy us, and we like to call it "Matrix testing".

Example with a function that returns zero when the given value is not a finite number:

interface IMatrix {
  inputValue: string | number | null | undefined;
  expectedReturnedValue: boolean;
}

describe('qpZeroWhenNotFiniteNumber()', (): void => {
  describe.each`
    inputValue   | expectedReturnedValue
    ${null}      | ${0}
    ${undefined} | ${0}
    ${'dummy'}   | ${0}
    ${-Infinity} | ${0}
    ${Infinity}  | ${0}
    ${NaN}       | ${0}
    ${888}       | ${888}
  `('when the given value is "$value"', ({ inputValue, expectedReturnedValue }: IMatrix): void => {
    it(`should return ${expectedReturnedValue}`, (): void => {
      expect.assertions(1);

      const result = qpZeroWhenNotFiniteNumber(inputValue);

      expect(result).toStrictEqual(expectedReturnedValue);
    });
  });
});

We think that it's helpful to have an eye-catchy list of inputs/outputs grouped together instead of having a bunch of unique tests. It's not always the best candidate in terms of syntax and should be use smartly, but it's definitely something we like.

System Under Test

As Front-End developers, we are most of the time facing documentations and articles with some examples or ways to test our code which does not reflect all the complexity we could have in our projects. Indeed, sometimes our tests and mocks are quite complicated due to a huge quantity of things to spy, values to mock, etc, and we end up with crappy tests. Besides, we love syntactic sugar, and we think that the code, like the tests, should be clean.

We found one way to achieve that with the notion of System Under Test. Basically, if we think that the tests are very hard to read due to all the spies and mocks we can create a class dedicated for the spec file that will do all the heavy lifting and just hide it through pretty method names (i.e: sugar). With that in mind, we can actually use these methods to "tell a story" of what we are going to do, and it's pretty nice if we can find good names and chain them for understandability.

Example:

describe('MyHipHopArtists', (): void => {
  let sut: SUT; 
  
  beforeEach((): void => {
    sut = new SUT();
  });
  
  describe('getList()', (): void => {
    describe('when there is only one artist in the list', (): void => {
      beforeEach((): void => {
        sut.addArtist('Travis Scott');
      });
      
      it('should return a list with only one artist', (): void => {
        expect.assertions(1);
    
        const result = sut.build().getList();
      
        expect(result).toStrictEqual(['Travis Scott']);
      });
    });
  });
});

// The class to simplify the tests
class SUT {
  private _myHipHopArtists: MyHipHopArtists = new MyHipHopArtists();
  
  public addArtist(artist: string): SUT {
    this._myHipHopArtists.list.push(artist);
    
    return this;
  }
  
  public build(): MyHipHopArtists {
    return this._myHipHopArtists;
  }
}

// Real code added here for this example
class MyHipHopArtists {
  public list: string[] = [];
  
  public getList(): string[] {
    return this.list;
  }
}

As you can see, the purpose of the SUT is to wrap the subject we wish to test into a more verbose API so that we can have reusable mocks, and we can reduce some boilerplate as well as making the tests a bit cleaner. The more complexity you have on your subject, the more useful it is to have a wrapper. It's also convenient when refactoring because most of the code that will require a change will be inside the SUT class instead of inside a bunch of it or beforeEach.

To go further

Now that we onboarded some clients, and we have more time to work on the performances, the quality and the technical debt, we are eager to start working with better practices and tools. Let's finish this story with some of them!

Snapshot testing

It's now a habit for us to enhance and refactor our code base, and we really need to do it in a way that can guarantee us more that we didn't break anything at all — especially on the UI side —. So, this is the feature we need the most right now. We already thought about it, and basically, we would like to start using Snapshot testing with Jest for the critical user workflow and Storyshots for our core components that have a story with Storybook.

We don't have any experience on these topics, so any advice would be helpful 😉

DOM testing

Ah, the (in)famous component fixture! Like I mentioned earlier, we decided to stick with unit and integration tests but without testing the DOM, at all. It means that we won't test the HTML. We don't really need end-to-end tests right now since our QAs are doing great work, and we plan to have some snapshots. But having integration tests involving the DOM would be so much more useful than just testing the TypeScript!

To test like a true Angular developer, we would simply use the fixture to achieve this. Nevertheless, it's not trivial, especially when we need to fit with complex components and the OnPush change detection. And this complexity may have a solution with Spectator. It provides a new way to write the tests which is scary and game changing, but we already used it as a proof of concept and well, it's very neat!

We will come with some news through a new topic once we made up our minds over the best choice.

Factory mocks generator

We already mentioned that we create a bunch of dedicated mock generators for our interfaces as well as dependencies models and classes. We also use Faker to simplify their creation and ensure flakiness. Nevertheless, it is still quite time-consuming to maintain and create them, and we would love to reduce the boilerplate here. One way to do it would be to use jest-ts-auto-mock. In that case, we can get rid of all the generators because this project looks at the definitions and creates random mock. It's not as good as a mock manually generated with Faker but with time, we hope it can get better and better to fit to our needs.

Linters

It's a habit to have fine-tuned linters in the code base but for now, we just have nothing to help us ensure that our tests are good for review. And sometimes it can be quite time-consuming and difficult to review the tests. So having some linters to leverage our code style would definitely help in that way. We already decided to use eslint-plugin-jest at some point, and we will probably do it this year.

Marble testing

We love RxJS at QIMA, except when it comes to the tests. It's difficult to cover the streams, and sometimes it would be very helpful especially for the complex ones to have testing validation. And the main issue comes from our side with the lack of marble tests. The marble API was created specifically for these kinds of tests. And so, one huge improvement would be to start creating our tests with the marble sauce — with or without new libraries —. One day, we will do that, and we will share our experience on this subject through this blog as well.

IDE boilerplates

Angular CLI is a great tool to create the initial files, but it's not enough for the small pieces of code that you can have when developing the same stuff over and over — which can be categorized as boilerplate —.

One way to provide more boilerplate tests is to use our beloved IDEs. It can be very helpful to increase the productivity and avoid minor differences between the code style. At QIMA, the battle is divided between Visual Studio Code and IntelliJ IDEA. Visual Studio Code provides snippets and IntelliJ IDEA provides live templates which are easy to configure and use. We already have some, yet we didn't take the time to find a way to share and version them between us.

Don't hesitate to create a bunch of them, it's really a worthy investment.

Improving Jest

Finally, there is some small improvements that could reduce our boilerplate. We could create our own matchers — which also help for readability —, create global files used inside the environment to provide helpers — could be a great replacement of for the mocks for instance since no import will be required — and discover all the fantastic dependencies around Jest.

Thank you!

This is the end but also the beginning of our stories, so thank you for the interest you gave us, and I hope it may help you!

At QIMA, we like to embrace change and grow together. If you have any suggestion or remark, please share them with us. We are always keen to discuss and learn!

By the way, as you may have understood it, we achieved a lot, but we also have many features to implement to help our clients improve their supply chain. We are currently hiring more Front-End developers! ✨

Written by TESTELIN Geoffrey, Front-End developer at QIMA. ✍️