Automated Testing (Part I)

Feb 2 2017

In this post, I wanted to talk about automated testing. It’s a recurrent issue that when working on a game, you implement some functionality that breaks something you did before. And even that bugs that you fixed at some point in the development start to arise again and again. For different reasons in the game industry we’re not used to doing automated tests to avoid regression. One of the reasons is that game development tools fail at providing a solid and easy way to implement them. Unity Test Tools was an attempt to fix this, but it sucks. Hard. It’s slow and broken.

After quite a few years of working under some agile principles, I developed a need for automated tests. Some people think this is wasting time and that it’s not necessary, I disagree. That’s why I decided to make automated test a first-class citizen in my games.

I had some requirements, though.

  1. Tests should be easy to implement, a simple TEST() method should do the trick.
  2. You should be able to simulate input.
  3. Tests should be isolated, the state that a test sets should be reverted after each test automatically.
  4. You should be able to wait for a certain amount of frames without blocking everything.
  5. Tests should run inside the game engine

So far, I’ve solved 1,2 and 4, but I haven’t worked on isolating tests yet, although I have some ideas about this.

Let’s take a look at the test I showed you in the previous post and let’s dissect it:

TEST(QuickCommands, ClosesWithEscape)
{
  Editor::memory->quick_commands->open = true;

  SET_FRAME(1);
  SET_KEY_DOWN(KEY_ESCAPE);
  END_STEP();

  SET_FRAME(2);
  SET_KEY_UP(KEY_ESCAPE);
  END_STEP();

  SIMULATE();

  WAIT_FOR_FRAMES(3);

  ASSERT_EQ(false, Editor::memory->quick_commands->open);
}

The TEST macro is used to create the test, and I provide two parameters: the test group (in this case QuickCommands), and the test name. The test group, for now, is not used in the code itself other than in the test name, but I’ll be using it soon to group tests in the UI.

I knew I wanted that API for the tests, and I knew it was possible to achieve since I’ve used Google Test before and they did the same thing. So I started thinking about how to implement it.

The main problem is that I wanted all these tests auto-register themselves when the program started, and it’s not evident how to do that.

First of all, I had to break one of the rules I had until now, and that is that all my memory allocations should reside in a MemoryArea.

If I wanted these tests to be loaded at start, they would have to be registered somewhere statically, or the macro won’t work.

I found a way to do this using a statically allocated struct with a constructor that runs some code to add the test to a tests array. Is this tests array the one that breaks the rules. Because I can’t have it defined in my Testing::Data* struct as (I’ve done with every other submodule) because it’s not allocated when I run the application, it’s gets allocated when all the submodules are.

Because of this tests must be allocated statically in the namespace:

namespace Editor
{
  namespace Testing
  {
    Data* data;

    // This array needs to be accessed before Testing::start is called, so I added it statically.
    Test tests[MAX_TEST_COUNT];
    int test_count = 0;
    [...]
  }
}

Now that this is clear, let’s see the hacky thing that I had to do to get the tests registering themselves in the test suite.

#define TEST(suite, name)                                                                          \
  U8 run_##suite##name(void);                                                                      \
  struct Test_##suite##name {                                                                      \
    Test_##suite##name()                                                                           \
    {                                                                                              \
      Editor::Testing::create_test("[" #suite "] " #name, __FILE__, __LINE__, &run_##suite##name); \
    }                                                                                              \
  };                                                                                               \
  Test_##suite##name _##suite##name;                                                               \
  U8 run_##suite##name(void)

I know, looks awful, but it’s not that bad if you know what’s going on. The first line is needed to define the structure of the test method itself. We define the body of that method after the call to the TEST macro (that’s why after the call we create a scope with {}).

After that definition, we create a struct that in its constructor, calls Editor::Testing::create_test, and register the tests, including the name, filename where it was created, the line it was created in (this two help when reporting a failed test!), and a pointer to the run_* method we defined before. The empty definition of the run_* method is done so that this pointer doesn’t generate an error on compile.

When we’re done with the struct we actually statically allocate an instance of this struct. This will run the constructor when the application starts, successfully adding the test to our tests array! Boom! Magic.

After that, we define our method name and parameters and leave it open, so we can define the body of the method after the macro call.

I think it’s clearer to see all of this with the macro expanded:

U8 run_QuickCommandsClosesWithEscape(void);

struct Test_QuickCommandsClosesWithEscape {
  Test_QuickCommandsClosesWithEscape()
  {
    Editor::Testing::create_test("[" #suite "] " #name, __FILE__, __LINE__, &run_QuickCommandsClosesWithEscape);
  }
};
Test_QuickCommandsClosesWithEscape _QuickCommandsClosesWithEscape;

U8 run_QuickCommandsClosesWithEscape(void)
{
  Editor::memory->quick_commands->open = true;

  SET_FRAME(1);
  SET_KEY_DOWN(KEY_ESCAPE);
  END_STEP();

  SET_FRAME(2);
  SET_KEY_UP(KEY_ESCAPE);
  END_STEP();

  SIMULATE();

  WAIT_FOR_FRAMES(3);

  ASSERT_EQ(false, Editor::memory->quick_commands->open);
}

In the next post, we’ll take a closer look at how the actual module is implemented and how assertions are handled.

Until next time!

comments powered by Disqus