As a financial product, Wealthfront runs on trust. Our clients expect the product to be there for them and work as they expect. Engineering at Wealthfront makes writing high quality tests a priority. For our clients to trust our product and brand, we must ensure we have that same confidence in our tests.
Our test runner, Mocha, has great built in support for promises. When you return a promise from a test the test will pass if the promise resolves, and fail if the promise rejects.
Since we didn’t return the promise, Mocha ignored the result of the promise and passed blindly. For our test to behave as intended, we need to return that promise so that Mocha knows whether to fail the test. The above test should be written like this:
We discovered this problem when we recently upgraded to Node 6. Node 6 logs a message on rejected promises that are never handled. For example, the following code causes Node to output a warning to the log.
Fixing our tests
There are two steps to solving this problem. We need to fix all of our incorrect tests, and we need an update to our testing platform to ensure this never happens again.
The most important thing is to make sure unhandled rejections fail our test suite. We can’t fail a specific test since an unhandled rejection means that the test has already finished running. Node and browsers don’t wait for promises to be handled before exiting, so tests at the end of your test suite with this problem may never be caught.
Because of this, we have to be happy with a solution that will:
- Provide enough information about the error so the developer can find and fix the test
- Make sure our test suite exits with a non-zero exit code, ensuring the rest of our suite fails out
Usng a Linter
We prefer using linters over unit tests since they are much faster to run than our entire test suite. So far we have been talking about unhandled rejections, but there is another case we should avoid: unreturned successful promises. For example:
The linter can only provide a best effort attempt since it has to make assumptions based on the existence of function calls of
.then. Since there are ways that would pass this static checking, we can’t rely solely on the linter. It is important that we also enforce this at runtime. We can use the following approaches to do that.
Node emits an event on
process when a promise that does not have a
.catch handler in it’s chain rejects. Using that to fail out of our test suite is relatively straightforward.
This causes our node test suite to exit with a status code of 1 and print to the logs:
Chrome 49 added an
unhandledrejection event and, at the time of writing this post, is the only browser to support this event. We use karma to run our unit tests in browsers. Since we run our test suite in multiple browsers, if we can cause the karma run to fail in Chrome, we can cause our overall test suite to fail.
Handling this in the browser is a unique challenge. Unlike Node, we don’t have a way to call
process.exit(1). There are a few valid options to be able to solve this problem, but they each have their tradeoffs.
Karma exits before Chrome’s
unhandledrejection event finishes, such that if your suite contains only a single test, karma will exit with no errors. Because of this, to demo the solution, we need another test to provide a short delay to the test suite exiting.
Rethrow and use
Since karma attaches an event listener for uncaught exceptions, we can rethrow the error from our
unhandledrejection handler which causes karma’s
onerror handler to be called.
This is a simple solution, but causes an unintended side effect. The error appears to come from inside the
should delay test. This approach could confuse engineers as they try to debug these issues since the error might not be anywhere near that test. It also prints the error message twice.
Fail using Karma’s internal api
Karma provides an object on
window that it uses to be able to
postmessage back to the running node process in order to share status. We can use this to directly cause the karma suite to fail.
Using this approach, it no longer reports the error coming from a test and only prints the error once. This is a huge improvement in that it keeps our engineers from chasing the wrong lead. Unfortunately, it requires us to use an internal api that could change at any time. Based on the tradeoffs, this is the approach we decided to take at Wealthfront.
Long term there will likely be better solutions as this event is more widely supported. Mocha is also looking to add support for unhandled rejections into the core.
Enforcing this in every project
We have written previously about how valuable it is to have a consistent test suite across all of our projects. Since we have a package that configures our test suite, we have a place to conditionally add the node or browser event listeners. This has enabled us to easily roll this change out to all of our test suites to make sure we don’t have unhandled promise rejections.