E2E tests are hard because they test everything. In a web application, they touch your UI, APIs, and databases. When there are so many touchpoints, something will go wrong. And let’s be clear, you want that. However, the hard part is figuring out how to structure your end-to-end tests to cover all of the various touchpoints. But for all of the moving pieces to get tested correctly, you need data.
Use a Live Database
- (Pros) Tests the entire stack
- (Cons) Depends on live data not changing
- (Cons) Requires fake data in live DB
A live database is dangerous but effective. The entire stack for the web application is tested. There’s no need to set up and maintain a separate test database. However, this will require that you have fake user accounts and data in the production database. Not to mention, you will have to filter or turn off web analytics tracking for test actions. You could set up and clean up test data for the end-to-end tests you are going to perform. But this will get complicated if the tests fail, and then you have to clean up manually or via a script.
Use a Test Database
- (Pros) Tests the entire stack
- (Pros) We can seed it however we want
- (Cons) Hard to set up and maintain
A test database is the most sensible approach for most web app development. Make sure you have an exceptional DBA and DevOps team to maintain a test database with sanitized data seeding the database. You can seed it with any account and interface. But be careful. All of my experiences in companies prove there’s a lack of long term maintenance for test databases. Not only do data models and schemas change, but the people that maintain such databases leave or are eliminated by organizations once projects finish. This approach may also not work if your application data depends on too many database sources; Elastic Search indexes, Mongo DB, Postgres DB, etc. Synchronizing so many data sources into a test environment is a lot of work.
Use no Database (mock data calls)
- (Pros) Easier to setup
- (Pros) We can mock it however we want
- (Cons) Doesn’t test the entire stack
Creating mock data calls is the approach you would follow if you are going to be testing things in isolation. Is this valid end-to-end testing? Hell no. But this is useful if you want to test particular functionality in the application without relying on long-running backend processes or testing for individual app UI states. If you have no database, or the DevOps team is still working on giving you a test database, then start with this approach.
Which approach to follow depends on your situation
It’s true. The correct answer as to which approach to follow is that it depends highly on your circumstance. The safe approach is to use a test database. But keep in mind that some teams cut corners and never implement the same data models in the test database. Worse, the test database is not sanitized to remove sensitive customer data. And when it comes to companies eliminating positions, the more likely to go are the database administrators and backend developers who were maintaining the test systems. I have been in many companies where the test database is entirely out of synch with the database infrastructure and data schema as the one in production. So keep that in mind and adapt end-to-end testing to your organization’s situation.
