Testing Philosophy

The Testing Pyramid

Many types of tests exist, such as Manual, E2E (end to end), Unit, Acceptance, Regression, and Integration. Where do you start though, and what importance should be placed on each? Enter the testing pyramid, the pyramid provides a visual representation of test type importance. It was developed to work with the agile methodology and is used industry wide The bottom of the pyramid represents the most importance, while obviously the top represents the least. All are important but with time constraints the provides a guideline to a clearer focus.

For every minute spent organizing, an hour is earned. - Anonymous

Unit Testing

Unit tests are typically written by the developers themselves, and are usually written to ensure specific method functions work as expected when changed or run. This allows the team to make changes without as much fear, and code more aggressively.

    PROS
  • Catcnes flaws much quicker
  • Increase confidence in changing/maintaining code
  • Makes code more reliable
  • Easier to debug issues
    CONS
  • Can miss bugs that only occur when integrated to the rest of the codebase
  • The time expense is large when implementing into an already existing codebase

Integration Testing

Integration tests are the next tier of the pyramid. Similar to unit testing, this is focused more on a specific area. As an example take an e-commerce site, now say a new feature wants to be made to how coupons work or function. This type of test would mean focusing on all coupon functionality ensuring it works as expected. In addition since coupons would be part of the checkout flow it should fall into the category of regression testing, where the entire flow is checked for unintended consequences. Depending on the case, the test could use automation, manual, or both.

    PROS
  • Catcn unintended consequences
  • Ensure bugs are appropriately addressed
  • Ensure new features run reliably
    CONS
  • The pitfall of automated testing; if something was added or the flow was changed the test would need to be updated adding to technical debt
  • Depending on the scale manual testing could take a large chunk of time, and needs to account for out of the box thinking for edge cases

End to End Testing

As the name suggests this type of test runs through typical customer flows from beginning to end. For example a user loads the homepage, searches for a product, add the product to the cart, and then checks out. This is essentially the final greenlight that a release is ready for production, and where automated testing really comes into play.

    PROS
  • Automated tests allow features to be tested much faster than manual testing
  • Tests can be set to run on a wide variety of devices and browsers
  • Automated tests can be set to run multiple times, and can continue outside of working hours
    CONS
  • Automated testing, and Selenium (A framework with a near monopoly on test automation), can be flakey. For example something as simple as changing the name or class of a button can cause a test to fail, while the process is acutally still working as intended. This means maintainance on these tests should be expected and planned for.
  • An automated test can tell you that an element is exists and is interactible, however it may not be able to tell you about any visual oddities

Manual Testing

The final manual walkthrough of the site, mainly looking for visual problems and oddities in behavior but also crawling through some basic flows.

    PROS
  • Catch UI problems missed by other tests
  • Assurance that an actual person has verified things
    CONS
  • Can be very time consuming