CST438 - Week 5

 This was our fifth week in CST438 - Software Engineering

Reflection

Last week we set up the backend for our group project, in the form of a RESTful API. This week we are setting up the front end of our project using React. My part was the instructor's assignment page, which allows instructors to view assignments for a section, and create, edit, or delete assignments. Overall, this week's assignment went much more smoothly than last week's assignment. This is largely because we didn't have to write any tests this week, we'll be doing that next week. The overall length of code for this assignment was also much shorter than last week's assignment, so that also helped.

This week we read about larger tests. I wrote down some of the main ideas below. As I believe I've mentioned before, my experience with tests is limited to unit tests and the integrated tests from a previous lab in this course. So it was interesting to read about larger tests, why they are done, and the challenges that they come with. While we will be dong more integrated tests using Chrome drivers, we won't be writing tests that involve multiple machines or SUTs. This makes me wonder if and when I will get to write this type of test. 

SAG Ch. 14 - Larger Tests

This chapter discusses tests that take up more than one thread or process, and may be split across multiple machines.

 

Larger tests have the following characteristics:

• They may be slow.
• They may be nonhermetic, or share resources with other tests and traffic.
• They may be nondeterministic, other tests or processes may interfere with it.

While unit tests ensure that individual methods, objects, or modules work, larger tests ensure that the overall system works as intended. To do this they focus on fidelity, how reflective a test is of the real behavior of the system.

Large tests are used to cover some of the weaknesses of unit tests:

• Unfaithful Doubles: Unit tests often use doubles or mocks to eliminate dependencies, but sometimes dependencies and their doubles can behave differently.
• Configuration Issues: Production code often has additional configurations which can sometimes cause problems. Unit tests are largely unable to catch these potential problems.
• Issues that arise under load: Unit tests are designed to be small and fast, and therefore cannot test how code may react when it is under high traffic.
• Unanticipated behaviors, inputs, and side effects: Unit tests can only test for anticipated behaviors, and different methods must be used to test for unanticipated behaviors.
• Emergent behaviors and the "vacuum effect": Unit tests work in a vacuum and thus have a limited scope. They cannot catch errors outside of this vacuum.

However, unlike unit tests, larger tests are much less reliable, fast, and scalable.

Large tests typically follow this structure:

• Obtain a system under test
• Seed necessary test data
• Perform actions using the system under test
• Verify behaviors

Tests involving SUTs have two main types of data:

• Seeded Data: Data preinitialized into the SUT at the start of the test.
• Test Traffic: Data sent to the SUT by the test during its execution.

Test data can be generated in the following ways:

• Handcrafted Data: Creating test data by hand.
• Copied Data: Copying data, typically from production.
• Samples Data: Copying samples of data to reduce the amount of test data compared to simply copying it.

We can verify SUT behavior using the following methods:

• Manual
• Assertions
• A/B Comparison: Running two copies of an SUT, sending the same data, and comparing the output. The output must be manually compared.

The book defines the following list of types of larger tests:

• Functional testing of one or more binaries: Uses an SUT that is single-machine hermetic or cloud-deployed isolated, handcrafted data, and assertions.
• Browser and device testing: Testing the UI and public APIs of browsers and mobile devices.
• Performance, load, and stress testing: Uses an SUT that is cloud-deployed isolated, data that is handcrafted or multiplexed from production, and A/B diff verification.
• Deployment Configuration Testing: Intended to see if an SUT crashes with a certain configuration or not.
• Exploratory testing: Focuses looking for questionable behavior by trying out new user scenarios.
• A/B diff (or regression) testing: Compare differences in old and new versions of services and determine whether the differences are anticipated or unanticipated (regression).
• User acceptance testing (UAT): Automated tests for a public API of a service.
• Probers and canary analysis: Check that a production environment is working. Canaries are used to test a release that is being pushed to the production environment.
• Disaster recovery and chaos engineering: Test how well system react to unexpected changes or failures.
• User evaluation: Changes are given to users (often a subset of users, sometimes without their knowing), and their reactions and evaluations and gathered and compared.