User Acceptance Testing: How to Run Effectively

What is User Acceptance Testing?

User Acceptance Testing (UAT) is the software testing phase during which the system is tested for acceptability and validated for the end-to-end business flow. User Acceptance Testing is conducted by the end-user or the client in a separate environment, similar to the production environment. During User Acceptance Testing, you test end-to-end scenarios and this typically involves running a suite of tests on the completed and integrated systems.

Some Key Points about User Acceptance Testing (UAT)?

1. This testing is to be conducted in the final stage of the Software Development Life Cycle (SDLC) before the system is delivered to a live environment.
2. Acceptance testing is regarded as “black box” testing, which means UAT users are not aware of the internal structure of the code, they specify the input to the system & check whether systems respond with correct results.
3. User Acceptance Testing is preferably performed after System Testing is done and signed off. All Sev1 and Sev2 defects have been fixed and retested.

Fundamental Best Practices for User Acceptance Testing

One of the critical pain points in UAT is how late in the cycle this testing takes place. Typically User Acceptance Testing is one of the last efforts to ensure the quality before the product goes live. Keeping this fact into consideration, the UAT must be carefully planned. The following workflow can come in handy and prove useful to ensure a smooth UAT cycle.

1. Planning
2. Execution
3. Documentation and Reporting
4. Evaluation

Planning the User Acceptance Testing

An important truth for any process where quality must be determined is the fact it should be thoughtfully planned and the same goes for UAT as well. To effectively plan the User Acceptance Testing phase, it needs to be included in the master project plan and scheduled across the life of the project. The following checklist gives an overview of what needs to be planned for a successful UAT cycle.

• Determine the point of contact for directing UAT testing.
• Determine the  UAT team. A mixed team comprising the end users, members from the business analysts team and the QA team is a good option.
• Gather all the requirements that will be verified during the UAT phase. If an Agile process is followed, the Acceptance Criteria can be the reference and usually provide the necessary details.
• Setting up the Environments for UAT is another key factor in UAT testing: Where will this test be conducted? Conducting the UAT in the regular development and QA environments should be avoided – these change frequently and any disruption in the development process could impact the schedule. Ideally, a separate environment for UAT would work best. It is crucial to set this up well in advance of the UAT to be sure everything is working. To iron out any inconsistencies, the QA team should do a smoke test before handing it to the customer for UAT.
• Determine the schedule and communicate the UAT testing dates. The UAT should be planned in a manner that gives some buffer for fixing any critical defects identified before the production release.

Execution of User Acceptance Testing

The execution phase of the User Acceptance Testing is the one that definitely holds the most anxiety of the UAT process. What if the customer doesn’t like it? What if there is a major issue? What if there is a requirement gap? And so on …. So if the implementation team is not prepared for the UAT and have questions about things going wrong, then cancel it!

The execution phase is an opportunity to get closer to your client and do not miss this to make client a raving fan of yours. This implies that full disclosure is important in UAT. If there is a major bug,  don’t hide it – openly discuss it and work around it.

It is helpful to have the basic test cases written, reviewed and signed off by the responsible stakeholders. The test cases should be made available in a test management system that tracks execution. This however, does not negate the importance of allowing customers to interact with the system in an exploratory manner but rather, provides a framework establishing the guidelines within which the UAT testing is conducted to ensure all the requirements have been reviewed and tested. Always remember: Clear expectations are key for a successful UAT. The following checklist gives an overview for executing UAT in a successful manner:

• Assure that the system is ready to be UAT’ed by the customer. At the minimum, all the Sev1 and Sev2 defects from the SIT cycle should have been closed. Any Sev3 or Sev4 defects would have to be communicated to the client in the SIT Test execution report.  
• A technology team/person should also be arranged to support the UAT activities. The technology team should be responsible for clarifying any queries raised regarding functionality or any bugs reported during the UAT execution. A developer and a senior QA from the implementation to start with at least.
• Track the test case execution and record all results and queries raised.
• Ensure the UAT team challenges the system by executing the scenarios outside of the happy path.
• Treat this execution as an opportunity to build relationships and trust with the client.

Documentation and Reporting

Ideally, the documentation and execution should happen in parallel. Recording and documenting the test results, findings, bugs and other relevant details is beneficial in evaluating the UAT execution. It is recommended to set up a system that lets you document all relevant information without losing any data. At the end of the UAT cycle, a test execution report should be prepared to document all the details:

• What was the scope and coverage of the UAT?
• How many testers were involved in completing the test execution complete the test cases?
• How many defects were encountered and the severity of each defect?
• What is the current status of those of those defects? Have those been fixed or deferred for the next release?
• Observations
• Requested changes
• Requested additions

It is worth mentioning here that the UAT test team must provide daily status updates to the stakeholders regarding the number of test cases executed, defects encountered and the current status of those defects. More importantly, any Sev1 or Sev2 defects should be reported and triaged immediately for the implementation team to provide the fix. This will ensure that there is minimal impact on the release schedule.

Evaluating Test Results

Once the test execution is completed and the test results documented, it is now time to evaluate and retrospect those results and processes. When evaluating the test results and findings, allows us to find answers to some of these questions.

• What went well during the UAT?
• What are processes that can be improved on?
• What challenges did we encounter and what could be the remedial actions?
• Has any test case failed? Why did the test case fail?
• Did the test case fail because there was a requirement gap or was missed during the SIT phase?
• What was our approach to the change control process? Did we accept improvements during the UAT phase that impacted the release schedule?
• What was the approach of each tester? Did he stick to the test cases or perform exploratory testing?
• What was the overall sentiment of the UAT team when doing the execution? Were they skeptical or were they comfortable during the execution?
• Answers to these questions will help in gathering insights and lessons learned which will help to improve your future UAT test cycles.

Conclusion

User Acceptance Testing isn’t easy or trivial. A successful UAT cycle depends on solid planning and execution throughout the project and not just at test time. But with the right approach and planning, the chances of UAT being a successful event increases significantly.

Get in touch with us if you have more questions about User Acceptance Testing!