QuestionQ66

VerifiedPerformance Measurement Fundamentals

Scenario —

You have been asked to test the performance of a software and system update to a North American mobile trivia quiz-game application. The quiz app is offered through major app stores to English-speaking users. The planned number of concurrent users is 5 million. Winning quiz participants are promised a portion of an allocated cash prize. After participants begin playing, their quiz sessions must not be lost, particularly when all previous questions have been answered correctly. The application owner wants to ensure that 99% of quiz sessions are retained (up to 4.95 million sessions) until the final answer has been successfully recorded.

You performed 100 test runs providing 5 million virtual concurrent users. You have measurements of participant concurrency at the final question for every test run. In your analysis, you sort all concurrency-test results by measured concurrency value from best to worst. You determine the 85%, 90%, 95%, and 100% values by selecting the 85th, 90th, 95th, and 100th (final and worst) measured values, counting list positions from the best value toward the worst until reaching the required position (for example, 85). Remember that results at lower list positions are as good as or better than the results selected for reporting.

After analyzing the performance-test results, you present the following results:

  • 85%: 4.92 million valid concurrent users
  • 90%: 4.90 million valid concurrent users
  • 95%: 4.0 million valid concurrent users
  • 100%: 3.8 million valid concurrent users

Based on these results, what should you report to the stakeholders?

Explanation

At an 85% reliability level, the 85th-best result must meet the required 4.95 million valid concurrent users for the 99%-of-5-million target to be reliably achieved. The reported 85th result is 4.92 million, so the target is not met at that reliability level; results ranked above it may still be better.

Community Discussion

No comments yet. Be the first to start the discussion!