The user story experiment Home

My colleagues in industry and universities praise user stories and epics, as the way to write requirements. But there is no agreement on how to use them and how well they cover requirements. The inventors of "user stories" were academics who had tried user stories on toy systems with students.

I had observed User Stories used in real-life projects and noticed that most of them were tiny issues such as as a user I want to see a list of customers. Alright, but what is the situation of use and who is the user? Without this information the user story is no better than a traditional requirement such as: The system shall be able to show a list of customers. Was State-of-the-art really so bad?

Mohammad Kuhail and I designed an experiment where we asked many professional analysts to write requirements for the same real-life project: A support system for a hotline. The system was described in an analysis report written when the original project started. It mentioned 30 functional issues the customer wanted the system to deal with. We encouraged the analysts to ask if they needed more information, but nobody did.

We were lucky to get replies from 8 analysts. In their replies, they used only user stories and epics. We asked the analysts whether they covered all requirements and to our surprise, they all said "yes". Actually, on average, their specifications covered only 10 of the 30 functional issues the customer had stated (ranging from 5 to 13 issues). Further, they hardly mentioned any of the non-functional requirements mentioned in the report, such as usability, data migration and phasing out. Many user stories were wrong: they specified something the customer definitely didn't want.

Below is the project description, which includes an invitation and the analysis report. Next are the 8 professional replies, A to H, trying to cover the project requirements with user-stories. For each reply we show the original reply and a tracking document that shows how well the reply covers each of the 30 issues.

For comparison, we also show a reply, S, based on problem-oriented requirements. It was reviewed and accepted by the hotline staff several years ago. It covers 29 of the 30 issues. Further it covers requirements not mentioned by stakeholders, but needed in most software acquisitions, for instance: ease of learning, response time, security, system maintenance, and phasing-out. None of these were mentioned in the 8 replies.

Details of the investigation and the results are published in Software 2022 - with improved formatting: User story quality in practice: A case study