Icsme2021

# The Impact of Continuous Code Quality Assessment on Defects

#### ICSME'21 - NIER Track

Helge Pfeiffer, Assistant Professor,<br>
[Research Center for Government IT](https://www.itu.dk/forskning/institutter/institut-for-datalogi/forskningscenter-for-offentlig-it),<br>
[IT University of Copenhagen, Denmark](https://www.itu.dk)<br>
`ropf@itu.dk`

---

## Aspects of SW Quality

Previous studies demonstrate the impact of various aspects on software quality.

* Organizational aspects impact defect rates:
    - organizational structure ([Nagappan et al. 2008](https://dl.acm.org/doi/pdf/10.1145/1368088.1368160)) 
    - code ownership ([Bird  et al.](https://dl.acm.org/doi/pdf/10.1145/2025113.2025119))
    - adherence to processes ([Herbsleb et al. 1994](https://apps.dtic.mil/sti/pdfs/ADA283848.pdf), [Diaz et al. 1997](http://faculty.salisbury.edu/~xswang/Courses/csc425_426/ReadingMaterial/Process_Improvement_Motorola_IEEE_Software_Sep1997.pdf))

* Socio-technical aspects impact defect rates, requests for assistance, or user-perceived quality:
    - code review coverage and code review participation ([McIntosh  et al. 2014](https://dl.acm.org/doi/pdf/10.1145/2597073.2597076))
    - applied branching strategies during development ([Shihab et al. 2012](https://dl.acm.org/doi/pdf/10.1145/2372251.2372305))
    - deployment schedules, hardware configurations, and software platforms ([Mockus et al. 2005](https://dl.acm.org/doi/pdf/10.1145/1062455.1062506))

* Technical aspects impact defects or customer perceived software quality:
    - size of software ([Lavallée et al. 2015](https://fac.ksu.edu.sa/sites/default/files/why_good_developers_write_bad_code-_an_observational_case_study_of_the_impacts_of_organizational_factors_on_software_quality.pdf))
    - application of certain design patterns ([Khomh et al. 2008](https://www.researchgate.net/profile/Yann-Gael-Gueheneuc/publication/4330216_Do_Design_Patterns_Impact_Software_Quality_Positively/links/561d229808ae50795afd7645/Do-Design-Patterns-Impact-Software-Quality-Positively.pdf))

---

## Perspectives of SW Quality

> * The **transcendental view** sees quality as something that can be recognized but not defined.
  > 
  > * The **user view** sees quality as fitness for purpose.
  > 
  > * The **manufacturing view** sees quality as conformance to specification.
  > 
  > * The **product view** sees quality as tied to inherent characteristics of the product.
  > 
  > * The **value-based view** sees quality as dependent on the amount a customer is willing t o pay for it.
  >
  > [Kitchenham et al. _"Software Quality: The Elusive Target"_ 1996](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.91.9555&rep=rep1&type=pdf)

---

## Motivation

> Imagine that your CEO’s aunt is also a customer. It’s not a big account; in fact, it’s tiny. But she makes his favorite pie, so her opinion matters more than it should. Unfortunately, the last release had a couple of bugs that mattered to her, so he’s been on the warpath ever since. He’s started ranting about quality and demanding numbers. He says that if you don’t come up with a way to measure quality and show improvement, he will. The glint in his eye says you won’t like it.
  >
  > Now what? Now it’s time for SonarQube, which will help you manage your code quality, instead of letting your code quality (and Aunt Betty) manage you.
  >
  > [Campbell et al. _"SonarQube in Action"_ 2014](https://www.manning.com/books/sonarqube-in-action)

---

## Motivation

Practitioners share this causality:

> It is well known that quality of code is in inversely proportional with Software bugs, as code quality goes down, the number of bugs increases. Thus, clean software is more likely to have less bugs than code of lower quality.
  >
  > [Bayer 2015](https://svenbayer.blog/2015/08/09/maintaining-high-code-quality-with-sonarqube/)

<br>

> [...] there is a principle that for all 30 rule infringements, 3 small and a significant Bug can be expected.
  >
  > [Gizycki](https://www.triology.de/en/blog-entries/statistical-code-analysis-with-sonarqube)

---

## Problem - Previous Work

Prior work presents inconsistent results regarding the impact of application of *automatic static analysis* tools to defects:

* Java development with FindBugs may decrease defects, e.g., [Hovemeyer et al. 2004,](https://dl.acm.org/doi/pdf/10.1145/1052883.1052895) 
[Zheng et al. 2006](https://ieeexplore.ieee.org/abstract/document/1628970)

Studies that link the product view (code quality) with user view (defect rates):

* Show either no impact of such tools, e.g., [Wagner et al. 2008](https://wwwbroy.in.tum.de/publ/papers/icst08evaltool.pdf)
  * Or they argue for their limited suitability since a large degree of defects are undetectable using code analysis, e.g., [Lauesen et al. 1998](https://ieeexplore.ieee.org/abstract/document/687949/).

---

## Research Question

Does the application of a Continuous Code Quality Assessment tool (SonarCloud) in five open-source projects actually reduce the number of defects that are reported for software products?

---

# Case Selection

---

# Case Selection

<table>
  <tr>
    <td style="width:20%">
      <a href="https://daffodil.apache.org">
        <img src="https://daffodil.apache.org/assets/themes/apache/img/apache-daffodil-logo.png", width="90%">
      </a>
    </td>
    <td>
      An implementation of the Data Format Description Language.
    </td>
  </tr>
  <tr>
    <td>
      <a href="https://www.groovy-lang.org/">
        <img src="https://upload.wikimedia.org/wikipedia/commons/thumb/3/36/Groovy-logo.svg/220px-Groovy-logo.svg.png", width="100%">
      </a>
    </td>
    <td>
       A dynamic and optionally typed JVM language.
    </td>
  </tr>
  <tr>
    <td>
      <a href="https://ozone.apache.org/">
        <img src="https://blog.cloudera.com/wp-content/uploads/2019/06/Logo-Color-TransparentBg.png", width="70%">
      </a>
    </td>
    <td>
       A distributed object store for Hadoop.
    </td>
  </tr>
  <tr>
    <td>
      <a href="https://karaf.apache.org/">
        <img src="https://www.apache.org/logos/res/karaf/karaf.png", width="90%">
      </a>
    </td>
    <td>
       An application container and runtime.
    </td>
  </tr>
  <tr>
    <td>
    <a href="https://karaf.apache.org/">
      <img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAXwAAACFCAMAAABv07OdAAAAllBMVEX///8AAABaydPFxcUcHBzQ0NDW1tZMTExlzNZQUFBTU1Os4eZcydTk5ORRx9LBwcH1+/zB6OzP7fBZWVm15epu0NjY8PPu+PqS2uHM7fD19fWI197h9fb39/ec3ePs7OyEhIRvb28oKCixsbGenp5+fn6Li4s5OTkhISHn5+e4uLibm5spKSkWFhZlZWWtra00NDRAQEDnwUGVAAALBklEQVR4nO2dh3biOhBAjWlOMC3EQExooQYSyv//3NNI4IJVxp3s0909OWRtjLmWR6PmNQyNRqPRaDQajUaj0UToTcZBZmWfz/+KUcf0sW2n7PP5XzHq2BMPR8svFFLy34wew5iZWn6REPlD7xctv1i0/BLR8ktEyy8RLb9Eelp+aYxatpZfDqOWZVpafhmQUk+wzP7njdFYt3CLoddyTNucfJhWAC2/CEYtk6gfD8Mda2bHKvvE/jQ9Cd5OIyj11gSCfe8jxFt5Z/4P0Ifo4ZA/8DeMebMP1SwJOEP5gTTx6ZP0haglfx5/WB0qn6gnryefZZ/ov0jfNN+5zEyQT6pZW5f6vOibHf6GN7LBq2Y1uSCRb84c07R0qc8PiXxLB5yckcnX1WzOyMKOLvU5I6twe/wtmqzQ8ktEyy8RLb9Enkf+d1VCYx73cPVqQ7zRlX1WhHrovfNqfSo+8rS+Wv5Ubrweq23ZOT6NfPe1IuH3ujzEOtyhUvkRf/Gl7LOiHB7eKzyV+fbl4a2X00F8qZ5GflXt4Bij+NfI/mvh1t948peBtzbI76/8o7Y33Hf/bkXn/Vzya1y8r7FHl/463V+4ef3F+Zi96Ax+6g9HbvKOOV3fZV8Hy+Vxuzw1v27/8nXmn8ZTyW+2Gzyq3dXp9jUWyKOxMtgVbud9UJu848rbECq4Ivnu7RRPh4brfUr3fi+cXM5b5L2aRVIV386UA/tyJ0ll59OOhgsE5B0vyp0E8r939BO3EcnrCyv8vOpfIt92+gWWfZV8w+jSSL3E2L9HAGmuESGF/CktGk2e4fmWhUxO2Zf2appOa6Q8m4xQyzfmA1q4EAcjpW3fRO7rk0L+RnZmDVqbNKOlRiYf5oqYs4J6NhHyDQOEViT5+40u2Wu9gps91ikkl9+Q10dtmkZHswVZzB99TGD8tpiRLJT8ObRfTspjLcle3+4v9wtLSC7/pDj5OdQI0ZRHke0Mxx06VUp5TqlByWcppKrof5N9NiwUqC9UgMTyacH/lr2rPdgvokFfmWqOZg5MZXjPO/jj5BsDRL55ZFkmvVBxuiUSy9+qM6spr+5H5Pmjd1L15l73IuUfyG4/8l2mV9a+mmIuVJDE8ndxA9yNPsko+ViBRlZ/0iH+cw3+SPlTkjn8yuMOrW7hxZm8uMQo+onl15TnxKcPVv1Fzf4ryw62cHtvYzp36i23zB8p37gqSxkp8DWmgp9jCEkqv02q9r005AsYzlo+lj3xXtnjVu9hT5jEk1vDCyt/yc0bAsz9ALwgL3f4U0gq/5vIv+A/RoBltm6vhpbtRCzT+bK2lU/wx8rfVmTdlQarbm89YfV4rdw08vexBxwe8eUbLdK8iu4weoeGl5NHwwsr/6iQDym1VwwHLOlEkibsJIr5IQLyjYnd6fP2+ZiQ0t/JPvPHyt8ows4heHGglfuDjsZpKtxE2U6IoPyRY1vcAt77HNMp4xkHf6x8aEyuJNubpLr1I01FsXeIxPIv8XtQowTlw6wp0aKfzxbMIncybXhh5e/8kM6jEW7WHuNUuakaWYguJzkh+ca7zQv7jN67A3P4Mwz+SPkuucVrkkByDF8bVHfEncTy6afE6sngEJZPwr7JDfuU3gfUvdn1usVo4V7Fm+HavAQ7UV5jVLmJ5bPB/5SB50E+CfumrGgPx6T0d5yPTII/Uj6k+Ufx5tXj5m6MDp7kvZpQJKTnheBBvjE07YnU7OeMdvu8Z6AfJ7+t6D+EIhjqPaSdudJ2gU9y+QYb41xyR2qRPMoXZPtBWOafQcMLJx8Sd8nwVD0ae49cWVxSyJ+zWUc18ZC9koh8YywL+zf6kPlbYxagZpMQ+GuCkr9WBBFIO6rhf6L3iiQ9CpBCvtFm4+SVQeKkJyp/RMKKUmDvbdKxbXMMDS8n0E9HKgR8NoSRTwO4JIZMKw/VLXBCZyJp5N/LPmnUHZL1NETlG0OSzyMiOgn+EH36vYnteA/dJI3kTEv+SpVTnHlBiV4x1ISTVPKNuTdfbbdN0sXJkU/Cvo16cin0upm2YwUqiaGZpfzpQpVN09GTSLFzm9hEJJ188hX86aaDVeziz5Mvz/aD9PrwkICA/M8M5burizKXrvN3OPKCEY+08smXGHj696d6PP9c+T1L0MnD4W2SSv5AtLGxpZNGFDkj3Pec/q05tspNL59sXnr6K7s4M3v58o0P25Rn+0GcNPJrg1ce95myG/l3AcncfpwNymo28sk9emz6/geyufxh+PKNmWmOPz2G0gsxSSNfykA1KrLmVbcArXIRYyrZyCeloH7yZ1a/VOU7ewjlW4FHHMlXiKYq+XIu8i7zaVOoGFq5iCo3K/kE9+xH/yaulSGQPzYDybvVyU9+bcDhdXfd30uRLO5AZ7KgPoZ7YqeucjOUT/je7O76UbmWSL7tDO98KNZGp5LPr3Dn80b1yO7jnSR4LCvC4SSYwvbY8uWQrXxy4uf76ixMh6ew5E+816GHmnJIJV+W51dBbuVLWPYbMHlGtHEhvLJBspZv+Kk/YllBFvJtv27uZ9rIou1XsUJhdXs/eE3Z65KDfGN6ZvbVYwqZyA8+KSzb7oUGrHESpfr70NjtI1+YyJuHfMNos9AvnWsEZCE//PTBbDvW4It/8Q3TKqMt4htaueL1cTfykW/MWWe/qsLPQP7wLQR+lAXVpbwRFuClMlNVz+7ISb7h0lWJqlUyGchPDEp+41dQgBsI98qO5bzkG23IlS+KZt7Ty6fDhLyK84iRr5rGkJt8NgikaGs9v3zIGXlVLsyWPFS7EqDaW8gPnp98AxF3EPJ75cpf8b8EzB5QNGRoyiffJUf5G/W7hPKd2f1/hBvbpcpv8y0v1Xf1N9R68pmDOcqH0lGT7yKUHxqXLVO+S/YaRFqLcEleVG3IDfetQXKUX1XfdwL5rfCMhHwW5uLlR5vqyjn7AE2IpBlHjvKh8yOZfNHDxTMlhXwXRhiVg0Z0hb60lf+U8gsBL7/52FaEgKruNlNP1s9Rfj1x2CkEvPzIAkNI/jHjRSBA1sWSUP4UMTgPg2mKZVt/U36bdzfwUK2PSyb//PurXn6xTJznFwJe/tdD6IDqFjVW1FC0cpPJh4Mq1wQlb+EWAk4+Z6CWPmYIN0dDUeUml6+yD+WD88iXEM8vn96/4cIL9ShyVQjUzIIuaSBh2KH2pZlu9VfZwPsT8qMrQS+Y2/6GvMpNWOEy+/ynqFHoRGnlkf+A/MPj9wAZV+zMJLh0e+HWpKkme/7Yj+gBmrR1u1cOYv4B+fQRRsEvslX3VvrQ8QBhxZc4z6+yWQonXr473WAqBeNPyGf9g/4tPged+BnZJ37HHCN5I2u6pIYrP6uHAt5gT7eTPGTS4y/Ipw11f+rgGV/d3j9FeK3StHAPt4ld+8G2O3fd6dR13frilT269oqZs1a2fMxsVla/eatBm4g8IoArWx+XqnvBvZVxxu4a+GWBqpLKlA/hGLeSFRqq3mMDr5KpUjzWcvnqM4DxWH5bwV3sKlH2G+RjT8qUT2pOdUbAWAcK+wq90JMxvVR+RC2yNWqxOjlPkc55d/mg/npGP3GmVPn/CPPucbugbM+46ck3tPwS0fJLRMsvES2/RLT8UnhnzzG1bg80xS281WRD67bsymY/tPwiaZnWzMPU8guFyPd/6Wj5haLll4iWXyJafolo+SWi5ZeIll8iWn6JaPklQuTfly8Ph7qFWyyt0CMTtPxCaXVCvJd9Pv8rRp8hCvu/QDUajUaj0Wg0Go1Go9FosPwHrzjHm6AETYgAAAAASUVORK5CYII=", width="100%">
      </a>
    </td>
    <td>
       An implementation of the Raft consensus algorithm.
    </td>
  </tr>
</table>
---

---