Quatic2021

class: middle

# Automatically Assessing Complexity of Contributions to Git Repositories

#### QUATIC'21 - Software Evolution Track

Helge Pfeiffer, Assistant Professor,<br>
[Research Center for Government IT](https://www.itu.dk/forskning/institutter/institut-for-datalogi/forskningscenter-for-offentlig-it),<br>
[IT University of Copenhagen, Denmark](https://www.itu.dk)<br>
`ropf@itu.dk`

---

# Motivation

Lehman's second law of software evolution says:

> As [software] is changed its complexity increases and [it] becomes more difficult to evolve unless work is done to maintain or reduce the complexity.
  > 
  > [Lehman et al. _"Rules and tools for software evolution planning and management"_ 2006](https://link.springer.com/article/10.1023/A:1012535017876)

Technical Debt is:
 
  > a collection of design or implementation constructs that are expedient in the short term, but set up a technical context that can make future changes more costly or impossible.
  >
  > [Avgeriou et al. _"Managing Technical Debt in Software Engineering (Dagstuhl Seminar 16162)"_ 2016](https://drops.dagstuhl.de/opus/volltexte/2016/6693/)

# Problem

How to assess if software _**"becomes more difficult to evolve"**_ or if _**"future changes [are] more costly"**_?

---

# Problem

Previous work assess change of complexity metrics over time and argue by induction that more complex systems are more complex to evolve.

<table style="white-space:nowrap;">
  <tr>
    <td>
      <img src="./images/related_work.png" width="90%">
      <p style="font-size:16px;">
        <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.433.4339&rep=rep1&type=pdf">Herraiz et al. <em>"The Evolution of the Laws of Software Evolution:<br>A Discussion Based on a Systematic Literature Review"</em> 2013</a>
      </p>
    </td>
    <td>
      <img src="./images/lehman.png" width="100%">
      <p style="font-size:16px;">
        <a href="https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.26.1214&rep=rep1&type=pdf">Lehman et al. <em>"Metrics  and  Laws  of  Software  Evolution<br> -  The  Nineties  View"</em></a>
      </p>
    </td>
  </tr>
</table>

---

# How does previous work assess complexity?

* Size-based metrics (LOC)
  * Cyclomatic (McCabe) complexity
  * Object-oriented metrics, e.g., method coupling

---

## Cyclomatic Complexity?

> There is, however, no other complexity metric that enjoys wider acceptance and is free of such criticisms.
  >
  > [Israeli et al. _"The Linux kernel as a case study in software evolution"_ 2010](https://www.sciencedirect.com/science/article/pii/S0164121209002519?casa_token=wU7n68t8nPQAAAAA:xWUJYgW7GwFSLxPDP5GGa_EZDYdbpD0CPvw2Zo-5Ku7vgHTC31f3TfOHplttODtYPSu4sbdepg)

[Shepperd _"A critique of cyclomatic complexity as a software metric"_ 1988](https://digital-library.theiet.org/content/journals/10.1049/sej.1988.0003)

> This critique demonstrates that it is based upon poor theoretical foundations and an inadequate model of software development. The argument that the metric provides the developer with a  useful engineering approximation is not borne out by the empirical evidence. Furthermore, it would appear that for a  large class of software it is no more than a  proxy for, and in many cases is outperformed by, lines of code.
  >
  > [Shepperd _"A critique of cyclomatic complexity as a software metric"_ 1988](https://digital-library.theiet.org/content/journals/10.1049/sej.1988.0003)

---

# What is complexity for us?

> ... the label we give to the existence of many independent variables in a given system. 
  > The more variables and the greater their interdependence, the greater that system's complexity.
  > Great complexity places high demands on a planner's capacities to gather information, integrate findings, and design effective actions.

[Dörner _"The Logic Of Failure"_ 1997](https://books.google.dk/books?id=HU59AAAAMAAJ&hl=da&source=gbs_book_other_versions)

> as a measure of the resources expended by a system while interacting with a piece of software to perform a given task.

[Basili _"Qualitative software complexity models: A summary"_](https://scholar.google.com/scholar?as_q=Qualitative+software+complexity+models%3A+A+summary+in+Tutorial+on+Models+and+Methods+for+Software+Management+and+Engineering&as_occt=title&hl=en&as_sdt=0%2C31)

* If the interacting system is a programmer and complexity is defined by the **difficulty of performing tasks** such as coding, debugging, testing, or modifying software

---

# Goal

Directly and automatically assess if the actual work of evolving a software system is getting more difficult/complex.

> As [software] is changed its complexity increases and [it] becomes more difficult to evolve unless work is done to maintain or reduce the complexity.
 
  > [TD is] a collection of design or implementation constructs that are expedient in the short term, but set up a technical context that can make future changes more costly or impossible.

---

# Terminology

* Issue tracker
  * Ticket
  * Commit
  * Modification
  * Contribution

---

## Example

A ticket with contributions in respectively Gaffer and Cassandra:

---

## Examples I

---

## Examples II

---

## Examples III

---

# How does it work?

Two-staged process. Per modification, we determine

* Number of lines added
  * Number of lines removed
  * Number of hunks
  * Number of modified methods
  * Modification kind

Map them via function to a _modification complexity_ of the discrete values values _low_, _moderate_, _medium_, _elevated_, or _high_.

---

# How does it work?

Per commit, we determine

* Number of modified files in commit
  * Number of lines in commit

---

# So many formulas?

Precise formulas and threshold values are configurable and the default values are provided in the paper.

The paper provides a mathematical notation for the implementation in [`contribution_complexity/metrics.py`](https://github.com/HelgeCPH/contribution-complexity/blob/master/contribution_complexity/metrics.py)

---

# Initial Evaluation

* Cassandra:
    - 16485 tickets
    - 14158 are resolved
    - 7866 are resolved with contribution
  * Gaffer: 
    - 2403 tickets
    - 2300 are resolved 
    - 820 are resolved with contribution

* Sample   48 contributions (25 from Cassandra and 23 from Gaffer) with 247 commits
  * Manually classify the CC scores for each of these 48 contributions
  * Compare manually classified CC scores with those created by the tool

* 36 cases (75%) manual classification is equal to the tool’s CC score
  * 8 cases (≈17%) manual assessment is one level higher than tool's score
  * 4 cases (≈ 8%) the tool’s assessment is one level higher than manual classification

---

# Initial Experiment

Contribution complexities over time for Cassandra and Gaffer
<table style="white-space:nowrap;">
  <tr>
    <td>
      <img src="./images/cassandra_iss_compls.png" width="100%">
    </td>
    <td>
      <img src="./images/gaffer_iss_compls.png" width="100%">
    </td>
  </tr>
</table>

---

# Related Work

* Code Change Model [Hassan _"Predicting Faults Using the 
Complexity of Code Change"_ 2009](https://ieeexplore.ieee.org/abstract/document/5070510/)
  * Delta Maintainability Model (DMM) [di Biase et al. _"The Delta Maintainability Model: Measuring Maintainability of Fine-Grained Code Changes"_ 2019](https://repository.tudelft.nl/islandora/object/uuid:6ff67dee-2781-47d7-916f-bd36c5b61beb/datastream/OBJ1/download)
  * Whitespace-complexity [Hindle et al. _"Reading Beside the Lines: Indentation as a Proxy for
Complexity Metrics"_ 2008](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.68.3558&rep=rep1&type=pdf)

---

# The Tool is Available Online

The _Contribution Complexity_ tool is:

* readily installable from PyPI: [https://pypi.org/project/contribution-complexity/](https://pypi.org/project/contribution-complexity/)
  * open-source (GPL3): [https://github.com/HelgeCPH/contribution-complexity](https://github.com/HelgeCPH/contribution-complexity)

It can be used as a Python library and as CLI tool.

---

# CLI Tool Demo

---

# Links

* Read the paper: [http://itu.dk/~ropf/blog/assets/quatic2021_pfeiffer.pdf](http://itu.dk/~ropf/blog/assets/quatic2021_pfeiffer.pdf)
  * Find the presentation: [http://itu.dk/~ropf/presentations/quatic21.html](http://itu.dk/~ropf/presentations/quatic21.html)
  * Check the tool: [https://github.com/HelgeCPH/contribution-complexity](https://github.com/HelgeCPH/contribution-complexity)

---

# Thank you for your attention

---

class: middle

# Automatically Assessing Complexity of Contributions to Git Repositories

#### QUATIC'21 - Software Evolution Track