Software development success despite uncertainty and complexity

Abstract

Good IT solutions are vital to the future of our civilization, so it is important to understand how to build them well. After more than half a century of software development, we still struggle with a basic dilemma: we cannot know at the outset what an IT solution is supposed to accomplish and how it is supposed to behave to do so. There are too many moving parts (complexity) and too many things that can change (uncertainty). The root of the problem is that development work happens in one environment — the development context — while the solution must serve goals in a quite different environment — the operational context. In the development context, people tend to treat complexity and uncertainty as something to be managed adaptively, whereas in the operational context, the aim is to minimize them as much as possible through administrative mechanisms. Enabling leadership is needed to bridge these two. The research findings presented here help address this gap. Experience from “do-or-die” projects during Covid 19 shows that setting difficult, attainable, and meaningful goals allows adaptive leadership to flourish even in environments that are usually run administratively. Studies on data sharing and artificial intelligence likewise demonstrate that adaptive approaches are essential for realizing the benefits of these technologies. However, both research and practice suffer from the lack of good conceptual models of adaptive development work. As a result, professionals and researchers often talk past each other, focusing on different concerns. The articles, therefore, also discuss ambiguities surrounding two phenomena that need clearer descriptions: agile practice and do-or-die projects. For each of these, we propose frameworks suitable for practical discussion as well as further research.