Nidumolu 1995: Residual Performance Risk

Full APA Reference

Nidumolu, S. (1995). The Effect of Coordination and Uncertainty on Software Project Performance: Residual Performance Risk as an Intervening Variable. Information Systems Research, 6(3), 191–219.

Other References

This particular article is jargon-heavy, so I’m writing out some notes to drive the definitions into my head.

• Coordination – “the linking together [of] different parts of an organization to accomplish a collective set of tasks” (Van de Ven et al. 1976, p. 322).
  • Vertical coordination mechanisms – between users and IS staff, undertaken by authorized entities such as project managers or steering committees.”
  • Horizontal coordination mechanisms – “through mutual adjustments and communications between users and IS staff.
• Performance Risk – “the extent of difficulty in estimating the performance-related outcomes of a project, regardless of the specific estimation technique used. These include outcomes such as the actual project cost, project completion time, system benefits, the system’s compatibility with its environment, and the technical performance of the resulting system (McFarlan 1981).”
  • Residual Performance Risk – “performance risk present in the later stages of the project after project planning and requirements analysis have been completed (i.e., during design, coding, testing and installation)”. Residual performance risk was operationalized/measured by asking about estimating the following late in the project: (1) the costs of the project, (2) the project completion time, (3) the benefits of the software, (4) whether the software would be compatible with its environment, (5) whether the software would meet user needs.
• Technology (Organization Theory) – the processes or tasks involved in transforming inputs into outputs
• Structural Contingency Perspective (Organization Theory) – “performance is determined by the match (or “fit”) between the uncertainty in the unit’s tasks and the ability of the unit’s structure to process the information required to cope with the uncertainty (Daft and Macintonsh 1981, Tushman and Nadler 1978). The better the match, the higher the performance. Consequently, the organizational structure needs to provide for this information processing, or else performance is adversely affected (Daft and Macintosh 1981, Galbraith 1973, 1977; Tushman and Nadler 1978). Because structures differ in their information processing capabilities (Galbraith 1973, 1977), the research problem is to identify the structure that maximizes performance for a given level of uncertainty.“
• Risk-Based Perspective (Software Engineering) – “The purpose of software engineering is … to control the software development process, particularly the sources of project uncertainty, and produce quality software for the user (Pressman 1992) … using methods, tools and procedures that bring more structure and formalism to the complex and uncertain task of developing software.”
• Project Uncertainty – Two key types in a software development project:
  • Requirements Uncertainty – the uncertainty about what the user requires
  • Technological Uncertainty –  the uncertainty concerning technologies such as tools and techniques needed to convert requirements to a software system.
• Performance – “It is described in terms of two key aspects emphasized in the IS literature on software project performance [examples scrubbed]: (1) process performance, which describes how well the software development process has been undertaken, and (2) product performance, which describes the performance of the system actually delivered to users. It is important to study both aspects because even though the software delivered by the project may be of high quality, the project itself may have significantly exceeded time and cost projections. On the other hand, well-managed projects which come in below time and cost budgets may deliver poor products.” [emphasis mine]
  • Process Performance –  “[includes] 1) Learning acquired during project, which describes the knowledge acquired by the firm [cf. scrubbed]; 2) Process control, which describes the extent to which the development process was under control [cf. scrubbed]; and 3) Quality of interactions, which describes the quality of interactions between IS staff and users during the development process.”
  • Product Performance – “[includes] 1) Operational efficiency of software, which describes the technical performance of the software (Mookerjee 1988); 2) Responsiveness of software, which describes how well the software responds to the needs of its users; and 3) Flexibility of software, which describes the software’s ability to adapt to changing business needs (Mookerjee 1988).”

“Requirements uncertainty has been widely studied in Information Systems research because of the difficulty in eliciting requirements from users. Requirements analysis is the most important stage in the software development process, because it has the greatest impact on future stages (Turner 1992). At the beginning of many projects, users and analysts do not have a clear sense of what is needed (Lucas 1982), leading to an ambiguous and incomplete set of requirements (Rowen 1990). This is especially true for applications that are new or involve unstructured tasks “(Zmud 1980, 1984).”

Findings & Discussion

“The introduction of residual performance risk […] makes two important theoretical contributions to research. First, it helps distinguish between project uncertainties (such as requirements uncertainty and technological uncertainty) and performance risks. Second, the findings show that residual performance risk is an intervening variable which explains the effect of vertical coordination on project performance. Rather than have a direct effect on performance, vertical coordination affects residual performance risk which in turn affects project performance. Moreover, the strong negative effect of residual performance risk on project performance provides considerable support for the arguments advanced by risk-based researchers, i.e., lack of information about expected performance outcomes is detrimental to the actual performance itself. Conversely, the ability to estimate outcomes enables project management to better plan and execute the project’s activities and assemble the needed resources. As suggested by risk-based research, an important goal of project management should therefore be to gain control over project risk by improving the ability to estimate project outcomes.”

Vertical Coordination (Project Management)

“Vertical coordination (at least through a project manager) had a significant role in
reducing risk drivers such as requirements and technological uncertainties.”

“A primary role for project management is to manage the performance risks in the project.”

“Vertical coordination did not have a direct effect on project performance,
which implies that it is important only to the extent that there is residual performance
risk in the system.”

“The role of a project manager becomes more important as the risk drivers increase. By contrast, structural contingency researchers would assert the opposite: vertical coordination through a project manager is more important for projects that are routine, more certain and therefore less risky; it should be less relied upon as the project becomes more complex, uncertain and risky. Overall, the results support the arguments made in risk-based research: because of schedule pressures that typically characterize software development projects, project coordination and control are of paramount importance. Vertical coordination ensures such control, particularly when the risk drivers are significant.”

Horizontal Coordination

“Horizontal coordination does seem to increase the quality of the system that is eventually developed […] perhaps […] because increased interactions between users and IS staff lead to a wider and better range of options that are considered, rather than focusing only on those that meet short-term needs.”

Joint Effects (Horizontal and Vertical)

“Projects perform best if high levels of both vertical and horizontal coordination are present in a project […] The two mechanisms of coordination are not alternative approaches […] High levels of both vertical and horizontal coordination should be present in the same project in order for it to be high-performing.”

Horizontal coordination […] directly increases project performance by improving the quality of the user-IS interactions and the flexibility of the developed software.”

“These arguments suggest tentatively that vertical and horizontal coordination may perhaps be fulfilling the roles of ensuring convergence and exploration respectively […] Because software development is typically an unfamiliar activity, there is need to sufficiently explore requirements and design alternatives by relying on increased user-IS interactions. At the same time, because projects typically have limited time and resources and are subject to many risk drivers, there is need to bring closure to the project by relying on a strong project manager or steering committee to avoid or resolve performance risks.”

Takeaways

This work is based on waterfall management of software systems. I should locate Agile papers which reference this work. Most modern game development calls itself Agile (though it’s often a smorgasbord of Agile sub-features modified to taste and called Agile, and does not indicate strict adherence to any formalized version of the term).

Unlike some other sources that define failure as missing schedule, budget, or spec, the author doesn’t directly define project failure and implies degrees of success or failure based on both process performance the project performance. I don’t believe I can infer the author’s take on their relative importance. Game teams can blow out all three sides of the Iron Triangle, though. When a team de-emphasizes long-term adherence to initial budget, schedule, and spec, what’s left to measure performance? How do we measure product performance if the intended outcomes change as much as they do with (many) game projects? Can we measure game dev process performance by the definition here, which used “learning acquired during project,” “process control,” and “quality of interactions with users?”

Game dev teams can be far more complex than the structures implied by the horizontal and vertical descriptions, where coordination is only needed between the “IS staff” and “users.” Large game projects require the coordination of multiple company teams (code, art, audio, qa/test, etc), 3rd-party contributors, and often include some form of user interaction through playtest.

Basically, I don’t know anyone who comes at software projects from the Organization Theory perspective. I don’t know anyone who would suggest that a project doesn’t need a manager because the project isn’t routine enough. Even in informal organizations, there’s often a sense of one person playing the role of project manager on any team software project involving more than a couple people, down to understanding I had to PM my own solo projects. It’s a role often fulfilled by someone with another title if the team isn’t big enough to have someone dedicated to project management. This includes my experience on both internal tools teams and projects for websites and consumer software (including games). The Project Management Institute was founded in 1969; what did they think they were needed for back then?