Using Empirical Studies to Evaluate the Maturity of a Technology for Use

by Forrest Shull, and Victor Basili

This tutorial concentrates on the role of empirical study in the assessment of the maturity of software technologies for use on live projects: What kinds of information empirical study can yield about technologies and their effectiveness in various contexts; what information can be fed back to technology developers who are seeking to improve their technologies; what we can say about the benefits and limitations of technologies based on multiple studies. We also discuss what advice we can give a project manager to reduce the risk of applying the technology.
We present options an experimenter might choose and offer some heuristics on what choices seem to go well together. Finally, we offer several example studies, showing how the type of study and choice of variables evolve with the maturity of the technology being studied.