In a study published in 1995, Tuchy et al. [1] surveyed 400 peer-reviewed research articles and concluded that computer scientists publish relatively few papers with experimentally validated results. In fact they cite that only 30% of CS papers devote at least one-fifth of their space to evaluation. This makes a strong case for the fact that journal articles cannot be perceived as the “holy grail” of publishing. Part of this process may be driven by the age-old adage “publish-or-perish” which plagues untenured faculty. However the “quality” of journal articles is often overlooked during the tenure process, or for that matter sometimes during the peer-review process.
Purely empirical work is often overlooked by journals due to its lack of “design” contribution
tenure committees must recognize that high-quality experimental CS needs time to produce validated results. Indeed, research can be of three types [wiki]:
1. Exploratory research: a new problem can be structured and identified.
2. Constructive research: a (new) solution to a problem can be developed
3. Empirical research: empirical evidence on the feasibility of an existing solution
to a problem can be provided
You would think computer scientists would value experimentation. Lets see… we write software and… we test it. Oops, there-in might lie the problem. Fifty years of programming and we still don’t have the ability to produce code with less then 10 errors per thousand LOC. Ahhh… now I understand, we don’t test our software nearly as well as we should, soooo why should experiments involving that software be any different?
Certainly food for thought.
[1] Tichy, W.F., Lukowicz, P., Prechelt, L., Heinz, E.A., “Experimental evaluation in computer science: A quantitative study,” Journal of Systems Software, 1995, Vol.28, pp.9-18.
