Speaking of constructive feedback, what about course evaluations? A nice idea, often implemented very poorly. Firstly, to review committees. Please LOOK at teaching dossier’s when evaluating someone’s teaching. Don’t just base you decision on a set of numbers (usually a mean), which indicates what the student thought of the professor and course. have seen instructors who teach a first year class of 600 compared directly with those that teach a fourth-year class with 35 students. These don’t equate, so you can’t compare them. Better statistical evaluations maybe? Anyone ever heard of the confidence interval? This can help alleviate some of the class-bias. In any manner it is wrong just to sum up a class by one or two numbers.

Yes, true, in the sciences teaching often doesn’t mean as much as it should. Professors who are good at research and *really* bad at teaching, get rewarded regardless. Again, research = grants + papers = good. teaching = sqrt(who cares)!

Sad, that we view things this way. Some of us love teaching, but get frustrated by a system which doesn’t care that much. We want to do innovative things, but many learn the leasson quickly that extra time taken in teaching pedagogy isn’t rewarded. Maybe things will change. I hope so. Many professors could learn a thing or too about pedagogical research. They might learn some new methods for teaching, and might be a little more empathetic towards students. Many forget that they were once students. Others think that a PhD entitles them to behave in a mightier-than-thou fashion. Some academics need to get off their pedastools and integrate into society. Students like profs who can relate to them.

I’m a normal person, and I don’t think I’m better than anyone. I’m have strong opinions, but that’s okay. They are my opinions, and not geared towards pleasing everyone. If you don’t like my opinion, that’s your right. All I want is for things to change, and for everyone to work together to provide a good education for students. That’s not much to ask… to properly educate the people who will one day be running the place is it? Some of us have kids who might one day attend university, and we all hope that they are taught by individuals who do as good a job as we hope we do.

Once upon a time there were three researchers, who worked in a medium-sized university and were happy. One day the three researchers were told that they should apply for grants. They were told that grants are good, but they weren’t given any help in writing these grants, so the three researchers went on their merry way to find grants.

The first researcher decided to apply for a SCREN grant. SCREN was considered the holy grail of grants. The problem for the first researcher was that he didn’t know where his research would fit in. His research project was titled “The evaluation of user-based design ontologies applied to robotic systems”. So he picked a field, wrote his application, submitted it and waited. And waited. Many months passed, and the researcher finally got a response. His grant had been denied. Confused, the researcher asked why? No good response was provided, so the researcher assumed his project was just too applied in nature. The reviewers comments were ridiculous and somewhat self-centred.

The second researcher also applied for a SCREN grant. This researcher had a huge grant from another source, and had applied for the covetted SCREN a few times before, always falling though the hole of peer-review. His research project was titled “Impact of computer-based technologies on learning”. His result also came back negative, even though he had numerous publications, a history of good industry-based funding and a project that had a real-life application. His project was well written, and he even had it peer-reviewed before it was submitted in order to identify and inadequacies. All to no avail.

The third researcher, also applied for a SCREN grant. The subject of his project was titled: “The control of vortex decision parameters in the design of ultra intelligent autonomous web-based search agents”. Surprisingly, the third research had his project approved, with a stipend of $29,000 annually. The other two researchers were stunned.

The moral of the story? If you can write a good story, and convince others that the research you are doing is beneficial (even if it is evidently not), you will get a grant. Many grant processes are tied to the same process of peer-review we know doesn’t work in paper-review. Except this time, people’s bias, lack of knowledge and leack of context actually costs others grants. Good, applied research with a strong possibility of an outcome will not usually get you a grant. People love theory, but don’t necessarily like applications. Some will say that these “applied” ideas should be industry funded. Yeah, maybe they’re right. But maybe, just maybe they should have the same right to be funded as the theory stuff. Funding by industry has strings attached. And any university that advocates it as a way of bringing in money is nuts!

Corporations have shareholders, and shareholders like to make money. Some of us have stocks in these companies, and it is fair to say that we like our stocks to go up, mainly so we can retire early! So, if you work with a company that is providing $ for your research, then they are probably entitled to a portion of the intellectual property. I mean it is only fair, isn’t it? Here’s the catch though. They probably don’t want you to publish the intricate details of your work. They probably want a patent of two, or just want to keep the whole thing hush. So you get $, but don’t get publications. Sort-of defeats the purpose don’t you think?

Too much money seems to be given out to support ridiculous research. Some of it never happens, or the results produced seem bogus. Where is the money to investigate practical projects, which impact both students and the greater community?

Who knows. Somebody should think about looking a little closer at grants and ask where the money goes. And for those that review grants, think about this. Review the grant without any bias, and ask yourself how this research could benefit others. Don’t write in the review that the author should include your work. It’s just ego-centric and unnecessary. And PLEASE provide some constructive feedback. Otherwise, some of us just view obtaining these grants as a crap-shoot.

In a study published in 1995, Tuchy et al. [1] surveyed 400 peer-reviewed research articles and concluded that computer scientists publish relatively few papers with experimentally validated results. In fact they cite that only 30% of CS papers devote at least one-fifth of their space to evaluation. This makes a strong case for the fact that journal articles cannot be perceived as the “holy grail” of publishing. Part of this process may be driven by the age-old adage “publish-or-perish” which plagues untenured faculty. However the “quality” of journal articles is often overlooked during the tenure process, or for that matter sometimes during the peer-review process.

Purely empirical work is often overlooked by journals due to its lack of “design” contribution
tenure committees must recognize that high-quality experimental CS needs time to produce validated results. Indeed, research can be of three types [wiki]:

1. Exploratory research: a new problem can be structured and identified.

2. Constructive research: a (new) solution to a problem can be developed

3. Empirical research: empirical evidence on the feasibility of an existing solution
to a problem can be provided

So, WHY, oh WHY isn’t empirical work and testing valued in computer science. Maybe because it is still a fledgling field? Maybe computer scientists don’t value experimental work. How many papers have you read where the experimental work is lacking, or inconsistent or statistically insignificant? If you’re going to write a paper, show some proper experiments, use some well thought out metrics, and PLEASE use some realistic data.

You would think computer scientists would value experimentation. Lets see… we write software and… we test it. Oops, there-in might lie the problem. Fifty years of programming and we still don’t have the ability to produce code with less then 10 errors per thousand LOC. Ahhh… now I understand, we don’t test our software nearly as well as we should, soooo why should experiments involving that software be any different?

Certainly food for thought.

[1] Tichy, W.F., Lukowicz, P., Prechelt, L., Heinz, E.A., “Experimental evaluation in computer science: A quantitative study,” Journal of Systems Software, 1995, Vol.28, pp.9-18.

Electronic journals always seem to get a good beating in academia. Especially in areas like computer science… which is ironic considering the one field that spawned the “net”, is very unwilling to view electronic journals as being equal to printed ones. *Maybe* they aren’t equal. Maybe, just maybe, electronic journals could actually be better. For a number of reasons. Firstly, many electronic journals are provided for free. Research is partially the “advancement of knowledge” which is driven by “driven by the researcher’s curiosity, interest, or hunch”. So, research is suppose to be somewhat exploratory. The insight you gain, ideally should be shared with others, so they too can benefit from this knowledge and further their research.

Alas, the number of papers I have read over the last 10 years which make no sense is astonishing. They sound alright, and the results are, well shall we say, somewhat convincing… but, and let’s face it you knew that was coming… when you go to “recreate” algorithm it doesn’t work. You try 101 different things in order to fix it, but to no avail. It just simply does not work the way it should. You may email the author to ask a question, and if you get a response, they often can’t find the original code. It could be that something was left out of the paper. It could be that the special parameters were just never explained, or even given for the examples shown. It makes you wonder if the algorithm ever really worked.

Some researchers publish their code, or make it available. I have a feeling, not all journals like this… some like proofs much better. I certainly don’t. Programming relies on discrete representations of problems. The problem with all of this is that if you are designing a technique and want to compare it against other technique to gauge it’s accuracy, this is almost impossible. Often the data they use is not publically available, or the algorithm is “proprietary”. Sometimes, I’m sure they don’t want their algorithm compared… it may just show the flaws that exist.

So what does all this have to do with electronic journals? Well consider this. A good electronic journal would allow you to submit the data, and the code associated with the work. It would then be archived for anyone to use. It would be accessible to many, and in most cases the author would still maintain copyright over the information. The growing trend towards electronic journals will continue, and the sooner academics realize that paper journals are not the “be all and end all”, the better. The electronic journals will also allow for comment from readers, allowing researchers to improve upon their work, and maybe make new collaborations. Constructive, open critisism is good. This is real peer review.
And if you don’t want to share, don’t bother publishing the algorithm. But then if that happened it wouldn’t be a vortex of nonsense would it?

Academia is rife with the notion of publishing. As academics we spend a good proportion of our time writing papers. Unlike earlier times, when academics had time to think, we are now confronted with an ever-increasing need to publish. Writing papers takes time, and usually time taken out of improving pedagogical quality. Of course not everyone is interested in teaching. Publishing comes in various strains, some more prestigious than others, but the main two are conferences and journals. The latter is a hot-bed of prestige. I mean articles, if accepted can take nearly 2 years to get published from initial submission. Oh, and paper journals are considered by some to be more prestigious then electronic journals. But what many academics obviously don’t realize is that publishing companies are commercial enterprises. As such they take papers, which essentially cost them nothing to create, publish them and sell the journals. Authors generally receive nothing. As such academics are
essentially a conduit for publishers to fill journals.

But does anybody realise this? Probably very few.

Journals work on the principle of peer-review. A nice idea… in principle. But what does peer review really entail? Very few academics really know. Sure, editorial boards are comprised of experienced academics who help ensure a fair review of the work. And in some journals this is the case, but without an open reviewing platform, how do we know who reviews papers? From experience, some papers I have seen rejected from journals were clearly reviewed by graduate students with little experience in the topic area. Sometimes it seems as though the paper has not even been properly read. Especially when reviewers make comments like “this is nothing original”, when comparable papers on the topic = 1.

Conferences I fear, are not much better. Conferences are a multi-million $ industry, and although better at releasing cutting-edge research, the largest conferences also tend to suffer from reviewer apathy. Sometimes a good paper doesn’t have to be bleeding edge. Truly, how many published papers are truly innovative and new? Some of the work produced is of a descriptive nature, some of the work is topical. Sometimes they don’t get accepted. That’s ok. Sometimes work is too controversial to be published. Sometimes reviewers say that the work lacks “experimental results” (yeah try and cram more into the 4 page prescribed limit!). Sometimes it is just too well written. It’s a shame that we think so little of research that judge its merits by the quantity of material published. It’s a shame that some consider research into pedagogy of a lesser kind. It’s a pity we don’t open conferences to the discussion of controversial topics. It’s a pity some academics couldn’t be more open minded.

Publish or perish? Nah, it’s just a vortex of nonsense.