It's pretty easy to find bona-fide empirical studies illustrating some of the major limitations of those student surveys that most of us are obliged to administer to our classes every semester (you can peruse some examples here and here).
Mind you, I'm glad that this kind of stuff is available, but it just confirms what long experience with these confounded surveys has taught me in any case. Thus, even if you're able to guess right and administer them on days when most students are in attendance, not too close to an exam, when term papers are returned, or some other factor that could skew their thinking, the results seem inherently unreliable.
An increasing number of students - at least in my classes - regard the surveys as a nuisance, and simply don't complete them. Others, although the allotted time is typically twenty minutes, seem to breeze through in less than five. Hard to believe that they've pondered the questionaire very deeply. Still others, as one of the studies linked above indicates, are confused by the terminology or format of the survey and return accordingly confused responses, or leave a signifiacnt portion of the questions unanswered. A few students do take the time to add written comments, and these usually reflect the polarities typical of RateMyProfessor: I'm either the best or the worst, with nothing in between. The surveys, depending on the questions they include, can also provide some straightforward information, such as an instructor's availability during office hours, punctuality in starting class or if he is timely in returning exams and written assignments. But if they're supposed to be a reliable measure of my "teaching effectivesness," then I'll take vanilla.
Still, all of this wouldn't matter if student evaluations weren't so often the gamebreaker in faculty personnel decisions. Even if all of your other ducks are in a row, a few negative evaluations can be all it takes to sink your application for tenure or promotion, or terminate your part-time teaching contract. Administrators in particular, ever concerned with student "success," don't like to hear that they're "unhappy." And nothing, finally, better cloaks ideological or personal animosity than appeals to purportedly neutral student evaluations.
Glad that I'm a full professor who doesn't have to walk that plank again, because you'd better believe that I'd otherwise worry a lot more about my grade distributions.
