 |
 Personal FAQ |
|
When a conference, workshop, symposium or other event desires to employ peer review, a program committee is selected to review the papers and make suggestions about whether to accept or reject papers. This process is occasionally complicated by the possibility that a member of the program committee might submit a paper to be judged. The question arises: What should be the disposition of such papers?
At issue is whether such a paper submission should be:
The concern driving this question is to optimize both the appearance and the practice of fairness in paper selection.
My opinion is that option c ("treated equally") is proper. This assumes, of course, that there are properly advertised guidelines for all committee members to recuse themselves from judging their own work, and that the committee members have explicitly agreed to abide by such guidelines.
I think the case for simply disallowing such submissions is internally consistent as well, but just overly harsh, so I don't address that below. The remarks below argue for why giving a higher bar of acceptance to program committee members would be inappropriate.
I think the proper role of a conference committee is not to weigh certain papers down. Let it tell people they may not submit if it wants to do that because that is fair-people can understand and reason about that effect since it is objectively applicable. Telling them their paper will be subjectively treated is not really fair to them because it cannot be objectively applied and it has the opposite effect of its intent, that is, it purports to send a signal that the conference will be fair, but it does so by telling some valid applicants that they should be assured of fair treatment because it is an advertised fact (assuming it even is) that other valid applicants will be given an unfair seat.
By the nature of the complex set of interlocking events Program Committee members are generally asked to commit to being such before the paper submission deadline. This means that anyone who might be in the unfortunate position of finding out that they have a need to submit a paper after they have accepted a program committee membership. This puts them in the awkward situation of having to opt out of being on the committee in order to gain fair treatment (an option they may not feel is open to them after the conference and their participation is advertised, or even after they have given their own good word that they would do it) or of not publishing in that forum (which might mean waiting a year or two, depending on whether there are other appropriate forums or how often the forum recurs).
Note that I use the term "need" advisedly, since in some cases paper submission is a casual luxury but we might as well admit that some people's career status hangs on the publication of papers.
The fact of co-authorship compounds the situation because it means one author's entanglement can upset others at one's place of employment. And this fact, in turn, makes signing up to be a member a potential liability both personal and professional because it might be regarded as undermining one's co-workers chances of submitting a paper. And while it's likely that such "undermining" occurs unintentionally, such a fact would just as likely be cold comfort to others held back from publishing by the situation.
if the committee member's name is on the paper because they are a principal investigator or a work supervisor, this can be especially unfair to an organizationally junior person because they may not have the option of separate publication or moving to another organization. In practice, that often requires being "well established" and the key may be publication. It may be no big deal for the senior member to find another venue in which to publish, including publishing on other topics, but the junior person may have initially a more narrow set of options and really need to be considered at this time in this venue in order to advance to a more first-class status.
It is important that we accord such people due respect not just for their own sake, which would be reason enough, but for our own. They are our future, and to hold them back is to hold us back. If their advancement hinges on being published and being published is hard because they are attached to someone who enjoys program committee work, then they can find themselves in an impossible bind that is ours as a community to loosen because they may be powerless to do so individually.
In some cases, too, senior members of the community, especially those conscious of how this effect of Co-Authorship Entanglement could hold back their colleagues, may be forced to opt out of peer review to avoid entangling their colleagues (i.e., their would-be co-authors). I don't see that that this pressure for them to voluntarily opt out serves the community. I see that as a loss to the community of a seasoned voice.
It has been observed by some, and I agree, that there ought not be an assumption that program committee members are pre-eminent practitioners. Indeed, it seems to me highly appropriate that there be a good mix of community opinions, both supporting and challenging established notions. Moreover, there is not even a uniquely determined notion of what makes a good practitioner. Nevertheless, it's one thing not to require that program committee members have established names, but it's quite another to create elements of the process that work against the presence of such people, especially if such elements are not properly justified.
The peer review process already has safeguards built into it, in terms of ethical guidelines, and it's important to see that additional mechanism may not necessarily make things better.
For example, except in the smallest of venues, it's likely that not all reviewers will read all papers. Indeed, if there are so few papers that everyone can afford to read all of the papers, the chances are that all of the papers will be accepted anyway, notwithstanding the reviews, just to fill the program. That's a problem in itself, and requires separate treatment, but for our purposes here it suffices to say that it is not made better by rules suggesting program committee members will observe a higher bar.
It seems sufficient to simply divide the reviewer from the ability to affect their own paper's judgment. One might claim that there is still an ability to mark others' papers down in hopes of creating extra slots for their own. But this ability exists anyway. Even at conferences where the reviewer does not submit a paper, someone bent on holding back others could grade down papers by others who are up-and-coming threats in hopes that they will nto succeed and therefore not survive to submit papers on other days. Our primary protection against schemes of this kind is not to say it does or doesn't happen, but to seek multiple opinions. By doing so, we reduce the problem to the question of whether the reviewers offer a representative interpretation of community opinion, and to the question of whether there are conspiracies.
On the matter of representative effect, it's hard to see how just picking other members of the committee to judge the paper does any worse when the committee knows the person being judged is on the committee. What matters is power relationships, and we already insist that no one with a power relationship be judging another's paper. Indeed, it might be the case that someone will give deference to another person on the committee, but surely they might give deference to that person anyway, even if they're not on the committee. Committee membership is only one of the various credentials that influence our opinions, even sometimes unconsciously, and is not the strongest of such credentials. Consequently, unless we're prepared to merely give lesser weight to anyone that has such a credential of any kind, which would seem perversely the opposite of what we seek to do, singling out this one kind of credential for special treatment seems odd. This principle is sometimes summed up as "a system is only as strong as its weakest link". This is not the weakest link, so optimizing it without addressing the weaker links is not helpful.
On the matter of conspiracies, it's very difficult to test for them. The cost of testing for them may be worse than the cost of sometimes tolerating them. About all you can do sometimes is invent strategies like double-entry bookkeeping and parity that exploit redundancy to detect anomalies. Peer review by multiple reviewers is of that kind—redundancy of opinion is the chief defense. Additional subjective requirements of weighing certain papers down is not provably helpful and might sometimes be injurious. Certainly they will be unevenly applied, and I would argue that the need for that unevenness has not been adequately demonstrated.
| Home • Contact • Donate • Legal Notices |
| Copyright © 2010 by Kent M. Pitman. All Rights Reserved. |