Saturday, January 10, 2009

Journals

A senior philosopher I respect once opined that there are only a few truly blind-reviewed journals: he listed Phil Review, Nous, and perhaps one or two others. Which journals are truly blind-reviewed, and (more interestingly) which journals are in practice not blind-reviewed?

20 comments:

bari said...

Nous and PPR are run by the same people; if one is blind, so is the other.

I've reviewed for Phil Quarterly, Nous, Synthese, and PPR. In every case, the manuscript was prepared for blind review, and the editor gave me no indication who the author was. (In one case, I recognized the paper.)

Unless my experience is atypical, I think that most journals do practice blind reviewing. However, with so many manuscripts online, finding the author's identity is often as easy as a simple google search.

Anonymous said...

... and, I'd add that I've more than once had refs from Nous and PPR use google to track down my identity while reviewing papers for that journal. (I think I've noticed that this has happened about four times with Nous/PPR and only twice with other journals. It happened once with Mind and once with Erkenntnis. In each case, I've seen people googling the title of a paper and following that to my webpage when the paper was under review.) I can't fault the people who run that journal for that, but it does strike me as strange that the people in my area seem to do this with more frequency than I've experienced with other journals and people in my area connected to the editor have an amazing number of pieces published with these journals. (I think it's odd when someone publishes near half of their work with PPR and Nous.) One possible explanation that isn't particularly sinister but isn't particularly good is that members from this group are for some reason disposed to expose themselves to the biases that result when review is unblinded.

There's my two cents.

Anonymous said...

I have to confess that I google the papers that I referee after I've read them and written the report. I'm not always all that confident in my own judgement, particularly when my report is negative, and I like to double-check that I haven't just dismissed the work of someone with a great reputation. On the other hand, I don't factor the name into my initial judgment, and if I thought the work was good, I would never change my mind based on what I found on the web...

Really?!?! said...

Hey anon 5:55, you're an asshole...and a coward.

Anonymous said...

Five years and blind review will be dead. Ten years, tops. For instance, everyone knows that Philosophy of Science has been an editorial disaster for a while now, so people simply post their papers (unreviewed!) to the phil sci archive. Soon, everyone will do something similar.

I doubt that the review process adds much. I've received as many intelligent comments as obtuse ones, but only just as many. You might worry that without blind reviews, there would be a surplus of mediocre work to sort through, but we have pretty good mechanisms for sorting through things (i.e., the google) that weren't available ten or twenty years ago. Most of what I read is determined by recommendations from others or by online/database searches, not by where it's published.

Anonymous said...

6:45-

How so? [I'm not 5:55, by the way]. No one is being harmed by the practice, only helped. It seems to me a sign of intellectual integrity to admit to the fact that one does not possess infinite wisdom and that one's own assessment of a philosophical work is not an infallible guide as to its worth. If we initially have a low view of a paper and see that it's authored by someone relatively unknown, it still may be the case that our assessment of the paper's worth is wrong, but it seems more likely that it's wrong if we discover that the paper is authored by someone we know to have produced a large quantity of very good work.
I think it would certainly be wrong change a positive assessment of a paper based on the author's reputation, but 5:55 isn't advocating or doing that. Noting the reputation of the author can simply be one more fail-safe that helps to ensure that we lower the rejection rate of philosophically interesting work. This is not going to ensure that no philosophically interesting work gets rejected, of course, or that one cannot reject the work of a superstar that's weaker than usual (it happens to us all, right?) but I don't think 5:55 meant either of these things.

Anonymous said...

In what ways other than googling is the blind review process compromised?

Anonymous said...

8.25: Anything which benefits big names in a way it does not benefit others is potentially a problem. There is very limited space in any journal, so if you positively evaluate some paper that you initially thought wasn't too hot, just because it's by a big name, that's likely to mean there's one less space for some non-big-shot to publish their work. That's not the kind of decision that should be made on the basis of anything other than quality of work.

However, the possibility remains that reputation can be *evidence* of quality. Whether or not one treats it as such will depend on how much faith one has in the profession.

(I'm not 6.45, incidentally)

chrono said...

i was going to post, but i find that my thoughts are perfectly captured by 11:41.

bari said...

So, no confirmation of the rumor that many journals don't really practice blind refereeing?

What exactly was alleged?

an ominous moderator said...

I remember the senior philosopher in question being somewhat vague when it came to exactly how blind review was supposed to be subverted -- if he offered anything by way of specifics, I don't remember.

It's always possible he was just bullshitting, though that would surprise me a bit.

Anonymous said...

There is also the role of the editor that needs to be considered. I take it that at some (most?) journals, not every submission is sent on to referees. I would also assume that the person doing the vetting knows the author of the submission.

Anonymous said...

There are different stages at which the review of an article can be blind. As people have pointed out above, many journals--my guess, nearly most--send out their manuscripts to referees blind. When I get a manuscript, there are no (or barely any) identifying markers. In my field(s), I generally don't recognize manuscripts. My refereeing is genuinely blind. Friends in more intimate fields have reported to me that they rarely see a manuscript they don't recognize (from talks, conferences, etc--all on the up and up). I suppose this sort of reviewing isn't completely blind.

The editor's job, however, represents another dimension along which the review process can be more or less "blind." Please take what follows with a grain of salt since most of it is hearsay and what passes in my mind for commonsense. An editor can sometimes make a very educated guess about what a reviewer will say about a submitted paper. Editors get to know particular reviewers and their tastes. They can also make generalizations based on the profiles of referees--e.g. graduate students tend to be REALLY critical, where crusty tenured professors tend to be softer (I emphasize 'tend to').

The upshot is that an editor can stack the deck against a paper. She can send a paper to reviewers that will give a paper a more favorable review. She could do this by design. But I suspect it is a side effect of other motives. That is, more likely, the editor will send papers submitted by noteworthy philosophers to noteworthy reviewers (who, by the way, are more likely to recognize the work), out of respect; and send paper submitted by philosophical plebes to other plebes. The fact is that, it's *highly* unlikely that an editor is going to send a paper of mine to Dretske to referee. You might think that's good; but based on my experience, I would MUCH rather have someone like Dretske referee my paper than the newly hired Assistant Professor at Central Eastern Blah-blah State U.

Anonymous said...

graduate students tend to be REALLY critical, where crusty tenured professors tend to be softer (I emphasize 'tend to').

This is true to the experience of many people in my generation, including me. It wasn't that I was any cockier. It's just that I was a little bit less open-minded and not as widely read as now. It was harder to see how a paper that didn't directly answer problems of concern to me could be interesting or important. And when I came across the inevitable flaw, it was easier to pronounce it fatal than to explain to the author how to correct it.

uturn said...

On several occasions, I have submitted a paper to a conference or journal and shortly thereafter found someone searching for the title on Google. Sometimes the searcher's IP address traces to the institution where the journal editor works or where the conference was to be held.

Because of this, I no longer post paper drafts to my website.

That said, I do not find 5:55's behavior problematic, so long as she only searches for a paper's author *after* submitting a referee report. Otherwise it is clearly unethical and likely to introduce unnecessary bias into the process--bias that blind review is specifically designed to ameliorate.

Anonymous said...

Anon 1:52 a.m. checking back in. I think Anon 7:59 p.m. made some really important observations. I think it points to the importance of having a greater number of individuals with editorial responsibilities and more clearly defined guidelines about what standards referees should use in evaluating a paper.

Anonymous said...

8:37 said--
"It wasn't that I was any cockier. It's just that I was a little bit less open-minded and not as widely read as now."

Maybe part of it is also that us younger philosophers are under constant pressure to say something "new and relevant" in order to get a job, get tenure, etc., while older philosophers don't have this pressure. It's actually kind of unfair, really. Younger philosophers are less developed, yet under more pressure to put out brilliant philosophical work. Seems like asking a college football team to win the Super Bowl. That kind of expectation, combined with our limited philosophical ability, tends to lend itself to constant criticism, which is much easier to sustain than positive projects, and through which many of us try to show we're smart, relevant, etc. I hate the negativity, too, but the way our employment system is set up seems to foster it.

Anonymous said...

Anonymous 1:52am: How were you able to discover that your referees were googling you, and how were you able to discover that they were your referees?

Thanks,
Igor

Anonymous said...

Hey Igor,

There might be various ways to do this. I'm not I could say I "knew" that papers were being googled, but it is not atypical for people to have devices built into webpages that tell the page's owner what browsers were used by visitors, what search strings were used to reach the page, the visitor's IP address, and the like. So, suppose on Monday I send a paper to one of those journals that tells you when referee's have agreed to referee the paper (e.g., Phil Studies). I see that a user from University of X reaches my page by googling the title of my paper in quotation marks within a day or two of a referee agreeing to review the piece. Suppose no one has prior to that ever reached my page by searching that string. Further confirmation comes from the fact that no one has since googled the paper to the page. That's some indication that one of your ref's has just discovered your identity. (And, if you look on the department's page the author might have discovered the ref's identity. Assuming, that is, that there are not loads of folks at that department that could be the ref for your paper.)

There might be other ways of discovering the identity of a referee, but that's one way I've done it in the past. The other is a former referee telling me drunkenly that they felt bad for having recommended a paper of mine for rejection at a conference. If anyone has any other ideas, I'd also love to know. From my past experience, some refs I've suspected knew my identity gave really bad reviews. Some, however, gave really good reviews and I suspect went out of their way to give good feedback and encouragement b/c they knew I was a nobody who could use a break. I think there's always reasons to try to preserve the integrity of blind review, but it seems some do know themselves well enough to know to not let that information bias their reviews.

Anonymous said...

Has anyone got the impression that journal reviewers have got more ridiculously harsh (like they weren't assholes already!!) since the beginning of the credit crisis??