As my first test-post, I thought I’d pass along a recent exchange on the Humanist email list. This doesn’t fall squarely within Victorian studies exactly, but it does play on some familiar Victorian themes like machine anxiety and the boundaries of automation. Note that although the given date of the first post is March 31st, it first showed up in mailboxes on April 1st…

Humanist Discussion Group, Vol. 24, No. 837.
Centre for Computing in the Humanities, King’s College London
http://www.digitalhumanities.org/humanist
Submit to: humanist@lists.digitalhumanities.org

Date: Thu, 31 Mar 2011 12:42:35 -0400
From: Julia Flanders <julia_flanders@brown.edu>
Subject: DHQ moves to all-automated peer review

Digital Humanities Quarterly is pleased to announce a move to Peer Review with Advanced Technology (PRAT): a new computational peer review system based on text analysis.

This new approach to peer review will enable DHQ to process vastly increased numbers of submissions, speeding up time to publication and ensuring consistency of review criteria, compared with results from human peer reviewers.

This move comes at a time when peer review mechanisms are coming under close scrutiny, as journals and other online publications experiment with alternatives to traditional peer review. DHQ has considered a variety of models, including crowd-sourced peer review based on reader comments, but we have determined that these require too great an investment of time and create unacceptable delays in production. Our new text-analysis-based method will eliminate human reviewers altogether and assess submissions based on a set of measures that may include:

1. Stylistic similarity to articles in the highest tier of citation rankings in a selected set of major journals, using state-of-the art measures that focus on vocabulary choice and patterns of clause construction.

2. Average and maximum word length, sentence length, and clause length. Submissions will be scored in all three categories and the results will be compared with a proprietary DHQ readability index to determine the submission’s suitability for publication. The highest-scoring articles will be published in a special “Expert” column.

3. Frequency of high-value technical terminology. Terms from specific domains will be weighted dynamically, based on current measures of usage of those terms in high-ranking peer journals, to ensure that DHQ’s published articles keep pace with the changing patterns of technical jargon in the broader community. For special issues, terms from specific topic areas will be given extra weight to ensure that submissions are related to the issue’s subject area.

4. Rate of citation of the most highly cited references in digital humanities. Submissions that cite the most highly cited references (or articles citing those) will be more highly ranked. Over time, we anticipate that this mechanism if properly used could result in the DHQ article corpus achieving complete self-referentiality.

The new system goes into effect with the special issue “From Lemons to Lemonade: Learning from project failures in the Digital Humanities”, to be published on April 1.

Best wishes, Julia, Melissa, and Wendell

Julia Flanders
Melissa Terras
Wendell Piez
General Editors, DHQ

_______________________________________________
List posts to: humanist@lists.digitalhumanities.org
List info and archives at at: http://digitalhumanities.org/humanist
Listmember interface at: http://digitalhumanities.org/humanist/Restricted/listmember_interface.php
Subscribe at: http://www.digitalhumanities.org/humanist/membership_form.php

Humanist Discussion Group, Vol. 24, No. 843.
Centre for Computing in the Humanities, King’s College London
http://www.digitalhumanities.org/humanist
Submit to: humanist@lists.digitalhumanities.org

[1]   From:    “Holly C. Shulman” <hcs8n@virginia.edu>                  (118)
Subject: Re: [Humanist] 24.839 DHQ: automate before the day is out!

[2]   From:    “Schlitz, Stephanie” <sschlitz@bloomu.edu>                 (5)
Subject: RE: [Humanist] 24.837 DHQ to automated peer-review

[3]   From:    David Golumbia <dgolumbia@gmail.com>                     (105)
Subject: Re: [Humanist] 24.839 DHQ: automate before the day is out!

[4]   From:    Alan Galey <galey.lists@gmail.com>                        (66)
Subject: Re: [Humanist] 24.837 DHQ to automated peer-review

–[1]————————————————————————
Date: Fri, 1 Apr 2011 12:31:48 -0400
From: “Holly C. Shulman” <hcs8n@virginia.edu>
Subject: Re: [Humanist] 24.839 DHQ: automate before the day is out!
In-Reply-To: <20110401155953.66D66126951@woodward.joyent.us>

I am writing in response to the move by the journal Digital Humanities
Quarterly to PRAT (Peer Review with Advanced Technology).

Before we all rush to accept – and copy – this innovation I would simply
like to inject a few words of caution, especially as a historian.  To begin
with, scholarly approaches to analysis, research, and texts vary between
disciplines, and within disciplines across sub-disciplinary requirements.
These differences at least in part determine the choice of style and
vocabulary employed by any given scholar.  In fact, the very meaning of a
word, especially a word used in a theoretical framework, may vary between
specializations.  Historians rarely deploy the same theoretical language as
literary critics.

These internal differences determine the style of any article. Thus the
introduction of automated review here will “privilege” some fields, and some
authors, over others, regardless of the merit of their contribution. This
approach to peer review will predetermine scholarship, and its expression,
in a way that will curtail innovation and diversity.

Holly C. Shulman


Holly C. Shulman
Editor, Dolley Madison Digital Edition
Founding Director, Documents Compass
Research Professor, Department of History
University of Virginia
434-243-8881
hcs8n@virginia.edu

–[2]————————————————————————
Date: Fri, 1 Apr 2011 12:30:29 -0400
From: “Schlitz, Stephanie” <sschlitz@bloomu.edu>
Subject: RE: [Humanist] 24.837 DHQ to automated peer-review
In-Reply-To: <20110401070110.57270126B97@woodward.joyent.us>

Peer Review with Advanced Technology (PRAT)… April Fools’?

— Stephanie

–[3]————————————————————————
Date: Fri, 1 Apr 2011 12:55:12 -0400
From: David Golumbia <dgolumbia@gmail.com>
Subject: Re: [Humanist] 24.839 DHQ: automate before the day is out!
In-Reply-To: <20110401155953.66D66126951@woodward.joyent.us>

#!/usr/bin/perl
$hpost = ”
http://www.digitalhumanities.org/cgi-bin/humanist/archive/archive_msg.cgi?file=/Humanist.vol24.txt&msgnum=832&start=102048&end=102155

$irony = EVAL_IN_HIST($hpost)
while(date = “today”) {
$irony = false }
else {
$irony = true }


David Golumbia
dgolumbia@gmail.com

–[4]————————————————————————
Date: Fri, 1 Apr 2011 15:22:12 -0400
From: Alan Galey <galey.lists@gmail.com>
Subject: Re: [Humanist] 24.837 DHQ to automated peer-review
In-Reply-To: <20110401070110.57270126B97@woodward.joyent.us>

Dear Julia, Melissa, and Wendell,

This is very welcome news. DHQ’s move to automated peer-review is a
wonderful idea, and a natural progression of our field.

I’m especially happy about your announcement because I’ve been working
on a complementary system to automate the writing of articles, based
on parameters nearly identical to your implementation of PRAT. The
system is tentatively called the Phonetic Iterator and Integrator (or
PhoneItIn), and should be able to produce 3-4 articles per week. It
can already detect and avoid duplication of Minimum Publishable Units.
I’m hoping to add an automated article submission and revision
component, which would mean that automated writing and review systems
would be interoperable, communicating asynchronously in a feedback
loop with no human supervision.

Unfortunately, every test-run of PhoneItIn so far only seems to
generate Eagles lyrics. Not sure why. Once I fix this problem I’ll
post the project on SourceForge.

Best of the season,
Alan


Alan Galey
Assistant Professor
University of Toronto
individual.utoronto.ca/alangaley/

_______________________________________________
List posts to: humanist@lists.digitalhumanities.org
List info and archives at at: http://digitalhumanities.org/humanist
Listmember interface at: http://digitalhumanities.org/humanist/Restricted/listmember_interface.php
Subscribe at: http://www.digitalhumanities.org/humanist/membership_form.php

Advertisements

4 thoughts on “peer-review and machine anxiety

  1. Hi-larious Alan. And since it borders on automatic writing and writing machines, I’m sure Gregory and Fiona will appreciate this too…
    p.s. Welcome to the Floating Academy!

  2. This is great, Alan! I love David Golumbia’s “encoded” irony (html as subtext), which I initially glossed over (like many imperfectly digitized humanities scholars, I’m sure) as immaterial computer-static.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s