Peer Review

Double-Blind Peer Review

Have you ever wondered about the vetting process for scholarly publication? Most respected academic journals and university presses subject manuscripts to a rigorous process of double-blind peer review. This process involves sending the anonymous manuscript to two experts in the field, who read and evaluate the work without knowing who produced it. The reviewers typically recommend one of three options:

  1. Accept and publish
  2. Revise and resubmit
  3. Reject

The editors share these anonymous reader’s reports wit the manuscript author(s). All of this work takes place anonymously behind the scenes, and it is a slow, arduous process.

Our site has been submitted and approved via a double blind peer review process conducted by ModNets, an organization that provides a vetting community for digital modernist scholarship.

An Experiment in Public Peer Review

Even as we submitted our site for a traditional process of double-blind peer review, we simultaneously conducted an experiment in a public peer review. We hoped this approach would make peer review more open, collaborative, efficient, and even enjoyable. In Spring 2019, we used—a free, web-annotating tool—to enable users to comment on our site. This powerful tool provided an easy to use method for making the peer review process public, and we recommend it highly. The Hypothesis team is also extraordinarily knowledgeable, helpful, and responsive, and worked with us to set up the tool and tailor it to our needs.

We received more than 80 annotations from our Advisory Board, but none in the Open Forum, despite an active social media campaign urging people to participate. Nevertheless, the feedback we received from our Advisory Board was insightful, informative, and helpful.

While the Mina Loy Baedeker: A Scholarly Guide for Digital Travelers has been peer reviewed by professional scholars in the field, other parts of the site, such as the student projects displayed in New Frequencies. Bios, Maps, and Timelines have not been vetted through the same rigorous process, in order to give students more freedom to experiment and to accommodate limitations of time and professional training.

screen shot of Hypothesis annotation in sidebar next to text from one of the scholarly chapters
Hypothesis annotation from Advisory Board member.

After responding to their comments and suggestions, we made an editorial decision to deactivate the Hypothesis plugin for several reasons:

  1. The tool is not yet accessible for vision impaired users, and we are trying to make our site meet web accessibility standards.1
  2. The yellow highlights that appear on annotated text distract from the site’s aesthetics, which are designed to promote an immersive reading experience.
  3. As the site expanded, its loading time slowed considerably, and we suspect that our use of multiple plugins may be a factor.

Although Hypothesis is no longer installed in the site, you can still install it on your browser and use it to annotate in public, or create your own private annotation group. We encourage yo to try out this great tool. Below, you can find screen shots and instructions for using the tool.

Instructions for Reading Expert Peer Review and Offering Commentary on Hypothesis

During the time period when we activated the Hypothesis plugin, a sidebar would appear on the top right of every page of our site:

screen shot with arrow indicating position of Hypothesis sidebar on upper right corner of screenUsers could open the Hypothesis sidebar by clicking on the top arrow <. At the top of the sidebar, they could select either Advisory Board PR or Open Forum.

Advisory Board PR

The Advisory Group PR is a group of ten experts in the fields of Mina Loy studies, modernist/avant-garde studies, and digital studies who agreed to support our project, writing letters of commitment for our NEH grant application. These experts agreed to peer review the site using Hypothesis in the spring of 2019. We created the Advisory Board PR group for them in recognition of the need for expert peer review to verify the authority, accuracy, and value of our scholarly work. Using this group, or Advisor Board members could comment on the site, as well as read and reply to one another’s annotations. Their comments were publicly visible, but the public could not join or participate in their group.

To read expert commentary, users could select the Advisory Board PR Group at the top of the sidebar. Their annotations appeared as page notes or highlighted text throughout the site.

Open Forum

We also created an Open Forum in recognition that public users also have expertise, insight, and meaningful feedback to share. Through this forum, users could also read and respond to one another’s comments. We welcome your comments and suggestions.

To participate in the Open Forum, users had to first create a free Hypothesis account, which they could do by choosing “Log in” in the Hypothesis sidebar, nd navigating to “Don’t have a Hypothesis account? Sign up.” Or they could create an account on the website. Once users created and verified your account, they could login and select the Open Forum group at the top of the Hypothesis sidebar to comment on the site.

Annotations and Page Notes

To annotate a specific portion of the text, users could highlight a selection of text on the page and click the “Annotate” button that pops up:

screen shot of selected text with annotation icon highlighted.

To comment on a page as a whole, users could click on the Page Note button on the annotation sidebar:

screen shot with arrow indicating page note button on right sidebar

Whether they created an annotation or a page note, users must click the “Post to Public Forum” button before moving on.

Notes about annotations and page notes

  • Once you have created your annotation, you can edit it by clicking on the pencil icon.
  • To delete your annotation, you can click on the trash can icon.
  • If you’d like to respond directly to your annotation or someone else’s, click on the reply arrow.
  • To report inappropriate content or abuse to the group moderator (Suzanne Churchill), click the flag icon.

screen shots of annotations window with errors pointing to toolbar options.Helpful Links


  1. For information about the accessibility of this tool, please consult Accessibility at Hypothesis.