In the spring of 2019, we conducted an experiment in public peer review using the free, online annotating tool Hypothesis. Hypothesis is a plug-in allows users to annotate any text on the internet. In the spirit of the en dehors garde, we wanted to turn outward and challenge the traditional double-blind peer review system, a long, slow process in which experts read manuscripts by unidentified authors and submit letters of evaluation, protected by the cloak of anonymity.
What would happen if we tried to bring a more open, collaborative spirit to the work of peer review? Could expert scholars provide honest criticism, highlighting both strengths and weaknesses, without the protection of anonymity? Would our readers be interested in seeing the process of peer review as it unfolded?
We considered creating a public forum open to all readers, from experts to enthusiasts—in other words, a crowd-sourcing approach to peer critique. While we recognize the value of the social construction and curation of knowledge as effected on an ever-expanding resource like Wikipedia, we realized we had only a limited time to receive and respond to feedback from the public. We also wanted to make sure experts vetted our site content and technical design, in order to make sure it meets the highest scholarly standards.
We explained our goals to the team at Hypothesis, which included Jeremy Dean, Heather Stains, Jon Udell, and Katelyn Lemay. Knowledgeable, responsive, flexible, and enthusiastic, they were terrific to work with. They helped us clarify our vision and understand what was technically possible. They also provided information, guidance, and expert instruction to enable us to achieve our goals. We installed the Hypothesis plugin on our site, and the Hypothesis team created a closed but public Advisory Board group. This group includes experts in the fields of Loy studies, avant-garde studies, and digital humanities. Their comments were publicly visible, and they could read and reply to each other’s comments. Because we wanted to include the public in the process, we asked the Hypothesis team to create an Open Forum group that anyone could join.
We invited our Advisory Board members to review and comment on the site between March 18 and May 28, 2019 (see our letter of invitation below). In the span of the peer review, we received more than 80 annotations on the site, ranging from macro insights to micro corrections. The critiques were as helpful as the expressions of enthusiasm, and we were able to make many changes and improvements to the site based on their suggestions. Unfortunately, despite our social media campaign encouraging the public to participate, we received no annotations in the Public Forum.
Based on our experience with peer review we highly recommend the Hypothesis tool and encourage you to use it. Although we consider our experimental use of it for public peer review a success, we have made an editorial decision to deactivate the Hypothesis plugin for two reasons:
- The tool is not yet accessible for vision impaired users, and we are trying to meet web accessibility standards.1
- The yellow highlighting that appears on annotated text distracts from the site’s aesthetics, which we believe are essential to an immersive reading experience.
Although you can no longer see the Advisory Board’s annotations, the good news is that you can still use Hypothesis to annotate our site in public, simply by going to their site, signing up for a free account, and installing Hypothesis in your browser. We’d love to hear from you!
Invitation to Advisory Board for Hypothesis Peer Review
Dear Advisory Board:
We are pleased to report that Mina Loy: Navigating the Avant-Garde is now ready for your peer review.
How to comment
Annotating with Hypothesis is simple. All you need to do is go to https://mina-loy.com, click on the upper right sidebar, login to Hypothesis, select the Advisory Board PR group, and begin commenting. You can find illustrated, step-by-step instructions in this Google doc (thanks, Katelyn Lemay at Hypothesis!).
Where to begin
We suggest you begin by navigating to About/Our Team and adding an author’s note to introduce yourself to the other Advisory Board members and to the public, who will be able to see but not respond to your comments. We hope that personalizing peer review in this way will make the process more open, collegial, and even enjoyable for you. Once you’ve introduced yourselves, you may feel more comfortable replying to one another’s comments so that a once solitary process can become a lively conversation.
How to proceed
You may roam wherever you like on the site, deciding which parts you want to read and comment on. We suggest that you spend time on your areas of expertise—e.g. Loy scholars may want to focus on the scholarly chapters, while digital humanists may want to respond to the site design or the theory of the en dehors garde and the digital flash mob. We do not expect you to offer “peer review” of student work in New Frequencies and Bios, though you are free to comment on these sections in a general way. Maps, Timelines, Artifacts, and Exhibits are still under development and will be completed this summer, though you’re welcome to offer suggestions. We don’t want to impose any restrictions on you, and we welcome feedback on any or all aspects of the project.
Adopting a DH scholarship mindset
As you respond, please keep in mind how digital humanities scholarship can productively differ from print scholarship. In writing for a digital environment, we incorporate a scholarly approach that we hope appeals to a broad audience, including scholars, students, and the interested public. We adopt prose styles appropriate to the medium and the digital reading/viewing environment. In re-mediating the conventions of print publication (reference-dense, extended discussions employing a scholarly lexicon), our scholarly writing for digital readers aims to be concise, direct, lively, substantive, and enriched with visual images.
Timeline for responding
We ask that you review the site between now and May 28, 2019. You can do so at your own pace, taking deep dives or occasional dips. We will email occasional reminders, encouragements, and thank you notes for your participation.
Joining the revolution
Okay, that’s a bit of a hyperbole, but we are not exaggerating our gratitude. Thank you so much for supporting our project and participating in this pilot program to test Hypothesis as a tool for public peer review. Your efforts will not only help us improve the content and design of our site, but also can transform peer review into a more open, collaborative process.
We look forward to receiving your comments and suggestions. Please let us know if you have any questions or concerns.
Kind regards,
Suzanne Churchill, suchurchill@davidson.edu
Linda Kinnahan, kinnahan@duq.edu
Susan Rosenbaum, srosenb@uga.edu
- For information about the accessibility of this tool, please consult Accessibility at Hypothesis.