同行评审方法

我们下面学习如何进行同行评审。我们经常需要评审其它同行的工作，提供评审意见。下面是哈佛大学老师的建议：

重要的是提供分析性的批评。具体指出你在论文中遇到的难题。你想的是“什么会让这篇文章更好？”即使你不能提供这个难题的解决办法，你也应该朝着改进你阅读的这个材料的方向努力。

你不必对文章讨论的话题有一些先验知识。你不知道，反而会有帮助。这让这篇文章的读者更加广泛。这会暴露作者以为是理所当然的细节、假设、连接其实不是那么理所当然，因此需要更加清晰的解释，才能让读者看懂。

不要害羞、或者感到不好意思，要求作者澄清、详细解释或进一步解释（clarification, elaboration, or further explanation）。这是你的责任，指出这些令人困惑的地方。

尽量具体地说出自己的困惑。比如：我猜你是想说这个，对吗？我是不是哪里没有理解？我能看到 X 和 Z 相关，但我看不出来 Y 和 X、Z 是怎么相关的。它们是什么关系？

聚焦清晰、解释、表达（clarity, explanation, and presentation）这三个方面，这些方面的问题会帮助作者改进，拼写、标点、语法的问题没有上面这三个方面重要。聚焦论文的基本元素：证据的结构、分析、和表达（structure, analysis, and presentation of evidence）。

阅读时，有4个技巧：

  1. 找出主论点（central argument）；

  2. 找出为了支持主论点而应该完成的各项任务：提供背景信息、阐述（expounding）理论、提出（develop）假设、描述（present）证据、在所提供的证据之间建立联系、论证证据与理论的因果关系、描述一种方法等；

  3. 评估每一段是否合适，标上 + 和 -；

  4. 评估每一章是否有效，包括：证据、可读性和透明度。

最后思考三个问题：

  1. 自己学会了什么；
  2. 还想知道什么；
  3. 作者在哪里表现出犹豫，哪里非常自信？

把这些都反馈给作者。

参考文献


Index Previous Next

附录 1：

附录1：政府学院同行评审指南

In the course of the Senior Thesis Writers’ Seminar, you will evaluate several of your peers’ draft chapters. The goal of this peer review is to expose you to collegial criticism of your work. This is exactly the kind of feedback that your professors and teaching fellows solicit when they ask colleagues to critique drafts of their work or send articles to peer-reviewed journals.

You are well qualified to undertake this sort of criticism because, as a writer of political research and analysis, you are attuned to those factors that make for good research writing. This process will hone your skills as your own editor; it will also assist your peers in honing their skills. Remember, the key to this process lies in offering analytic criticism; you need to be specific in identifying the nature of the problems you encounter, and your mindset should be, “What would make this piece better?” Even if you cannot offer a solution to your peer’s problems, you should seek to improve what you read.

Knowing something about the topic that you read about is not really necessary. It may, in fact, be helpful that you do not know much about it—you will be able to give advice and feedback that will make the work under consideration more accessible to a wider range of political science readers. You are the expert on your own topic, and that can sometimes lead to a myopia about the subject of your labors. Peer reviews are opportunities to assess whether details, assumptions, and connections you take for granted need to be made clearer for the sake of your readers.

Everyone shares responsibility for achieving the goals of peer reviews, so all participants must adhere to the following guidelines and instructions.

When preparing feedback, do not be shy or embarrassed about asking for clarification, elaboration, or further explanation. As a peer reviewer, it’s your responsibility to signal to your partners that passages or points are potentially confusing. This is not done due to the writer having chosen a bad topic or approach—you must offer advice on how a friendly but skeptical reader may approach the piece. You are an ideal thesis reader because you are a thesis writer.

If you are still perplexed after reading a passage over and over, do not be afraid to flag that for the writer. Assume the challenge of helping the writer convey the point to the reader.

Being a good reader requires a lot of effort, almost as much as writing the passage in the first place. You may have been frustrated in the past by the number or quality of the comments you have received (or, more usually, did not) on class papers; now that you can see how difficult that task is, make sure that your thesis writing peer gets comments at least as good as those you hope to get yourself. Be specific in your comments. “This passage was unclear—can you explain it to me?” would be much less helpful than:

As a peer reviewer, your job lies in motivating your thesis partners to improve the drafts that have come out of the research they have done. By focusing on issues of clarity, explanation, and presentation, you help them to present themselves as self-aware, confident researchers. Do not, however, “line-edit.” Checking spelling, punctuation, and grammar are less helpful at this stage than you might think. If your thesis partners take your suggestions seriously, they will engage in some substantial revisions, and such suggestions will be largely irrelevant. The focus of peer review is on fundamental elements of chapters: structure, analysis, and presentation of evidence.

Remember the above when receiving comments from thesis writing partners. The tone and content of feedback should be positive, even when critical. As you receive your comments from your partners and seminar leaders, you will help yourself much more by listening than arguing or defending your writing.

Tips for reading chapters

When you read a classmate’s chapter, you should follow the general approach outlined below. Your Gov 99 workshop leader will provide specific instructions for the format of comments for your classmates.

Locate the central argument

What is the central argument of the work in front of you? There should be some passage that the author uses to indicate the main idea the author wants to convey over the next 25 pages or so. Double underline that passage.

Identifying tasks

In carrying out the central idea of the chapter, the author should indicate a series of tasks or steps that she or he will fulfill to sustain that argument. These tasks can include: offering background information, expounding on the theory, developing hypotheses, presenting evidence, drawing connections between pieces of evidence presented, arguing for the causal connection of evidence to theory, describing a methodology, and so forth. For this exercise, find (up to) the four most important tasks the author sets out; underline and number those tasks. Assess whether the author accomplishes those tasks (or whether there are tasks that the author should have undertaken but did not), and label each passage with the number of the task it helps to fulfill.

Paragraph assessment

Every paragraph in the draft chapter should accomplish some piece of the tasks that the chapter seeks to fulfill. Most will do this in a fairly adequate fashion, and so they will not need special attention from you. Others, however, will strike you in some way, and you need to mark them so the author knows that he or she should pay special attention to those paragraphs in the process of revision. Mark paragraphs like these with a + or - and elaborate upon the reasons for doing so in a marginal note.

§ Paragraphs that do not help to accomplish the stated goals of the chapter are likely to be superfluous, and they might need to be removed. BUT they might simply be misplaced. Consider, for example, an interesting point or piece of evidence that does not seem to be related to the stated tasks of the chapter. It may simply be that the author has left out material that would connect this material to the main ideas and tasks of the chapter. Help the author to think about how to connect this lonesome bit of prose to the larger chapter.

§ Perhaps it really is filler material. Explain why you think this to be the case.

§ It may be repetitious or wordy. Help the author to make efficient use of his or her prose.

§ It might be an interesting point, but perhaps it detracts from the main flow of the text and argument. You could suggest that the author consider moving it to a footnote or another location where it will prove less of an impediment to the larger goals of the chapter.

§ The paragraph presents material vital to the central argument of the chapter.

§ The paragraph links together tasks or crucial ideas.

§ The paragraph fulfills one of the tasks you identified above.

Topics to consider

In all cases, there are some “objective” criteria that all draft pieces of writing must fulfill if they are to be truly effective. No matter how polished the prose, sophisticated the methods, or brilliant the theory, the chapter can still fail if the author does not do the following.

There are several tasks regarding evidence that an author must fulfill. First, is there enough convincing evidence or evidentiary logic to sustain the central argument? If the author does not provide a representative sample or does not show how the philosophical principle under discussion matters in the “real world,” then there may be sufficient reason to doubt the conclusions. Second, does the evidence have sufficient explanation or analysis? In this case, if the author does not provide a full discussion of what the evidence can indicate, the chapter may again fail to fulfill its purpose or tasks. Third, not all components of the chapter or tasks are equal in importance. The space an author dedicates to a task should be proportionate to that component’s importance in advancing the central argument.

This topic covers several matters. First, the material must be accessible to the reader. As thesis writers, you may often get so involved in the subject that you forget what “basic” knowledge you need to provide for your reader. All key terms, especially ones that are specialized or used in very particular ways, should be defined. The narrative should move smoothly from one point to the next; transitions and the pattern of logic should be clear. Is there enough background material to make the context of the central point and the tasks clear? Second, the material should be transparent. Do you detect any subtle or explicit biases? Does the author consider all the evidence, or just that which upholds one side of the story?

Final thoughts on peer review

When you finish reading a peer’s chapter, consider the following questions:

Finally, write down three reflections about the chapter, perhaps highlighting elements you found intriguing or suggestions for general improvement.

Reviewing Your Own Chapter

In large part, the way you review another’s chapter is how you should review your own. This is certainly not easy, but you must critique your writing if you wish to improve it. Follow the instructions for locating the central argument, identifying your tasks, assessing paragraphs, and the final instructions in the previous section.

附录 2：历史系同行评审指南

和哈佛政府学院的内容差不多。

Peer reviews in seminar are intended to empower you to offer and receive collegial criticism, a fundamental part of good historical scholarship. The History Department emphasizes these skills throughout the tutorial program. As you learned when exchanging “packet papers” in History 97, reviewing others’ work and providing feedback is a good way to develop your talents as both a reader and a writer of history. The process helps you become your own best—that is, most demanding— editor. At the same time, you are also helping your peers improve their own projects, thereby strengthening the historical writing produced by your cohort of concentrators.

The exercise is less about the particular histories being written and more about the general art of writing history. Therefore, you do not need to worry about how much (or little) you know of your partners’ topics or what they know of yours. In fact, it can be advantageous to have partners who know little about your specific topic, but a lot about the Department’s curriculum and its goals. Feedback from such readers can help you make the presentation of your evidence and argument accessible to a wider range of historians. After all, you should always assume that you are the expert on your chosen topic and know more about it than your readers. Peer reviews are opportunities to assess whether details, assumptions, and connections you take for granted need to be made clearer for readers. Since your adviser may be too familiar with your research to anticipate the questions of general readers, your partners in seminar could be the best sources of this kind of scholarly support.

Guidelines

Everyone shares responsibility for achieving the goals of peer reviews so all participants must adhere to the following guidelines and instructions.

When preparing feedback, do not be shy or embarrassed about asking for clarification, elaboration, or explanation of your partners’ writing. As a peer reviewer, it is your responsibility to alert your partners to confusing points and/or passages; this is the only way to make the exercise productive and stimulating for all participants. In doing so, you are not telling authors that they have picked a bad topic, done a poor job, or taken the wrong approach; rather, you are offering them valuable insights into how a reader might interpret their writing. You are the ideal thesis reader because you are a thesis writer and, thus, are acutely aware of the value of good historical writing. If you are still perplexed after reading a passage over and over, do not conclude that your reading comprehension has failed you; instead, assume the challenge of helping the author convey his or her point to readers.

Peer reviews are opportunities to assess whether details, assumptions, and connections you take for granted need to be made clearer for readers.

You are the ideal thesis reader because you are a thesis writer and, thus, are acutely aware of the value of good historical writing.

Being a good reader requires considerable effort, almost as much effort as writing a paper yourself. Be prepared to devote a significant amount of time and thought to the exercise. Remember that you are counting on your partners to make the same investment in your chapter. Consider your questions carefully and phrase them constructively; this means being as specific as possible. For example, “This passage was unclear. Can you explain it to me?” is less useful than alternatives like the following:

Keeping in mind the thesis submission date, your job as a peer reviewer is to motivate your partners to do what is possible to improve the drafts that have emerged from their research. By alerting your partners to aspects of their presentation that they might not have been conscious of, you will help them come across as selfaware authors when they submit their final theses.

Peer reviews are not blanket invitations for peer editing. Details like grammar, vocabulary, and punctuation are better left for final stage revisions; they are distractions from the primary goal of the exercise, which is to ensure that students are doing the best historical writing possible. The focus of peer reviews is on fundamental elements of chapters, such as structure, analysis, and presentation of evidence. The instructions below should help you focus on these fundamental elements.

In this exercise, everyone is both a reader and an author so think about the above guidelines when you are receiving feedback. The tone and content of feedback should be positive, even when critical. When it comes to receiving comments from your partners, you can help yourself more by listening than by talking or defending your writing.

Preparing for Seminar

Everyone will be assigned to a group of three. Group members will be responsible for reading each other’s work in advance and coming to seminar prepared to discuss them. Groups of three require twice the work of pairs, but they double both feedback and exposure to different techniques (organizational, interpretive, and stylistic) employed by peers. Whatever the number, it only works if each pulls his or her weight. The deadline for submitting your peer-review chapter is non-negotiable.

Skim your partners’ chapters without taking any notes or making any marks. When done, jot down whatever you took away from the chapters. Then, read the chapters more carefully and follow the instructions for readers (below). For your own chapter, follow the instructions for authors (further below). Some of the instructions require actually making marks and notes, but many involve thinking about what you have read and marked in preparation for the seminar discussion with your partners. You can mark up the texts on screen with word processing tools (e.g., Insert/Comment, underline) or on paper with a writing implement. However you approach the task, you must bring hard copies of all three chapters to seminar and arrive ready to discuss all three.

Note: Since everyone has different writing styles, the unit of measurement for this exercise is flexible. The term “passage” is used to indicate a single sentence or string of 2-3 sentences.

Instructions for Reviewing Peers’ Chapters

Locating the Central Argument

What is the central argument of the chapter? There should be a passage that indicates to you the most important point that the author is trying to convey with the evidence and analysis offered in the chapter. While essential, the central argument will not necessarily appear at the beginning of the chapter.

Double underline the passage (only one) that represents what you consider to be the central argument of the chapter.

Identifying Tasks

What tasks does the author indicate will be fulfilled in order to sustain the central argument of the chapter? Common tasks to advance the central argument include providing necessary background (historical or theoretical), presenting bodies of evidence, comparing pieces of evidence, and demonstrating causal links. For the purposes of this exercise, tasks are limited to four to encourage you to identify the connections between different elements of the chapter.

Underline the passages (up to four) that signal the tasks to be accomplished.

Does the author fulfill these tasks over the course of the chapter?

Topics to Consider

Evidence

Is there enough evidence included to sustain the central argument of the chapter? Or, does the author

Are pieces of evidence sufficiently explained and/or analyzed to be of maximum value for making the central argument of the chapter? Or, does the author

Is space devoted to the evidence proportionate to its importance to the central argument of the chapter? Or, does the author

Accessibility

Are essential terms defined? Would more background at any point help a non-expert reader? Are there any areas of ambiguity about the narrative or argument?

Transparency

Do you detect anything from reading between the lines?

An implicit bias or sympathy in the author’s presentation? A lack of consideration of different kinds of sources or viewpoints? A subtle point that the author should make more explicit?

Transitions

Is there a logical progression from one topic or section to the next?

If a transition seems abrupt, what is missing?

A simple transition sentence? An adequate conclusion of the previous topic? A smooth introduction of the next topic?

Final Thoughts

As soon as you finish reading the chapter, consider the following questions:

Estimate the ratio of summary (background, discussion of literature) to original argument (presentation of primary sources, analysis).

Write down three reflections.

These can highlight the elements you found most intriguing or take the form of suggestions for addressing any of the above questions.

Instructions for Reviewing Your Own Chapter

Locating the Central Argument

What is the central argument of your chapter?

There should be a passage that indicates to readers the most important point that you are trying to convey with the evidence and analysis offered in the chapter. While essential, your central argument does not have to appear at the beginning of the chapter. Double underline the passage (only one) that represents what you consider to be the central argument of the chapter.

Identifying Tasks

What tasks do you indicate will be fulfilled in order to sustain the central argument of the chapter? Common tasks to advance the central argument include providing necessary background (historical or theoretical), presenting bodies of evidence, comparing pieces of evidence, and demonstrating causal links. For the purposes of this exercise, tasks are limited to four to encourage you to identify the connections between different elements of the chapter.

Underline the passages (up to four) that signal the tasks to be accomplished.

How do you fulfill these tasks over the course of the chapter?

Final Thoughts

Consider the following questions about your chapter:

Write down your answers

Comparing them with those of your peer reviewers might shed light on assumptions or approaches that characterize your project as a whole.

Estimate the ratio of summary (background, discussion of literature) to original argument (presentation of primary sources, analysis).

Discussing Chapters in Seminar

With three partners per group, there will be 15-18 minutes to discuss each chapter in seminar. This may seem like a lot of time, but it will go by quickly. Since every chapter is unique and group dynamics will vary, you are left to shape the discussion of each chapter. The instructions above will give you a framework for getting the conversation started. The following is a prioritized list of topics to cover:

1) Readers should indicate what they consider to be the central argument of the chapter. Discuss any discrepancies between the two readers or a reader and the author.

2) Partners should compare notes about tasks and how they are fulfilled. Discuss any discrepancies between the two readers or a reader and the author.

3) As time permits, readers should raise matters from “Topics to Consider” and “Final Thoughts.” It is likely that each member of the group will be particularly interested in discussing one or two of these questions. Partners should come ready to address any of the items in “Final Thoughts” and may wish to write up brief answers in advance.

Following Up with Your Adviser

Work in seminar is most valuable when integrated with the adviser-advisee relationship, so you should meet with your adviser soon after peer reviews. Your partners’ suggestions might lead you to do extra research, reorganize your writing, and/or change the emphasis of your chapter. Since any of these tasks could alter your work schedule and final product, your adviser must be part of the decision-making process.

You should not walk away from seminar feeling like you need to conceive your thesis anew or do weeks of additional research. Regardless of the feedback you receive, you still know the content of your project better than anyone else. As you decide how to follow up on suggestions from your partners, think about how you can do so primarily with sources you already have. At this stage, it is wise to tailor your writing to your research because there is little time to do additional research. Your goal should be to produce a final version of the thesis that is polished and reflects your understanding of its achievements and limitations.

附录 3：Wilson 老师的同行评审指南

https://wilson.fas.harvard.edu/files/jeffreywilson/files/jeffrey_r._wilson_commenting_worksheet.docx

Writers, fill in question #6.

Writers, give your paper to your commenter(s). The following steps are for commenters.

  1. Put yourself in the proper frame of mind.

Don’t think to yourself, “I’m going to fix this paper.” Instead think, “I’m going to try to learn something.” Then, as you read, take note of two obstacles to your leaning: (1) the moments when ineffective writing hampers the communication of ideas, and (2) the times when your understanding of the subject is incompatible with the paper’s representation of that subject.

  1. Scan the paper.

One of the trickiest things about commenting on papers is determining whether or not the paper at hand is a serious paper, whether the writer gave his or her best effort on the paper, or he or she just shat out a crap paper without much thought or effort. To determine whether or not the paper you’re about to read is a serious paper, start by scanning through it and asking some basic questions:

• Does the paper fulfill the assignment? • Is the paper properly formatted? • Is there a fairly obvious thesis statement? • Is there a fairly clear organization to the paper (introduction, body, conclusion)? • Do the sentences seem to be well written? • Is the paper a serious paper?

It is rare to see a paper, however serious it might be, that is so good it needs no improvement whatsoever; it is also rare to see a non-serious paper that holds no promise at all. In general, a serious paper will allow you to engage with the ideas in play – to learn something, to wrestle with an argument, to agree or disagree – but a non-serious paper will require you to direct your energy to understanding what the writer is trying to say and helping that writer actually articulate his or her ideas.

  1. Read the Paper.

Without making any marks, read the paper from start to finish, aiming not to correct but to understand the paper. Thus, you must read slowly. As with all reading – as, indeed, with life – you will get yourself into serious and irreparable trouble if you try to fix something you don’t understand. If you must make marks as you read, maybe put a ✓ for things that are good that you want to comment on later and an ✖ for things that are bad that you want to comment on later.

On Comments in General: As you start your commentary, resist the urge to circle or correct misspellings, punctuation errors, and so forth. Instead, comment first on ideas and organization: encourage the writer to solve these higher-order problems before turning to lower-order problems. In your comments, be fair and charitable, but be honest, and be aware that silence can be seen as endorsement. That is, failing to comment on weak aspects of a paper may make the writer think that, because no comment was made, that aspect of the paper is just fine, even though it is not. Also, be specific. Point out specific strengths and weaknesses. Give very specific comments, criticisms, and compliments. Be sure to identify (by using page numbers, quotations, clear references to the paper.) the exact sections of the paper that you are critiquing. Give very specific recommendations and raise specific objections if they will be helpful.

  1. Comment on the Introduction, Body, and Conclusion.

For the moment, set aside the overall argument of the paper. Instead, answer these questions:

What are the most important and interesting points made in the body of the paper?

What are the least interesting or important?

Which ideas need to be further developed, and how might the writer go about doing so?

Are there any moments when the writer plays it too loose with the evidence?

How can the paper be more effectively organized?

At the very least, point your writer to specific passages and say, “Do more of this: it’s good,” or, “Don’t do this: it’s bad.”

Identify what’s missing from the paper, what needs to be explained more fully.

Also identify what can be cut.

  1. Write a General Comment.

Commenting on the introduction, body, and conclusion will help you fully grasp the ideas put forth in the paper, and once you’ve done so you should use this now mature understanding to write a general comment. In your general comment, you may find yourself repeating points made in your more targeted commentary on the introduction, body, and conclusion, which is fine. Write your general comment by pausing to consider the author’s intent: what is he or she trying to accomplish here? Perhaps the most helpful thing you can do as a commentator is to try to understand what the paper is trying to say, and to help the writer say it more effectively. In your view, what is or should be the main point, and how can it be better (i.e. more clearly and more concisely) articulated? If the paper is already a good paper – i.e. one in which the writing does not get in the way of the communication of ideas – then ask yourself whether or not you find the paper convincing in its argument: why or why not? If, in your estimation, the paper is poorly written, your task as a commentator is to try to draw out the argument and to articulate it for the writer. Often an argument is more apparent on the outside looking in. That’s right: try to write a thesis statement for the writer. Doing so will allow that writer to revise the paper– that is, to rewrite the entire paper – using your version of the thesis as a starting point and guideline.

  1. Writers, note any particular problem areas that you’d like help on. Everyone knows that you want feedback on your argument and your language errors, but cite any particular areas of the paper you’d like the commenter to focus on, and ask specific questions.

  2. Commenters, respond to these notes with specific advice, not on what needs to be improved (that’s already known), but on how to improve it.

  3. Edit the Paper.

The extent to which you should mark up the paper with line-level edits will vary, sometimes depending on the circumstances of your commentary, and sometimes on your determination of whether or not the paper is a serious paper. If someone has clearly given a good-faith effort on a paper, and obvious language errors still occur, it is incumbent on you the commentator to fix those errors. If it is not a serious paper, however, it is probably wiser to list the kinds of language errors that plague the paper and invite the writer to workshop the paper with yourself or someone else (fixing the errors in a non-serious paper could well be a waste of your time, and it is a significant time-commitment, if those edits are never going to be looked at). In sum, you are not the writer’s proof-reader or copy-editor. If the writer has trouble with writing standard academic English sentences, you might mark one section of the paper with suggestions, but it is their responsibility to get help on all the rest.

附录 4：斯坦福 CS324 大语言课程论文评审指南

链接：https://stanford-cs324.github.io/winter2022/paper-reviews/

Paper reviews

The goal of the panel reviews is to critically read and analyze the paper(s). Reviewing is a central practice in the academic community; our goal for the course is for you to have a better grasp on what constitues a strong review and how to write strong reviews for a variety of different papers. At the top level, reviews should address:

For this course, reviews should be a few paragraphs (in the style of a conference review, say for ACL or NeurIPS). To provide further guidance, please refer to the ACL Reviewer Tutorial and the NeurIPS Reviewer Guidelines.

In addition, while there are many ways to write a good review, we provide two examples:

https://openreview.net/forum?id=jznizqvr15J&noteId=ftk5QwCAJLi https://openreview.net/forum?id=de11dbHzAMF&noteId=ZLlS_WgGyHH

附录 5：斯坦福 CS324 大语言课程论文讨论组织

链接：https://stanford-cs324.github.io/winter2022/paper-discussions/

Paper discussions

The goal of the panel discussion is to analyze the paper from a variety of different vantage points. During each panel discussion, there is a panel of 4-5 students, each with an assigned role who will provide one particular perspective. Everyone in the class should participate by commenting and asking questions. The panel discussion will be moderated by one of the instructors.

Each panel role covers one aspect of critically assessing an academic paper. Many of the roles are taken from or inspired by Colin Raffel and Alec Jacobsen’s role-playing paper-reading seminars.