academic-reviewer

name: academic-reviewer description: "Review systems conference papers for OSDI, NSDI, SIGCOMM, MOBICOM, SOSP, and FAST. Produces structured HotCRP-style reviews with paper summary, strengths, weaknesses, detailed comments, questions for authors, and overall merit/confidence scoring. Covers evaluation criteria (novelty, soundness, significance, evaluation quality, clarity, relevance, reproducibility), venue-specific expectations, common paper weaknesses, constructive feedback tone, one-shot revision reviews, and reviewer ethics. Use when asked to review, critique, or provide feedback on a systems conference paper submission."

Systems Conference Paper Reviewing

Core Capabilities

1. Review Structure and Format

All reviews follow the HotCRP form structure used by OSDI, NSDI, SIGCOMM, MOBICOM, SOSP, and FAST:

For detailed field-by-field guidance and complete example reviews, load references/review_form_templates.md.

2. Evaluation Dimensions

Evaluate every paper along seven dimensions. Weight them according to venue-specific priorities (see Section 3).

Evaluation quality is the most common area of weakness in systems paper submissions. Check it rigorously: map each numbered contribution to specific experiments and flag any unsupported claims.

For detailed rubrics (5-point scale per dimension), venue-specific weights, and specific checks, load references/review_criteria.md.

3. Venue-Specific Review Expectations

Each venue has a distinct review culture. Adjust your emphasis accordingly.

Key venue-specific notes:

• OSDI: Would you enthusiastically champion this paper? "Exciting and imperfect" can beat "boring and thorough."
• NSDI: Could this system actually be deployed? Production experience and operational lessons get extra credit. One-shot revision (OSR) available for borderline papers.
• SIGCOMM: Scores of 3+ generally indicate acceptability. Papers need a strong advocate. The networking contribution must be clear and substantial.
• MOBICOM: Real-world wireless experiments strongly preferred over simulation. Multiple environments and conditions required. Signal-level analysis expected.
• SOSP: Must present new abstractions or principles, not just "faster X." Artifact evaluation is mandatory. The system must be real and substantial.
• FAST: Deep storage/file systems expertise valued. Evaluation must use real storage workloads or production traces. Deployed-systems papers get credit for operational lessons. Short papers (6 pages) evaluated for completeness within their scope. Double-blind review. One-shot revision (OSR) available for borderline papers.

For full venue-specific rubrics, load references/review_criteria.md.

4. Review Writing Principles

Follow these seven principles when writing reviews:

1. Be specific: Reference exact sections, figures, equations, and claims. Never write "the evaluation is weak" without saying what is missing.
2. Explain impact: For each weakness, state why it matters: "This undermines the main throughput claim because..."
3. Suggest remedies: Where possible, indicate what would fix the problem: "Comparing against [baseline] on [workload] would address this."
4. Prioritize by severity: Label weaknesses as Fatal / Major / Minor. Fatal = fundamental flaw; Major = significant but potentially fixable; Minor = does not affect the contribution.
5. Stay constructive: Critique the work, not the authors. Frame negatives as paths to improvement. Ask: "Would my review help the authors write a better version?"
6. Match tone to score: Enthusiastic language for score 4-5, balanced for 3, respectful-but-clear for 1-2. Mismatched tone confuses everyone.
7. Be honest about confidence: Set reviewer confidence to your actual expertise level. Flag areas where a more expert reviewer should weigh in.

For detailed tone calibration, constructive rephrasing examples, and the Roscoe/Levin-Redell evaluation frameworks, load references/review_ethics_and_tone.md.

5. Common Paper Weaknesses

Weaknesses are classified by paper section with standardized IDs for consistent reference:

Severity classification:

• Fatal: Fundamental flaw that cannot be fixed without redesigning the system or rethinking the approach. Examples: core mechanism is incorrect; problem is already solved; evaluation contradicts claims.
• Major: Significant issue that affects the assessment but could be addressed with additional work. Examples: missing important baseline; hidden assumption; limited scale.
• Minor: Does not affect the core contribution. Examples: typos; figure formatting; missing citation.

For the complete weakness catalog with constructive phrasing templates, load references/common_paper_weaknesses.md.

6. Scoring Guidelines

Overall Merit (1-5):

Reviewer Confidence (1-4):

Calibration reminders:

• At ~15-20% acceptance, score 3 means "uncertain" — not "acceptable"
• Most accepted papers need at least one score of 4 or 5
• Use the full range: a clearly flawed paper should get 1 or 2, not a generous 3
• ~15-20% of papers you review should receive 4 or higher

For full scoring rubrics and example reviews at each score level, load references/review_form_templates.md.

7. One-Shot Revision (OSR) Reviews

Available at NSDI, SIGCOMM, MOBICOM, and FAST. OSR means a conditional accept with specific required changes.

When to recommend OSR:

• Core contribution is sound and significant
• Weaknesses are fixable within 4-6 weeks
• Required changes are concrete and verifiable
• The paper would likely be accepted if changes are made

When NOT to recommend OSR:

• Fundamental approach is flawed
• Novelty is questionable (cannot be added in revision)
• Changes require new system design or major re-implementation

OSR requirements must be:

• Specific: "Add experiment comparing against X on workload Y"
• Achievable: Feasible within the revision period
• Verifiable: The PC can check whether each requirement is met

For OSR templates and venue-specific guidelines, load references/review_form_templates.md.

8. Review Anti-Patterns

Avoid these six common reviewer behaviors:

For detailed descriptions and examples, load references/review_ethics_and_tone.md.

9. Multi-Perspective Review Generation

When asked to review a paper from multiple perspectives (or to simulate a PC sub-committee), generate four independent reviews — one at each confidence level. This surfaces issues that any single reviewer would miss.

Output structure:

Key behavioral differences:

• Expert: Cites specific prior work, uses full scoring range, assesses all 7 dimensions with authority
• Knowledgeable: Hedges on niche novelty claims, provides balanced coverage, flags areas where expertise is limited
• Some familiarity: Defers on novelty and baseline appropriateness, excels at catching clarity and methodology issues
• Limited knowledge: Constrains scope to presentation and methodology, explicitly defers on domain-specific dimensions

Each review must:

• Use the full HotCRP form (summary, strengths, weaknesses, detailed comments, questions, merit, confidence)
• Set the overall merit score independently based on that reviewer's perspective and assessable dimensions
• State what the reviewer can and cannot assess at their confidence level
• Use hedging language appropriate to the confidence level (see reference below)
• Be an independent perspective — NOT a degraded copy of the expert review

For detailed per-level behavioral profiles, dimension weight shifts, scoring behavior, worked examples, and anti-patterns, load references/confidence_level_profiles.md.

Review Workflow

Single vs. Multi-Perspective Mode: The workflow below produces one review. In multi-perspective mode (Section 9), execute Stages 1-4 four times — once per confidence level (4, 3, 2, 1). The Expert review is generated first; each subsequent review is a fresh perspective, not a degradation of the expert review.

Follow this four-stage workflow to produce a thorough, calibrated review.

Stage 1: First Read (understand the claims)

1. Read the title, abstract, and introduction. Identify:
   • The stated problem and why it matters
   • The key insight or approach
   • The numbered contributions (if any)
   • The target venue and its expectations
2. Skim the design and evaluation sections to understand the system architecture and experimental approach.
3. Read the conclusion to see if the narrative is coherent from introduction to conclusion.
4. Write the Paper Summary immediately after the first read — this forces you to articulate your understanding before diving into critique.

Stage 2: Detailed Analysis (evaluate seven dimensions)

1. Re-read the paper carefully, section by section.
2. For each of the seven evaluation dimensions, make specific notes:
   • Novelty: What is the key insight? Is it genuinely new? Cite specific prior work if not.
   • Soundness: Are the design choices internally consistent? Check assumptions and failure modes.
   • Significance: If everything works as claimed, how much does it matter?
   • Evaluation: Map each contribution to experiments. Check baselines, workloads, metrics, statistical rigor, ablation.
   • Presentation: Note specific clarity issues, figure problems, or organizational weaknesses.
   • Relevance: Is this the right venue?
   • Reproducibility: Could you reproduce the results from what is described?
3. Cross-check: do the evaluation results actually support the claims in the introduction?
4. Identify the most common weakness type using the taxonomy in references/common_paper_weaknesses.md.

Stage 3: Write the Review

Write the review fields in this order:

1. Paper Summary: Neutral, factual, in your own words. 3-5 sentences.
2. Strengths: 3-6 specific positive observations, most important first. Reference sections and figures.
3. Weaknesses: 3-6 specific concerns with severity labels (Fatal/Major/Minor), most severe first. Each must include: what the problem is, why it matters, and how it could be addressed.
4. Detailed Comments: Elaborate on key points. Provide technical feedback that helps regardless of accept/reject.
5. Questions for Authors: 3-5 numbered questions whose answers could change your score. Flag which are critical.
6. Overall Merit: Score 1-5 based on the dimension assessments, calibrated to venue acceptance rate.
7. Reviewer Confidence: Honest self-assessment of your expertise.
8. Minor Issues: Typos, formatting, editorial items (optional).

Stage 4: Calibration (self-check before submission)

Run through the 8-item self-assessment checklist:

1. Does my summary demonstrate I understood the paper?
2. Does every strength and weakness reference specific sections or claims?
3. Would my review help the authors improve, even if rejected?
4. Does the tone match the score?
5. Have I evaluated the paper for what it is, not what I wish it were?
6. Have I addressed all seven dimensions?
7. Is my summary neutral and factual?
8. Does my confidence accurately reflect my expertise?

If any answer is "no," revise before submitting.

References

This skill includes comprehensive reference files covering specific aspects of systems paper reviewing:

• references/review_criteria.md: Detailed rubrics for all 7 evaluation dimensions, venue-specific review priorities and weights, and scoring calibration guidelines
• references/review_form_templates.md: HotCRP field-by-field guide, 3 complete example reviews (score 5, 3, and 2), and one-shot revision templates
• references/review_ethics_and_tone.md: Roscoe and Levin-Redell frameworks, constructive vs. destructive critique examples, tone calibration by score, 6 anti-patterns, self-assessment checklist, and edge case handling
• references/common_paper_weaknesses.md: Catalog of ~27 common weaknesses by section (motivation, design, evaluation, presentation, scope), severity classification, and constructive phrasing templates
• references/confidence_level_profiles.md: Per-confidence-level behavioral profiles, dimension weight shifts, scoring behavior, worked 4-review example, and multi-perspective anti-patterns

Load these references as needed when working on specific aspects of paper reviewing.