cihr-project-grant-audit

name: cihr-project-grant-audit description: Audit non-RCT CIHR Project Grant applications (registry, cohort, prediction model, biobank, observational, AI/tech) for aim-hypothesis-endpoint-methods traceability, cross-references, abbreviations, garbled text, and terminology consistency. Use when auditing, reviewing, or QCing a non-RCT CIHR grant.

CIHR Project Grant Audit (Non-RCT)

Overview

Systematic audit of a CIHR Project Grant application for a non-RCT study (registry, cohort, prediction model, biobank, AI/technology, observational). Produces a structured checklist (markdown), applies tracked changes for fixable issues, and adds comments for items requiring investigator judgment.

Non-RCT CIHR grants typically organize around Specific Aims, each with its own rationale, hypothesis, endpoints, and analysis plan — rather than the numbered RCT section structure (1.1–3.3). This audit is designed for that architecture.

The audit has seven parts. Each part can surface issues that require either tracked changes (fixable problems) or comments (judgment calls).

Prerequisites

Before starting the audit:

1. Extract the .docx text using python-docx or direct XML parsing
2. Identify the document's section structure (headings, subheadings, Roman numerals, etc.)
3. Build a section index mapping heading text to paragraph positions
4. If a budget file is provided, extract its text as well
5. Note: .docx text extraction loses formatting (italics, superscripts, subscripts). Flag items requiring .docx-level verification (gene name italicization, superscript reference numbers, subscript terms)

Part A: Aim-Objective-Hypothesis-Endpoint-Methods Traceability Matrix

Every Specific Aim — and every sub-aim (1A, 1B, 1C, 2A, 2B, etc.) — must have a complete chain:

1. Aim statement — a high-level goal (e.g., "Aim 1: Arrhythmic risk prediction")
2. Rationale — why this aim matters, grounded in cited literature
3. Hypothesis — a testable prediction (must appear explicitly as "We hypothesize that...")
4. Endpoints/Outcomes — primary and secondary outcomes with exact definitions (measurement, timing, data source, adjudication plan)
5. Analysis plan — statistical method, model structure, covariates, validation approach, pre-specified subgroups
6. Sample size/feasibility — event counts, required N, recruitment evidence
7. Expected results — anticipated clinical impact and deliverables

Procedure

1. Extract each Specific Aim from the document (look for "Aim 1", "Aim 2", "Specific Aim", or equivalent heading patterns). Also identify sub-aims (1A, 1B, 1C, etc.) — each sub-aim is treated as a separate traceability chain.

2. For each aim and sub-aim, locate and verify:

   a. Hypothesis present? Search for explicit hypothesis statement. Flag if missing or vague. b. Endpoints defined? For each stated outcome:

   • Is the outcome precisely defined (not just named)?
   • Is the measurement method specified (instrument, data source, timing)?
   • Are primary vs. secondary vs. exploratory outcomes clearly distinguished?
   • Adjudication: For composite endpoints, is each component defined? Who adjudicates events? What criteria are used? Is adjudication blinded? If different aims have different endpoint types (e.g., arrhythmic vs. HF), verify that adjudicators have relevant expertise for each aim's endpoints. c. Analysis plan complete? For each endpoint:
   • Is the statistical method named (e.g., Cox regression, logistic regression, C-statistic)?
   • Are covariates/candidate predictors listed?
   • Is the validation strategy described (internal: bootstrapping/cross-validation; external: named cohort)?
   • For each named external validation cohort: does the text confirm which required predictor variables are available in that cohort?
   • Are pre-specified subgroups listed (sex, age, genotype, site)?
   • Is handling of missing data described (e.g., multiple imputation)?
   • Are competing risks addressed if relevant? For prediction models, Fine-Gray subdistribution hazards are appropriate. For etiologic/causal aims, cause-specific hazards may be more informative. Verify the chosen approach matches the aim's purpose.
   • For exploratory analyses (PCA, dimension reduction, "we will also explore..."): is there a pre-specified stopping rule, multiple testing strategy, and acknowledgment that results are hypothesis-generating? d. Sample size justified?
   • Is the events-per-variable rule applied (e.g., 10 EPV)?
   • Is the assumed event rate cited with source?
   • Does the recruitment projection support the required N?
   • Is a worst-case scenario considered?
   • For sub-aims: is sample size separately justified (not just inherited from the parent aim)?
   • For AI/deep learning sub-aims: is the training set size justified relative to model complexity? Is the number of events in the test set sufficient for reliable performance estimation (typically >50–100 events per TRIPOD-AI)? Flag when test sets have very few events.
   • For recruitment feasibility projections: verify that the reference population (registry, clinic volume) matches the proposed study population. Extrapolations from registries with different disease populations should be flagged. e. Literature-methods alignment? (see Part G for full procedure)
   • Does the cited literature actually support the chosen methods and candidate predictors?

3. Cross-aim consistency check: After auditing each aim individually, compare across aims:

   • Do gene lists, predictor lists, or variable definitions differ between aims? If so, is the rationale for the difference explicitly stated?
   • Is missing data handling described for every aim (not just the first)? A cross-reference such as "as per Aim 1" is acceptable only if the referenced approach fully applies. If the new aim has different variables, different cohort characteristics, or different missingness patterns, the strategy should be independently described.
   • Are endpoint definitions consistent where they should be (e.g., "sustained VA" in background vs. Aim 1)?
   • If aims share a cohort, is the derivation/validation split described for each aim?

4. Design-specific bias check: For ambispective/retrospective-prospective designs, verify:

   • Is time zero clearly defined?
   • Is immortal time bias addressed?
   • Is informative censoring discussed?
   • Are secular trends in treatment acknowledged?
   • For retrospective components: is ascertainment bias addressed?
   • For registry/cohort studies with treatment changes over follow-up: is time-varying treatment handled appropriately (time-dependent covariates, landmark analysis, or marginal structural models)? A single baseline measurement of medication may introduce bias.

5. Flag gaps:

   • GAP: Component entirely missing (e.g., no hypothesis for Aim 2, no adjudication plan)
   • PARTIAL: Component present but incomplete (e.g., analysis plan says "Cox regression" without specifying covariates, competing risks, or validation)
   • MISMATCH: Component present but inconsistent with another part of the chain (e.g., hypothesis mentions "outperform LVEF alone" but analysis plan doesn't include a comparison to LVEF-only model)
   • OK: Complete and internally consistent

Output Format

Common Issues

• Hypothesis is stated in the background/rationale but not repeated in the aim-specific methods section
• Exploratory analyses mentioned in passing but with no formal analysis plan
• Validation cohort referenced by name but without confirming it has the same variables
• Candidate predictors in the analysis plan don't fully match those in the rationale or pilot data
• Composite endpoints list components without defining each or specifying adjudication
• AI/ML aims describe model architecture but lack a plan for comparing to a simpler baseline model
• Event rate assumptions drawn from studies with different inclusion criteria
• Sub-aims inherit sample size from parent aim without separate justification
• Different gene lists across aims without explicit rationale for the difference
• Missing data strategy described in one aim but not carried through to subsequent aims

Part B: Cross-Reference Verification

Every internal reference ("Section X", "Aim 1A", "see above", "as described in...", "see support letter from...") must point to real content.

Procedure

1. Use regex to find all internal references. Patterns include:

   • Section [IVX]+ or Section \d+
   • Aim \d+[A-C]?
   • see (above|below|Section|Aim|Table|Figure)
   • as described (in|above|below|previously)
   • see (support |reference )?letter(s)? from
   • presented in Section
   • Parenthetical references like (Section II) or (see Aim 1A)
   • Role-based references: co-A [Name], co-PA [Name], NPA, the NPA
   • Empty parentheses () or ( ) — likely stripped URLs from .docx export

2. For each reference:

   • Note the source location (which section makes the reference)
   • Note the target (what is being referenced)
   • Verify the target exists and contains the claimed content

3. For every person named in a "support letter from Dr. X" reference, verify they appear in the Expertise section with a role description.

4. Flag issues:

   • OK: Referenced content exists and matches the claim
   • BROKEN: Target section/figure/table does not exist
   • STALE: Target exists but content doesn't match what's claimed
   • VAGUE: Reference is too imprecise to verify (e.g., "as mentioned above" without specifying where)
   • STRIPPED: Empty parentheses indicating a removed URL/hyperlink

Common Issues

• Support letters referenced by name but the letter author's role isn't described in the team section
• "See preliminary results" without specifying which subsection of pilot data
• Figure/Table references that don't match actual numbering
• Cross-references between aims that claim shared methods but the methods differ in detail
• "As described above" spanning multiple pages — reader cannot locate the referent
• Empty parentheses () where URLs were stripped during document conversion
• A person named as co-A for a specific task (e.g., event adjudication) who does not appear in the site PI list or team expertise section

Output Format

Part C: Non-RCT Grant Section Completeness

Non-RCT CIHR Project Grants do not follow the mandatory RCT heading structure (1.1–3.3). Instead, check against the expected content areas for a competitive application. Compare the document against the reference structure in references/cihr-non-rct-sections.md.

Procedure

1. Extract all section headings from the document
2. Map each heading to the expected content areas
3. Flag missing content areas (not just missing headings — the content may exist under a different heading)
4. Verify that each content area has substantive treatment (not just a sentence)

Expected Content Areas

Common Issues

• SGBA+ reduced to a single paragraph stating "sex will be included as a covariate" — should pervade aims
• Patient engagement section names a foundation but doesn't describe specific contributions to the study design
• Knowledge translation limited to "conferences and publications" without naming guideline bodies or clinical tools
• No explicit mention of EDI considerations beyond sex/gender
• Challenges section lists problems but mitigation strategies are vague or absent
• No timeline or milestones — CIHR expects year-by-year deliverables
• Ethics section limited to one sentence about REB approval at a single site; no discussion of multi-site harmonization
• Trainees mentioned in passing ("trainees are expected to contribute") but no structured training plan

Part D: Budget-Protocol Alignment

If a budget file is provided, verify bidirectional alignment between budget and protocol.

If no budget file is provided, list all protocol commitments that imply costs (personnel, equipment, biobanking, genotyping, imaging transfers, core labs, travel, data platforms) and flag them as UNVERIFIABLE. This serves as a checklist for when the budget becomes available.

Procedure

1. Extract budget line items with amounts and justifications
2. For each budget item, verify it maps to a protocol commitment:
   • Personnel: role described in the team/expertise section
   • Equipment: needed for procedures described in the protocol
   • Biobanking/genotyping: matches the sample processing described
   • Data linkages/transfers: required for the endpoints and data sources described
   • Core labs: match the imaging/analysis infrastructure described
   • Travel: justified by multi-site coordination needs
3. For each protocol commitment, verify budget support:
   • Number of sites: personnel and coordination budgets cover all sites?
   • Sample sizes: biobanking and genotyping budgets match target enrollment?
   • Imaging transfers: platform fees budgeted?
   • Training/student support: matches described trainee involvement?

Output Format

Part E: Content Issues (Garbled Text, Duplicates, Formatting)

Scan the full document text for garbled text, missing spaces, duplicate fragments, and formatting errors.

Procedure

1. Garbled text detection: Search for patterns indicating splice errors from tracked-change acceptance:

   • Period immediately followed by lowercase letter with no space: \.\w (excluding known abbreviations like "e.g.", "i.e.", "et al.", "vs.", decimal numbers, URLs)
   • Orphan fragments: short phrases (< 5 words) that don't connect grammatically to surrounding text
   • Possessive markers without antecedent: 's preceded by whitespace or punctuation instead of a noun
   • Sentence fragments ending abruptly mid-thought

2. Missing spaces: Search for:

   • Lowercase immediately followed by uppercase: [a-z][A-Z] (e.g., "patientsThe" should be "patients. The")
   • Digit immediately followed by letter in non-standard ways: \d[a-zA-Z] (excluding units like "3D", "p53", "12-lead", known abbreviations)
   • Period followed by uppercase with no space: \.[A-Z] (excluding abbreviations)
   • Reference number fused with following text: \d{1,3}[A-Z][a-z] (e.g., "15We" — superscript reference merged with next word)

3. Duplicate fragments: Search for:

   • Near-identical sentences or phrases within 500 characters of each other
   • Paragraphs that restate the same information in slightly different words (content duplication across sections)
   • Repeated references to the same fact in close proximity (e.g., enrollment count stated in both pilot data and methods)

4. Reference number issues:

   • Establish a baseline: if all references appear as inline numbers (e.g., "individuals.1"), this is a .docx text extraction artifact, not a document error. Note this once and do not flag each instance individually.
   • Flag genuinely inconsistent reference formatting (some superscripted, some inline)
   • Duplicate reference citations

5. Empty URL placeholders: Search for empty parentheses \(\s*\) — these typically indicate stripped hyperlinks from .docx export. Flag each and note the context (platform name, website, etc.).

6. Grammar errors: Beyond formatting issues, check for grammatical errors in critical sentences (hypothesis statements, method descriptions, endpoint definitions):

   • Subject-verb agreement errors
   • Missing words (e.g., "nor does there evidence" → missing "exist")
   • Incomplete sentences

7. Formatting consistency:

   • Section reference format: "Section II" vs "section II" vs "Sec II" — pick canonical and flag deviations
   • List formatting: mixing numbered lists (1-, 2-) with bullet points within the same section
   • Parenthetical style: "(see X)" vs "— see X" — flag inconsistencies
   • Hyphenation: "non-ischemic" vs "nonischemic" — flag inconsistencies

8. Paragraph numbering gaps: In .docx text extraction, non-contiguous paragraph numbers indicate removed elements (figures, tables, text boxes). Flag these as potential missing content and note that figures/tables should be reviewed in the original .docx.

Part F: Terminology and Abbreviation Consistency

Automatically discover and audit all abbreviations, key terms, cohort names, gene names, and recurring terminology in the document. This part is grant-agnostic — it builds registries dynamically from the document text rather than checking against a hardcoded list.

Step 1: Auto-Discover Abbreviations

Scan the entire document to build an abbreviation registry using these detection patterns:

1. Parenthetical definitions (most reliable): full term (ABBR) pattern

   • Regex: ([A-Za-z][\w\s\-/]+)\s*\(([A-Z][A-Za-z0-9\-+/]{1,15})\)
   • Captures: the full term + its abbreviation
   • Example: "non-ischemic cardiomyopathy (NICM)" → registers NICM, defined at this location

2. Undefined uppercase sequences: Find all tokens matching [A-Z]{2,}[a-z0-9+\-]* that were NOT captured by pattern 1

   • These are abbreviations used without a parenthetical definition in the document
   • Filter out: section headings in all-caps, Roman numerals (I, II, III, IV, V, VI), single-letter variables, reference numbers
   • Each needs investigation: is it a universally known abbreviation? Or does it need a definition?

3. Lowercase/mixed-case abbreviations: Catch patterns like eGFR, eCRF, co-A, co-PA, NT-proBNP

   • Regex: \b[a-z]+[A-Z][A-Za-z]*\b for camelCase
   • Regex: \b[a-z]+-[A-Z]+[a-z]*\b for hyphenated role abbreviations
   • Regex: \b[A-Z]{1,3}-[a-z]+[A-Z][A-Za-z]*\b for compound biomarker names

Step 2: Audit Each Discovered Abbreviation

For every abbreviation found in Step 1, check:

"Universally known" threshold: For CIHR grants, assume reviewers include at least one non-clinical methodologist and one patient partner. Abbreviations not universally known outside the specific clinical field should be defined at first use. Examples that likely need definition: disease-specific abbreviations (NICM, ARVC, HCM), imaging modalities (CMR, TTE, LGE), clinical scores (NYHA), biomarkers (NT-proBNP, eGFR). Examples that likely do NOT need definition: DNA, RNA, URL, PhD, MD.

CIHR-specific role abbreviations: NPA (Nominated Principal Applicant), co-PA (Co-Principal Applicant), co-A (Co-Applicant) are standard CIHR terms. They should still be defined at first use in the grant body, but their absence is a minor issue rather than a critical gap.

Step 3: Auto-Discover Key Terms and Check Consistency

Identify recurring domain-specific terms and check for naming drift:

1. Cohort/study names: Find all capitalized proper nouns or named entities that appear 3+ times. For each:

   • List all variant forms (e.g., "CaNICM", "CaNICM registry", "the CaNICM study")
   • Verify referent is always unambiguous
   • If multiple registries/cohorts exist, verify each reference clearly identifies which one

2. Endpoint/outcome terms: Extract all phrases near "outcome", "endpoint", "primary", "secondary", "composite", "event". For each unique endpoint:

   • Collect every variant phrasing across the document
   • Flag if the same outcome uses different wording in different sections
   • Flag if composite endpoint components differ between where they are listed

3. Statistical method terms: Extract all named statistical methods (search for "regression", "model", "analysis", "test", "statistic", "score"). For each:

   • Collect all variant forms
   • Identify the most complete form as canonical
   • Flag naming inconsistencies
   • Check proper noun spelling: For named methods (Harrell's C, Kaplan-Meier, Cox, Akaike, Fine-Gray, Bayesian), verify consistent and correct spelling across the document

4. Gene/protein names: Find gene-like tokens (2-6 uppercase letters, optionally followed by digits). For each:

   • Check if italicized (gene names should be italicized per HUGO/HGNC convention) — note: this check requires the original .docx; text extraction loses italics formatting. Flag for .docx-level verification.
   • Check if gene lists are consistent across sections (e.g., if "high-risk genes" is defined as a specific set, does the set stay the same across aims? If different, is the rationale for different lists explicitly stated?)
   • Check variant nomenclature consistency (e.g., "p.Arg14del" vs "R14del")

5. Person/institution names: Extract all named individuals and institutions. For each:

   • Collect all variant forms
   • Verify consistent role labeling ("co-A" vs "co-PA" vs "collaborator")
   • Verify all site PIs mentioned in methods also appear in the site list section
   • Verify all individuals referenced as "support letter from Dr. X" are listed in the team section
   • Verify all individuals assigned specific tasks (e.g., "event adjudication by co-A Roberts and Rivard") appear in the expertise section

Output Format

Part G: Literature-Methods Alignment

Verify that cited literature actually supports the methodological choices AND epidemiological claims made in the grant.

Procedure

1. For each aim's candidate predictors/variables:

   • Is each predictor justified by cited literature (meta-analysis, prior study, or own pilot data)?
   • Does the cited study actually support the claimed association, or is it tangential?
   • Are effect sizes from cited literature consistent with what's claimed in the text?

2. For each aim's statistical approach:

   • Is the chosen method appropriate for the data structure (time-to-event → survival analysis, binary → logistic regression)?
   • If a specific approach is cited as precedent (e.g., "following a similar approach as X"), does the cited study actually use that approach?
   • Are validation methods consistent with cited methodological standards?
   • Methodology currency: Is the chosen approach current best practice? For prediction models, check compliance with TRIPOD (Transparent Reporting of a Multivariable Prediction Model) guidelines and PROBAST (Prediction model Risk Of Bias ASsessment Tool). For AI/ML prediction models, check TRIPOD-AI (data preprocessing described, model interpretability addressed via saliency maps/SHAP, prospective validation plan). Flag traditional approaches that contemporary guidelines advise against (e.g., backward stepwise selection vs. penalized regression like LASSO/elastic net) unless explicitly justified.

3. For sample size justifications:

   • Are event rates cited from studies with comparable populations?
   • If multiple event rate estimates are cited, is the range honestly represented?
   • Does the chosen "conservative" estimate actually come from the most comparable study?

4. For pilot data claims:

   • Are statistics correctly reported (HR, CI, p-values match what's stated)?
   • Are pilot results from the applicant's own work, or repurposed from collaborators?
   • Is the pilot population comparable to the proposed study population?
   • Confidence interval width: Are CIs around pilot estimates narrow enough to support the claims being built on them? Flag wide CIs (e.g., AUC 0.87 [0.73–0.97] spans from "fair" to "near-perfect") and note the uncertainty.

5. For epidemiological/rationale claims: Check that prevalence figures, event rates, and burden-of-disease statistics in the background/rationale section are supported by citations. An unsupported prevalence figure is as problematic as an unsupported statistical choice.

6. Evidence quality check: For each cited reference supporting a key claim, note whether it is:

   • Peer-reviewed: published in a journal
   • Preprint: medRxiv, bioRxiv, SSRN, etc.
   • Unpublished: "unpublished data", "manuscript under review"
   • Flag claims where major methodological decisions rest on non-peer-reviewed evidence.

7. Flag issues:

   • SUPPORTED: Literature clearly supports the claim (peer-reviewed)
   • WEAK: Literature is tangential, from a substantially different population, or based on preprint/unpublished data
   • OVERCLAIMED: The text overstates what the cited literature shows
   • UNSUPPORTED: No citation provided for a key claim (methodological or epidemiological)
   • INCONSISTENT: Cited effect size doesn't match what's stated in the text

Output Format

Applying Fixes

After completing the audit:

Tracked Changes (for fixable issues)

Use the body-swap serialization approach for Word XML manipulation:

1. Parse document.xml with lxml
2. Modify the <w:body> element (add <w:del> and <w:ins> elements)
3. Serialize only the body: etree.tostring(body, encoding='unicode')
4. Replace the <w:body>...</w:body> region in the original XML string
5. This preserves namespace declarations that lxml would otherwise mangle

For text replacements that span multiple <w:r> elements:

• Collect all non-deleted runs, concatenate their text
• Find the match position in the concatenated string
• Map back to affected runs
• Remove affected runs, insert: before-text run + <w:del> + <w:ins> + after-text run

Use author "Claude (Audit)" and a fixed date for all changes.

Comments (for judgment calls)

For issues requiring investigator review:

1. Add <w:commentRangeStart> before the target paragraph's first run
2. Add <w:commentRangeEnd> and <w:commentReference> after the last run
3. Add the comment text to word/comments.xml
4. Add the author to word/people.xml (check namespace prefix — may be w15: not w:)
5. Escape any < or > characters in comment text

Output Checklist

Save the audit results as a markdown checklist file alongside the document. Structure:

• Part A: Traceability matrix (including cross-aim consistency)
• Part B: Cross-reference table
• Part C: Section completeness table
• Part D: Budget alignment table (if budget provided) or cost-implied items list
• Part E: Content issues list with fix status
• Part F: Terminology consistency table
• Part G: Literature-methods alignment table (including evidence quality)

Quick Reference: What Gets a Tracked Change vs a Comment

Quick Reference: Non-RCT vs RCT Audit Differences