Copilot PR Prompt Analysis - 2025-12-24

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2025-12-24

Summary

Analysis Period: Last 30 days
Total PRs: 1000 | Merged: 741 (74.1%) | Closed: 255 (25.5%) | Open: 4
Success Rate (Merged/Completed): 74.4%

Key Findings

The analysis of 1000 Copilot-generated PRs reveals strong patterns in prompt effectiveness:

• Testing-focused prompts have the highest success rate at 76.0%
• Bug fix prompts perform well at 75.1% success rate
• Prompt length matters: Merged PRs average 475 words vs 433 words for closed PRs
• General/vague prompts ("other" category) show only 45.5% success rate

Prompt Categories and Success Rates

Category	Total	Merged	Closed	Success Rate
Testing	659	501	158	76.0%
Bug Fix	742	557	185	75.1%
Feature	798	598	200	74.9%
Documentation	551	398	153	72.2%
Refactor	190	137	53	72.1%
Other	22	10	12	45.5%

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 475 words
• Most common keywords: copilot, agent, github, coding, workflow, details, https
• Categories with highest success: Testing (76.0%), Bug Fix (75.1%), Feature (74.9%)

Key success factors identified:

1. Specificity: Prompts that reference specific files, functions, or error messages
2. Testing focus: Including test coverage or test cases increases success
3. Clear action verbs: "Fix", "Add test for", "Implement" are more successful than vague terms

Example successful prompts:

1. PR Export main function instead of auto-executing in GitHub Actions scripts #7465: Export main function instead of auto-executing in GitHub Actions scripts → Merged

   • Preview: Several JavaScript files used in GitHub Actions workflows were auto-executing main() on require, causing execution before workflows were ready. Work...

2. PR Fix check_workflow_timestamp_api.cjs missing module.exports #7462: Fix check_workflow_timestamp_api.cjs missing module.exports → Merged

   • Preview: The check_workflow_timestamp_api.cjs script was invoked with const { main } = require(...) in compiled workflows but lacked a module.exports sta...

3. PR Update JavaScript developer instructions to clarify export pattern for top-level .cjs files #7461: Update JavaScript developer instructions to clarify export pattern for top-level → Merged

   • Preview: The JavaScript refactoring skill documentation incorrectly showed top-level .cjs scripts calling await main() directly. The bundler automatically ...

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 433 words (slightly shorter than successful prompts)
• Most common keywords: copilot, agent, coding, github, workflow, details, start
• Lower success categories: General/Other (45.5%), Refactor (72.1%)

Common issues identified:

1. Lack of specificity: General improvement requests without clear scope
2. Missing context: Not explaining why a change is needed
3. Overly broad scope: Trying to address too many things at once

Example unsuccessful prompts:

1. PR [WIP] Fix JavaScript to support require main and run #7475: [WIP] Fix JavaScript to support require main and run → Closed
   • Preview: _## Fix JavaScript to support require main and run

Plan

• Understand the issue: Scripts need to support both `const { main } = require('./scr..._

2. PR Add validation test for JavaScript main exports in lock files #7463: Add validation test for JavaScript main exports in lock files → Closed
   • Preview: _Validates that JavaScript scripts compiled into lock files export main functions and are properly packaged in custom actions.

Changes

• **Added..._

3. PR Add discussions permission to GitHub App token for safe-outputs #7455: Revert incorrect discussions permission fix - not supported by actions/create-gi → Closed
   • Preview: After reviewing the source code of actions/create-github-app-token, the original fix was incorrect. The action does NOT support `permission-disc...

Key Insights

• Testing-oriented prompts perform best: Including test coverage or test cases in the prompt increases merge probability by ~3%
• Bug fixes are highly successful: When prompts clearly identify a bug with reproduction steps or error messages, success rate reaches 75%
• Specificity matters more than length: While merged PRs are slightly longer on average (~42 words), the key differentiator is concrete details rather than verbosity
• Avoid vague improvement requests: General refactoring or improvement prompts without specific goals have the lowest success rate

Recommendations

Based on today's analysis of 1000 PRs:

1. DO: Include specific file paths, function names, or error messages in your prompt
2. DO: Add testing requirements or test cases to your prompt - this category shows the highest success rate
3. DO: For bug fixes, include reproduction steps or error messages
4. DO: Use clear action verbs: "Fix X in file.go", "Add test for Y", "Implement Z feature"
5. AVOID: Vague requests like "improve the code" or "make it better"
6. AVOID: Overly broad scope - break large changes into focused, specific prompts

Prompt Template Suggestions

High-Success Bug Fix Template

Fix [specific error/bug] in [file/function]

Error: [error message or description]
Reproduction: [steps to reproduce]
Expected: [expected behavior]

High-Success Feature Template

Add [specific feature] to [component/file]

Requirements:
- [requirement 1]
- [requirement 2]

Include tests for:
- [test case 1]
- [test case 2]

High-Success Test Template

Add tests for [specific function/feature] in [file]

Coverage needed:
- [test case 1]
- [test case 2]
- [edge case]

Statistical Summary

• Sample size: 1000 PRs (741 merged, 255 closed, 4 open)
• Overall success rate: 74.4%
• Analysis confidence: High (large sample size)
• Most successful category: Testing (76.0%)
• Least successful category: Other/General (45.5%)

---

Data Source: Copilot-generated PRs from githubnext/gh-aw repository (last 30 days)
Analysis Method: Keyword extraction, categorization, and statistical correlation
Next Analysis: Scheduled for tomorrow

References:

• §20485670847

AI generated by Copilot PR Prompt Pattern Analysis

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 3 days ago.