GitHub Copilot Evals explained: what it changes this month
What GitHub shipped and what changes for your team
GitHub put Copilot Evals into public preview on November 17, 2026, to run repeatable checks on the skills, custom agents, and MCP tools in your Copilot setup, according to GitHub Changelog. For a small B2B team that customizes Copilot, it changes how you verify an edit after you make it, but the launch post announces no price change and does not publish a per-run credit cost.
The category decision is this. If your question is whether a changed skill, agent, or MCP tool still behaves the way it did before, this launch is the native fit. If your question is whether a pull request should fail when a prompt regresses, the closer match is a CI-first tool such as Promptfoo, which we compare below.
GitHub defines the core concept in one sentence: an eval is "a repeatable test for AI behavior." You give an agent a realistic task, describe what a good result looks like, and check both the result and how the agent got there (GitHub Changelog). GitHub's own point is that one good answer does not show a customization works, because AI output varies from run to run.
The launch names Vally, which it describes as an extensible local eval tool, as the evaluation engine. The changelog says the preview covers Individual, Business, and Enterprise plans and does not list the Free plan. Each run waits for your approval before it starts, which gives you a control point on spend.
No product in this launch was renamed, merged, or discontinued. The change buyers must track is in billing, and it came earlier. GitHub's billing announcement says premium request units were replaced by GitHub AI Credits starting June 1, 2026, and that all Copilot plans moved to usage-based billing on that date (GitHub billing announcement). Any eval run is therefore priced in that new credit model, not in the old premium request count.
Know the base price before you plan anything. Pro plan on Copilot: $10 per user per month according to GitHub Copilot plans (2026).
Key Takeaways
GitHub Copilot Evals entered public preview on November 17, 2026, for Individual, Business, and Enterprise plans, and it covers skills, custom agents, and MCP tools.
The engine is Vally, which GitHub calls an extensible local eval tool. The changelog does not describe a hosted dashboard.
Cost is the open question. GitHub's Copilot Evals docs say task and grading requests "may use tokens/credits; costs are uncapped."
Copilot moved to usage-based billing on June 1, 2026, so eval runs are priced in credits, not in a flat seat fee alone.
For before-and-after checks on pull requests that edit prompts, Promptfoo's GitHub Action is the documented option. Blocking a merge would take your own workflow or branch-protection setup, and GitHub has not published a CI integration for Copilot Evals.
Act this month only if you already maintain custom skills, agents, or MCP tools. If you do not, the launch changes nothing in your routine.
TL;DR: Copilot Evals is a preview check for Copilot customizations, not a merge gate. Run it on one customization you already depend on, then decide after you see what a run costs in credits.
Who this is for and a short glossary
Good fit: a team that writes and edits its own Copilot skills, custom agents, or MCP tool setups, and needs to know after each change whether behavior held.
Not a fit: a team that uses Copilot mainly for autocomplete, or one that needs an automatic pull request block this quarter.
Red flags: (1) your Copilot use is mostly code completion, which this launch does not cover; (2) you need failed checks to block merges, and GitHub has not published a CI integration; (3) your budget cannot absorb uncapped credit use until you have seen a real per-run cost.
Industry context matters here too. Teams running offline evals on test sets: 52.4% according to LangChain's State of Agent Engineering (2026). LangChain sells agent-engineering tools, so treat the figure as vendor-run research, and as a sign that offline testing is common rather than universal.
Glossary of terms used below:
Eval: a repeatable test for AI behavior that checks the result and the path the agent took, per GitHub.
Skill: a packaged set of instructions in a Copilot project. Copilot Evals can target it.
Custom agent: a configured Copilot agent in your project. Copilot Evals can target it.
MCP tool: a tool exposed through the Model Context Protocol that Copilot can call. Copilot Evals can check its behavior.
AI Credits: GitHub's metered unit. GitHub's plans page says one credit equals $0.01.
Flex allotment: extra usage on top of a plan's base credits. GitHub says flex amounts will vary over time.
Public preview: a stage before general availability. GitHub asks for feedback through issues and discussions.
Regression: a result that gets worse after a change. Rerunning the same eval is how you catch one.
Decision checklist before you act:
List every skill, agent, and MCP tool your team edited in a typical month.
Confirm which plan each person uses, since the preview covers Individual, Business, and Enterprise.
Run one eval on your most-used customization and record the credits it consumed, if the run output reports them.
Decide whether a pull request check is required. If yes, evaluate Promptfoo's action.
Assign one person to review credit use each month.
How we evaluated these two options
We compared the two options a small B2B buyer is most likely to weigh: GitHub Copilot Evals, which is the launch, and Promptfoo, a dedicated eval tool with a documented GitHub Actions integration. We weighted six criteria by how much each one changes a buying decision this month. The weights are our judgment for a buyer, not vendor data, so change them if your priorities differ. This post does not score the tools by running them. It reads each vendor's own documentation and the independent sources cited in this article.
| Criterion | Weight | Why a small B2B buyer cares |
|---|---|---|
| Cost predictability | 25% | Uncapped credit use can grow with every run |
| Fit with current Copilot setup | 20% | Less switching if tests run where the team already works |
| Regression checks after edits | 20% | The core job: rerun the same check after each change |
| Pull request gating | 15% | Stops a bad prompt before it merges |
| Setup effort and prerequisites | 10% | Node.js 22.15.0 for Vally versus 22.22.0 for the Promptfoo action |
| Vendor maturity and evidence | 10% | Preview status and how much independent coverage exists |
Feature matrix: what each option does and does not publish
The matrix uses only what each vendor's pages state. Where a page is silent, the cell says so.
| Feature | GitHub Copilot Evals (preview) | Promptfoo (GitHub Action) |
|---|---|---|
| Launch status | Public preview from November 17, 2026 | Status not stated on the action page |
| Plans or access | Individual, Business, and Enterprise | Pricing tiers: Community, Enterprise, On-Premise |
| Where it runs | VS Code extension, Copilot CLI plugin, GitHub Copilot app | GitHub Actions workflow on pull requests |
| What it checks | Skills, custom agents, MCP tools | Edited prompts, before and after the change |
| Comparing runs | Yes, before and after a customization change | Yes, before versus after on edited prompts |
| Result reporting | Pass and fail per check, with reasons for failures | Pull request comment with a link to a web viewer |
| Merge blocking | Vendor has not published this | Not described on the page |
| Cost disclosure | Docs: task and grading requests may use credits, and costs are uncapped | Provider keys are your own; no per-run cost stated |
| Setup | Marketplace install in VS Code, CLI plugin, or the GitHub Copilot app | Config file, github-token input, Node.js on the runner |
| Feedback path | GitHub issue or GitHub Community discussion | Not stated on the pages checked |
The runtime and access lines come from the vendors' own setup pages. According to Vally runtime setup, the Vally CLI needs Node.js 22.15.0 or later, plus a GitHub account with an active Copilot subscription. According to Promptfoo GitHub Action docs, the action needs Node.js 22.22.0 or later on the runner and a required config file, and it posts results as a pull request comment.
The matrix shows a difference in shape more than in quality. Copilot Evals lives where your customizations already live, and it has no published merge gate. Promptfoo lives in your repository's pipeline, and it does not describe Copilot skills or agents.
Pricing and total cost of ownership
Pricing checked November 17, 2026. Copilot's prices below come from GitHub's plans page, and Promptfoo's come from its pricing page, which shows no numeric price for its paid tiers.
| Vendor | Plan | Monthly price | Included usage |
|---|---|---|---|
| GitHub Copilot | Free | $0 | 2,000 completions per month; limited chat and agent usage |
| GitHub Copilot | Pro | $10 per user per month | $15 total monthly credits |
| GitHub Copilot | Pro+ | $39 per user per month | $70 total monthly credits |
| GitHub Copilot | Max | $100 per user per month | $200 total monthly credits |
| GitHub Copilot | Business | Quote-based | Admins set usage limits |
| GitHub Copilot | Enterprise | Quote-based | Admins set usage limits |
| Promptfoo | Community | Free Forever | Red-teaming limit listed; no eval-run limit stated |
| Promptfoo | Enterprise | Quote-based | Custom limits; no numeric eval-run limit stated |
| Promptfoo | On-Premise | Quote-based | Dedicated runner; no numeric limits stated |
Those Copilot prices and credit totals are from GitHub's plans page, whose FAQ says one credit equals $0.01. The Pro plan's $15 splits into $10 of base credits and $5 of flex, per GitHub's individual plans post. GitHub says base credits are "matched 1:1 with your subscription price," while flex allotments "will vary over time." Business and Enterprise prices are quote-based on the plans page, so a company purchase will not match the individual figures above.
For a five-seat team, here is the arithmetic from those published prices and the $0.01 credit rate. This is not a quote.
| Plan | Subscription for 5 seats per month | Included credits for 5 seats per month | Credit units at $0.01 each |
|---|---|---|---|
| Pro | $50 | $75 | 7,500 |
| Pro+ | $195 | $350 | 35,000 |
| Max | $500 | $1,000 | 100,000 |
Take a five-person team that trials Copilot Evals on individual Pro+ plans. Five seats at $39 per user per month come to $195 in subscription fees, and each seat carries $70 in monthly credits, so the team holds $350 in credits, or 35,000 credits at $0.01 each. The arithmetic stops there, because GitHub's docs do not say how many credits one eval suite uses, and they describe the costs as uncapped. If the team runs Vally with a token rather than a reused CLI login, Vally's setup page says the fine-grained token needs the Copilot requests account permission and is exposed through COPILOT_GITHUB_TOKEN, so decide which account owns that token before the first run. If a run's output reports credit use, divide the monthly pool by that per-run figure to size the month.
For the cost side of a broader automation stack, our post on GPT-5.6 Sol alternatives covers how model choice changes what a run costs, and our guide to GPT-6 Astra alternatives when access is closed is a related read on planning around gated model access.
Vendor profiles: GitHub Copilot Evals and Promptfoo
GitHub Copilot Evals
Best fit: teams with many edited Copilot skills, custom agents, or MCP tools that already work in VS Code, the Copilot CLI, or the GitHub Copilot app. Choose it if your risk is a customization that quietly changes behavior after an edit.
Limitations: the product is in public preview, GitHub has not published a CI or pull request gating integration, and the docs say costs are uncapped. The launch does not publish a per-run price. The changelog does not list the Free plan, and the preview covers only skills, agents, and MCP tools.
Implementation: install the GitHub Copilot Evals extension from the Visual Studio Marketplace for VS Code, install the plugin for the Copilot CLI, or add the github/evals marketplace in the GitHub Copilot app's Customize panel. Then ask Copilot in plain words or use the /eval command to create, run, and compare evals. Approve each run before it starts.
Evidence: the GitHub changelog carries the launch facts. The Copilot Evals docs say that for task and grading requests, "costs are uncapped," in the approval prompt of a CLI example, and that the setup skill mentions choosing "CI checks" without describing how to set them up.
Disqualifier: not yet, if you need automatic merge blocking or a hard spending cap.
Promptfoo
Best fit: teams whose main risk is a prompt change in a repository that should be checked before merge. Choose it if before-and-after pull request checks are a requirement this quarter; blocking merges takes your own workflow or branch-protection setup.
Limitations: the action page describes a GitHub workflow, not checks inside Copilot's chat. Its example workflow watches a prompts/ folder, so your repository layout must match it or be edited. The action page does not state a license, and the paid tiers are quote-based.
Implementation: the action needs a config file, a github-token input, and Node.js on the runner. The example workflow requests write access so it can post comments. For OpenAI, set the OPENAI_API_KEY secret in the repository.
Evidence: the Promptfoo GitHub Action docs describe a before-and-after comparison of edited prompts on every pull request that modifies a prompt.
Disqualifier: not a fit if the eval must run inside Copilot's own chat or check Copilot skills directly.
Signal vs speculation: what is confirmed and what is our read
Confirmed by sources we opened:
The launch date, the plans in preview, the install paths, and the Vally engine come from GitHub's changelog.
GitHub's Copilot Evals docs say that task and grading requests "may use tokens/credits; costs are uncapped."
Premium request units were replaced by AI Credits starting June 1, 2026, per GitHub's billing announcement.
Developers who distrust AI output accuracy: 46% according to Stack Overflow's 2025 Developer Survey (2025), up from 31% the year before. The survey drew over 49,000 responses from 177 countries.
Teams naming quality as top barrier: 32% according to LangChain's State of Agent Engineering (2026). The survey had 1,340 respondents, with fieldwork from November 18 to December 2, 2025.
One developer reported that a single change to their project cost them more than $6 in credits, according to The Register (2026). That is one account, not a measured average.
Our read, not established by the sources:
Copilot Evals is worth its credit cost mainly for a team with many customizations. With only a few, the check adds spend with little payoff.
Because costs are uncapped, a finance owner should set the first budget ceiling, not an engineer's judgment alone.
The install paths in VS Code and the CLI suggest GitHub wants checks used in daily work. We have no evidence yet that a CI integration is coming, and the vendor has not announced one.
Business and Enterprise buyers will face different economics, and we cannot price that without a quote.
Changes to a small team's weekly routine
For the engineering side, each customization edit becomes a short routine: note which skill, agent, or MCP tool changed, rerun the same eval, read which checks failed, and decide whether the change ships. For the ticket side of that handoff, our guide on connecting Jira to GitHub automation is a starting point.
A proposed US Tech Automations workflow could take over the repetitive part. The trigger is a pull request that changes a file defining a skill, custom agent, or MCP tool in the team's repository. The action asks the named engineer to approve the eval run, because each run waits for approval before it starts, and then starts the approved suite. The output is a summary with pass and fail counts posted to the team's channel. The prerequisites are repository webhook access and a way to start an eval run from a script. GitHub's changelog does not describe a CLI or API entry point for that, so the team must confirm it before the design is real. The human review point is an engineer deciding whether a failed check blocks the merge.
The second proposed workflow sits on the cost side. The trigger is a finished eval run. The action reads any credit-usage field in the run output and appends it to a monthly sheet the finance owner reviews. Vally's 0.11.0 release notes say it reports Copilot AI Credit usage per stimulus, in a totalNanoAiu field (Vally 0.11.0 release notes). The same notes do not say that eval runs consume credits, so the field needs confirmation before anyone relies on it. The output is a monthly credit total per customization. The human review point is the finance owner setting the monthly ceiling. On Business and Enterprise accounts, admins set usage limits and decide whether additional paid usage is allowed, per GitHub's plans page, so that is the control point to design around.
The alternative most small teams weigh is stitching this together in Zapier, Make, or n8n, or scripting it in-house. Those tools can keep run histories, retries, error branches, and audit trails when someone configures them. The buyer still has to design and own observability, idempotency so a retried run does not post twice, escalation paths, access controls for the Copilot token, and maintenance when GitHub changes its interfaces. A proposed US Tech Automations design would configure the same steps with those fields named at the start: run history, retry and error branches, an escalation contact, and an audit record. Its prerequisites match a DIY build, and so do its review points.
When NOT to use US Tech Automations
US Tech Automations is the wrong layer in three cases. If you edit one skill a quarter, running the eval by hand in VS Code costs less than building a workflow around it. If your only need is a pull request check, Promptfoo's action is the documented route and needs no extra layer. If your Copilot use is mostly code completion, the launch covers skills, agents, and MCP tools, so there is little for an eval layer to check.
Frequently asked questions
Is GitHub Copilot Evals generally available?
No. GitHub announced it as in public preview on November 17, 2026, and the changelog does not describe a general availability date.
Does Copilot Evals cost extra?
The vendor has not published a separate price for Copilot Evals. Its docs say task and grading requests "may use tokens/credits; costs are uncapped," so the real cost depends on how many credits each run consumes.
Which plans can use it?
The changelog says the preview is available for Individual, Business, and Enterprise plans. The Free plan is not listed.
Does it run in CI or on pull requests?
The vendor has not published a CI or pull request integration for Copilot Evals. The docs mention "CI checks" only in the description of the setup skill.
Should a small team switch to it this month?
Only if it already maintains custom skills, agents, or MCP tools and can absorb uncapped credit use while it sees real per-run costs. Otherwise, wait for the cost details.
Can Promptfoo check Copilot skills?
The Promptfoo action page does not describe Copilot skills. It describes a before-and-after comparison of edited prompts in a GitHub workflow, so it answers a different question.
Conclusion
GitHub Copilot Evals gives a team a documented way to check whether an edit to a Copilot skill, custom agent, or MCP tool still behaves as intended. It launched in public preview on November 17, 2026, with no published merge gate and no per-run cost. Decide on two things: whether your team changes enough customizations to need the check, and whether uncapped credit use is acceptable before you have seen one run's cost. If you need before-and-after pull request checks today, Promptfoo's action is the documented route, and blocking merges takes your own workflow or branch-protection setup. For the workflow layer around either choice, see how US Tech Automations configures this.
About the Author

Helping businesses leverage automation for operational efficiency.