Skip to content
AI & Automation

GitHub Copilot Evals explained: what it changes this month

Nov 18, 2026

What GitHub shipped and what changes for your team

GitHub put Copilot Evals into public preview on November 17, 2026, to run repeatable checks on the skills, custom agents, and MCP tools in your Copilot setup, according to GitHub Changelog. For a small B2B team that customizes Copilot, it changes how you verify an edit after you make it, but the launch post announces no price change and does not publish a per-run credit cost.

The category decision is this. If your question is whether a changed skill, agent, or MCP tool still behaves the way it did before, this launch is the native fit. If your question is whether a pull request should fail when a prompt regresses, the closer match is a CI-first tool such as Promptfoo, which we compare below.

GitHub defines the core concept in one sentence: an eval is "a repeatable test for AI behavior." You give an agent a realistic task, describe what a good result looks like, and check both the result and how the agent got there (GitHub Changelog). GitHub's own point is that one good answer does not show a customization works, because AI output varies from run to run.

The launch names Vally, which it describes as an extensible local eval tool, as the evaluation engine. The changelog says the preview covers Individual, Business, and Enterprise plans and does not list the Free plan. Each run waits for your approval before it starts, which gives you a control point on spend.

No product in this launch was renamed, merged, or discontinued. The change buyers must track is in billing, and it came earlier. GitHub's billing announcement says premium request units were replaced by GitHub AI Credits starting June 1, 2026, and that all Copilot plans moved to usage-based billing on that date (GitHub billing announcement). Any eval run is therefore priced in that new credit model, not in the old premium request count.

Know the base price before you plan anything. Pro plan on Copilot: $10 per user per month according to GitHub Copilot plans (2026).

Key Takeaways

  • GitHub Copilot Evals entered public preview on November 17, 2026, for Individual, Business, and Enterprise plans, and it covers skills, custom agents, and MCP tools.

  • The engine is Vally, which GitHub calls an extensible local eval tool. The changelog does not describe a hosted dashboard.

  • Cost is the open question. GitHub's Copilot Evals docs say task and grading requests "may use tokens/credits; costs are uncapped."

  • Copilot moved to usage-based billing on June 1, 2026, so eval runs are priced in credits, not in a flat seat fee alone.

  • For before-and-after checks on pull requests that edit prompts, Promptfoo's GitHub Action is the documented option. Blocking a merge would take your own workflow or branch-protection setup, and GitHub has not published a CI integration for Copilot Evals.

  • Act this month only if you already maintain custom skills, agents, or MCP tools. If you do not, the launch changes nothing in your routine.

TL;DR: Copilot Evals is a preview check for Copilot customizations, not a merge gate. Run it on one customization you already depend on, then decide after you see what a run costs in credits.

Who this is for and a short glossary

Good fit: a team that writes and edits its own Copilot skills, custom agents, or MCP tool setups, and needs to know after each change whether behavior held.

Not a fit: a team that uses Copilot mainly for autocomplete, or one that needs an automatic pull request block this quarter.

Red flags: (1) your Copilot use is mostly code completion, which this launch does not cover; (2) you need failed checks to block merges, and GitHub has not published a CI integration; (3) your budget cannot absorb uncapped credit use until you have seen a real per-run cost.

Industry context matters here too. Teams running offline evals on test sets: 52.4% according to LangChain's State of Agent Engineering (2026). LangChain sells agent-engineering tools, so treat the figure as vendor-run research, and as a sign that offline testing is common rather than universal.

Glossary of terms used below:

  • Eval: a repeatable test for AI behavior that checks the result and the path the agent took, per GitHub.

  • Skill: a packaged set of instructions in a Copilot project. Copilot Evals can target it.

  • Custom agent: a configured Copilot agent in your project. Copilot Evals can target it.

  • MCP tool: a tool exposed through the Model Context Protocol that Copilot can call. Copilot Evals can check its behavior.

  • AI Credits: GitHub's metered unit. GitHub's plans page says one credit equals $0.01.

  • Flex allotment: extra usage on top of a plan's base credits. GitHub says flex amounts will vary over time.

  • Public preview: a stage before general availability. GitHub asks for feedback through issues and discussions.

  • Regression: a result that gets worse after a change. Rerunning the same eval is how you catch one.

Decision checklist before you act:

  1. List every skill, agent, and MCP tool your team edited in a typical month.

  2. Confirm which plan each person uses, since the preview covers Individual, Business, and Enterprise.

  3. Run one eval on your most-used customization and record the credits it consumed, if the run output reports them.

  4. Decide whether a pull request check is required. If yes, evaluate Promptfoo's action.

  5. Assign one person to review credit use each month.

How we evaluated these two options

We compared the two options a small B2B buyer is most likely to weigh: GitHub Copilot Evals, which is the launch, and Promptfoo, a dedicated eval tool with a documented GitHub Actions integration. We weighted six criteria by how much each one changes a buying decision this month. The weights are our judgment for a buyer, not vendor data, so change them if your priorities differ. This post does not score the tools by running them. It reads each vendor's own documentation and the independent sources cited in this article.

CriterionWeightWhy a small B2B buyer cares
Cost predictability25%Uncapped credit use can grow with every run
Fit with current Copilot setup20%Less switching if tests run where the team already works
Regression checks after edits20%The core job: rerun the same check after each change
Pull request gating15%Stops a bad prompt before it merges
Setup effort and prerequisites10%Node.js 22.15.0 for Vally versus 22.22.0 for the Promptfoo action
Vendor maturity and evidence10%Preview status and how much independent coverage exists

Feature matrix: what each option does and does not publish

The matrix uses only what each vendor's pages state. Where a page is silent, the cell says so.

FeatureGitHub Copilot Evals (preview)Promptfoo (GitHub Action)
Launch statusPublic preview from November 17, 2026Status not stated on the action page
Plans or accessIndividual, Business, and EnterprisePricing tiers: Community, Enterprise, On-Premise
Where it runsVS Code extension, Copilot CLI plugin, GitHub Copilot appGitHub Actions workflow on pull requests
What it checksSkills, custom agents, MCP toolsEdited prompts, before and after the change
Comparing runsYes, before and after a customization changeYes, before versus after on edited prompts
Result reportingPass and fail per check, with reasons for failuresPull request comment with a link to a web viewer
Merge blockingVendor has not published thisNot described on the page
Cost disclosureDocs: task and grading requests may use credits, and costs are uncappedProvider keys are your own; no per-run cost stated
SetupMarketplace install in VS Code, CLI plugin, or the GitHub Copilot appConfig file, github-token input, Node.js on the runner
Feedback pathGitHub issue or GitHub Community discussionNot stated on the pages checked

The runtime and access lines come from the vendors' own setup pages. According to Vally runtime setup, the Vally CLI needs Node.js 22.15.0 or later, plus a GitHub account with an active Copilot subscription. According to Promptfoo GitHub Action docs, the action needs Node.js 22.22.0 or later on the runner and a required config file, and it posts results as a pull request comment.

The matrix shows a difference in shape more than in quality. Copilot Evals lives where your customizations already live, and it has no published merge gate. Promptfoo lives in your repository's pipeline, and it does not describe Copilot skills or agents.

Pricing and total cost of ownership

Pricing checked November 17, 2026. Copilot's prices below come from GitHub's plans page, and Promptfoo's come from its pricing page, which shows no numeric price for its paid tiers.

VendorPlanMonthly priceIncluded usage
GitHub CopilotFree$02,000 completions per month; limited chat and agent usage
GitHub CopilotPro$10 per user per month$15 total monthly credits
GitHub CopilotPro+$39 per user per month$70 total monthly credits
GitHub CopilotMax$100 per user per month$200 total monthly credits
GitHub CopilotBusinessQuote-basedAdmins set usage limits
GitHub CopilotEnterpriseQuote-basedAdmins set usage limits
PromptfooCommunityFree ForeverRed-teaming limit listed; no eval-run limit stated
PromptfooEnterpriseQuote-basedCustom limits; no numeric eval-run limit stated
PromptfooOn-PremiseQuote-basedDedicated runner; no numeric limits stated

Those Copilot prices and credit totals are from GitHub's plans page, whose FAQ says one credit equals $0.01. The Pro plan's $15 splits into $10 of base credits and $5 of flex, per GitHub's individual plans post. GitHub says base credits are "matched 1:1 with your subscription price," while flex allotments "will vary over time." Business and Enterprise prices are quote-based on the plans page, so a company purchase will not match the individual figures above.

For a five-seat team, here is the arithmetic from those published prices and the $0.01 credit rate. This is not a quote.

PlanSubscription for 5 seats per monthIncluded credits for 5 seats per monthCredit units at $0.01 each
Pro$50$757,500
Pro+$195$35035,000
Max$500$1,000100,000

Take a five-person team that trials Copilot Evals on individual Pro+ plans. Five seats at $39 per user per month come to $195 in subscription fees, and each seat carries $70 in monthly credits, so the team holds $350 in credits, or 35,000 credits at $0.01 each. The arithmetic stops there, because GitHub's docs do not say how many credits one eval suite uses, and they describe the costs as uncapped. If the team runs Vally with a token rather than a reused CLI login, Vally's setup page says the fine-grained token needs the Copilot requests account permission and is exposed through COPILOT_GITHUB_TOKEN, so decide which account owns that token before the first run. If a run's output reports credit use, divide the monthly pool by that per-run figure to size the month.

For the cost side of a broader automation stack, our post on GPT-5.6 Sol alternatives covers how model choice changes what a run costs, and our guide to GPT-6 Astra alternatives when access is closed is a related read on planning around gated model access.

Vendor profiles: GitHub Copilot Evals and Promptfoo

GitHub Copilot Evals

Best fit: teams with many edited Copilot skills, custom agents, or MCP tools that already work in VS Code, the Copilot CLI, or the GitHub Copilot app. Choose it if your risk is a customization that quietly changes behavior after an edit.

Limitations: the product is in public preview, GitHub has not published a CI or pull request gating integration, and the docs say costs are uncapped. The launch does not publish a per-run price. The changelog does not list the Free plan, and the preview covers only skills, agents, and MCP tools.

Implementation: install the GitHub Copilot Evals extension from the Visual Studio Marketplace for VS Code, install the plugin for the Copilot CLI, or add the github/evals marketplace in the GitHub Copilot app's Customize panel. Then ask Copilot in plain words or use the /eval command to create, run, and compare evals. Approve each run before it starts.

Evidence: the GitHub changelog carries the launch facts. The Copilot Evals docs say that for task and grading requests, "costs are uncapped," in the approval prompt of a CLI example, and that the setup skill mentions choosing "CI checks" without describing how to set them up.

Disqualifier: not yet, if you need automatic merge blocking or a hard spending cap.

Promptfoo

Best fit: teams whose main risk is a prompt change in a repository that should be checked before merge. Choose it if before-and-after pull request checks are a requirement this quarter; blocking merges takes your own workflow or branch-protection setup.

Limitations: the action page describes a GitHub workflow, not checks inside Copilot's chat. Its example workflow watches a prompts/ folder, so your repository layout must match it or be edited. The action page does not state a license, and the paid tiers are quote-based.

Implementation: the action needs a config file, a github-token input, and Node.js on the runner. The example workflow requests write access so it can post comments. For OpenAI, set the OPENAI_API_KEY secret in the repository.

Evidence: the Promptfoo GitHub Action docs describe a before-and-after comparison of edited prompts on every pull request that modifies a prompt.

Disqualifier: not a fit if the eval must run inside Copilot's own chat or check Copilot skills directly.

Signal vs speculation: what is confirmed and what is our read

Confirmed by sources we opened:

  • The launch date, the plans in preview, the install paths, and the Vally engine come from GitHub's changelog.

  • GitHub's Copilot Evals docs say that task and grading requests "may use tokens/credits; costs are uncapped."

  • Premium request units were replaced by AI Credits starting June 1, 2026, per GitHub's billing announcement.

  • Developers who distrust AI output accuracy: 46% according to Stack Overflow's 2025 Developer Survey (2025), up from 31% the year before. The survey drew over 49,000 responses from 177 countries.

  • Teams naming quality as top barrier: 32% according to LangChain's State of Agent Engineering (2026). The survey had 1,340 respondents, with fieldwork from November 18 to December 2, 2025.

  • One developer reported that a single change to their project cost them more than $6 in credits, according to The Register (2026). That is one account, not a measured average.

Our read, not established by the sources:

  • Copilot Evals is worth its credit cost mainly for a team with many customizations. With only a few, the check adds spend with little payoff.

  • Because costs are uncapped, a finance owner should set the first budget ceiling, not an engineer's judgment alone.

  • The install paths in VS Code and the CLI suggest GitHub wants checks used in daily work. We have no evidence yet that a CI integration is coming, and the vendor has not announced one.

  • Business and Enterprise buyers will face different economics, and we cannot price that without a quote.

Changes to a small team's weekly routine

For the engineering side, each customization edit becomes a short routine: note which skill, agent, or MCP tool changed, rerun the same eval, read which checks failed, and decide whether the change ships. For the ticket side of that handoff, our guide on connecting Jira to GitHub automation is a starting point.

A proposed US Tech Automations workflow could take over the repetitive part. The trigger is a pull request that changes a file defining a skill, custom agent, or MCP tool in the team's repository. The action asks the named engineer to approve the eval run, because each run waits for approval before it starts, and then starts the approved suite. The output is a summary with pass and fail counts posted to the team's channel. The prerequisites are repository webhook access and a way to start an eval run from a script. GitHub's changelog does not describe a CLI or API entry point for that, so the team must confirm it before the design is real. The human review point is an engineer deciding whether a failed check blocks the merge.

The second proposed workflow sits on the cost side. The trigger is a finished eval run. The action reads any credit-usage field in the run output and appends it to a monthly sheet the finance owner reviews. Vally's 0.11.0 release notes say it reports Copilot AI Credit usage per stimulus, in a totalNanoAiu field (Vally 0.11.0 release notes). The same notes do not say that eval runs consume credits, so the field needs confirmation before anyone relies on it. The output is a monthly credit total per customization. The human review point is the finance owner setting the monthly ceiling. On Business and Enterprise accounts, admins set usage limits and decide whether additional paid usage is allowed, per GitHub's plans page, so that is the control point to design around.

The alternative most small teams weigh is stitching this together in Zapier, Make, or n8n, or scripting it in-house. Those tools can keep run histories, retries, error branches, and audit trails when someone configures them. The buyer still has to design and own observability, idempotency so a retried run does not post twice, escalation paths, access controls for the Copilot token, and maintenance when GitHub changes its interfaces. A proposed US Tech Automations design would configure the same steps with those fields named at the start: run history, retry and error branches, an escalation contact, and an audit record. Its prerequisites match a DIY build, and so do its review points.

When NOT to use US Tech Automations

US Tech Automations is the wrong layer in three cases. If you edit one skill a quarter, running the eval by hand in VS Code costs less than building a workflow around it. If your only need is a pull request check, Promptfoo's action is the documented route and needs no extra layer. If your Copilot use is mostly code completion, the launch covers skills, agents, and MCP tools, so there is little for an eval layer to check.

Frequently asked questions

Is GitHub Copilot Evals generally available?

No. GitHub announced it as in public preview on November 17, 2026, and the changelog does not describe a general availability date.

Does Copilot Evals cost extra?

The vendor has not published a separate price for Copilot Evals. Its docs say task and grading requests "may use tokens/credits; costs are uncapped," so the real cost depends on how many credits each run consumes.

Which plans can use it?

The changelog says the preview is available for Individual, Business, and Enterprise plans. The Free plan is not listed.

Does it run in CI or on pull requests?

The vendor has not published a CI or pull request integration for Copilot Evals. The docs mention "CI checks" only in the description of the setup skill.

Should a small team switch to it this month?

Only if it already maintains custom skills, agents, or MCP tools and can absorb uncapped credit use while it sees real per-run costs. Otherwise, wait for the cost details.

Can Promptfoo check Copilot skills?

The Promptfoo action page does not describe Copilot skills. It describes a before-and-after comparison of edited prompts in a GitHub workflow, so it answers a different question.

Conclusion

GitHub Copilot Evals gives a team a documented way to check whether an edit to a Copilot skill, custom agent, or MCP tool still behaves as intended. It launched in public preview on November 17, 2026, with no published merge gate and no per-run cost. Decide on two things: whether your team changes enough customizations to need the check, and whether uncapped credit use is acceptable before you have seen one run's cost. If you need before-and-after pull request checks today, Promptfoo's action is the documented route, and blocking merges takes your own workflow or branch-protection setup. For the workflow layer around either choice, see how US Tech Automations configures this.

About the Author

Garrett Mullins
Garrett Mullins
Workflow Specialist

Helping businesses leverage automation for operational efficiency.