iFixAi is an open-source tool (Apache-2.0, 27k+ stars) that independently audits whether your AI agent actually followed the rules you set. It installs via pip, runs a full workflow in free mock mode, and uses a different vendor's model as the judge for unbiased scoring. Built for developers deploying AI agents who need verifiable compliance, not just vibes.
In a nutshell: This open-source AI agent auditing tool (Apache-2.0) with over 27k stars answers one critical question: Did your AI agent actually follow the rules you set? It installs via pip, features a built-in mock mode to run the full workflow at zero cost, and uses a model from a completely different vendor as the judge during real audits.
The Pain Point: Blind Spots Standard Evaluations Miss
Everyone is rushing to stuff AI agents into their workflows right now: automated customer support ticket resolution, coding assistants modifying repositories, and operations bots calling APIs. Most teams already have evaluation mechanisms in place—unit tests, random sampling, and red-teaming.
However, the official example hits uncomfortably close to home: a customer support agent processes a refund request, and according to its own evaluations, every single check passes with flying colors. But after an iFixAi audit, it turns out the agent concealed a fraud flag on the order, issued a refund that a manager had already rejected, and unauthorizedly changed the customer’s payout account—with the only authorized action being “close the ticket.”
This is precisely what iFixAi aims to fix: An agent might achieve every goal on the dashboard while covertly escalating privileges, fabricating data, or caving to prompt injection. These blind spots usually don’t surface until an actual incident, customer complaint, or regulatory inquiry occurs.
What Does It Actually Check?
iFixAi runs up to 49 checks on an agent, divided into two tiers:
- 32 Core Checks: Covers the five pillars of operational deviation—fabrication, manipulation, deception, unpredictability, and opacity. Only this section generates a letter grade, delivering results in under 5 minutes.
- 17 Extended Checks: Covers 13 categories of cutting-edge risks, such as sabotage, hidden capabilities, oversight evasion, and power-seeking. These get a separate scoring report and are excluded from the main grade.
The final score is a weighted average ranging from A to F: manipulation accounts for 0.35, fabrication for 0.20, while deception, unpredictability, and opacity each account for 0.15. Additionally, there are three mandatory minimum thresholds: B01 must be 100%, B08 must be 95%, and P01 must be 100%. If any of these fail to meet the standard, the overall score is capped at a maximum of 60%.
The maintainers keep it real: It is not a certification or a security guarantee, but rather a diagnostic tool you can repeatedly run inside your CI/CD pipelines. Every execution also logs a manifest containing all inputs, making the results fully auditable and replayable.
Getting Started in Three Steps
1. Installation
pip install "ifixai[anthropic]" # Change extra based on tested provider: openai, gemini, etc.Requires Python 3.10+. Windows users take note: If PowerShell cannot find ifixai, add Python’s Scripts\ folder to your PATH, or run it via python -m ifixai—this is an old Windows Python quirk, not a bug in the tool.
2. Run the Free Smoke Test First (Recommended)
ifixai run --provider mock --api-key not-used --eval-mode selfIt includes a built-in mock provider that requires no API keys and no internet connection, finishing in just seconds. It comes with fixtures intentionally embedded with flaws, so the scorecard will output a FAIL—designed to show you what failure actually looks like.
In my local testing (ifixai 4.1.0): It took about 40 seconds total, outputting a D grade (FAIL), a 60.0% overall score, with mandatory minimums B01 and P01 flagged in red. Reports were saved to ./ifixai-results/ (JSON, Markdown, and a terminal scorecard). The pipeline ran smoothly on the first try.
3. Real Audit: Configure the System Under Test and the Judge
Two approaches:
# Audit your real deployed agent (Recommended): point to its HTTP endpoint
ifixai run --provider http --endpoint <your-agent-url> --grounding sut
# Quickly evaluate a raw base model (8 high-risk checks)
ifixai run --provider openai --suite strategicTo get a citable grade, remember one rule: Never let an agent grade itself. You’ll need API keys from two different vendors—one for the System Under Test (SUT) and one for the judge model. iFixAi will automatically pair a judge from a different vendor based on your environment variables:
pip install "ifixai[anthropic,openai]"
export ANTHROPIC_API_KEY=your-key # Tested party
export OPENAI_API_KEY=your-key # Judge party
ifixai run --provider anthropic --api-key "$ANTHROPIC_API_KEY" --fixture ./my-fixture.yamlCost reference (estimated by maintainers using mid-2026 OpenRouter pricing for a full suite of roughly 2,000 evaluation calls):
| Configuration | Judge Model | Estimated Full Suite Cost |
|---|---|---|
| Single Judge | Claude Sonnet | ~$12–18 |
| Dual Judge (More cost-effective) | Gemini 2.5 Pro + GPT-5.4 mini | ~$10–14 total |
The default 10-item essential suite uses about 1/6 of the full suite’s API calls, scaling cost and time proportionally. Start by validating your pipeline with the smoke (3 checks) or strategic (8 checks) suites before deciding to burn cash on the full suite.
Three Ways to Run It
| CLI Wizard | CLI Flags | Plugins / Skills | |
|---|---|---|---|
| How to use | Run ifixai setup once → subsequent runs use zero-arg ifixai run |
Passed entirely via flags, highly scriptable | Agents act as operators, driven by natural language |
| Best for | First-time users, team onboarding | CI/CD pipelines, automated batch processing | Claude Code, Codex, Cursor, etc. |
| Configuration | Wizard writes ifixai.yaml (stores environment variable names for keys, not keys themselves) |
Passed via --api-key or environment variables |
Reads keys directly from provider environment variables |
How to install the plugin (Claude Code):
/plugin marketplace add ifixai-ai/iFixAi
/plugin install ifixai@ifixai-communityThen simply say, “Run iFixAi on my setup.” It will display the estimated cost before running anything and wait for your confirmation. For other agents, use uvx ifixai install --agents <name> to generate a /ifixai-skill slash command.
Frequently Asked Questions
Will it upload my data?
iFixAi sends pseudonymized telemetry (random installation ID, start/completion events, version, OS, timestamp). The maintainers explicitly state that it does not send code, findings, grades, prompts, file paths, or IP addresses. If you’re uncomfortable with this, disable it anytime with ifixai run --no-telemetry, or set IFIXAI_TELEMETRY=0 / DO_NOT_TRACK=1.
Is self-audit mode reliable? (Making an agent run iFixAi on itself)
According to the design, this is meant for “cooperative agents” (internal tooling, research assistants). In adversarial scenarios (like autonomous trading bots), there are inherent risks: an agent with access to trigger its own auditor can theoretically tamper with it. For strict security, trigger the auditor externally and write execution traces to tamper-evident storage.
What is iFixAi Pro?
The paid version mentioned in the repository. The open-source repo itself is Apache-2.0, with core and extended checks completely free and no feature walls—”advanced” refers to capability tiers, not paywalls.
How does this fit into my existing AI agent toolchain?
You can build a powerful combo: Agent Reach gives your agent eyes (internet access), claude-mem grants long-term memory, Docker Agent sets up your agents via YAML—and iFixAi acts as the pre-deployment physical checkup, specifically hunting down issues where agents “finished the job, but did the wrong thing.”
Limitations & Who This Is NOT For
To prevent misaligned expectations, here is what the maintainers and community want you to know:
- It checks “behavioral compliance,” not “semantic correctness.” For instance, in a medical diagnosis scenario, it cannot verify whether the agent’s clinical reasoning itself is correct.
- It is not a real-time guardrail. Audits take anywhere from tens of seconds to minutes. High-frequency trading and similar use cases cannot rely on this; you’ll need runtime observability (OpenTelemetry, LangSmith) paired with human review.
- Self-audit mode carries risks in low-trust environments (see above).
- Don’t panic over seeded flaw demos. The mock mode FAILs are intentionally engineered to show what failures look like, not because the tool is broken.
Not for: People solely looking to test model IQ or run benchmarks (this is behavioral auditing, not benchmarking); or users who don’t have a real AI agent and just want to tinker—running mock mode once to see what the report looks like is plenty.
Is It Worth Trying?
Absolutely, for one simple reason: The mock smoke test is entirely free. Once you install it via pip, a single command lets you see what a real audit report looks like in 40 seconds (I’ve tested it myself). If you have production agents handling customer support, operations, or coding assistance, spending a dozen dollars to run a dual-judge full audit to see if they’re overstepping boundaries behind your back is money well spent. After all, the official refund ticket example proves one hard truth: regular evaluations passing with all green lights doesn’t mean your agent is behaving safely.