MCP Observatory
4m ago Tracked 59,479 Last 4m ago 24h 2,575
github Python analyzed 9ff1b96

UiPath/coder_eval

github

Playwright for coding agents. Test that your skills, MCP servers, and CLIs actually work when an agent uses them — sandboxed YAML suites, A/B experiments, CI gates.

maintainer
UiPath
licence
Apache-2.0
first seen
2026-09-14
last seen
2026-09-17
releases · 30d
11
short id

Drift inferred · capture-to-capture

No drift recorded — single capability capture; advisories appear once its surface changes.

capabilities 0 tools
transport stdio counts 0 tools · 0 res · 0 prompts permission surface via code analysis

No tools enumerated yet for this server.

skills & danger signals github-tarball
prompt-surface shipped agent-instruction files + hidden-content / dangerous-code findings — quoted from the analyzed source

analyzed commit 9ff1b96 · analyzer v33 · 1h ago

skills & prompt files 14

danger signals3

code evidence vv0.12.2 · github-tarball
evidence-backed findings quoted directly from the published source artifact — not inferred

code files: 628

filesystem 200

  • fs (weak) UiPath-coder_eval-9ff1b96/.github/scripts/harbor_e2e.py :21 import shutil
  • fs (weak) UiPath-coder_eval-9ff1b96/.github/scripts/release_notes.py :28 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/litellm/cost_logger.py :187 with open(path, "a", encoding="utf-8") as fh:
  • fs (weak) UiPath-coder_eval-9ff1b96/scripts/timing/decompose_run.py :34 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/agents/_skills.py :14 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/agents/antigravity_agent.py :27 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/agents/claude_code_agent.py :12 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/agents/codex_agent.py :9 import shutil
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/agents/opencode_agent.py :25 import shutil
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/agents/pi_agent.py :32 import shutil
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/cli/evaluate_command.py :8 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/cli/evaluate_target.py :22 from pathlib import Path
show 28 more
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/cli/execute_command.py :20 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/cli/export_command.py :21 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/cli/harbor_command.py :12 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/cli/plan_command.py :4 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/cli/report_command.py :3 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/cli/run_command.py :6 import shutil
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/cli/run_helpers.py :4 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/cli/run_task_internal_command.py :20 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/cli/utils.py :3 import shutil
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/config.py :8 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/criteria/base.py :19 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/evaluation/checker.py :11 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/evaluation/judge_context.py :18 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/evaluation/judge_persistence.py :22 from pathlib import PurePosixPath, PureWindowsPath
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/evaluation/sub_agent.py :13 import shutil
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/fs_permissions.py :40 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/harbor/atif_emit.py :20 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/harbor/experiment_packager.py :22 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/harbor/packager.py :24 import shutil
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/harbor/reward.py :20 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/invocation_log.py :112 with open(LOG_PATH, "a", encoding="utf-8", newline="\\n") as handle:
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/isolation/docker_runner.py :17 import shutil
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/litellm_cost.py :24 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/logging_config.py :21 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/models/criteria.py :11 from pathlib import PurePosixPath
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/models/enums.py :135 MOVE_ON_WRITE = "MOVE_ON_WRITE" # Run in a tempdir, shutil.move into run_dir/artifacts at the end.
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/models/experiment.py :6 from pathlib import Path
  • fs UiPath-coder_eval-9ff1b96/src/coder_eval/orchestration/batch.py :14 import shutil

160 more not shown — this panel samples each group; the count above is the real total.

shell / exec 29

  • shell (weak) UiPath-coder_eval-9ff1b96/.github/scripts/harbor_e2e.py :22 import subprocess
  • shell UiPath-coder_eval-9ff1b96/src/coder_eval/isolation/docker_runner.py :18 import subprocess
  • shell UiPath-coder_eval-9ff1b96/src/coder_eval/models/sandbox.py :22 ``subprocess.run(timeout=...)``); other fields are accepted but not
  • shell UiPath-coder_eval-9ff1b96/src/coder_eval/sandbox.py :9 import subprocess
  • shell UiPath-coder_eval-9ff1b96/src/coder_eval/utils.py :21 import subprocess
  • shell UiPath-coder_eval-9ff1b96/tasks/samples/skillsbench/court-form-filling/verifier/test_outputs.py :13 import subprocess
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/lint/prose_budget.py :20 import subprocess
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/lint/rules/no_blocking_io_in_async.py :3 Calling shutil.rmtree(), open(), subprocess.run(), etc. directly from an
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/lint/rules/subprocess_run_explicit_encoding.py :1 """CE010: ``subprocess.run(..., text=True, ...)`` must pass ``encoding=``.
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/test_action_inputs.py :22 import subprocess
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/test_codex_agent.py :498 import subprocess # noqa: E402
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/test_custom_lint.py :15 import subprocess
show 17 more
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/test_docker_runner_mounts.py :676 import subprocess
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/test_docker_workdir_live.py :14 import subprocess
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/test_git_clone_failure.py :6 import subprocess
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/test_image_from_dockerfiles.py :19 import subprocess
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/test_image_skew_refusal.py :6 import subprocess
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/test_lint_runner.py :321 source = "import subprocess\n\ndef f():\n subprocess.run(['ls'], text=True, capture_output=True)\n"
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/test_litellm_config.py :22 import subprocess
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/test_logging.py :524 sandbox.run_command("echo 'Test output from subprocess'")
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/test_optional_dependencies.py :12 import subprocess
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/test_prose_budget.py :10 import subprocess
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/test_reports.py :1532 import subprocess
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/test_reports_package.py :8 import subprocess
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/test_sandbox_record_cli.py :11 import subprocess
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/test_sandbox_venv_live.py :16 import subprocess
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/test_tags.py :4 import subprocess
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/test_utils.py :4 import subprocess
  • shell (weak) UiPath-coder_eval-9ff1b96/tests/test_verify_published_workflow.py :23 import subprocess

network 10

  • net UiPath-coder_eval-9ff1b96/src/coder_eval/agents/codex_agent.py :16 from urllib.parse import urlparse
  • net UiPath-coder_eval-9ff1b96/src/coder_eval/cli/run_command.py :8 import urllib.error
  • net UiPath-coder_eval-9ff1b96/src/coder_eval/config.py :10 from urllib.parse import urlsplit
  • net UiPath-coder_eval-9ff1b96/src/coder_eval/isolation/docker_runner.py :91 from urllib.parse import urlsplit, urlunsplit
  • net UiPath-coder_eval-9ff1b96/src/coder_eval/logging_config.py :109 Non-coder_eval loggers (e.g. ``aiohttp.*``) pass through unchanged.
  • net UiPath-coder_eval-9ff1b96/src/coder_eval/orchestrator.py :17 from urllib.parse import urlparse
  • net (weak) UiPath-coder_eval-9ff1b96/tests/test_litellm_route.py :10 import urllib.error
  • net (weak) UiPath-coder_eval-9ff1b96/tests/test_logging.py :298 record = _make_record(name="aiohttp.access", msg="GET /")
  • net (weak) UiPath-coder_eval-9ff1b96/tests/test_sandbox.py :497 exit_code, stdout, stderr = sandbox.run_command('python -c "import requests; print(requests.__version__)"')
  • net (weak) UiPath-coder_eval-9ff1b96/tests/test_sandbox_adopt.py :112 exit_code, stdout, stderr = sandbox.run_command('python -c "import requests; print(requests.__version__)"')

secrets 7

  • secrets UiPath-coder_eval-9ff1b96/src/coder_eval/agents/antigravity_agent.py :339 api_key = os.getenv("GEMINI_API_KEY") or None
  • secrets UiPath-coder_eval-9ff1b96/src/coder_eval/agents/codex_agent.py :841 api_key = os.getenv("CODEX_API_KEY")
  • secrets (weak) UiPath-coder_eval-9ff1b96/tests/test_agent_judge_criterion.py :1113 api_key = os.environ.get("ANTHROPIC_API_KEY", "")
  • secrets (weak) UiPath-coder_eval-9ff1b96/tests/test_byoa_plugin_live.py :32 return bool(os.getenv("ANTHROPIC_API_KEY"))
  • secrets (weak) UiPath-coder_eval-9ff1b96/tests/test_codex_agent_live.py :32 return bool(os.getenv("CODEX_API_KEY"))
  • secrets (weak) UiPath-coder_eval-9ff1b96/tests/test_judge_burn_in_live.py :75 api_key = os.environ.get("ANTHROPIC_API_KEY", "")
  • secrets (weak) UiPath-coder_eval-9ff1b96/tests/test_litellm_judge_live.py :37 return bool(os.getenv("CODEX_API_KEY") and os.getenv("CODEX_BASE_URL") and os.getenv("CODEX_MODEL"))

declared dependencies 27

  • mcp@>=1.28.1
  • pydantic@>=2.12.5
  • pydantic-settings@>=2.14.2
  • pyyaml@>=6.0.3
  • tomli-w@>=1.0.0
  • typer@>=0.24.1
  • click@>=8.3.3
  • rich@>=14.3.3
  • python-dotenv@>=1.2.2
  • anthropic@>=1.0.0,<2.0.0
  • s@own bound on it (`httpx2<3,>=2.0.0`).
  • pytest-asyncio@>=1.3.0
show 15 more
  • pytest@>=9.0.2
  • pytest-mock@>=3.15.1
  • pytest-cov@>=7.1.0
  • pytest-xdist@>=3.0.0
  • ruff@>=0.15.7
  • pyright@>=1.1.408
  • pip-audit@>=2.10.0
  • bandit@>=1.9.4
  • pre-commit@>=4.5.1
  • defusedxml@>=0.7.1
  • uipath@>=2.10.31
  • litellm@>=1.95.0,<2.0.0
  • openai-codex@>=0.144.4
  • google-antigravity@==0.1.8
  • harbor@==0.22.0