security-guidance: pip --target fallback when venv can't bootstrap pip (2.0.4 → 2.0.5)

Option A, the data-gated fix for venv_ensurepip_fail (#2154 follow-up).

v2.0.4 telemetry made the call: of the venv_ensurepip_fail cohort, ~95%
HAVE pip (sdk_has_pip=true) and run Python 3.11–3.14 — so it's not the
Apple-3.9 problem; it's modern interpreters where `python -m venv` can't
bootstrap pip (Debian python3-venv absent, or python.org/pyenv builds
without ensurepip) but pip itself works. `pip install --target` needs only
pip, so it recovers the agentic reviewer for them instead of degrading to
pattern + single-shot review.

Producer (ensure_agent_sdk.py):
  - New outcomes BUILT_TARGET=7, NOOP_TARGET=8; new phase pip_target=5.
  - _build_via_target(): `pip install --target <state>/agent-sdk-libs
    --upgrade --prefer-binary claude-agent-sdk`. Failures categorized via
    _pip_err_from_stderr (sibling of main()'s pip chain — kept separate to
    avoid disturbing the working venv categorizer); errno embedded for
    OSError-family exceptions.
  - _target_sdk_importable(): probes a prior target install → NOOP_TARGET.
    Dir-check short-circuits before any subprocess, and it's only reached
    when there's no working venv, so the 81% NOOP_VENV cohort never pays.
  - main() falls through to the target build ONLY on venv_ensurepip_fail;
    every other venv/pip failure stays terminal BUILD_FAILED. The sentinel
    is released before the target build so a retry isn't seen as SKIP_SENTINEL.

Consumer (llm.py):
  - _inject_agent_sdk_venv_into_syspath() adds the flat agent-sdk-libs dir
    (packages sit directly in it, not under site-packages). The existing
    pywin32 .pth bootstrap applies (target installs don't run .pth either).

No change to the happy path — the new branch is taken only on the
ensurepip failure, and the extra candidate dir is a no-op when absent.

Verified locally on macOS Python 3.13:
  - py_compile clean.
  - 30 new tests (test_venv_target_fallback.py): outcome/phase codes
    (append-only, 4 stays retired), _pip_err_from_stderr categories,
    _build_via_target success/CalledProcessError/timeout/exc+errno (mocked
    subprocess), _target_sdk_importable dir-short-circuit, main() wiring
    (ensurepip→target fallthrough + NOOP_TARGET probe + sentinel release),
    consumer adds the flat dir. Full suite 533/533 pass + 2 skipped.
  - END-TO-END harness (real install, simulated ensurepip failure):
    main() → BUILT_TARGET, target dir has claude_agent_sdk; 2nd run →
    NOOP_TARGET; consumer _inject → `import claude_agent_sdk` resolves
    FROM the --target dir. Full chain proven without needing a
    broken-ensurepip box.
  - Real `pip install --target` + import confirmed independently (exit 0,
    SDK imports from the flat layout).

NOT validated in tmux: the ensurepip failure can't be reproduced on macOS
(working ensurepip), so the fallback was proven via the real-install
harness above instead. The happy path (NOOP_VENV / normal agentic review)
is unchanged and covered by the existing hook-smoke suite.

Version 2.0.4 -> 2.0.5 per the per-PR-bump policy (#2114).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

.claude-plugin/marketplace.json

@@ -2387,7 +2387,7 @@
     {
       "name": "security-guidance",
       "description": "Security review for Claude-generated code. Pattern-based warnings on edits, LLM-powered diff review on Stop, and an agentic commit reviewer that catches injection, XSS, SSRF, hardcoded secrets, and 25+ other vulnerability classes.",
-      "version": "2.0.4",
+      "version": "2.0.5",
       "author": {
         "name": "Anthropic",
         "email": "support@anthropic.com"

plugins/security-guidance/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "security-guidance",
-  "version": "2.0.4",
+  "version": "2.0.5",
   "description": "Security review for Claude-generated code. Pattern-based warnings on edits, LLM-powered diff review on Stop, and an agentic commit reviewer that catches injection, XSS, SSRF, hardcoded secrets, and 25+ other vulnerability classes.",
   "author": {
     "name": "David Dworken",

plugins/security-guidance/hooks/ensure_agent_sdk.py

@@ -40,6 +40,15 @@ BUILD_FAILED = 3     # venv create or pip install raised/timed out
 SKIP_SENTINEL = 5    # another SessionStart is currently building
 HOOK_PY_INCOMPATIBLE = 6  # hook interpreter is <3.10 — SDK syntax can't load
                           # here no matter how the venv was built. See #2071.
+# --target fallback: when `python -m venv` can't bootstrap pip (ensurepip
+# missing — Debian python3-venv not installed, or a python.org/pyenv build
+# without ensurepip), fall back to `pip install --target <dir>` which needs
+# only the system pip, not venv/ensurepip. Telemetry (v2.0.4 sdk_has_pip
+# probe) confirmed ~95% of venv_ensurepip_fail users HAVE pip, so this
+# recovers the agentic reviewer for them instead of degrading to pattern +
+# single-shot review. See #2154 follow-up.
+BUILT_TARGET = 7     # venv ensurepip failed → SDK pip-installed via --target
+NOOP_TARGET = 8      # --target libs already present and importable
 
 
 # Phase + err-kind integer encoding for sdk_bootstrap_phase / sdk_bootstrap_err.
@@ -63,6 +72,7 @@ SDK_BOOTSTRAP_PHASE_CODES = {
     "venv": 2,  # python -m venv --clear
     "pip":  3,  # pip install
     "main": 4,  # uncaught exception above main()
+    "pip_target": 5,  # `pip install --target` fallback (venv ensurepip failed)
 }
 SDK_BOOTSTRAP_ERR_CODES = {
     "pip_no_match":         1,
@@ -242,6 +252,96 @@ def _probe_has_pip() -> bool:
         return False
 
 
+def _pip_err_from_stderr(stderr_b):
+    """Categorize a pip-install stderr into a known err_kind (the pip subset
+    of SDK_BOOTSTRAP_ERR_CODES). Used by the --target fallback; mirrors the
+    pip branches of main()'s inline categorizer. Kept as a sibling rather
+    than extracting main()'s chain (which also has venv-phase branches) to
+    avoid disturbing the working venv categorization."""
+    if isinstance(stderr_b, bytes):
+        s = stderr_b.decode("utf-8", errors="replace")
+    else:
+        s = str(stderr_b or "")
+    low = s.lower()
+    if "no matching distribution" in low or "could not find a version" in low:
+        return "pip_no_match"
+    if ("name or service not known" in low or "name resolution" in low
+            or "nodename nor servname" in low or "temporary failure in name" in low):
+        return "dns_fail"
+    if "connection refused" in low or "connection reset" in low:
+        return "conn_refused"
+    if "ssl" in low and ("verify" in low or "certificate" in low):
+        return "ssl_verify"
+    if "permission denied" in low or "read-only file system" in low:
+        return "perm_denied"
+    if "no module named pip" in low or "no module named ensurepip" in low:
+        return "no_pip"
+    if "no space left" in low or "disk quota" in low:
+        return "disk_full"
+    if "proxy" in low and ("authent" in low or "tunnel" in low or "407" in low):
+        return "proxy_auth"
+    if "timeout" in low or "timed out" in low:
+        return "stderr_timeout"
+    tail = next((ln.strip() for ln in reversed(s.splitlines()) if ln.strip()), "")[:60]
+    return f"other:{tail}" if tail else "other"
+
+
+def _target_dir(state_dir) -> Path:
+    return Path(state_dir) / "agent-sdk-libs"
+
+
+def _target_sdk_importable(state_dir) -> bool:
+    """True iff the --target libs dir has an importable claude_agent_sdk,
+    probed with THIS interpreter (the one llm.py will import it from) and the
+    target dir prepended to sys.path. Cheap dir-check first to avoid a
+    subprocess on the common no-target path."""
+    target = _target_dir(state_dir)
+    if not (target / "claude_agent_sdk").is_dir():
+        return False
+    try:
+        r = subprocess.run(
+            [sys.executable, "-c",
+             "import sys; sys.path.insert(0, sys.argv[1]); import claude_agent_sdk",
+             str(target)],
+            capture_output=True, timeout=10,
+        )
+        return r.returncode == 0
+    except Exception:
+        return False
+
+
+def _build_via_target(state_dir) -> tuple[int, str, str]:
+    """Fallback install when `python -m venv` can't bootstrap pip (ensurepip
+    missing — Debian python3-venv absent, or a python.org/pyenv build without
+    ensurepip). `pip install --target <dir>` needs only the system pip, not
+    venv/ensurepip. v2.0.4 telemetry (sdk_has_pip) confirmed ~95% of
+    venv_ensurepip_fail users have pip. The consumer (llm.py) adds this flat
+    dir to sys.path. Returns (outcome, err_phase, err_kind).
+
+    --upgrade so a stale/partial target dir from a prior failed attempt
+    doesn't make pip refuse; --prefer-binary mirrors the venv path's wheel
+    preference (ARM64 Windows cryptography)."""
+    target = _target_dir(state_dir)
+    try:
+        subprocess.run(
+            [sys.executable, "-m", "pip", "install",
+             "--target", str(target), "--upgrade",
+             "--disable-pip-version-check", "--prefer-binary",
+             "claude-agent-sdk"],
+            capture_output=True, timeout=120, check=True,
+        )
+        return BUILT_TARGET, "", ""
+    except subprocess.CalledProcessError as e:
+        return BUILD_FAILED, "pip_target", _pip_err_from_stderr(e.stderr)
+    except subprocess.TimeoutExpired:
+        return BUILD_FAILED, "pip_target", "subprocess_timeout"
+    except Exception as e:
+        errno = getattr(e, "errno", None)
+        if isinstance(errno, int):
+            return BUILD_FAILED, "pip_target", f"exc:{type(e).__name__}:{errno}"
+        return BUILD_FAILED, "pip_target", f"exc:{type(e).__name__}"
+
+
 def _sdk_on_syspath() -> bool:
     # find_spec is ~10ms; actually importing the SDK pulls in
     # transitive deps and costs ~800ms — too heavy for a
@@ -330,6 +430,12 @@ def main() -> tuple[int, str, str]:
         except Exception:
             pass  # broken venv; rebuild below
 
+    # If a prior run installed the SDK via the --target fallback (ensurepip
+    # path), reuse it. Only reached when there's no working venv, so healthy
+    # NOOP_VENV users never pay for this probe.
+    if _target_sdk_importable(state_dir):
+        return NOOP_TARGET, "", ""
+
     err_phase = ""
     err_kind = ""
     we_own_sentinel = False
@@ -444,6 +550,16 @@ def main() -> tuple[int, str, str]:
                 "",
             )[:60]
             err_kind = f"other:{tail}" if tail else "other"
+        # venv couldn't bootstrap pip (ensurepip missing) but pip itself may
+        # work — fall back to a flat `pip install --target`. Only this one
+        # category falls through; every other venv/pip failure is terminal.
+        # The finally block unlinks our sentinel first (so the target build
+        # isn't blocked by it); _build_via_target does the target install.
+        if err_kind == "venv_ensurepip_fail":
+            if we_own_sentinel:
+                sentinel.unlink(missing_ok=True)
+                we_own_sentinel = False
+            return _build_via_target(state_dir)
         return BUILD_FAILED, err_phase, err_kind
     except subprocess.TimeoutExpired:
         return BUILD_FAILED, err_phase, "subprocess_timeout"

plugins/security-guidance/hooks/llm.py

@@ -55,6 +55,12 @@ def _inject_agent_sdk_venv_into_syspath(state_dir):
     candidates = (
         glob.glob(os.path.join(venv_root, "lib", "python*", "site-packages"))
         + glob.glob(os.path.join(venv_root, "Lib", "site-packages"))
+        # `pip install --target` fallback (ensure_agent_sdk BUILT_TARGET, used
+        # when venv can't bootstrap pip): a FLAT layout — packages sit directly
+        # in agent-sdk-libs/, not under a site-packages subdir. See #2154
+        # follow-up. The pywin32 .pth bootstrap below applies here too (target
+        # installs don't process .pth at runtime, same as a manual venv insert).
+        + [os.path.join(state_dir, "agent-sdk-libs")]
     )
     added = False
     for sp in candidates: