Skip to content

Commit 13cbfa7

Browse files
committed
test(e2e): safety hardening suite + CI matrix + invariant fixups
Adds four end-to-end integration test files exercising the safety primitives through the binary, plus shared `tests/common/mod.rs` helpers, plus two existing-test contract updates. **Suites added (20 new tests):** - `e2e_safety_lock.rs` (6 tests, non-ignored). Test holds the same `.socket/apply.lock` the binary uses via `fs2` directly, then spawns `socket-patch apply` and asserts the second process exits with `error.code == "lock_held"`. Zero production-code hooks. - `e2e_safety_yarn_pnp.rs` (5 tests, non-ignored). Yarn-berry PnP markers (`.pnp.cjs`, `.pnp.loader.mjs`) trigger `error.code == "yarn_pnp_unsupported"`. Negative control: plain npm layout does NOT trigger the refusal. - `e2e_safety_cargo_build.rs` (5 tests, `#[ignore]` + `--features cargo`). Three synthetic-vendor tests: 1. Baseline `cargo check --offline --frozen` succeeds. 2. Negative control — mutating the source WITHOUT the sidecar fixup makes cargo refuse with "checksum changed". Proves cargo actually verifies, which is what makes the positive test meaningful. 3. Sidecar fixup makes `cargo check` pass; `.cargo-checksum.json` is rewritten and the `package` field is preserved. 4. JSON envelope contract: `.cargo-checksum.json` appears in `event.details.sidecarsUpdated`. Plus `traitobject_real_socket_patch_round_trip` — the cargo layer-2+3 combined test: `cargo fetch traitobject@0.0.1` from crates.io → `socket-patch get b15f2b7f-d5cb-43c9-b793-80f71682188f` from patches-api.socket.dev → assert `.cargo-checksum.json` rewritten + `cargo check` succeeds against the real, production Socket patch. - `e2e_safety_pnpm.rs` (4 tests, `#[ignore]`). Two projects share a pnpm content store via `--config.package-import-method=hardlink`. `socket-patch get` in project A patches A; project B + store entry stay byte-identical. `pnpm install --frozen-lockfile` in B afterwards does not revert A. Exercises CoW against a real pnpm install rather than a hand-rolled hardlink. **`tests/common/mod.rs`** — shared helpers (`binary`, `run`, `assert_run_ok`, `git_sha256`, `sha256_hex`, `pnpm_run`, `cargo_run`, `write_minimal_manifest`, `write_blob`, `parse_json_envelope`, `envelope_error_code`, `envelope_error_message`) lifted from the duplicated copies in `e2e_npm.rs` etc. Additive; existing suites keep their inlined copies for now. **CI matrix** in `.github/workflows/ci.yml`: - `e2e_safety_cargo_build` on ubuntu + macos + windows - `e2e_safety_pnpm` on ubuntu + macos + windows (pnpm-on-Windows uses junctions + copies by default, so the CoW invariant holds vacuously; the test still runs to verify apply doesn't error on Windows. Semantic Windows nlink coverage is a follow-up — `std::fs::Metadata` doesn't expose nlink on Windows without `GetFileInformationByHandle` via `windows-sys`.) - New `Setup pnpm` step (`npm install -g pnpm@10`) gated on the pnpm suite. The fast non-ignored suites (`e2e_safety_lock`, `e2e_safety_yarn_pnp`) run via the standard `test` job on all three platforms. **Existing-test contract updates** (these tests were pinning the old, broken behavior; both still describe correct invariants — their assertions just needed to track the rebased semantics): - `tests/apply_invariants.rs`: `dir_hash` excludes `apply.lock`. The lock file is deliberate ephemeral session state, not patch content; the "apply is read-only against .socket/" invariant is about manifest + blobs + diffs + packages. - `tests/in_process_edge_cases.rs`: `apply_blob_after_hash_mismatch_reports_failure` now asserts the atomic-write contract — the target file is byte-identical to its pre-call state on the hash-mismatch failure path, no half-written corruption. Assisted-by: Claude Code:claude-opus-4-7
1 parent 39a2321 commit 13cbfa7

8 files changed

Lines changed: 1636 additions & 11 deletions

File tree

.github/workflows/ci.yml

Lines changed: 34 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -411,6 +411,30 @@ jobs:
411411
suite: e2e_scan
412412
- os: macos-latest
413413
suite: e2e_scan
414+
# Safety-hardening e2e suites. The fast non-ignored ones
415+
# (e2e_safety_lock, e2e_safety_yarn_pnp) run via the
416+
# standard `test` job above on all three platforms, so no
417+
# matrix entry is needed for them. The two below need real
418+
# toolchains and are #[ignore]-gated.
419+
- os: ubuntu-latest
420+
suite: e2e_safety_cargo_build
421+
- os: macos-latest
422+
suite: e2e_safety_cargo_build
423+
- os: windows-latest
424+
suite: e2e_safety_cargo_build
425+
- os: ubuntu-latest
426+
suite: e2e_safety_pnpm
427+
- os: macos-latest
428+
suite: e2e_safety_pnpm
429+
# pnpm-on-Windows uses junctions for symlinks and copies
430+
# (not hardlinks) by default, so the CoW invariant holds
431+
# vacuously. Test still runs to verify apply doesn't error
432+
# on Windows — semantic Windows nlink coverage is a
433+
# follow-up (`std::fs::Metadata` doesn't expose nlink on
434+
# Windows; needs `GetFileInformationByHandle` via
435+
# `windows-sys`).
436+
- os: windows-latest
437+
suite: e2e_safety_pnpm
414438
runs-on: ${{ matrix.os }}
415439
steps:
416440
- name: Checkout
@@ -436,11 +460,20 @@ jobs:
436460
restore-keys: ${{ matrix.os }}-cargo-e2e-
437461

438462
- name: Setup Node.js
439-
if: matrix.suite == 'e2e_npm' || matrix.suite == 'e2e_scan'
463+
if: matrix.suite == 'e2e_npm' || matrix.suite == 'e2e_scan' || matrix.suite == 'e2e_safety_pnpm'
440464
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
441465
with:
442466
node-version: '20.20.2'
443467

468+
- name: Setup pnpm
469+
if: matrix.suite == 'e2e_safety_pnpm'
470+
# Pin the major version so the store layout the test
471+
# asserts on stays stable. `npm install -g` is the simplest
472+
# cross-platform install path (works on ubuntu, macos,
473+
# windows-runners — they all ship a usable npm via
474+
# actions/setup-node).
475+
run: npm install -g pnpm@10
476+
444477
- name: Setup Python
445478
if: matrix.suite == 'e2e_pypi'
446479
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5

crates/socket-patch-cli/tests/apply_invariants.rs

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,9 +75,18 @@ fn write_project(root: &Path) {
7575
/// Recursive, stable hash of every regular file under `dir`. Combines
7676
/// each file's relative path and bytes into a single SHA-256 so any
7777
/// change — adding, removing, or rewriting a file — flips the digest.
78+
///
79+
/// Excludes `apply.lock` (advisory lock file created by `apply` /
80+
/// `rollback` / `repair` / `remove`). That file is deliberate
81+
/// ephemeral session state — not patch content — and persists by
82+
/// design so subsequent runs can re-flock the same inode without a
83+
/// create race. The "apply is read-only against .socket/" invariant
84+
/// is about the patch payload (manifest, blobs, diffs, packages),
85+
/// not session metadata.
7886
fn dir_hash(dir: &Path) -> String {
7987
let mut files: Vec<(PathBuf, Vec<u8>)> = Vec::new();
8088
collect_files(dir, dir, &mut files);
89+
files.retain(|(rel, _)| rel.file_name().and_then(|n| n.to_str()) != Some("apply.lock"));
8190
files.sort_by(|a, b| a.0.cmp(&b.0));
8291
let mut hasher = Sha256::new();
8392
for (rel, bytes) in files {
Lines changed: 263 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,263 @@
1+
//! Helpers shared across the e2e-safety test suites.
2+
//!
3+
//! The original e2e files (`e2e_npm.rs`, `e2e_pypi.rs`, `e2e_gem.rs`)
4+
//! each carry their own copy of the same `binary` / `run` /
5+
//! `assert_run_ok` / `git_sha256` helpers. Rather than refactor those
6+
//! files in this PR, this module is an additive landing place for the
7+
//! same surface plus the new helpers the safety suites need
8+
//! (synthetic manifest writers, pnpm runners, cargo runners). Existing
9+
//! suites can migrate in a follow-up.
10+
//!
11+
//! Each test file pulls this in with `#[path = "common/mod.rs"] mod common;`.
12+
//!
13+
//! `#![allow(dead_code)]` because each test file uses a different
14+
//! subset of these helpers; the unused ones would otherwise produce
15+
//! warnings under `-D warnings`.
16+
17+
#![allow(dead_code)]
18+
19+
use std::collections::HashMap;
20+
use std::path::{Path, PathBuf};
21+
use std::process::{Command, Output};
22+
23+
use sha2::{Digest, Sha256};
24+
25+
// ── Binary discovery + invocation ─────────────────────────────────────
26+
27+
/// Absolute path to the built `socket-patch` binary that cargo
28+
/// provides via the `CARGO_BIN_EXE_*` env var. Available because
29+
/// these tests live in the same crate that produces the binary.
30+
pub fn binary() -> PathBuf {
31+
env!("CARGO_BIN_EXE_socket-patch").into()
32+
}
33+
34+
/// Quick check whether `cmd` is on PATH. Used to soft-skip
35+
/// toolchain-dependent tests when the toolchain isn't installed
36+
/// (CI gates the toolchain at the workflow level; this is a
37+
/// belt-and-braces guard for local runs).
38+
pub fn has_command(cmd: &str) -> bool {
39+
Command::new(cmd)
40+
.arg("--version")
41+
.stdout(std::process::Stdio::null())
42+
.stderr(std::process::Stdio::null())
43+
.status()
44+
.is_ok()
45+
}
46+
47+
/// Run the CLI binary with `args`, working dir `cwd`. Returns
48+
/// `(exit_code, stdout, stderr)`. Strips `SOCKET_API_TOKEN` from the
49+
/// environment so apply paths default to the public proxy and tests
50+
/// don't accidentally exercise authed endpoints.
51+
pub fn run(cwd: &Path, args: &[&str]) -> (i32, String, String) {
52+
let out: Output = Command::new(binary())
53+
.args(args)
54+
.current_dir(cwd)
55+
.env_remove("SOCKET_API_TOKEN")
56+
.output()
57+
.expect("failed to execute socket-patch binary");
58+
let code = out.status.code().unwrap_or(-1);
59+
let stdout = String::from_utf8_lossy(&out.stdout).to_string();
60+
let stderr = String::from_utf8_lossy(&out.stderr).to_string();
61+
(code, stdout, stderr)
62+
}
63+
64+
/// `run` + assertion that exit code is 0. Returns `(stdout, stderr)`
65+
/// on success; panics with a context message + both streams on
66+
/// failure (so test logs show exactly what the binary printed).
67+
pub fn assert_run_ok(cwd: &Path, args: &[&str], context: &str) -> (String, String) {
68+
let (code, stdout, stderr) = run(cwd, args);
69+
assert_eq!(
70+
code, 0,
71+
"{context} failed (exit {code}).\nstdout:\n{stdout}\nstderr:\n{stderr}"
72+
);
73+
(stdout, stderr)
74+
}
75+
76+
// ── Hashing ───────────────────────────────────────────────────────────
77+
78+
/// Compute Git-flavored SHA-256: `SHA256("blob <len>\0" ++ content)`.
79+
/// This is the hash socket-patch records in manifests under
80+
/// `before_hash` / `after_hash`.
81+
pub fn git_sha256(content: &[u8]) -> String {
82+
let header = format!("blob {}\0", content.len());
83+
let mut hasher = Sha256::new();
84+
hasher.update(header.as_bytes());
85+
hasher.update(content);
86+
hex::encode(hasher.finalize())
87+
}
88+
89+
/// Git-SHA-256 of the file at `path`. Panics if the file can't be
90+
/// read — tests use this on paths they know exist.
91+
pub fn git_sha256_file(path: &Path) -> String {
92+
let content =
93+
std::fs::read(path).unwrap_or_else(|e| panic!("read {}: {e}", path.display()));
94+
git_sha256(&content)
95+
}
96+
97+
/// Raw lowercase-hex SHA-256 (no Git blob framing). Used by the
98+
/// Cargo sidecar which embeds plain digests in
99+
/// `.cargo-checksum.json`.
100+
pub fn sha256_hex(content: &[u8]) -> String {
101+
let mut hasher = Sha256::new();
102+
hasher.update(content);
103+
format!("{:x}", hasher.finalize())
104+
}
105+
106+
// ── Toolchain runners ─────────────────────────────────────────────────
107+
108+
/// Run `npm` in `cwd`, panic on non-zero exit with full output.
109+
pub fn npm_run(cwd: &Path, args: &[&str]) {
110+
run_toolchain(cwd, "npm", args, &[]);
111+
}
112+
113+
/// Run `pnpm` in `cwd`. Same shape as `npm_run`; `extra_env` lets
114+
/// the caller force store-dir overrides etc.
115+
pub fn pnpm_run(cwd: &Path, args: &[&str], extra_env: &[(&str, &str)]) {
116+
run_toolchain(cwd, "pnpm", args, extra_env);
117+
}
118+
119+
/// Run `cargo` in `cwd`. Returns the raw Output so callers can
120+
/// inspect stdout/stderr/exit on either pass or fail — the cargo
121+
/// e2e test wants both passing and failing cases (negative control).
122+
pub fn cargo_run(cwd: &Path, args: &[&str], extra_env: &[(&str, &str)]) -> Output {
123+
let mut cmd = Command::new("cargo");
124+
cmd.args(args).current_dir(cwd);
125+
for (k, v) in extra_env {
126+
cmd.env(k, v);
127+
}
128+
cmd.output().expect("failed to run cargo")
129+
}
130+
131+
fn run_toolchain(cwd: &Path, exe: &str, args: &[&str], extra_env: &[(&str, &str)]) {
132+
let mut cmd = Command::new(exe);
133+
cmd.args(args).current_dir(cwd);
134+
for (k, v) in extra_env {
135+
cmd.env(k, v);
136+
}
137+
let out = cmd
138+
.output()
139+
.unwrap_or_else(|e| panic!("failed to run {exe}: {e}"));
140+
assert!(
141+
out.status.success(),
142+
"{exe} {args:?} failed (exit {:?}).\nstdout:\n{}\nstderr:\n{}",
143+
out.status.code(),
144+
String::from_utf8_lossy(&out.stdout),
145+
String::from_utf8_lossy(&out.stderr),
146+
);
147+
}
148+
149+
// ── Project scaffolding ───────────────────────────────────────────────
150+
151+
/// Write a minimal package.json. Avoids `npm init -y` which rejects
152+
/// temp dir names that start with `.` or contain invalid chars.
153+
pub fn write_package_json(cwd: &Path) {
154+
std::fs::write(
155+
cwd.join("package.json"),
156+
r#"{"name":"e2e-test","version":"0.0.0","private":true}"#,
157+
)
158+
.expect("write package.json");
159+
}
160+
161+
// ── Synthetic manifest + blob construction ────────────────────────────
162+
163+
/// Describe a single patched-file row in a synthetic manifest.
164+
pub struct PatchEntry<'a> {
165+
/// File path as recorded by the manifest (may include the
166+
/// `package/` prefix used by the API; apply strips it before
167+
/// resolving against pkg_path).
168+
pub file_name: &'a str,
169+
pub before_hash: &'a str,
170+
pub after_hash: &'a str,
171+
}
172+
173+
/// Write a minimal `.socket/manifest.json` at `socket_dir/manifest.json`
174+
/// describing one patch for `purl` with the given `uuid` and `files`.
175+
///
176+
/// Returns the path to the manifest file.
177+
///
178+
/// Does NOT write the `after_hash` blobs — that's `write_blob`'s
179+
/// job, and the test gets to decide which blobs to omit (e.g. to
180+
/// force an offline-apply failure).
181+
pub fn write_minimal_manifest(
182+
socket_dir: &Path,
183+
purl: &str,
184+
uuid: &str,
185+
files: &[PatchEntry<'_>],
186+
) -> PathBuf {
187+
std::fs::create_dir_all(socket_dir).expect("create .socket dir");
188+
let mut files_map = serde_json::Map::new();
189+
for f in files {
190+
files_map.insert(
191+
f.file_name.to_string(),
192+
serde_json::json!({
193+
"beforeHash": f.before_hash,
194+
"afterHash": f.after_hash,
195+
}),
196+
);
197+
}
198+
let manifest = serde_json::json!({
199+
"patches": {
200+
purl: {
201+
"uuid": uuid,
202+
"exportedAt": "2026-01-01T00:00:00Z",
203+
"files": files_map,
204+
"vulnerabilities": {},
205+
"description": "synthetic test patch",
206+
"license": "MIT",
207+
"tier": "free",
208+
}
209+
}
210+
});
211+
let path = socket_dir.join("manifest.json");
212+
std::fs::write(&path, serde_json::to_string_pretty(&manifest).unwrap())
213+
.expect("write manifest.json");
214+
path
215+
}
216+
217+
/// Drop `content` at `<socket_dir>/blobs/<hash>`. Used to stage the
218+
/// `after_hash` blob a synthetic manifest references so apply can
219+
/// run fully offline.
220+
pub fn write_blob(socket_dir: &Path, hash: &str, content: &[u8]) {
221+
let blobs = socket_dir.join("blobs");
222+
std::fs::create_dir_all(&blobs).expect("create .socket/blobs");
223+
std::fs::write(blobs.join(hash), content).expect("write blob");
224+
}
225+
226+
/// Parse `--json` apply output, returning the top-level JSON object
227+
/// or panicking with the raw text on parse failure. Most safety tests
228+
/// want to assert on specific fields (`errorCode`, `status`, etc.).
229+
pub fn parse_json_envelope(stdout: &str) -> serde_json::Value {
230+
serde_json::from_str(stdout)
231+
.unwrap_or_else(|e| panic!("failed to parse JSON envelope: {e}\nstdout:\n{stdout}"))
232+
}
233+
234+
/// Extract a stringified field from a parsed JSON envelope, or None
235+
/// if the field is missing / not a string. Convenience for the
236+
/// `status` checks the safety tests do repeatedly.
237+
pub fn json_string<'a>(env: &'a serde_json::Value, key: &str) -> Option<&'a str> {
238+
env.get(key).and_then(|v| v.as_str())
239+
}
240+
241+
/// Extract `env.error.code` from a parsed envelope. The v3.0
242+
/// envelope shape nests the error under a top-level `error` object
243+
/// (`{"error": {"code": "lock_held", "message": "..."}}`), not at
244+
/// the top level. This helper centralises that lookup so individual
245+
/// tests can stay terse.
246+
pub fn envelope_error_code(env: &serde_json::Value) -> Option<&str> {
247+
env.get("error")?.get("code")?.as_str()
248+
}
249+
250+
/// Extract `env.error.message` from a parsed envelope. Companion to
251+
/// [`envelope_error_code`].
252+
pub fn envelope_error_message(env: &serde_json::Value) -> Option<&str> {
253+
env.get("error")?.get("message")?.as_str()
254+
}
255+
256+
/// Map a slice of `(env-var-name, env-var-value)` tuples into a
257+
/// HashMap for callers that want a stable container.
258+
pub fn env_map(pairs: &[(&str, &str)]) -> HashMap<String, String> {
259+
pairs
260+
.iter()
261+
.map(|(k, v)| ((*k).to_string(), (*v).to_string()))
262+
.collect()
263+
}

0 commit comments

Comments
 (0)