forked from WHC/Hermit
Usability pass: tool registry, TUI polish, saved keys and settings
Tools - Table-driven tool registry with permission classes and default/minimal profiles; new tools: edit_file, list_dir, glob, grep, delete_file, move_file, todo_write, web_fetch, ask_user (plus read_file paging). - Built-in Myers diff; write tools return unified diffs. - Persistent permission rules (.agent/permissions.json), per-turn undo snapshots (.agent/undo), AGENTS.md project notes, auto-compact at 80% of the model context, cancellable run_shell with process-group kill. TUI - Table-driven slash commands: /provider /key /effort /status /tools /permissions /yolo /undo /retry /diff /export /copy /rename /delete /init /max-steps /todo, with / and @ completion. - Permission modal with diff and prefix rules, ask_user modal, todo panel, header with model/effort/context, bracketed paste, line editing, wide-char cursor, NO_COLOR and ASCII fallback; Esc never quits. Startup and configuration - API keys are entered in the TUI (hidden) and saved per provider in <config>/credentials (0600); env variables are only a fallback. - Last provider, per-provider model and reasoning effort are remembered in <config>/settings.json; --effort/-e and /effort send reasoning_effort (low|medium|high|max) to models that accept it. - DeepSeek preset uses deepseek-flash / deepseek-v4-pro. - Session flags --new/--continue/--resume/--list-sessions, --yolo, --confirm. Compatibility and tests - ncurses detection, macOS/Homebrew support, make install, XDG config dir, POSIX cleanups; Gitea CI (gcc/clang/asan/musl/macOS). - Unit tests for diff, fs tools, registry, undo/perms and config; e2e fixtures multi_file_refactor and ask_or_todo; vendored cJSON.
This commit is contained in:
Executable
+63
@@ -0,0 +1,63 @@
|
||||
#!/usr/bin/env bash
|
||||
# Experiment matrix: protocol x model x fixture x N runs, then summarize.
|
||||
#
|
||||
# Usage:
|
||||
# scripts/run_matrix.sh [-p "native-eos native-inc ctl ctl-heredoc xml"]
|
||||
# [-m "deepseek:deepseek-flash openai:gpt-4.1-mini"]
|
||||
# [-f "mini_calc makefile_tab big_file test_fix_test"]
|
||||
# [-n 3] [-o out/matrix-<stamp>] [-r] [-t minimal|default]
|
||||
#
|
||||
# -m provider:model pairs (model may be empty: "deepseek:")
|
||||
# -r also record raw SSE streams per run (for replay experiments)
|
||||
# -t tool profile (default minimal, as in the paper runs)
|
||||
#
|
||||
# Each run is scripts/e2e_fixture.sh; results land in <out>/<run>/ and the
|
||||
# summary in <out>/results.csv and <out>/summary.md.
|
||||
set -uo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
PROTOCOLS="native-eos native-inc ctl ctl-heredoc xml"
|
||||
MODELS="deepseek:deepseek-flash"
|
||||
FIXTURES="mini_calc makefile_tab big_file test_fix_test"
|
||||
N=3
|
||||
OUT="$ROOT/out/matrix-$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
RECORD=()
|
||||
TOOLS=minimal
|
||||
|
||||
while getopts "p:m:f:n:o:rt:h" opt; do
|
||||
case "$opt" in
|
||||
p) PROTOCOLS="$OPTARG" ;;
|
||||
m) MODELS="$OPTARG" ;;
|
||||
f) FIXTURES="$OPTARG" ;;
|
||||
n) N="$OPTARG" ;;
|
||||
o) OUT="$OPTARG" ;;
|
||||
r) RECORD=(--record) ;;
|
||||
t) TOOLS="$OPTARG" ;;
|
||||
h|*) sed -n '2,16p' "$0"; exit 0 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
mkdir -p "$OUT"
|
||||
if [[ -z "${CST_AGENT_BIN:-}" ]]; then
|
||||
make -s -C "$ROOT" cst-agent >/dev/null || { echo "build failed" >&2; exit 65; }
|
||||
fi
|
||||
|
||||
total=0
|
||||
for pm in $MODELS; do
|
||||
for proto in $PROTOCOLS; do
|
||||
for fx in $FIXTURES; do
|
||||
for ((i = 1; i <= N; i++)); do
|
||||
total=$((total + 1))
|
||||
provider="${pm%%:*}"
|
||||
model="${pm#*:}"
|
||||
margs=(--provider "$provider")
|
||||
[[ -n "$model" ]] && margs+=(--model "$model")
|
||||
printf '[%d] %-10s %-12s %-14s #%d ' "$total" "$provider" "$proto" "$fx" "$i"
|
||||
"$ROOT/scripts/e2e_fixture.sh" "$fx" --protocol "$proto" --tools "$TOOLS" "${margs[@]}" \
|
||||
--out "$OUT" "${RECORD[@]}" | sed 's/ -> .*//'
|
||||
done
|
||||
done
|
||||
done
|
||||
done
|
||||
|
||||
python3 "$ROOT/scripts/summarize.py" "$OUT"
|
||||
Reference in New Issue
Block a user