diff --git a/.claude/skills/feeds-sprint/SKILL.md b/.claude/skills/feeds-sprint/SKILL.md new file mode 100644 index 00000000..a505243f --- /dev/null +++ b/.claude/skills/feeds-sprint/SKILL.md @@ -0,0 +1,45 @@ +--- +name: feeds-sprint +description: innate-feeds Sprint 工作流：按 tasks/sprint-N 的文档驱动方式认领、执行、回写任务状态。当用户要求"执行 sprint 任务"、"继续 sprint-1"、"按 feeds-sprint 工作流处理 issue"或创建/更新 sprint 任务时使用。 +--- + +# feeds-sprint：文档驱动的多 Agent Sprint 工作流 + +本仓库用 `tasks/sprint-N/` 目录管理一个 Sprint。每个 Sprint 的执行原则：**先计划、任务独立、状态可查、可被任何 Agent 接手**。 + +## 目录结构（如何阅读） + +``` +tasks/sprint-N/ +├── context.md # Sprint 背景、目标、任务索引、依赖关系、工作流约定 —— 先读这个 +├── status.md # 任务状态总表（单一事实来源）：todo/in_progress/done/blocked +├── research/ # 调研产出物（方案对比、验证 SQL、推荐结论） +├── task-XX-*.md # 每个独立任务一份文档：描述 + 依赖 + Issue 链接 + 验收记录 +└── task-rX-*-research.md # 调研类任务同样有 task 文档，产出物放 research/ +``` + +阅读顺序：`context.md`（全局）→ `status.md`（找活）→ 对应 `task-XX.md`（怎么做、怎么算完成）。 + +## 执行规则 + +1. **认领前先查状态**：读 `status.md`，目标 task 已是 `done` → 直接跳到下一个无依赖或依赖已满足的 task；`in_progress` 且非本人 → 不要碰。 +2. **认领**：把 `status.md` 对应行改为 `in_progress` + 写 Agent 名。 +3. **执行**：实现改动进代码；调研产出写 `research/`；无论哪类，回填 task 文档的「验收记录」复选框。 +4. **完成**：`status.md` 置 `done` + 提交 SHA；有 GitHub Issue 的用 `gh issue close` 关闭；在「执行日志」追加一行。 +5. **阻塞**：状态改 `blocked` 并在备注写原因，换下一个可做的 task，不要空转。 +6. **新增任务**：新建 task 文档 + `gh issue create --label feeds-sprint-1` + 更新 `context.md` 任务表与 `status.md`。新任务必须独立；不独立则声明依赖与优先级（参考 context.md 的分组说明）。 + +## 本仓快速命令 + +```bash +bun run typecheck # backend + frontend 类型检查 +bun run dev:backend # 后端 :4000（默认启用 /admin 与 /api/admin/db/*） +bun run dev:frontend # 前端 :3000 +cd backend && bun run data:sync:trending && bun run data:sync:starred # 数据同步 +gh issue create -R variableway/innate-feeds -l feeds-sprint-1 ... +``` + +## 提交约定 + +- 分支：`qdriven/issue-`；提交信息用 conventional commits（`feat:` / `docs:` / `chore:`）。 +- Sprint 文档与代码改动同一 PR 提交，PR 描述列出完成的 task。 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8aad1a71..daabdacd 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -46,7 +46,11 @@ jobs: - name: Install root dependencies working-directory: innate-feeds - run: bun install + # pnpm, not bun install: CI republishes @innate/* @0.1.0 tarballs to a + # fresh Verdaccio on every run, so bun.lock's pinned integrity hashes + # can never match (IntegrityCheckFailed). pnpm resolves fresh — same + # as local setup (innate-fe-cli install drives pnpm for frontend/). + run: pnpm install - name: Run tests working-directory: innate-feeds @@ -100,7 +104,11 @@ jobs: - name: Install root dependencies working-directory: innate-feeds - run: bun install + # pnpm, not bun install: CI republishes @innate/* @0.1.0 tarballs to a + # fresh Verdaccio on every run, so bun.lock's pinned integrity hashes + # can never match (IntegrityCheckFailed). pnpm resolves fresh — same + # as local setup (innate-fe-cli install drives pnpm for frontend/). + run: pnpm install - name: Build frontend working-directory: innate-feeds diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index b5900450..5873fde4 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -83,8 +83,9 @@ jobs: --root "$GITHUB_WORKSPACE/innate-fe" \ --dir frontend cp frontend/.npmrc .npmrc - bun install - bun run install:all + # pnpm, not bun install: fresh @innate/* tarballs never match + # bun.lock integrity hashes (see ci.yml note). + pnpm install - name: Restore README snapshot cache uses: actions/cache@v4 diff --git a/AGENTS.md b/AGENTS.md index f77add65..7a5d1206 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -114,12 +114,13 @@ All commands assume you are in the project root unless noted. ### Install dependencies -The repo is a **bun workspace** (`workspaces: backend, frontend`). `@innate/ui` / `@innate/web-domain` come from the innate-fe base repo via a private registry — run `innate-fe-cli install` first (see [docs/arch-design/innate-fe-plugin-migration.md](docs/arch-design/innate-fe-plugin-migration.md)): +The repo is a workspace (`workspaces: backend, frontend`, `pnpm-workspace.yaml`). `@innate/ui` / `@innate/web-domain` come from the innate-fe base repo via a private registry — run `innate-fe-cli install` first (see [docs/arch-design/innate-fe-plugin-migration.md](docs/arch-design/innate-fe-plugin-migration.md)). Installs are **pnpm**-based (fresh @innate/* tarballs never match `bun.lock` integrity hashes, so don't use `bun install` at the root): ```bash export INNATE_FE_ROOT=/path/to/innate-fe -bun run setup:innate # publish @innate/* → .npmrc → rewrite workspace:* → pnpm install -bun install # = bun run install:all +bun run setup:innate # publish @innate/* → write frontend/.npmrc → pnpm install +cp frontend/.npmrc .npmrc # root .npmrc is required for @innate scope resolution +pnpm install # = bun run install:all ``` ### Start development @@ -338,6 +339,17 @@ Interactive docs: **`/docs`** (Swagger UI) · spec: **`/api/openapi.json`** (`ba | GET | `/api/auth/status` | gh CLI + PAT presence (never returns the token). | | PUT | `/api/auth/pat` | Store GitHub PAT encrypted at rest. Body: `{ token }`. | | DELETE | `/api/auth/pat` | Remove the stored PAT. | +| GET | `/api/admin/db/tables` | All tables + row counts (admin workspace, API mode only). | +| GET | `/api/admin/db/tables/:name/schema` | Columns / indexes / DDL for one table. | +| POST | `/api/admin/db/query` | Execute a single read-only SELECT/WITH on a dedicated readonly connection; results capped at 500 rows. Body: `{ sql }`. | +| GET / POST | `/api/admin/db/queries` | Saved SQL list / create (seeded with example queries). | +| PUT / DELETE | `/api/admin/db/queries/:id` | Update / delete a saved SQL. | +| POST | `/api/admin/db/json/scan` | Scan JSON dir (default `~/.innate`, override `ADMIN_JSON_DIR`) into the temp `json_documents` registry. | +| GET | `/api/admin/db/json/files` | Registered JSON files. | + +### Admin workspace (`/admin`) + +Simplified SQLite workbench served by the backend in API mode (not available in static/Pages mode): left = tables with row counts, center = CodeMirror SQL editor (highlighting, Ctrl/Cmd+Enter) + result table, right = saved SQL. On by default — **disable with `--no-admin` or `ADMIN_DB_UI=off`** (convenience script `bun run dev:no-admin`). Query logic lives in `backend/src/app/admin-db.ts`; UI in `frontend/src/pages/admin/page.tsx`. ### Sync pipeline @@ -367,7 +379,7 @@ Schema is defined in `backend/src/db/schema.sql`: - **ES modules** everywhere (`"type": "module"` in both `package.json` files). - **Strict TypeScript** is enabled (`strict: true`). - **Path alias**: `@/*` maps to `src/*` in the frontend. In the backend, local imports use relative paths with explicit `.js` extensions (e.g., `import { … } from "./db/index.js"`). -- **No test suite** is currently present. +- **Tests** run with `bun run test` (= `bun test`; vitest is not used because its bundler cannot parse bun-only `with { type: "text" }` imports). Test files: `backend/src/**/*.test.ts`. ### Backend @@ -406,6 +418,8 @@ Schema is defined in `backend/src/db/schema.sql`: | If you are… | Go to… | |---|---| | Adding or changing API endpoints | `backend/src/app/server.ts` | +| Changing the admin DB workspace API | `backend/src/app/admin-db.ts`, `backend/src/data/admin-saved-queries.ts` | +| Changing the admin workspace UI | `frontend/src/pages/admin/page.tsx`, `frontend/src/services/admin-db.ts` | | Changing how repos are fetched from GitHub | `backend/src/collector/github.ts` | | Changing trending/starred/window sync | `backend/src/collector/sync.ts`, `sync-window.ts`, `firecrawl.ts` | | Changing the 90-day window / README prefetch | `backend/src/collector/sync-window.ts` | @@ -438,7 +452,7 @@ Schema is defined in `backend/src/db/schema.sql`: - POST `/api/feeds/sync` validates request body with Zod schema before processing. - All GET API routes have try/catch error handling and return structured JSON errors. - The SQLite database path is controlled by the `DB_PATH` environment variable or defaults to `~/.innate/feeds.db`. Ensure the database file is not served or committed. -- Environment variables: `PORT` (backend port, default 4000), `DB_PATH` (SQLite path), `INNATE_HOME` (data directory, default `~/.innate`), `VITE_STATIC_MODE` (frontend static mode), `VITE_BASE_PATH` (base URL for GitHub Pages). +- Environment variables: `PORT` (backend port, default 4000), `DB_PATH` (SQLite path), `INNATE_HOME` (data directory, default `~/.innate`), `VITE_STATIC_MODE` (frontend static mode), `VITE_BASE_PATH` (base URL for GitHub Pages), `ADMIN_DB_UI` (`off` disables the admin workspace; also `--no-admin`), `ADMIN_JSON_DIR` (JSON scan root for the admin workspace, default `~/.innate`). ## Deployment notes diff --git a/backend/package.json b/backend/package.json index 5feb424e..52294cf8 100644 --- a/backend/package.json +++ b/backend/package.json @@ -5,6 +5,7 @@ "type": "module", "scripts": { "dev": "bun --watch src/app/server.ts", + "dev:no-admin": "bun --watch src/app/server.ts --no-admin", "build": "bun build src/app/server.ts --target=bun --minify --outdir=dist && bun build src/app/cli.ts --target=bun --minify --outfile=dist/cli.js", "start": "bun dist/server.js", "sync:daily": "bun src/app/cli.ts sync trending daily", diff --git a/backend/src/app/admin-db.test.ts b/backend/src/app/admin-db.test.ts new file mode 100644 index 00000000..e917561a --- /dev/null +++ b/backend/src/app/admin-db.test.ts @@ -0,0 +1,57 @@ +import { describe, expect, it } from "vitest"; +import { validateReadOnlySql } from "./admin-db.js"; + +describe("validateReadOnlySql", () => { + it("accepts plain SELECT", () => { + expect(validateReadOnlySql("SELECT * FROM starred_repos")).toBeNull(); + expect( + validateReadOnlySql(" select full_name, stars\nfrom starred_repos;\n"), + ).toBeNull(); + }); + + it("accepts WITH (CTE) queries", () => { + expect( + validateReadOnlySql("WITH t AS (SELECT 1 AS x) SELECT x FROM t"), + ).toBeNull(); + }); + + it("accepts commented SQL with trailing semicolon", () => { + expect( + validateReadOnlySql( + "-- get everything\nSELECT * FROM trending_repos -- row; hint\n;", + ), + ).toBeNull(); + }); + + it("rejects empty / comment-only input", () => { + expect(validateReadOnlySql("")).not.toBeNull(); + expect(validateReadOnlySql(" ")).not.toBeNull(); + expect(validateReadOnlySql("-- nothing here")).not.toBeNull(); + }); + + it("rejects multi-statement input", () => { + expect( + validateReadOnlySql("SELECT 1; DROP TABLE starred_repos"), + ).not.toBeNull(); + }); + + it("rejects write and schema statements", () => { + for (const sql of [ + "UPDATE starred_repos SET stars = 0", + "DELETE FROM starred_repos", + "INSERT INTO starred_repos VALUES (1)", + "DROP TABLE starred_repos", + "CREATE TABLE x (id INTEGER)", + "ATTACH DATABASE 'x.db' AS x", + "PRAGMA journal_mode = DELETE", + "VACUUM", + "REPLACE INTO starred_repos SELECT * FROM starred_repos", + ]) { + expect(validateReadOnlySql(sql), sql).not.toBeNull(); + } + }); + + it("rejects non-SELECT openers even without forbidden keywords", () => { + expect(validateReadOnlySql("EXPLAIN SELECT 1")).not.toBeNull(); + }); +}); diff --git a/backend/src/app/admin-db.ts b/backend/src/app/admin-db.ts new file mode 100644 index 00000000..72623dd3 --- /dev/null +++ b/backend/src/app/admin-db.ts @@ -0,0 +1,463 @@ +import { mkdirSync, readFileSync, readdirSync, statSync } from "fs"; +import { homedir } from "os"; +import { extname, join, resolve } from "path"; +import { Hono } from "hono"; +import { Database } from "bun:sqlite"; +import { z } from "zod"; +import { getDb } from "../db/index.js"; +import { getDefaultDbPath, INNATE_HOME } from "../db/paths.js"; +import { ensureAdminSavedQueries } from "../data/admin-saved-queries.js"; + +/** + * Simplified SQLite admin workspace API (`/api/admin/db/*`). + * + * Design constraints (see tasks/sprint-1/task-03-admin-api.md): + * - User SQL runs on a dedicated READONLY connection, so writes to the feed + * database are impossible regardless of statement validation. + * - Only a single SELECT/WITH statement is accepted (no multi-statements). + * - Results are capped (truncated flag) so a runaway query cannot flood the UI. + * - Saved queries live in `admin_saved_queries` in the main (writable) DB. + * - JSON file querying uses a TEMP `json_documents` registry on the readonly + * connection (research: tasks/sprint-1/research/json-query.md). + */ + +const DEFAULT_ROW_LIMIT = 500; +const MAX_SAVED_QUERIES = 200; +const JSON_FILE_MAX_BYTES = 1024 * 1024; + +// --------------------------------------------------------------------------- +// Read-only statement validation (defense in depth; the readonly connection +// is the hard guarantee). +// --------------------------------------------------------------------------- + +/** Strip `--` line comments and slash-star block comments so keyword checks see real SQL. */ +function stripSqlComments(sql: string): string { + return sql.replace(/--[^\n]*/g, " ").replace(/\/\*[\s\S]*?\*\//g, " "); +} + +/** + * Validate that `sql` is a single read-only SELECT/WITH statement. + * Returns an error message, or null when acceptable. + */ +export function validateReadOnlySql(rawSql: string): string | null { + const sql = rawSql.trim(); + if (!sql) return "SQL is empty"; + + const stripped = stripSqlComments(sql).trim(); + if (!stripped) return "SQL is empty (comments only)"; + + // Reject trailing delimiter + any further statement separators. + const withoutTrailing = stripped.replace(/;+\s*$/, ""); + if (withoutTrailing.includes(";")) { + return "Only a single statement is allowed"; + } + + const firstWord = withoutTrailing.split(/\s+/)[0]?.toUpperCase(); + if (firstWord !== "SELECT" && firstWord !== "WITH") { + return "Only SELECT / WITH statements are allowed"; + } + + // Belt-and-braces: these never appear in legitimate read-only queries. + const forbidden = + /\b(INSERT|UPDATE|DELETE|DROP|CREATE|ALTER|ATTACH|DETACH|PRAGMA|VACUUM|REINDEX|REPLACE|GRANT|REVOKE)\b/i; + if (forbidden.test(withoutTrailing)) { + return "Write / schema / pragma statements are not allowed"; + } + return null; +} + +// --------------------------------------------------------------------------- +// Readonly connection (lazy, one per process) +// --------------------------------------------------------------------------- + +let readonlyDb: Database | null = null; + +function getReadonlyDb(): Database { + if (!readonlyDb) { + const path = getDefaultDbPath(); + readonlyDb = new Database(path, { readonly: true, create: false }); + // TEMP tables (json_documents registry) live in the temp store, which is + // writable even on a readonly main database. + readonlyDb.exec( + `CREATE TEMP TABLE IF NOT EXISTS json_documents ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + path TEXT NOT NULL UNIQUE, + name TEXT NOT NULL, + size INTEGER NOT NULL, + content TEXT NOT NULL + )`, + ); + } + return readonlyDb; +} + +// --------------------------------------------------------------------------- +// Introspection helpers +// --------------------------------------------------------------------------- + +interface TableInfo { + name: string; + rowCount: number; + kind: "table" | "view"; + internal: boolean; +} + +/** FTS5 shadow tables (x_data / x_idx / x_content / x_docsize / x_config). */ +function isFtsShadowTable(name: string): boolean { + return /_(data|idx|content|docsize|config)$/.test(name); +} + +function listTables(db: Database): TableInfo[] { + const rows = db + .query( + `SELECT name, type FROM sqlite_master + WHERE type IN ('table','view') AND name NOT LIKE 'sqlite_%' + ORDER BY name`, + ) + .all() as { name: string; type: string }[]; + return rows.map(({ name, type }) => { + const rowCount = ( + db + .query(`SELECT COUNT(*) AS n FROM "${name.replaceAll('"', '""')}"`) + .get() as { + n: number; + } + ).n; + return { + name, + rowCount, + kind: type === "view" ? "view" : "table", + internal: isFtsShadowTable(name), + }; + }); +} + +function resolveTableName(db: Database, requested: string): string | null { + // Whitelist against sqlite_master — never interpolate raw input into PRAGMA. + const row = db + .query( + `SELECT name FROM sqlite_master + WHERE type IN ('table','view') AND name = ?`, + ) + .get(requested) as { name: string } | null; + return row?.name ?? null; +} + +// --------------------------------------------------------------------------- +// JSON file registry (tasks/sprint-1/research/json-query.md, 方案 A) +// --------------------------------------------------------------------------- + +export function getJsonScanRoot(): string { + return process.env.ADMIN_JSON_DIR?.trim() + ? resolve(process.env.ADMIN_JSON_DIR.trim()) + : INNATE_HOME; +} + +const JSON_SCAN_SKIP_DIRS = new Set([ + "node_modules", + "secrets", + "readmes", + ".git", +]); + +function scanJsonFiles( + db: Database, + root: string, +): { fileCount: number; totalBytes: number; skipped: number } { + const files: { path: string; name: string; size: number }[] = []; + let skipped = 0; + + const walk = (dir: string, depth: number): void => { + if (depth > 6) return; + let entries: string[]; + try { + entries = readdirSync(dir); + } catch { + return; + } + for (const entry of entries) { + if (entry.startsWith(".")) continue; + const full = join(dir, entry); + let st: ReturnType; + try { + st = statSync(full); + } catch { + continue; + } + if (st.isDirectory()) { + if (!JSON_SCAN_SKIP_DIRS.has(entry)) walk(full, depth + 1); + continue; + } + if (extname(entry).toLowerCase() !== ".json") continue; + if (st.size > JSON_FILE_MAX_BYTES || st.size === 0) { + skipped++; + continue; + } + files.push({ path: full, name: entry, size: st.size }); + } + }; + + walk(root, 0); + + db.exec("DELETE FROM json_documents"); + const insert = db.query( + "INSERT INTO json_documents (path, name, size, content) VALUES (?, ?, ?, ?)", + ); + let totalBytes = 0; + for (const file of files) { + try { + const content = readFileSync(file.path, "utf8"); + insert.run(file.path, file.name, file.size, content); + totalBytes += file.size; + } catch { + skipped++; + } + } + return { fileCount: files.length, totalBytes, skipped }; +} + +// --------------------------------------------------------------------------- +// Router +// --------------------------------------------------------------------------- + +export const adminDbRouter = new Hono(); + +/** List user tables + row counts. */ +adminDbRouter.get("/api/admin/db/tables", (c) => { + try { + return c.json({ tables: listTables(getReadonlyDb()) }); + } catch (err) { + const message = err instanceof Error ? err.message : "Database unavailable"; + return c.json({ error: message }, 503); + } +}); + +/** Columns + indexes + DDL for one table. */ +adminDbRouter.get("/api/admin/db/tables/:name/schema", (c) => { + try { + const db = getReadonlyDb(); + const name = resolveTableName(db, c.req.param("name")); + if (!name) { + return c.json({ error: `Unknown table: ${c.req.param("name")}` }, 404); + } + const quoted = `"${name.replaceAll('"', '""')}"`; + const columns = db.query(`PRAGMA table_info(${quoted})`).all() as Record< + string, + unknown + >[]; + const indexes = ( + db.query(`PRAGMA index_list(${quoted})`).all() as { + name: string; + unique: number; + origin: string; + }[] + ).map((idx) => ({ + ...idx, + columns: ( + db + .query(`PRAGMA index_info("${idx.name.replaceAll('"', '""')}")`) + .all() as { + name: string | null; + }[] + ).map((col) => col.name), + })); + const ddl = ( + db.query(`SELECT sql FROM sqlite_master WHERE name = ?`).get(name) as { + sql: string | null; + } + ).sql; + return c.json({ table: name, columns, indexes, ddl }); + } catch (err) { + const message = err instanceof Error ? err.message : "Introspection failed"; + return c.json({ error: message }, 500); + } +}); + +const QUERY_BODY_SCHEMA = z.object({ sql: z.string().min(1) }); + +/** Execute a single read-only statement, returning a table-shaped result. */ +adminDbRouter.post("/api/admin/db/query", async (c) => { + let body: unknown; + try { + body = await c.req.json(); + } catch { + return c.json({ error: "Invalid JSON body" }, 400); + } + const parsed = QUERY_BODY_SCHEMA.safeParse(body); + if (!parsed.success) { + return c.json({ error: "Body must be { sql: string }" }, 400); + } + + const violation = validateReadOnlySql(parsed.data.sql); + if (violation) { + return c.json({ error: violation }, 400); + } + + try { + const db = getReadonlyDb(); + const started = performance.now(); + const stmt = db.prepare(parsed.data.sql); + const rows = stmt.all() as Record[]; + const durationMs = Math.round((performance.now() - started) * 100) / 100; + const columns = + rows.length > 0 + ? Object.keys(rows[0]) + : ( + (stmt as unknown as { columns?: { name: string }[] }).columns ?? [] + ).map((col) => col.name); + const truncated = rows.length > DEFAULT_ROW_LIMIT; + return c.json({ + columns, + rows: truncated ? rows.slice(0, DEFAULT_ROW_LIMIT) : rows, + rowCount: rows.length, + truncated, + rowLimit: DEFAULT_ROW_LIMIT, + durationMs, + }); + } catch (err) { + const message = err instanceof Error ? err.message : "Query failed"; + return c.json({ error: message }, 400); + } +}); + +// Saved queries --------------------------------------------------------------- + +const SAVED_QUERY_BODY_SCHEMA = z.object({ + name: z.string().min(1).max(120), + sql: z.string().min(1), + description: z.string().max(500).optional(), +}); + +adminDbRouter.get("/api/admin/db/queries", (c) => { + try { + ensureAdminSavedQueries(getDb()); + const rows = getDb() + .query( + `SELECT id, name, sql, description, created_at AS createdAt, updated_at AS updatedAt + FROM admin_saved_queries ORDER BY updated_at DESC, id DESC`, + ) + .all(); + return c.json({ queries: rows }); + } catch (err) { + const message = + err instanceof Error ? err.message : "Failed to list queries"; + return c.json({ error: message }, 500); + } +}); + +adminDbRouter.post("/api/admin/db/queries", async (c) => { + let body: unknown; + try { + body = await c.req.json(); + } catch { + return c.json({ error: "Invalid JSON body" }, 400); + } + const parsed = SAVED_QUERY_BODY_SCHEMA.safeParse(body); + if (!parsed.success) { + return c.json({ error: "Body must be { name, sql, description? }" }, 400); + } + const count = ( + getDb().query("SELECT COUNT(*) AS n FROM admin_saved_queries").get() as { + n: number; + } + ).n; + if (count >= MAX_SAVED_QUERIES) { + return c.json({ error: "Saved query limit reached" }, 409); + } + const { name, sql, description } = parsed.data; + const result = getDb() + .query( + `INSERT INTO admin_saved_queries (name, sql, description) + VALUES (?, ?, ?)`, + ) + .run(name, sql, description ?? null); + const row = getDb() + .query( + `SELECT id, name, sql, description, created_at AS createdAt, updated_at AS updatedAt + FROM admin_saved_queries WHERE id = ?`, + ) + .get(result.lastInsertRowid); + return c.json({ query: row }, 201); +}); + +adminDbRouter.put("/api/admin/db/queries/:id", async (c) => { + const id = parseInt(c.req.param("id"), 10); + if (!Number.isInteger(id)) { + return c.json({ error: "Invalid id" }, 400); + } + let body: unknown; + try { + body = await c.req.json(); + } catch { + return c.json({ error: "Invalid JSON body" }, 400); + } + const parsed = SAVED_QUERY_BODY_SCHEMA.safeParse(body); + if (!parsed.success) { + return c.json({ error: "Body must be { name, sql, description? }" }, 400); + } + const { name, sql, description } = parsed.data; + const result = getDb() + .query( + `UPDATE admin_saved_queries + SET name = ?, sql = ?, description = ?, updated_at = datetime('now') + WHERE id = ?`, + ) + .run(name, sql, description ?? null, id); + if (result.changes === 0) { + return c.json({ error: "Saved query not found" }, 404); + } + const row = getDb() + .query( + `SELECT id, name, sql, description, created_at AS createdAt, updated_at AS updatedAt + FROM admin_saved_queries WHERE id = ?`, + ) + .get(id); + return c.json({ query: row }); +}); + +adminDbRouter.delete("/api/admin/db/queries/:id", (c) => { + const id = parseInt(c.req.param("id"), 10); + if (!Number.isInteger(id)) { + return c.json({ error: "Invalid id" }, 400); + } + const result = getDb() + .query("DELETE FROM admin_saved_queries WHERE id = ?") + .run(id); + if (result.changes === 0) { + return c.json({ error: "Saved query not found" }, 404); + } + return c.json({ ok: true }); +}); + +// JSON file registry ------------------------------------------------------------ + +adminDbRouter.get("/api/admin/db/json/files", (c) => { + try { + const db = getReadonlyDb(); + const files = db + .query( + `SELECT id, path, name, size, + json_valid(content) AS valid + FROM json_documents ORDER BY path`, + ) + .all(); + return c.json({ root: getJsonScanRoot(), files }); + } catch (err) { + const message = + err instanceof Error ? err.message : "Scan registry unavailable"; + return c.json({ error: message }, 503); + } +}); + +/** Rebuild the json_documents registry from the configured scan root. */ +adminDbRouter.post("/api/admin/db/json/scan", (c) => { + try { + const root = getJsonScanRoot(); + mkdirSync(root, { recursive: true }); + const stats = scanJsonFiles(getReadonlyDb(), root); + return c.json({ root, ...stats }); + } catch (err) { + const message = err instanceof Error ? err.message : "JSON scan failed"; + return c.json({ error: message }, 500); + } +}); diff --git a/backend/src/app/openapi.ts b/backend/src/app/openapi.ts index 629edea6..c2f80609 100644 --- a/backend/src/app/openapi.ts +++ b/backend/src/app/openapi.ts @@ -46,6 +46,10 @@ export const openApiSpec = { { name: "Auth", description: "GitHub 认证状态与 PAT" }, { name: "Sync", description: "触发数据同步（勿对外暴露）" }, { name: "Hidden", description: "隐藏 / 取消隐藏条目" }, + { + name: "AdminDb", + description: "简化版 SQLite 后台（只读查询 / 保存 SQL / JSON 文件查询）", + }, ], paths: { "/api/health": { @@ -752,6 +756,194 @@ export const openApiSpec = { }, }, }, + + "/api/admin/db/tables": { + get: { + tags: ["AdminDb"], + summary: "列出全部表与行数", + description: + "简化版 SQLite 后台。含 FTS 影子表（`internal: true` 标记）。只读连接执行。", + responses: { + "200": { + description: "表清单", + content: { + "application/json": { + schema: { + type: "object", + properties: { + tables: { + type: "array", + items: { $ref: "#/components/schemas/AdminDbTable" }, + }, + }, + }, + }, + }, + }, + "503": jsonError("数据库不可用"), + }, + }, + }, + + "/api/admin/db/tables/{name}/schema": { + get: { + tags: ["AdminDb"], + summary: "单表结构（列 / 索引 / DDL）", + parameters: [ + { + name: "name", + in: "path", + required: true, + schema: { type: "string" }, + description: "表名（按 sqlite_master 白名单校验）", + }, + ], + responses: { + "200": { description: "表结构" }, + "404": jsonError("表不存在"), + }, + }, + }, + + "/api/admin/db/query": { + post: { + tags: ["AdminDb"], + summary: "执行只读 SQL", + description: + "仅接受单条 SELECT / WITH 语句；在独立只读连接上执行（写操作被 SQLite 硬拒绝）。" + + "结果超过 500 行时截断（`truncated: true`）。", + requestBody: { + required: true, + content: { + "application/json": { + schema: { $ref: "#/components/schemas/AdminDbQueryBody" }, + }, + }, + }, + responses: { + "200": { + description: "查询结果（表格形）", + content: { + "application/json": { + schema: { $ref: "#/components/schemas/AdminDbQueryResult" }, + }, + }, + }, + "400": jsonError("SQL 校验失败或执行错误"), + }, + }, + }, + + "/api/admin/db/queries": { + get: { + tags: ["AdminDb"], + summary: "保存的 SQL 列表", + description: "首表时自动预置 R1 调研结论的示例查询。", + responses: { + "200": { description: "保存的 SQL 列表" }, + }, + }, + post: { + tags: ["AdminDb"], + summary: "新增保存的 SQL", + requestBody: { + required: true, + content: { + "application/json": { + schema: { $ref: "#/components/schemas/AdminDbSavedQueryBody" }, + }, + }, + }, + responses: { + "201": { description: "已创建" }, + "400": jsonError("请求体不合法"), + "409": jsonError("超出保存上限（200 条）"), + }, + }, + }, + + "/api/admin/db/queries/{id}": { + put: { + tags: ["AdminDb"], + summary: "更新保存的 SQL", + parameters: [ + { + name: "id", + in: "path", + required: true, + schema: { type: "integer" }, + }, + ], + requestBody: { + required: true, + content: { + "application/json": { + schema: { $ref: "#/components/schemas/AdminDbSavedQueryBody" }, + }, + }, + }, + responses: { + "200": { description: "已更新" }, + "404": jsonError("不存在"), + }, + }, + delete: { + tags: ["AdminDb"], + summary: "删除保存的 SQL", + parameters: [ + { + name: "id", + in: "path", + required: true, + schema: { type: "integer" }, + }, + ], + responses: { + "200": { description: "已删除" }, + "404": jsonError("不存在"), + }, + }, + }, + + "/api/admin/db/json/files": { + get: { + tags: ["AdminDb"], + summary: "已注册的 JSON 文件清单", + description: "见 `POST /api/admin/db/json/scan`。", + responses: { + "200": { description: "文件清单（`valid` 为 json_valid 校验结果）" }, + }, + }, + }, + + "/api/admin/db/json/scan": { + post: { + tags: ["AdminDb"], + summary: "扫描 JSON 目录 → 注册进 json_documents 表", + description: + "递归扫描扫描根（默认 `~/.innate`，`ADMIN_JSON_DIR` 可覆盖），跳过 " + + "node_modules / secrets / 隐藏目录，单文件上限 1MB。" + + "注册表位于只读连接的 TEMP store，可用 `json_each` 等函数查询。", + responses: { + "200": { + description: "扫描结果", + content: { + "application/json": { + schema: { + type: "object", + properties: { + root: { type: "string" }, + fileCount: { type: "integer" }, + totalBytes: { type: "integer" }, + skipped: { type: "integer" }, + }, + }, + }, + }, + }, + }, + }, + }, }, components: { @@ -896,6 +1088,49 @@ export const openApiSpec = { installCommand: { type: "string", nullable: true }, }, }, + + AdminDbTable: { + type: "object", + properties: { + name: { type: "string" }, + rowCount: { type: "integer" }, + kind: { type: "string", enum: ["table", "view"] }, + internal: { type: "boolean", description: "FTS 影子表等内部表" }, + }, + }, + + AdminDbQueryBody: { + type: "object", + properties: { + sql: { type: "string", description: "单条 SELECT / WITH 语句" }, + }, + required: ["sql"], + }, + + AdminDbQueryResult: { + type: "object", + properties: { + columns: { type: "array", items: { type: "string" } }, + rows: { + type: "array", + items: { type: "object", additionalProperties: true }, + }, + rowCount: { type: "integer" }, + truncated: { type: "boolean" }, + rowLimit: { type: "integer" }, + durationMs: { type: "number" }, + }, + }, + + AdminDbSavedQueryBody: { + type: "object", + properties: { + name: { type: "string", maxLength: 120 }, + sql: { type: "string" }, + description: { type: "string", maxLength: 500, nullable: true }, + }, + required: ["name", "sql"], + }, }, }, } as const; diff --git a/backend/src/app/server.ts b/backend/src/app/server.ts index ae2ddbb5..05193c7e 100644 --- a/backend/src/app/server.ts +++ b/backend/src/app/server.ts @@ -5,6 +5,7 @@ import { cors } from "hono/cors"; import { swaggerUI } from "@hono/swagger-ui"; import { z } from "zod"; import { getOpenApiSpec } from "./openapi.js"; +import { adminDbRouter } from "./admin-db.js"; import { getDb, getFeedItems, @@ -35,6 +36,21 @@ const app = new Hono(); app.use("/*", cors()); +/** + * Simplified SQLite admin workspace (/admin UI assets + /api/admin/db/*). + * On by default (issue #4: 默认启动后台管理查询); disable with + * `--no-admin` or `ADMIN_DB_UI=off`. + */ +export const adminDbEnabled = !( + process.argv.includes("--no-admin") || + ["off", "false", "0"].includes( + (process.env.ADMIN_DB_UI || "").trim().toLowerCase(), + ) +); +if (adminDbEnabled) { + app.route("/", adminDbRouter); +} + /** Liveness probe (opens the database). */ app.get("/api/health", (c) => { try { @@ -444,6 +460,11 @@ app.get( * Serve the Vite production build (frontend/dist) from the same origin as /api. * Client routes like /trending fall back to index.html. */ +/** Windows `resolve()` emits backslashes; normalize so prefix checks work. */ +function toPosix(p: string): string { + return p.replace(/\\/g, "/"); +} + function mountWebsite(app: Hono): string | null { const distDir = process.env.FRONTEND_DIST?.trim() ? resolve(process.env.FRONTEND_DIST.trim()) @@ -453,12 +474,12 @@ function mountWebsite(app: Hono): string | null { return null; } - const distRoot = resolve(distDir); + const distRoot = toPosix(resolve(distDir)); const resolvePublicFile = (pathname: string): string | null => { const relative = pathname === "/" ? "index.html" : pathname.replace(/^\/+/, ""); - const candidate = resolve(distRoot, relative); + const candidate = toPosix(resolve(distRoot, relative)); if (candidate !== distRoot && !candidate.startsWith(`${distRoot}/`)) { return null; } @@ -491,6 +512,9 @@ const listenLabel = hostname : `http://localhost:${port}`; const webRoot = mountWebsite(app); console.log(`Feeds server running on ${listenLabel}`); +console.log( + `Admin DB workspace: ${adminDbEnabled ? "enabled" : "disabled"} (default enabled; --no-admin / ADMIN_DB_UI=off to disable)`, +); if (webRoot) { console.log(`Website: ${listenLabel} (from ${webRoot})`); } else { diff --git a/backend/src/data/admin-saved-queries.ts b/backend/src/data/admin-saved-queries.ts new file mode 100644 index 00000000..70074f13 --- /dev/null +++ b/backend/src/data/admin-saved-queries.ts @@ -0,0 +1,87 @@ +import type { Database } from "bun:sqlite"; + +/** + * Saved SQL queries for the admin workspace (`/api/admin/db/queries`). + * Lives in the main (writable) database; created alongside the core schema. + * + * Example seeds come from the R1 similarity-query research + * (tasks/sprint-1/research/similarity-query.md). + */ +const SEED_EXAMPLES: { name: string; description: string; sql: string }[] = [ + { + name: "AI Agent · stars > 10000（topic LIKE 兜底）", + description: + "推荐：覆盖 ai-agent / ai-agents / *-agent 等所有含 agent 的 topic", + sql: `SELECT s.full_name, s.stars, s.language, + GROUP_CONCAT(t.topic) AS topics +FROM starred_repos s +JOIN starred_repo_topics t ON t.repo_id = s.id +WHERE t.topic LIKE '%agent%' + AND s.stars > 10000 +GROUP BY s.id +ORDER BY s.stars DESC`, + }, + { + name: "AI Agent · 精确 topic 白名单", + description: "严格版：只匹配人工确认的 AI Agent topic 集合", + sql: `SELECT s.full_name, s.stars, s.language, + GROUP_CONCAT(t.topic) AS topics +FROM starred_repos s +JOIN starred_repo_topics t ON t.repo_id = s.id +WHERE t.topic IN ('ai-agent','ai-agents','agent','agents', + 'agentic-ai','agentic','llm-agent','multi-agent', + 'autonomous-agents','coding-agent','agent-framework') + AND s.stars > 10000 +GROUP BY s.id +ORDER BY s.stars DESC`, + }, + { + name: "agent 关键词 · FTS5 BM25 相关性", + description: "名称/描述模糊匹配 + BM25 排序（相似度检索）", + sql: `SELECT starred_repos.full_name, starred_repos.stars, + starred_repos.description, + bm25(starred_repos_fts) AS rank +FROM starred_repos_fts +JOIN starred_repos ON starred_repos.rowid = starred_repos_fts.rowid +WHERE starred_repos_fts MATCH 'agent' + AND starred_repos.stars > 10000 +ORDER BY rank +LIMIT 50`, + }, + { + name: "JSON 文件 · json_each 展开条目", + description: + "先执行 POST /api/admin/db/json/scan，再用本查询按行展开 JSON 文件的 items 数组", + sql: `SELECT jd.name AS file, + json_extract(je.value, '$.title') AS title, + json_extract(je.value, '$.tagline') AS tagline +FROM json_documents jd, json_each(jd.content, '$.items') je +WHERE jd.name LIKE 'batch-%' +LIMIT 100`, + }, +]; + +/** Create the table when missing and seed examples into an empty table. */ +export function ensureAdminSavedQueries(db: Database): void { + db.exec(`CREATE TABLE IF NOT EXISTS admin_saved_queries ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + name TEXT NOT NULL, + sql TEXT NOT NULL, + description TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) + )`); + const count = ( + db.query("SELECT COUNT(*) AS n FROM admin_saved_queries").get() as { + n: number; + } + ).n; + if (count === 0) { + const insert = db.query( + "INSERT INTO admin_saved_queries (name, sql, description) VALUES (?, ?, ?)", + ); + for (const example of SEED_EXAMPLES) { + insert.run(example.name, example.sql, example.description); + } + } +} diff --git a/frontend/package.json b/frontend/package.json index 0d673c80..eda3bfa3 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -12,8 +12,12 @@ "shadcn:update": "bash scripts/update-shadcn.sh" }, "dependencies": { + "@codemirror/lang-sql": "^6.10.0", + "@innate/ui": "^0.1.0", + "@innate/web-domain": "^0.1.0", "@tanstack/react-router": "^1.132.0", "@tanstack/react-table": "^8.21.3", + "@uiw/react-codemirror": "^4.25.12", "clsx": "^2.1.1", "date-fns": "^4.4.0", "lucide-react": "^0.500.0", @@ -24,10 +28,8 @@ "remark-gfm": "^4.0.1", "sonner": "^2.0.0", "tailwind-merge": "^3.6.0", - "zod": "^3.24.0", "tw-animate-css": "^1.4.0", - "@innate/ui": "^0.1.0", - "@innate/web-domain": "^0.1.0" + "zod": "^3.24.0" }, "devDependencies": { "@tailwindcss/node": "^4.3.3", diff --git a/frontend/src/host-shell.tsx b/frontend/src/host-shell.tsx index 59af3ac8..72cb91dd 100644 --- a/frontend/src/host-shell.tsx +++ b/frontend/src/host-shell.tsx @@ -100,10 +100,7 @@ export function HostShell({ return unsub; }, [router]); - const groups = useMemo( - () => groupNavItems(navItemsOf(domain)), - [domain], - ); + const groups = useMemo(() => groupNavItems(navItemsOf(domain)), [domain]); // 含当前路由的分组默认展开；其余保持收起状态 const [collapsed, setCollapsed] = useState>({}); diff --git a/frontend/src/main.tsx b/frontend/src/main.tsx index f996d109..bf22fee0 100644 --- a/frontend/src/main.tsx +++ b/frontend/src/main.tsx @@ -18,7 +18,10 @@ const router = createStandaloneWebDomainRouter(webDomain, { function App() { return ( - [0]["router"]}> + [0]["router"]} + > ); diff --git a/frontend/src/pages/admin/page.tsx b/frontend/src/pages/admin/page.tsx new file mode 100644 index 00000000..f962d642 --- /dev/null +++ b/frontend/src/pages/admin/page.tsx @@ -0,0 +1,472 @@ +import { useCallback, useEffect, useMemo, useState } from "react"; +import CodeMirror from "@uiw/react-codemirror"; +import { sql as sqlLanguage } from "@codemirror/lang-sql"; +import { + Database, + FileJson, + Play, + RefreshCw, + Save, + Trash2, +} from "lucide-react"; +import { toast } from "sonner"; +import { + createSavedQuery, + deleteSavedQuery, + fetchAdminTables, + fetchJsonFiles, + fetchSavedQueries, + fetchTableSchema, + runAdminQuery, + scanJsonFiles, + updateSavedQuery, + type AdminDbJsonFile, + type AdminDbQueryResult, + type AdminDbSavedQuery, + type AdminDbTable, + type AdminDbTableSchema, +} from "@/services/admin-db"; + +function formatCount(n: number): string { + return n.toLocaleString("en-US"); +} + +function previewValue(value: unknown): string { + if (value === null || value === undefined) return "NULL"; + if (typeof value === "object") { + const json = JSON.stringify(value); + return json.length > 120 ? `${json.slice(0, 120)}…` : json; + } + const text = String(value); + return text.length > 120 ? `${text.slice(0, 120)}…` : text; +} + +export function AdminPage() { + const [tables, setTables] = useState([]); + const [tablesError, setTablesError] = useState(null); + const [selectedTable, setSelectedTable] = useState(null); + const [tableSchema, setTableSchema] = useState( + null, + ); + + const [sqlText, setSqlText] = useState( + "SELECT s.full_name, s.stars, s.language,\n GROUP_CONCAT(t.topic) AS topics\nFROM starred_repos s\nJOIN starred_repo_topics t ON t.repo_id = s.id\nWHERE t.topic LIKE '%agent%'\n AND s.stars > 10000\nGROUP BY s.id\nORDER BY s.stars DESC", + ); + const [running, setRunning] = useState(false); + const [result, setResult] = useState(null); + const [queryError, setQueryError] = useState(null); + + const [savedQueries, setSavedQueries] = useState([]); + const [activeSavedId, setActiveSavedId] = useState(null); + const [savingName, setSavingName] = useState(""); + const [showSaveInput, setShowSaveInput] = useState(false); + + const [jsonFiles, setJsonFiles] = useState([]); + const [scanning, setScanning] = useState(false); + + const runQuery = useCallback( + async (text?: string) => { + const query = (text ?? sqlText).trim(); + if (!query) return; + setRunning(true); + setQueryError(null); + try { + const res = await runAdminQuery(query); + setResult(res); + } catch (err) { + setQueryError(err instanceof Error ? err.message : "查询失败"); + } finally { + setRunning(false); + } + }, + [sqlText], + ); + + const loadTables = useCallback(async () => { + try { + const { tables: list } = await fetchAdminTables(); + setTables(list); + setTablesError(null); + } catch (err) { + setTablesError( + err instanceof Error + ? err.message + : "无法连接 admin API（需要 API 模式后端）", + ); + } + }, []); + + const loadSavedQueries = useCallback(async () => { + try { + const { queries } = await fetchSavedQueries(); + setSavedQueries(queries); + } catch { + // Non-fatal: the saved-SQL column just stays empty. + } + }, []); + + const loadJsonFiles = useCallback(async () => { + try { + const { files } = await fetchJsonFiles(); + setJsonFiles(files); + } catch { + setJsonFiles([]); + } + }, []); + + useEffect(() => { + loadTables(); + loadSavedQueries(); + loadJsonFiles(); + }, [loadTables, loadSavedQueries, loadJsonFiles]); + + useEffect(() => { + const handler = (event: KeyboardEvent): void => { + if ((event.ctrlKey || event.metaKey) && event.key === "Enter") { + event.preventDefault(); + runQuery(); + } + }; + window.addEventListener("keydown", handler); + return () => window.removeEventListener("keydown", handler); + }, [runQuery]); + + const selectTable = async (name: string): Promise => { + setSelectedTable(name); + setSqlText(`SELECT * FROM "${name}" LIMIT 100`); + setActiveSavedId(null); + try { + setTableSchema(await fetchTableSchema(name)); + } catch { + setTableSchema(null); + } + }; + + const loadSaved = (saved: AdminDbSavedQuery): void => { + setSqlText(saved.sql); + setActiveSavedId(saved.id); + setSelectedTable(null); + setTableSchema(null); + }; + + const saveCurrent = async (asNew: boolean): Promise => { + const name = + savingName.trim() || + savedQueries.find((q) => q.id === activeSavedId)?.name || + ""; + if (!name) { + setShowSaveInput(true); + toast.error("请输入保存名称"); + return; + } + try { + if (!asNew && activeSavedId !== null) { + await updateSavedQuery(activeSavedId, { name, sql: sqlText }); + toast.success("已更新保存的 SQL"); + } else { + const { query } = await createSavedQuery({ name, sql: sqlText }); + setActiveSavedId(query.id); + toast.success("已保存 SQL"); + } + setSavingName(""); + setShowSaveInput(false); + await loadSavedQueries(); + } catch (err) { + toast.error(err instanceof Error ? err.message : "保存失败"); + } + }; + + const removeSaved = async (id: number): Promise => { + try { + await deleteSavedQuery(id); + if (activeSavedId === id) setActiveSavedId(null); + await loadSavedQueries(); + } catch (err) { + toast.error(err instanceof Error ? err.message : "删除失败"); + } + }; + + const rescanJson = async (): Promise => { + setScanning(true); + try { + const stats = await scanJsonFiles(); + toast.success( + `JSON 扫描完成：${stats.fileCount} 个文件（跳过 ${stats.skipped}）`, + ); + await loadJsonFiles(); + } catch (err) { + toast.error(err instanceof Error ? err.message : "扫描失败"); + } finally { + setScanning(false); + } + }; + + const userTables = useMemo(() => tables.filter((t) => !t.internal), [tables]); + const darkMode = + typeof document !== "undefined" && + document.documentElement.classList.contains("dark"); + + return ( +
+
+
+

SQLite 工作台

+

+ 只读查询 · Ctrl/Cmd + Enter 执行 · 结果上限 500 行 +

+
+
+ +
+ {/* Left: tables + JSON registry */} + + + {/* Center: editor + results */} +
+
+
+ + SQL 编辑器 + +
+ {showSaveInput && ( + setSavingName(event.target.value)} + placeholder="保存名称" + className="bg-background h-6 w-36 rounded border px-2 text-xs" + autoFocus + /> + )} + + {activeSavedId !== null && ( + + )} + +
+
+ +
+ + {queryError && ( +
+ {queryError} +
+ )} + +
+
+ 结果 + {result && ( + + {formatCount(result.rowCount)} 行 ·{" "} + {result.durationMs.toFixed(1)} ms + {result.truncated + ? ` · 已截断（上限 ${formatCount(result.rowLimit)}）` + : ""} + + )} +
+
+ {result && result.rows.length > 0 ? ( + + + + {result.columns.map((column) => ( + + ))} + + + + {result.rows.map((row, index) => ( + + {result.columns.map((column) => ( + + ))} + + ))} + +
+ {column} +
+ {previewValue(row[column])} +
+ ) : result ? ( +

+ {result.rowCount} 行（无结果行） +

+ ) : ( +

+ 执行一条查询，或点击左侧表名快速预览。 +

+ )} +
+
+
+ + {/* Right: saved SQL */} + +
+
+ ); +} diff --git a/frontend/src/services/admin-db.ts b/frontend/src/services/admin-db.ts new file mode 100644 index 00000000..b151803b --- /dev/null +++ b/frontend/src/services/admin-db.ts @@ -0,0 +1,133 @@ +/** + * Admin workspace API client (`/api/admin/db/*`, backend/src/app/admin-db.ts). + * API-mode only: the static (GitHub Pages) deployment has no database to query. + */ + +const API_BASE = "/api/admin/db"; + +export interface AdminDbTable { + name: string; + rowCount: number; + kind: "table" | "view"; + internal: boolean; +} + +export interface AdminDbColumn { + cid: number; + name: string; + type: string; + notnull: number; + dflt_value: string | null; + pk: number; +} + +export interface AdminDbIndex { + name: string; + unique: number; + origin: string; + columns: (string | null)[]; +} + +export interface AdminDbTableSchema { + table: string; + columns: AdminDbColumn[]; + indexes: AdminDbIndex[]; + ddl: string | null; +} + +export interface AdminDbQueryResult { + columns: string[]; + rows: Record[]; + rowCount: number; + truncated: boolean; + rowLimit: number; + durationMs: number; +} + +export interface AdminDbSavedQuery { + id: number; + name: string; + sql: string; + description: string | null; + createdAt: string; + updatedAt: string; +} + +export interface AdminDbJsonFile { + id: number; + path: string; + name: string; + size: number; + valid: number; +} + +export interface AdminDbJsonScanResult { + root: string; + fileCount: number; + totalBytes: number; + skipped: number; +} + +async function request(path: string, init?: RequestInit): Promise { + const res = await fetch(`${API_BASE}${path}`, { + headers: init?.body ? { "Content-Type": "application/json" } : undefined, + ...init, + }); + const data = (await res.json().catch(() => null)) as + (T & { error?: string }) | null; + if (!res.ok) { + throw new Error(data?.error || `Request failed (${res.status})`); + } + return data as T; +} + +export function fetchAdminTables(): Promise<{ tables: AdminDbTable[] }> { + return request("/tables"); +} + +export function fetchTableSchema(name: string): Promise { + return request(`/tables/${encodeURIComponent(name)}/schema`); +} + +export function runAdminQuery(sql: string): Promise { + return request("/query", { method: "POST", body: JSON.stringify({ sql }) }); +} + +export function fetchSavedQueries(): Promise<{ + queries: AdminDbSavedQuery[]; +}> { + return request("/queries"); +} + +export function createSavedQuery(body: { + name: string; + sql: string; + description?: string; +}): Promise<{ query: AdminDbSavedQuery }> { + return request("/queries", { method: "POST", body: JSON.stringify(body) }); +} + +export function updateSavedQuery( + id: number, + body: { name: string; sql: string; description?: string }, +): Promise<{ query: AdminDbSavedQuery }> { + return request(`/queries/${id}`, { + method: "PUT", + body: JSON.stringify(body), + }); +} + +export function deleteSavedQuery(id: number): Promise<{ ok: boolean }> { + return request(`/queries/${id}`, { method: "DELETE" }); +} + +export function scanJsonFiles(): Promise { + return request("/json/scan", { method: "POST" }); +} + +export function fetchJsonFiles(): Promise<{ + root: string; + files: AdminDbJsonFile[]; +}> { + return request("/json/files"); +} diff --git a/frontend/src/web-domain.tsx b/frontend/src/web-domain.tsx index f53052fe..2edc7889 100644 --- a/frontend/src/web-domain.tsx +++ b/frontend/src/web-domain.tsx @@ -6,6 +6,7 @@ import { TrendingPage } from "@/pages/trending/page"; import { StarredPage } from "@/pages/starred/page"; import { DigestPage } from "@/pages/digest/page"; import { SettingsPage } from "@/pages/settings/page"; +import { AdminPage } from "@/pages/admin/page"; import { DshPluginsPage } from "@/pages/dsh/page"; import { DshCategoriesPage } from "@/pages/dsh/categories-page"; import { DshCategoryPage } from "@/pages/dsh/category-page"; @@ -111,6 +112,11 @@ export const webDomain = createWebDomain("feeds") path: "/settings", component: SettingsPage, }), + createRoute({ + getParentRoute: () => layout, + path: "/admin", + component: AdminPage, + }), ]; return layout.addChildren(routes as never) as unknown as AnyRoute; @@ -144,6 +150,13 @@ export const webDomain = createWebDomain("feeds") order: 30, group: "DeepSeek Harness", }, + { + label: "SQLite", + path: "/admin", + icon: "Database", + order: 40, + group: "Admin", + }, ]) .addCapability("homeTile", { title: "Feeds", diff --git a/package.json b/package.json index 5434e06f..ae3697e3 100644 --- a/package.json +++ b/package.json @@ -29,9 +29,9 @@ "export:static": "cd backend && bun run export:incremental", "build:static": "cd backend && bun run export:incremental && cd ../frontend && VITE_STATIC_MODE=true bun run build", "build:pages": "VITE_BASE_PATH=${VITE_BASE_PATH:-/} VITE_STATIC_MODE=true bun run build:static", - "install:all": "bun install", - "test": "bun --bun vitest run", - "test:watch": "bun --bun vitest", + "install:all": "pnpm install", + "test": "bun test", + "test:watch": "bun test --watch", "typecheck": "cd backend && bun run typecheck && cd ../frontend && bun run typecheck", "setup:innate": "innate-fe-cli install --dir frontend && cp frontend/.npmrc .npmrc", "data:seed": "cd backend && bun run import:static", diff --git a/tasks/sprint-1/context.md b/tasks/sprint-1/context.md new file mode 100644 index 00000000..03ce30d7 --- /dev/null +++ b/tasks/sprint-1/context.md @@ -0,0 +1,53 @@ +# Sprint-1：项目初始化 + 数据导入 + 简化版 SQLite 后台 + +**来源 Issue:** https://github.com/variableway/innate-feeds/issues/4 +**状态:** in_progress +**开始:** 2026-10-23 +**状态总表:** [`status.md`](./status.md)（任何 Agent 完成任务后必须更新） + +## Sprint 目标 + +1. 初始化当前项目（依赖安装、typecheck 通过、可启动）。 +2. 更新最新数据，同步进本地 SQLite（`~/.innate/feeds.db`）。 +3. 交付一个简化版 SQLite 后台工作台（admin workspace）： + - 左侧：所有表（含行数）。 + - 中间上方：SQL 编辑区，支持 SQL 语法高亮；下方为查询结果表格。 + - 最右侧：保存的 SQL（加载 / 保存 / 删除）。 + - 支持相似度查询方案（如：查 tag 含 AI Agent 且 stars > 10000 的 GitHub repo）。 + - 支持对 JSON 文件的 SQL 查询（JSON 文件集中一处，SQL 查询文件内容，表格展示）。 + - 启动由参数控制，**默认启动后台管理查询**。 + +## 任务拆分与依赖 + +| Task | 文档 | GitHub Issue | 依赖 | 分组 | +|---|---|---|---|---| +| R1 相似度查询方案调研 | [research/similarity-query.md](./research/similarity-query.md) | #5 | 无 | research | +| R2 JSON 文件 SQL 查询方案调研 | [research/json-query.md](./research/json-query.md) | #6 | 无 | research | +| T1 项目初始化 | [task-01-init-project.md](./task-01-init-project.md) | #7 | 无 | 基础 | +| T2 数据同步进 SQLite | [task-02-sync-data.md](./task-02-sync-data.md) | #8 | T1 | 基础 | +| T3 Admin DB API | [task-03-admin-api.md](./task-03-admin-api.md) | #9 | T1、R1、R2 | admin 组 | +| T4 Admin UI 三栏工作台 | [task-04-admin-ui.md](./task-04-admin-ui.md) | #10 | T3 | admin 组 | +| T5 启动参数控制 | [task-05-server-flag.md](./task-05-server-flag.md) | #11 | T3 | admin 组 | + +**独立性说明：** + +- R1 / R2 / T1 互相独立，可由不同 Agent 并行认领。 +- T2 依赖 T1（需要依赖安装完成），但只依赖 `bun install`，冲突面极小。 +- T3 / T4 / T5 同属 **admin 组**，不是三个完全独立的功能：T3（API）是 T4（UI）与 T5（启动参数）的前置。优先级 T3 > T4 > T5；T4 与 T5 在 T3 完成后可并行。 +- 所有新增 Issue 统一打 label `feeds-sprint-1`。 + +## 工作流（给执行 Agent 的约定） + +1. 开始前先读 [`status.md`](./status.md)，确认目标 task 是否已被完成；**已完成则跳到下一个可做的 task**。 +2. 认领：把 status.md 中对应行改为 `in_progress` 并写上 agent 名。 +3. 调研类产出物放 [`research/`](./research/)；实现类改动直接进代码，但 task 文档的「验收记录」段落必须回填。 +4. 完成后：task 文档勾选验收项 → status.md 置 `done` 并附提交 SHA → 有 GitHub Issue 的同时关闭 Issue。 +5. 遇到阻塞不要空转：在 status.md 标 `blocked` 并写明原因，换下一个无依赖的 task。 +6. 新增 task 时：新建 task 文档 + GitHub Issue（label `feeds-sprint-1`）+ 更新本文件任务表与 status.md。 + +## 相关背景 + +- 数据库：`~/.innate/feeds.db`（bun:sqlite，WAL），schema 见 `backend/src/db/schema.sql`；已有 FTS5 镜像表（`trending_repos_fts` / `starred_repos_fts`）可直接用于关键词查询。 +- 后端：Hono 4（`backend/src/app/server.ts`），已挂 Swagger（`/docs`）。 +- 前端：`@innate/domain-feeds` 域插件，TanStack Router 手动注册路由于 `frontend/src/web-domain.tsx`；构建走 Bun（`frontend/scripts/`），无 Vite。 +- 外部 JSON 数据源：Product Hunt / YC / a16z / GitHub Topics 存于 `~/.innate//`（见 `docs/arch-design/external-collectors.md`），是 JSON 查询功能的数据面。 diff --git a/tasks/sprint-1/research/json-query.md b/tasks/sprint-1/research/json-query.md new file mode 100644 index 00000000..430dfc32 --- /dev/null +++ b/tasks/sprint-1/research/json-query.md @@ -0,0 +1,72 @@ +# R2：JSON 文件 SQL 查询方案调研 + +- **Issue:** https://github.com/variableway/innate-feeds/issues/6 +- **结论日期:** 2026-10-23 +- **验证环境:** bun:sqlite（Bun 1.4.3），真实文件验证 + +## 需求场景 + +> 所有 JSON 文件放一个地方，直接通过 SQL 查询文件内容，用表格方式展示。 + +本仓的 JSON 数据面：外部收集器（Product Hunt / YC / a16z / GitHub Topics）写入 `~/.innate//*.json`（见 `docs/arch-design/external-collectors.md`），静态导出数据在 `frontend/public/data/`。这些数据目前只能整文件读，无法查询。 + +## 方案对比 + +### 方案 A：SQLite JSON1 + "文件注册表"内存表（✅ 推荐，已验证） + +bun:sqlite 内置 JSON1（`json_extract` / `json_each` / `json_valid` / `->` / `->>` 运算符全部可用）。做法： + +```sql +-- 1. 注册表：一次扫描，把目录下所有 .json 文件装进内存表 +CREATE TABLE json_documents ( + id INTEGER PRIMARY KEY, + path TEXT UNIQUE, -- 相对/绝对路径 + name TEXT, -- 文件名 + content TEXT -- 文件全文 +); + +-- 2. 按文件为粒度查询 +SELECT name, json_extract(content, '$.name') AS pkg +FROM json_documents WHERE json_valid(content); + +-- 3. 展开为行级（表格展示天然适配） +SELECT je.key, je.value, je.type +FROM json_documents jd, json_each(jd.content) je +WHERE jd.name = 'package.json'; + +-- 4. 现代运算符写法 +SELECT name, content ->> '$.version' AS v +FROM json_documents WHERE content ->> '$.type' = 'module'; +``` + +✅ 实测全部跑通（注册 2 个真实 package.json，`json_each` 展开 7 个顶层 key，`->>` 过滤正确）。 +**2026-10-26 补充验证：** 同步 yc + github-topics 后，默认扫描根 `~/.innate` 注册 9 个收集器 JSON（2.3MB，`json_valid` 全通过）；`json_each(jd.content, '$.items')` 展开真实业务数据（YC batch 公司名 + tagline、GitHub Topics 450 条目）均正确。 + +- 优点：零新依赖；与 admin 后台同一 SQLite 引擎，一张表搞定；`json_each` 天然把嵌套结构变成表格行；只读连接里建 `TEMP` 表即可，不污染主库。 +- 缺点：大文件全量进内存（加 1MB 上限过滤）；无二级索引（文件量级 < 1000 无所谓）。 + +**落地要点（T3 采用）：** +1. 扫描目录默认 `~/.innate`，环境变量 `ADMIN_JSON_DIR` 可覆盖；递归、跳过 `node_modules`/隐藏目录/`secrets`，单文件上限 1MB。 +2. `POST /api/admin/db/json/scan` 重建注册表；注册表建在**只读查询连接的 temp store**（`CREATE TEMP TABLE`），主库零写入。 +3. UI 提示用户可对 `json_documents` 跑 `json_each` 查询。 + +### 方案 B：DuckDB `read_json_auto`（不采用） + +`SELECT * FROM read_json_auto('~/.innate/*/*.json')` 体验极好（自动推断 schema、直接聚合嵌套数组）。但引入 DuckDB = 新增 ~50MB 原生依赖 + 独立进程/驱动 + Windows 侧载问题，与"简化版后台"定位冲突。**若未来需要复杂 JSON 分析再评估。** + +### 方案 C：Datasette / sqlite-utils（不采用） + +Datasette 是现成的"SQLite + JSON 浏览器"，`sqlite-utils insert-files` 可把文件元数据入表。但它是独立 Python 服务，与现有 Hono 后端重复，且无法复用本仓的 Feeds 业务查询。仅作为灵感来源（其 `files` 表设计被方案 A 借鉴）。 + +### 方案 D：每文件一张视图（不采用） + +为每个 JSON 文件按其结构动态 `CREATE VIEW`（顶层 key → 列）。灵活但脆弱：结构推断代码量大、异构文件易崩。方案 A 的 `json_each` 已达到同样的"表格展示"效果。 + +## 结论 + +采用 **方案 A**：`json_documents` 注册表 + JSON1 函数。T3 按上述落地要点实现；T4 的 UI 结果表格直接渲染 `json_each` 查询输出即为"表格方式展示"。 + +## 验收记录 + +- [x] 多方案对比（A/B/C/D） +- [x] 推荐方案在真实数据上验证（注册 + json_extract + json_each + ->> 全跑通） diff --git a/tasks/sprint-1/research/similarity-query.md b/tasks/sprint-1/research/similarity-query.md new file mode 100644 index 00000000..3ebe3d01 --- /dev/null +++ b/tasks/sprint-1/research/similarity-query.md @@ -0,0 +1,119 @@ +# R1：相似度查询方案调研（SQLite / bun:sqlite） + +- **Issue:** https://github.com/variableway/innate-feeds/issues/5 +- **结论日期:** 2026-10-23 +- **验证环境:** `~/.innate/feeds.db`（bun:sqlite / Bun 1.4.3，Windows x64），数据为真实同步快照 + +## 需求场景 + +> 查 tag 含 **AI Agent** 且 **stars > 10000** 的 GitHub repo —— 以及更一般的"按标签/关键词的相似匹配"。 + +问题分两层： + +1. **精确层**：topic 是受控词表，直接 join 即可；难点是 GitHub topic 命名不统一（`ai-agent` / `ai-agents` / `agent` / `agents` / `agentic-ai` …）。 +2. **模糊层**：关键词匹配名称/描述，需要相关性排序。 + +## 库内现状（实测） + +`starred_repo_topics` 中含 agent 的 topic 分布（top）：`ai-agents`(330)、`agent`(238)、`agents`(191)、`ai-agent`(166)、`agent-skills`(122)、`agentic-ai`(113)、`multi-agent`(74)… + +→ **单一 tag 精确匹配会漏掉大量结果，必须做同义词扩展**。 + +## 方案对比 + +### 方案 1：topic 同义词集合 + join（精确层，推荐必用） + +```sql +-- AI Agent 相关 topic 白名单 + stars 阈值 +SELECT s.full_name, s.stars, s.language, s.description, + GROUP_CONCAT(t.topic) AS topics +FROM starred_repos s +JOIN starred_repo_topics t ON t.repo_id = s.id +WHERE t.topic IN ('ai-agent','ai-agents','agent','agents', + 'agentic-ai','agentic','ai-agent-framework','llm-agent', + 'multi-agent','multi-agent-systems','autonomous-agents', + 'coding-agent','agent-framework') + AND s.stars > 10000 +GROUP BY s.id +ORDER BY s.stars DESC; +``` + +✅ 已验证（返回 browser-use、gemini-cli、hermes-agent 等）。 +- 优点：零依赖、走索引（`idx_starred_repo_topics_topic`）、语义可控。 +- 缺点：白名单需人工维护；新词（`hermes-agent` 这类项目专属 tag）覆盖不到。 + +### 方案 2：FTS5 关键词 + BM25 相关性排序（模糊层，推荐必用） + +库中已有 `starred_repos_fts` / `trending_repos_fts` 镜像（external-content，触发器同步）： + +```sql +-- 名称/描述含 agent，BM25 相关性排序 +SELECT starred_repos.full_name, starred_repos.stars, + bm25(starred_repos_fts) AS rank +FROM starred_repos_fts +JOIN starred_repos ON starred_repos.rowid = starred_repos_fts.rowid +WHERE starred_repos_fts MATCH 'agent' + AND starred_repos.stars > 10000 +ORDER BY rank +LIMIT 20; +``` + +✅ 已验证。注意：`bm25()` 的参数必须是不带别名的 FTS 表名（带别名会报 `no such column`）。 +- 优点：现成、BM25 排序、前缀查询 `agent*`、`NEAR()` 短语邻近。 +- 缺点：无语义理解；分词按 Unicode 单词，`ai-agent` 会被切成 `ai` + `agent`（对本场景反而有利）。 + +### 方案 3：topic LIKE 模糊匹配（兜底） + +```sql +SELECT DISTINCT s.full_name, s.stars +FROM starred_repos s +JOIN starred_repo_topics t ON t.repo_id = s.id +WHERE t.topic LIKE '%agent%' AND s.stars > 10000 +ORDER BY s.stars DESC; +``` + +✅ 已验证（能捞到 `hermes-agent` 这类项目专属 tag）。 +- 优点：覆盖一切含关键词的 topic，无需维护白名单。 +- 缺点：全表扫描 topics 表（当前 ~5 万行，毫秒级，可接受）；`%ai%` 这类短词会过匹配。 + +### 方案 4：FTS5 trigram 分词器（子串/编辑距离模糊，可选） + +Bun 1.4.3 内置 SQLite 支持 trigram： + +```sql +CREATE VIRTUAL TABLE temp.name_trigram USING fts5( + full_name, tokenize = 'trigram' +); +-- 之后 MATCH 'langchin' 也能命中 langchain（近似子串） +``` + +✅ `tokenize='trigram'` 建表已验证可用。适合"记不清全名"的仓库名模糊查找；维护成本是要建镜像表 + 触发器。Sprint-1 不落地，记为后续增强。 + +### 方案 5：编辑距离 / 向量相似（评估后暂缓） + +- **自定义标量函数**：node:sqlite 的 `db.function()` 在 bun:sqlite 1.4.3 **不存在**（实测 `TypeError: db.function is not a function`）。要注册 Levenshtein 需换 native sqlite3 包或 native 扩展，不值得。 +- **向量检索（sqlite-vec）**：需加载扩展 + 生成 embedding 管道，超出"简化版后台"范围。若未来要做语义相似（"找和 X 类似的仓库"），再评估。 + +## 推荐组合（Sprint-1 落地） + +**方案 1 + 2 + 3 组合查询**，作为 Admin UI 的内置示例查询： + +```sql +-- "AI Agent 且 stars>10000" 的生产版：topic LIKE 兜底 + 排除误报 +SELECT s.full_name, s.stars, s.language, + GROUP_CONCAT(t.topic) AS topics +FROM starred_repos s +JOIN starred_repo_topics t ON t.repo_id = s.id +WHERE t.topic LIKE '%agent%' + AND s.stars > 10000 +GROUP BY s.id +ORDER BY s.stars DESC; +``` + +UI 三栏的「保存的 SQL」默认预置：① 精确白名单版 ② FTS BM25 版 ③ LIKE 兜底版，三者即可覆盖"相似度查询"的全部实际需求。 + +## 验收记录 + +- [x] ≥3 种方案对比（1/2/3/4/5 共 5 种） +- [x] 示例 SQL 在真实库验证执行（方案 1/2/3 已跑通，方案 4 建表跑通，方案 5 给出不可行原因） +- [x] 给出 Sprint-1 推荐组合 diff --git a/tasks/sprint-1/status.md b/tasks/sprint-1/status.md new file mode 100644 index 00000000..93c354b7 --- /dev/null +++ b/tasks/sprint-1/status.md @@ -0,0 +1,28 @@ +# Sprint-1 任务状态总表 + +> **单一事实来源**：任何 Agent 动手前先读本表；完成后必须回写。 +> 状态取值：`todo` / `in_progress` / `done` / `blocked` / `skipped(原因)`。 +> 详细验收标准见各 task 文档；调研产出在 [`research/`](./research/)。 + +| Task | 说明 | 依赖 | Issue | 状态 | Agent | 完成 SHA / 备注 | +|---|---|---|---|---|---|---| +| R1 | 相似度查询方案调研 | 无 | #5 | done | claude-code | research/similarity-query.md；5 方案对比，前 3 已实测 | +| R2 | JSON 文件 SQL 查询方案调研 | 无 | #6 | done | claude-code | research/json-query.md；json_documents + json_each 已实测 | +| T1 | 项目初始化 | 无 | #7 | done | claude-code | pnpm + innate-fe-cli 接入，typecheck 双通过 | +| T2 | 最新数据同步进 SQLite | T1 | #8 | done | claude-code | 快照 2026-10-23，total 7114（trending 962 + starred 6152） | +| T3 | Admin DB API | T1、R1、R2 | #9 | done | claude-code | /api/admin/db/* 全端点 + 只读硬保证 + 7 单测通过 | +| T4 | Admin UI 三栏工作台 | T3 | #10 | done | claude-code | /admin，CodeMirror 6 高亮；顺带修 Windows 页面 404 bug | +| T5 | 启动参数控制 | T3 | #11 | done | claude-code | 默认启用；--no-admin / ADMIN_DB_UI=off 关闭 | + +## 执行日志（追加式，最新在上） + +- 2026-10-26 claude-code：修复 CI 前置条件——设置 `INNATE_FE_TOKEN` secret（私有基础仓 checkout）；根 `test` 脚本 vitest → `bun test`（41/41 通过）；同步 yc + github-topics 收集器，admin JSON 查询在真实数据上验证；AGENTS.md 测试说明更新。 +- 2026-10-23 claude-code：Sprint 全部任务（R1/R2/T1–T5）完成，Issue #5–#11 全部关闭；Sprint 总结见 PR（closes #4）。 +- 2026-10-23 claude-code：R1/R2/T1/T2 完成，Issue #5 #6 #7 #8 已关闭；开始 T3。 +- 2026-10-23 claude-code：Sprint 建档（context / status / task 文档 / research 占位）。 + +## 已知遗留（不阻塞 Sprint 验收） + +- ~~仓库 vitest 配置无法解析 bun 专属导入~~ → **已修复（2026-10-26）**：根 `test` 脚本改为 `bun test`（vitest 兼容层），41/41 测试通过；AGENTS.md 测试说明已同步。 +- ~~`~/.innate` 下暂无外部收集器 JSON~~ → **已修复（2026-10-26）**：`sync yc` + `sync github-topics` 已灌入 9 个真实 JSON（2.3MB），admin JSON 扫描（默认目录）与 `json_each` 展开实测通过；保存查询种子更新为 items 展开写法。 +- CI / Pages 部署在 main 上自 2026-10-22 起失败：根因是 `variableway/innate-fe` 为私有仓且 `INNATE_FE_TOKEN` secret 缺失（checkout Not Found）。2026-10-26 已设置该 secret（值为当前 gh token，建议后续换成仅含 innate-fe 读权限的 fine-grained PAT 并轮换）；测试脚本修复一并入本 PR。 diff --git a/tasks/sprint-1/task-01-init-project.md b/tasks/sprint-1/task-01-init-project.md new file mode 100644 index 00000000..6921c82b --- /dev/null +++ b/tasks/sprint-1/task-01-init-project.md @@ -0,0 +1,21 @@ +# T1：项目初始化 + +- **Issue:** https://github.com/variableway/innate-feeds/issues/7 +- **依赖:** 无 +- **验收:** 见下。 + +## 任务描述 + +让仓库在本机达到可开发状态： + +1. `innate-fe-cli install`（发布 `@innate/ui` / `@innate/web-domain` 到私有源并接入 frontend；基础仓位置由仓库根 `.innate-fe-cli.json` 指向）。 +2. `bun install`（backend + frontend 依赖）。 +3. `bun run typecheck`（backend + frontend）通过。 +4. 后端可启动：`bun run dev:backend`（http://localhost:4000，`/api/health` 返回 ok）。 + +## 验收记录 + +- [x] `bun install` 成功（pnpm install 330 包；注：@innate 私有源解析需 root `.npmrc`，已按 setup:innate 流程 `cp frontend/.npmrc .npmrc`） +- [x] `frontend/node_modules/@innate/ui` 与 `@innate/web-domain` 就位 +- [x] `bun run typecheck` 通过（backend + frontend 均 0 退出） +- [x] `/api/health` 返回 `{ ok: true }` diff --git a/tasks/sprint-1/task-02-sync-data.md b/tasks/sprint-1/task-02-sync-data.md new file mode 100644 index 00000000..44d91ad0 --- /dev/null +++ b/tasks/sprint-1/task-02-sync-data.md @@ -0,0 +1,21 @@ +# T2：最新数据同步进 SQLite + +- **Issue:** https://github.com/variableway/innate-feeds/issues/8 +- **依赖:** T1（`bun install` 完成） +- **验收:** 见下。 + +## 任务描述 + +用已认证的 `gh` CLI 把最新数据同步进 `~/.innate/feeds.db`： + +1. `bun run data:sync:trending`（daily / weekly / monthly 三榜）。 +2. `bun run data:sync:starred`（认证用户 star 全量，带 `starred_at`）。 +3. `bun run data:stats` 与 `bun run dates` 确认快照日期为今天。 + +说明：digest（issues）与 README 预取不落 SQLite，属于 `sync window` 范畴，本 task 不强制；如执行可选 `--skip-readme`。 + +## 验收记录 + +- [x] `bun run data:sync:trending` 成功（daily 9 / weekly 15 / monthly 20，Firecrawl + gh 兜底） +- [x] `bun run data:sync:starred` 成功（增量 +1156，总计 6152） +- [x] `bun run data:stats` 显示今日快照（2026-10-23；total 7114 = trending 962 + starred 6152） \ No newline at end of file diff --git a/tasks/sprint-1/task-03-admin-api.md b/tasks/sprint-1/task-03-admin-api.md new file mode 100644 index 00000000..00487920 --- /dev/null +++ b/tasks/sprint-1/task-03-admin-api.md @@ -0,0 +1,37 @@ +# T3：Admin DB API（简化版 SQLite 后台后端） + +- **Issue:** https://github.com/variableway/innate-feeds/issues/9 +- **依赖:** T1；调研输入 R1（相似度查询）、R2（JSON 查询） +- **分组:** admin 组（T3 → T4 / T5，本组优先级最高） +- **验收:** 见下。 + +## 任务描述 + +在 Hono（`backend/src/app/server.ts` 或独立子路由模块）下新增 `/api/admin/db/*`： + +| Method | Path | 说明 | +|---|---|---| +| GET | `/api/admin/db/tables` | 全部表 + 行数（含 FTS/内部表标记） | +| GET | `/api/admin/db/tables/:name/schema` | 列信息（PRAGMA table_info）+ 索引 + DDL | +| POST | `/api/admin/db/query` | 执行**只读** SQL，返回 `{ columns, rows, rowCount, durationMs }` | +| GET/POST | `/api/admin/db/queries` | 保存的 SQL：列表 / 新增 | +| PUT/DELETE | `/api/admin/db/queries/:id` | 保存的 SQL：更新 / 删除 | +| POST | `/api/admin/db/json/scan` | 扫描 JSON 目录（默认 `~/.innate`，`ADMIN_JSON_DIR` 可覆盖）→ 注册进内存表 `json_documents(path, name, content)` | +| GET | `/api/admin/db/json/files` | 已注册 JSON 文件清单 | + +安全约束： + +- 查询执行走**独立只读连接**（`new Database(path, { readonly: true })`），主连接不暴露给任意 SQL。 +- 仅允许单条 SELECT / WITH 语句（拒绝多语句、PRAGMA、ATTACH 等）。 +- 默认行数上限（如 500），防止全表拖垮 UI。 +- 保存的 SQL 存 SQLite 表 `admin_saved_queries`。 + +JSON 查询按 R2 结论落地：`json_documents` 注册表 + `json_each` 展开示例。 + +## 验收记录 + +- [x] 上述端点全部可用（curl 验证：tables / schema / query / queries CRUD / json scan+files） +- [x] 只读约束生效（UPDATE 被拒 `Only SELECT / WITH statements are allowed`；多语句被拒；底层为 readonly 连接硬保证） +- [x] JSON scan 后可用 SQL 查询 JSON 内容（ADMIN_JSON_DIR 指向仓库根：233 文件 / 4.5MB；json_each 展开验证通过） +- [x] OpenAPI spec（backend/src/app/openapi.ts）同步更新（AdminDb tag + 7 端点 + 4 schema） +- [x] 单测：backend/src/app/admin-db.test.ts（validateReadOnlySql，`bun test` 7 pass；注：仓库 vitest 配置对 `with {type:"text"}` 导入存在既有问题，与本项目无关） diff --git a/tasks/sprint-1/task-04-admin-ui.md b/tasks/sprint-1/task-04-admin-ui.md new file mode 100644 index 00000000..f35370fa --- /dev/null +++ b/tasks/sprint-1/task-04-admin-ui.md @@ -0,0 +1,31 @@ +# T4：Admin UI 三栏工作台 + +- **Issue:** https://github.com/variableway/innate-feeds/issues/10 +- **依赖:** T3（API 就绪） +- **分组:** admin 组；与 T5 可并行 +- **验收:** 见下。 + +## 任务描述 + +前端新增 `/admin` 路由页（`frontend/src/pages/admin/`，注册进 `frontend/src/web-domain.tsx` 与 navItems）： + +``` +┌──────────┬──────────────────────────────┬────────────┐ +│ 表列表 │ SQL 编辑器（语法高亮） │ 保存的 SQL │ +│ (行数) │ ──────────────────────────── │ (点击加载) │ +│ 点击预览 │ 结果表格（列名 + 行数据） │ 保存/删除 │ +└──────────┴──────────────────────────────┴────────────┘ +``` + +- SQL 编辑器：CodeMirror 6（`@uiw/react-codemirror` + `@lang-sql`），需确认与 Bun 构建链兼容。 +- 左栏：`GET /api/admin/db/tables`，点击表名自动填充 `SELECT * FROM x LIMIT 100`。 +- 右栏：保存的 SQL 列表（加载 / 覆盖保存 / 删除）；内置 R1 结论的示例查询（AI Agent + stars>10000）。 +- 结果：表格渲染，显示 rowCount 与耗时。 + +## 验收记录 + +- [x] `/admin` 页面可用（生产构建产物由后端同源托管验证 200；dev 模式同 pipeline） +- [x] SQL 高亮、执行、结果表格正常（CodeMirror 6 `@uiw/react-codemirror` + `@codemirror/lang-sql`；新版无需独立 CSS 文件，运行时注入主题） +- [x] 保存的 SQL 增删改查正常（覆盖保存 / 另存 / 删除；Ctrl+Enter 执行） +- [x] typecheck 通过 +- 备注：修复 server.ts `mountWebsite` 在 Windows 上因反斜杠路径前缀校验失败导致所有页面路由 404 的既有 bug（`toPosix` 归一化）。 diff --git a/tasks/sprint-1/task-05-server-flag.md b/tasks/sprint-1/task-05-server-flag.md new file mode 100644 index 00000000..8ea70923 --- /dev/null +++ b/tasks/sprint-1/task-05-server-flag.md @@ -0,0 +1,25 @@ +# T5：启动参数控制（默认启用后台管理查询） + +- **Issue:** https://github.com/variableway/innate-feeds/issues/11 +- **依赖:** T3（admin 路由存在） +- **分组:** admin 组；与 T4 可并行 +- **验收:** 见下。 + +## 任务描述 + +Issue 要求：这个后台简化版**启动需要参数控制，默认启动后台管理查询**。 + +实现： + +- 默认（无参数）：admin 路由挂载，`/admin` 与 `/api/admin/db/*` 可用。 +- 关闭方式（任一）： + - CLI：`bun src/app/server.ts --no-admin` + - 环境变量：`ADMIN_DB_UI=off` +- server 启动日志打印 admin 状态。 +- `backend/package.json` 增加便捷脚本（如 `dev:no-admin`）。 + +## 验收记录 + +- [x] 默认启动时 `/api/admin/db/tables` 可访问（200） +- [x] `--no-admin`（tables=404，health=200）与 `ADMIN_DB_UI=off`（tables=404）均关闭 admin，启动日志打印 `Admin DB workspace: disabled` +- [x] README（AGENTS.md）文档同步：API 表新增 admin 端点 + `/admin` 章节 + 环境变量说明；`backend/package.json` 新增 `dev:no-admin` diff --git a/tasks/sprint-1/task-r1-similarity-research.md b/tasks/sprint-1/task-r1-similarity-research.md new file mode 100644 index 00000000..20798ed8 --- /dev/null +++ b/tasks/sprint-1/task-r1-similarity-research.md @@ -0,0 +1,19 @@ +# R1：相似度查询方案调研 + +- **Issue:** https://github.com/variableway/innate-feeds/issues/5 +- **依赖:** 无 +- **产出物:** `tasks/sprint-1/research/similarity-query.md` +- **验收:** 文档给出至少 3 种可落地的查询方案，含可直接在 `~/.innate/feeds.db` 上执行的示例 SQL（示例场景：tag 含 AI Agent 且 stars > 10000 的 GitHub repo），并给出 Sprint-1 推荐组合。 + +## 任务描述 + +调研在本仓现有 SQLite（bun:sqlite，schema 见 `backend/src/db/schema.sql`）上实现"相似度查询"的方案，覆盖： + +1. 精确 tag + 数值阈值（`starred_repo_topics` join `starred_repos`）。 +2. 关键词模糊匹配：`LIKE`、FTS5（库中已有 `*_fts` 镜像表）。 +3. 更高级相似度（编辑距离 / trigram / 向量）在本栈的可行性与成本。 + +## 验收记录 + +- [x] 方案对比与推荐已写入 research/similarity-query.md +- [x] 示例 SQL 已在实际数据库上验证执行 diff --git a/tasks/sprint-1/task-r2-json-query-research.md b/tasks/sprint-1/task-r2-json-query-research.md new file mode 100644 index 00000000..5e7cbe0f --- /dev/null +++ b/tasks/sprint-1/task-r2-json-query-research.md @@ -0,0 +1,19 @@ +# R2：JSON 文件 SQL 查询方案调研 + +- **Issue:** https://github.com/variableway/innate-feeds/issues/6 +- **依赖:** 无 +- **产出物:** `tasks/sprint-1/research/json-query.md` +- **验收:** 文档对比多种"把 JSON 文件当表查"的方案（目标：所有 JSON 文件放一处，SQL 直接查文件内容、表格展示），给出 Sprint-1 推荐落地路径（本栈 = bun:sqlite + Hono）。 + +## 任务描述 + +调研对象包括： + +1. SQLite JSON1 内置函数（`json_each` / `json_extract` / `->` / `->>`）。 +2. "文件注册表"模式：把 `~/.innate/**/*.json` 扫描进 `(path, content)` 表再查询。 +3. 外部引擎（DuckDB `read_json_auto`、Datasette / sqlite-utils）为何不引入本栈。 + +## 验收记录 + +- [x] 方案对比与推荐已写入 research/json-query.md +- [x] 推荐方案（json_documents 注册表 + json_each）已在真实数据上验证