MakeSense for GitHub — Make Sense of Any GitHub PR, with full cognitive coverage of every change.
getnao/nao#1360 — 43 files, 60 review comments
Standardize LLM Model Configuration, Track Inference, and Display Story Refresh Failures
Overview
This change centralizes LLM model configuration within projects, allowing for category-specific overrides. It introduces comprehensive tracking of LLM inference usage and provides users with visibility into story data refresh failures. Database schema updates support these new configurations and tracking mechanisms.
Technical Highlights
Database Schema (`0059_defauls_models.sql`, `pg-schema.ts`, `sqlite-schema.ts`): Centralized LLM model configuration into a `jsonb`/`json` column (`default_models`) on the `project` table; removed old `model_provider` and `model_id` columns from `context_recommendation_config`.
Schema Snapshots (`meta/0059_snapshot.json`): Reflects the `default_models` change and introduces numerous new tables for LLM inference logging, OAuth, activity tracking, story management (folders, versions, caching), and message parts.
Shared Utilities (`background-models.ts`, `llm.ts`): Introduced `BackgroundModelCategory` enum and `resolveDefaultModelSelection` for hierarchical, category-specific LLM model resolution; provided `selectBackgroundModel` and `setBackgroundModelForCategory` for managing these settings.
Backend API (`project.queries.ts`, `project.routes.ts`): Added query and TRPC procedures to fetch and update project-specific `default_models` settings.
Frontend UI (`default-models-section.tsx`, `settings.project.models.tsx`): Implemented a new administrative interface for configuring default LLM models, supporting global and category-specific assignments.
LLM Service Integration (`agent.ts`, `automation-title.ts`, `compaction.ts`, `context-recommendations.service.ts`, `cron-nlp.ts`, `live-story.ts`, `memory.ts`): Updated various services to utilize the new `resolveDefaultModelSelection` for dynamic LLM model resolution based on task category.
LLM Inference Tracking (`agent.ts`, `live-story.ts`, `usage.queries.ts`, `llm.ts` types): Implemented tracking and recording of LLM token usage for title generation and live story refreshes; unified usage metrics by combining `chatMessage` and `llmInference` data.
Story Refresh Failure Display (`activity.queries.ts`, `shared-story.routes.ts`, `story.routes.ts`, `use-story-viewer-content.ts`, `story-header.tsx`, `story-page-header.tsx`): Added backend query and frontend components to fetch and display `lastRefreshFailure` information for stories and shared stories.
Impact
Functionality: Users can now precisely control which LLM models are used for different background tasks within a project, and gain visibility into LLM inference costs and story data refresh issues.
Risk: The extensive database schema changes and new LLM model resolution logic require thorough testing to prevent regressions in existing LLM-powered features; the snapshot also reveals a potential security risk with `google_client_secret` stored directly in the `organization` table.
Files reviewed
- apps/backend/migrations-postgres/0059_defauls_models.sql — 2 comment(s)
- apps/backend/migrations-postgres/meta/0059_snapshot.json — 21 comment(s)
- apps/backend/migrations-postgres/meta/_journal.json — 1 comment(s)
- apps/backend/migrations-sqlite/0059_defauls_models.sql — 2 comment(s)
- apps/backend/migrations-sqlite/meta/0059_snapshot.json — 25 comment(s)
- apps/backend/migrations-sqlite/meta/_journal.json — 0 comment(s)
- apps/backend/src/db/pg-schema.ts — 1 comment(s)
- apps/backend/src/db/sqlite-schema.ts — 1 comment(s)
- apps/backend/src/queries/activity.queries.ts — 2 comment(s)
- apps/backend/src/queries/project.queries.ts — 1 comment(s)
- apps/backend/src/queries/usage.queries.ts — 4 comment(s)
- apps/backend/src/services/agent.ts — 0 comment(s)
- apps/backend/src/services/automation-title.ts — 0 comment(s)
- apps/backend/src/services/compaction.ts — 0 comment(s)
- apps/backend/src/services/context-recommendations.service.ts — 0 comment(s)
- apps/backend/src/services/cron-nlp.ts — 0 comment(s)
- apps/backend/src/services/live-story.ts — 0 comment(s)
- apps/backend/src/services/memory.ts — 0 comment(s)
- apps/backend/src/trpc/context-recommendation.routes.ts — 0 comment(s)
- apps/backend/src/trpc/project.routes.ts — 0 comment(s)
- apps/backend/src/trpc/shared-story.routes.ts — 0 comment(s)
- apps/backend/src/trpc/story.routes.ts — 0 comment(s)
- apps/backend/src/types/llm.ts — 0 comment(s)
- apps/backend/src/utils/llm.ts — 0 comment(s)
- apps/backend/tests/compaction.test.ts — 0 comment(s)
- apps/backend/tests/usage-queries.test.ts — 0 comment(s)
- apps/frontend/src/components/settings-search-index.ts — 0 comment(s)
- apps/frontend/src/components/settings/default-models-section.tsx — 0 comment(s)
- apps/frontend/src/components/side-panel/hooks/use-story-viewer-content.ts — 0 comment(s)
- apps/frontend/src/components/side-panel/hooks/use-story-viewer-live-settings.ts — 0 comment(s)
- apps/frontend/src/components/side-panel/story-header.tsx — 0 comment(s)
- apps/frontend/src/components/side-panel/story-viewer.tsx — 0 comment(s)
- apps/frontend/src/components/story-page-header.tsx — 0 comment(s)
- apps/frontend/src/hooks/use-llm-providers.ts — 0 comment(s)
- apps/frontend/src/hooks/use-time-ago.ts — 0 comment(s)
- apps/frontend/src/routes/_sidebar-layout.settings.project.models.tsx — 0 comment(s)
- apps/frontend/src/routes/_sidebar-layout.settings.recommendations.tsx — 0 comment(s)
- apps/frontend/src/routes/_sidebar-layout.settings.usage.tsx — 0 comment(s)
- apps/frontend/src/routes/_sidebar-layout.stories.preview.$chatId.$storySlug.tsx — 0 comment(s)
- apps/frontend/src/routes/_sidebar-layout.stories.shared.$shareId.tsx — 0 comment(s)
- apps/frontend/src/routes/_sidebar-layout.stories.standalone.$storyId.tsx — 0 comment(s)
- apps/shared/src/background-models.ts — 0 comment(s)
- apps/shared/src/index.ts — 0 comment(s)
View the full interactive review (JavaScript required).
