MakeSense for GitHub — Make Sense of Any GitHub PR, with full cognitive coverage of every change.
getnao/nao#1360 — 44 files, 60 review comments
Implement Project-Level Default LLM Models
Overview
This change centralizes LLM model configuration by migrating existing settings into a new `default_models` column on the `project` table. It introduces a flexible system for project administrators to select specific LLM models for various background tasks. Multiple backend services now use this new project-level model selection logic.
Technical Highlights
Database Schema: Adds a `default_models` JSON column to the `project` table. Removes `model_provider` and `model_id` from `context_recommendation_config`.
Data Migration: Migrates existing model configuration data from `context_recommendation_config` to `project.default_models`.
LLM Model Selection: Introduces `BackgroundModelSettings` and `resolveDefaultModelSelection` for robust, project-level default LLM model management.
Backend Services: Updates multiple services (e.g., agent, compaction, live-story) to use the new project-level default LLM model selection.
API and UI: Adds TRPC endpoints and a new frontend component (`DefaultModelsSection`) for configuring default LLM models.
LLM Usage Tracking: Expands LLM inference tracking for specific tasks (e.g., title generation, live story refresh). Refactors usage queries to combine `chatMessage` and `llmInference` data.
Story UI: Integrates `lastRefreshFailure` display into story-related UI components.
Testing: Shifts usage query tests to use an in-memory SQLite database for integration testing.
Impact
Functionality: Project administrators can configure specific LLM models for background tasks. Enhanced LLM usage visibility and cost attribution. Users see story data refresh failures.
Risk: Data migration from `context_recommendation_config` to `project.default_models` requires careful validation. New LLM model resolution logic increases configuration complexity. Potential for breaking changes in services relying on old model selection.
Files reviewed
- apps/backend/migrations-postgres/0063_default_models.sql — 1 comment(s)
- apps/backend/migrations-postgres/meta/0063_snapshot.json — 30 comment(s)
- apps/backend/migrations-postgres/meta/_journal.json — 0 comment(s)
- apps/backend/migrations-sqlite/0063_default_models.sql — 3 comment(s)
- apps/backend/migrations-sqlite/meta/0063_snapshot.json — 26 comment(s)
- apps/backend/migrations-sqlite/meta/_journal.json — 0 comment(s)
- apps/backend/src/db/pg-schema.ts — 0 comment(s)
- apps/backend/src/db/sqlite-schema.ts — 0 comment(s)
- apps/backend/src/queries/activity.queries.ts — 0 comment(s)
- apps/backend/src/queries/project.queries.ts — 0 comment(s)
- apps/backend/src/queries/usage.queries.ts — 0 comment(s)
- apps/backend/src/services/agent.ts — 0 comment(s)
- apps/backend/src/services/automation-title.ts — 0 comment(s)
- apps/backend/src/services/compaction.ts — 0 comment(s)
- apps/backend/src/services/context-recommendations.service.ts — 0 comment(s)
- apps/backend/src/services/cron-nlp.ts — 0 comment(s)
- apps/backend/src/services/live-story.ts — 0 comment(s)
- apps/backend/src/services/memory.ts — 0 comment(s)
- apps/backend/src/trpc/context-recommendation.routes.ts — 0 comment(s)
- apps/backend/src/trpc/project.routes.ts — 0 comment(s)
- apps/backend/src/trpc/shared-story.routes.ts — 0 comment(s)
- apps/backend/src/trpc/story.routes.ts — 0 comment(s)
- apps/backend/src/types/llm.ts — 0 comment(s)
- apps/backend/src/utils/llm.ts — 0 comment(s)
- apps/backend/tests/compaction.test.ts — 0 comment(s)
- apps/backend/tests/usage-queries.test.ts — 0 comment(s)
- apps/frontend/src/components/settings-search-index.ts — 0 comment(s)
- apps/frontend/src/components/settings/default-models-section.tsx — 0 comment(s)
- apps/frontend/src/components/side-panel/hooks/use-story-viewer-content.ts — 0 comment(s)
- apps/frontend/src/components/side-panel/hooks/use-story-viewer-live-settings.ts — 0 comment(s)
- apps/frontend/src/components/side-panel/story-header.tsx — 0 comment(s)
- apps/frontend/src/components/side-panel/story-viewer.tsx — 0 comment(s)
- apps/frontend/src/components/story-page-header.tsx — 0 comment(s)
- apps/frontend/src/hooks/use-llm-providers.ts — 0 comment(s)
- apps/frontend/src/hooks/use-time-ago.ts — 0 comment(s)
- apps/frontend/src/routes/_sidebar-layout.settings.project.models.tsx — 0 comment(s)
- apps/frontend/src/routes/_sidebar-layout.settings.recommendations.tsx — 0 comment(s)
- apps/frontend/src/routes/_sidebar-layout.settings.usage.tsx — 0 comment(s)
- apps/frontend/src/routes/_sidebar-layout.stories.preview.$chatId.$storySlug.tsx — 0 comment(s)
- apps/frontend/src/routes/_sidebar-layout.stories.shared.$shareId.tsx — 0 comment(s)
- apps/frontend/src/routes/_sidebar-layout.stories.standalone.$storyId.tsx — 0 comment(s)
- apps/shared/src/background-models.ts — 0 comment(s)
- apps/shared/src/index.ts — 0 comment(s)
- apps/shared/tests/background-models.test.ts — 0 comment(s)
View the full interactive review (JavaScript required).
