Assessment Architecture¶

Status: Design - umbrella roadmap for reconciling CMS-authored practice widgets with LMS graded quiz entries.

This document defines the target assessment mechanic for the Open Learning Hub LMS. It connects the existing Quiz System, Analytics, Entry Pagination System, and CMS widget model into one course-creator and learner experience.

The decision is intentional:

• Canonical graded assessment: first-class quiz entries, server-graded through the LMS.
• Formative practice: existing assessment widgets stay available inside lessons for client-side practice and immediate feedback, but they are not persisted, scored, or used for entry gating.

Current State¶

The codebase currently has two assessment models that do not line up.

The LMS also infers an entry as a quiz when a CMS page contains one of several legacy assessment widget types in src/lib/course/entry-classifier.ts. That inference made early prototypes easy, but it now creates ambiguity: a page with a practice widget becomes a graded entry even though no server-graded assessment exists.

Target Model¶

Graded quizzes are explicit. A CMS page becomes a graded LMS quiz entry only when it contains one dedicated structured quiz widget. Legacy assessment widgets no longer classify a page as a quiz entry.

flowchart TB
  subgraph cms [CMS Authoring]
    Page["Leaf page in course outline"]
    QuizWidget["structured-quiz widget"]
    PracticeWidgets["legacy assessment widgets"]
  end
  subgraph wire [Shared Wire Contract]
    QuizContent["quizWidgetContentSchema"]
    WidgetContent["legacy widget content schemas"]
  end
  subgraph lms [LMS Runtime]
    Adapter["getEntry quiz adapter"]
    FullQuiz["Quiz server-only"]
    PublicQuiz["QuizPublic client-safe"]
    QuizPlayer["QuizPlayer"]
    SubmitAction["submitQuizAttemptAction"]
    Attempts["quiz_attempts"]
    Progress["entry_progress"]
    Reports["learner and admin reports"]
    LessonPlayer["LessonPlayer"]
  end

  Page --> QuizWidget --> QuizContent --> Adapter
  Adapter --> FullQuiz
  FullQuiz --> PublicQuiz --> QuizPlayer
  QuizPlayer -->|"answers only"| SubmitAction
  SubmitAction --> Attempts
  SubmitAction --> Progress
  Attempts --> Reports
  Progress --> Reports
  PracticeWidgets --> WidgetContent --> LessonPlayer

The hard boundary is that answer keys never reach the browser for graded quizzes. The server holds the full quiz, strips it to QuizPublic for rendering, and re-fetches or re-validates the authoritative CMS quiz on submission before grading.

Authoring Vehicle¶

Recommendation¶

Add a new dedicated structured-quiz widget in the CMS and shared widget wire schemas. One structured-quiz widget on one leaf CMS page represents one graded LMS quiz entry.

This is the recommended V1 because it matches the current CMS model:

• Courses are CMS projects.
• Course structure is a nested page tree.
• Entry bodies are authored as blocks and widgets.
• There is no page type, page template, or separate quiz document model today.

Adding a widget gives course creators a clear "graded quiz" affordance without requiring a broader CMS page-type migration.

Author Workflow¶

1. Create or select a leaf page in a course project.
2. Add one structured-quiz widget from the assessment category.
3. Configure quiz settings:
   • passThreshold, stored as a fraction from 0 to 1.
   • maxAttempts, integer >= 1.
   • shuffleQuestions, default false.
   • showPerQuestionFeedback, default true.
4. Add questions:
   • multiple_choice
   • true_false
   • short_answer
5. Add optional explanation copy for review mode.
6. Publish the project. The LMS treats the page as a quiz entry.

The editor must surface validation inline:

One Quiz Per Page¶

V1 enforces one structured-quiz widget per page. A page may contain explanatory text, images, or other non-assessment content around the quiz, but it must not contain multiple graded quiz widgets.

If an author needs multiple graded checkpoints, they should create multiple quiz pages in the course outline. This keeps attempts, reporting, gating, and analytics scoped to one (courseSlug, entrySlug) pair.

Forward Path¶

If the CMS later needs many explicit entry types, promote quiz intent and quiz settings to page-level metadata:

• pages.properties.entryType = "lesson" | "quiz" | "assignment"
• pages.properties.quizSettings = { passThreshold, maxAttempts, ... }

That larger page metadata model is not required for V1 and should not block the structured quiz widget.

Wire Contract¶

The new widget content schema should align with the existing LMS quiz schemas rather than invent a parallel shape.

type StructuredQuizContent = {
  instructions?: PortableText;
  passThreshold: number;
  maxAttempts: number;
  shuffleQuestions?: boolean;
  showPerQuestionFeedback?: boolean;
  questions: QuizQuestion[];
};

QuizQuestion should remain the shape already documented in quiz-system.md:

• multiple_choice
• true_false
• short_answer

The shared package should expose a schema such as structuredQuizContentSchema or quizWidgetContentSchema from @open-learning-hub/widget-wire-schemas. The LMS adapter then maps one widget instance plus page metadata into the existing Quiz domain shape:

type Quiz = {
  _type: "quiz";
  _order: number;
  slug: string;
  title: string;
  instructions?: PortableText;
  passThreshold: number;
  maxAttempts: number;
  shuffleQuestions?: boolean;
  showPerQuestionFeedback?: boolean;
  questions: QuizQuestion[];
};

The server-only Quiz includes correct flags and acceptedAnswers. The client receives only QuizPublic from toQuizPublic(quiz).

LMS Entry Classification¶

The target classifier rule is:

• structured-quiz present: entry _type is quiz.
• Otherwise: entry remains lesson, unless a future explicit assignment marker exists.

Legacy widgets should be removed from QUIZ_WIDGET_TYPES during the implementation. That behavior change must be called out in release notes because a page with only form, trueorfalse, quick-questions, quick-questions-multiple-choice, or rubric will stop being gated as a graded quiz and will render as a lesson with formative practice.

The implementation should keep the classifier narrow and explicit. Do not classify by broad widget category, because the assessment category contains formative and self-reflection widgets such as checklist and rubric.

Grading and Submission Flow¶

The grading flow follows quiz-system.md, with the authoring vehicle clarified here.

sequenceDiagram
  participant Student
  participant QuizPlayer
  participant Action as submitQuizAttemptAction
  participant CMS
  participant DB as Kysely DB

  Student->>QuizPlayer: Complete QuizPublic form
  QuizPlayer->>Action: Submit courseSlug, entrySlug, answers, startedAt
  Action->>Action: Auth, access, rate limit, preview check
  Action->>CMS: Re-fetch structured quiz content
  Action->>Action: Validate answers against server-only Quiz
  Action->>Action: Grade score and pass/fail
  Action->>DB: Insert quiz_attempts
  alt passed or attempts exhausted
    Action->>DB: Upsert entry_progress.completed_at
  end
  Action->>QuizPlayer: Return score, attempt, feedback, remaining attempts

Submission requirements:

1. Resolve the authenticated user.
2. Reject preview mode without writes.
3. Enforce enrollment, course access, and gating.
4. Rate-limit by authenticated user using the existing quizSubmit policy.
5. Re-fetch the CMS quiz and grade against the server-only payload.
6. Validate every answer shape by question type.
7. Refuse attempts when the learner has already passed or exhausted attempts.
8. Insert quiz_attempts for every valid submission.
9. Set entry_progress.completed_at when the learner passes or exhausts maxAttempts.
10. Revalidate course and progress state.

The action must never trust score, passed, correct, acceptedAnswers, or passThreshold from the client.

Learner Experience¶

The learner experience should be a single-page quiz form with one submission per attempt.

Core states:

The results view must provide:

• Score line (correctCount / totalCount plus percent).
• Pass/fail banner with focus management.
• Per-question feedback rows when enabled.
• Learner answer echo.
• Explanation copy only in review mode.
• Retry when failed and attempts remain.
• Next-entry affordance only when the quiz entry is complete.

Accessibility follows quiz-system.md: fieldsets and legends for question groups, form error summary with role="alert", focus moved to the outcome banner after submit, and translation-keyed UI chrome.

Scoring Rules¶

V1 scoring stays intentionally simple:

• Equal-weight questions.
• All-or-nothing correctness per question.
• No partial credit.
• No negative marking.
• No timers.
• No essays or file-upload questions.

Question-specific grading:

The persisted score is correctCount / totalCount. The attempt passes when score >= passThreshold.

Formative Widgets¶

Existing assessment widgets continue to serve a useful role as low-friction practice:

• trueorfalse
• quick-questions
• quick-questions-multiple-choice
• form
• guess
• rubric
• checklist
• select-media-files

Their target role is formative, not graded:

• They render in lessons through LessonPlayer.
• They may self-grade or provide local feedback.
• They do not write quiz_attempts.
• They do not write entry_progress.
• They do not appear in learner or admin assessment reports.
• Their answer keys may remain in client payloads because they are not trusted assessment records.

If a course creator wants a gradebook-visible result, they must use structured-quiz.

Reporting¶

Assessment reporting reads from LMS-owned persistence, not from widget-local state.

Learner Reports¶

Learner reports use quiz_attempts, entry_progress, and the CMS hierarchy for labels and ordering.

V1 learner surfaces:

Learner queries must always scope by tenantId and authenticated userId. They must not use shared cache across users.

Admin Reports¶

Admin reports follow analytics.md.

Access rules:

• Course admins see assigned courses only.
• Tenant admins see their tenant only.
• Super admins must use the existing tenant override pattern for cross-tenant views.
• Aggregate reports should avoid exposing learner names or emails unless the route is explicitly a roster or student-detail surface.

Initial query helpers should read OLTP tables directly. Materialized views are a future optimization and must preserve the helper return types.

Data Sources¶

The source of truth is split by concern:

Do not add an analytics event table in V1. The existing domain tables are enough for launch volume.

Migration Notes¶

The move from widget-sniff quizzes to explicit structured quizzes is a behavior change.

Existing content with only legacy assessment widgets will become lesson content with formative practice. This is the desired target, but teams should audit existing courses before shipping the classifier change.

Recommended migration path:

1. Add the structured-quiz widget and leave the old classifier behavior temporarily available behind an implementation flag or branch-only compatibility note.
2. Audit CMS pages that currently classify as quiz because they contain legacy assessment widgets.
3. For pages that need gradebook-visible results, add a structured-quiz widget or split the content into a dedicated quiz page.
4. Remove legacy widget types from QUIZ_WIDGET_TYPES.
5. Update author docs to explain graded quiz vs formative practice.

Do not attempt to auto-convert every legacy widget. The semantics do not map cleanly:

• form.passRate uses percent while Quiz.passThreshold uses a fraction.
• quick-questions uses index-based answers, not stable option ids.
• rubric is self-assessment, not an answer-key quiz.
• checklist can be progress tracking, not correctness.

Implementation Roadmap¶

Phase A - CMS Authoring and Wire Contract¶

• Add structured-quiz schema to @open-learning-hub/widget-wire-schemas.
• Add CMS registry entry and editor component.
• Add authoring validation and one-widget-per-page guidance.
• Add defaults that match QuizSchema.
• Document author-facing graded vs formative distinction.

Phase B - LMS Quiz Extraction and Secure Grading¶

• Add an LMS adapter that extracts structured-quiz content into server-only Quiz.
• Pass only QuizPublic to QuizPlayer.
• Change entry classification to structured-quiz only.
• Rebuild submitQuizAttemptAction to accept answers only and grade server-side.
• Re-fetch the authoritative CMS quiz during submission.
• Persist real scores and feedback.

Phase C - Attempts and Results UX¶

• Replace the stub QuizPlayer submit flow.
• Add QuizResults, pass/fail banner, score line, feedback rows, retry, review mode, and next-entry affordance.
• Enforce maxAttempts and passed-terminal behavior.
• Complete entries on pass or attempts exhausted.
• Add preview-mode read-only behavior.

Phase D - Learner Reports¶

• Add quiz-attempt query helpers scoped by user and tenant.
• Add recent scores and progress to /learn/dashboard.
• Add current-entry review/history surfaces.
• Keep per-user data uncached across users.

Phase E - Admin Analytics¶

• Add analytics query helpers from analytics.md.
• Wire /admin/courses/\[slug\]/dashboard to completion, funnel, and pass-rate cards.
• Add /admin/courses/\[slug\]/dashboard/quizzes for question and attempts-to-pass drilldowns.
• Component-test empty, zero-attempt, and populated states.
• E2E-test analytics after enrollment, progress completion, and quiz submission.

CSV export is intentionally out of scope for this roadmap. Add it after the reporting query contracts settle.

Documentation Corrections¶

The implementation should update adjacent docs when code lands:

• quiz-system.md should clarify that CMS authoring is via structured-quiz widget in V1.
• analytics.md remains the query contract but should link back to this architecture doc.
• planning_milestone_3.md currently overstates completion of Phases 8 and 9 relative to the shipped code; update it or add an audit note when implementation starts.
• hierarchy-structure.md says ordering is CMS-authored and 1-indexed, while current mapping renumbers entries 0-based; avoid relying on raw _order for assessment persistence.

Decisions¶

These were previously open; they are now resolved.

• Page composition: structured-quiz may share a page with non-assessment widgets (explanatory text, images) placed above or below it. Only the quiz widget is graded, and the one-graded-quiz-widget-per-page constraint still holds.
• Review source attempt: review mode shows the learner's latest attempt. A best/highest-scoring view is deferred until a future policy allows retaking already-passed quizzes.
• Admin report granularity: admin reports aggregate first and avoid named learner rows outside explicit roster or student-detail contexts. Dedicated student-detail surfaces can be added later.
• Legacy migration: no migration tooling is provided and no automatic or manual data migration of legacy widget content is performed. The only required change is that the database schema supports structured quiz persistence (the existing quiz_attempts table, extended only if grading needs a new column). Authors recreate any graded checks as new structured-quiz content where desired; existing legacy widgets simply become formative practice.

Related Documents¶

• Quiz System - server grading, retry policy, results UI, Quiz and QuizPublic.
• Analytics - course-admin and tenant-admin query contracts.
• Entry Pagination System - completion and gating handoff.
• Hierarchy Structure - CMS course outline projection.
• Content Rendering - shared widget rendering in LMS pages.