Three AI features now call one shared layer. This page reports the local run: the same inputs on the old code and the new code, on five models, with the numbers and every answer.
Each AI provider had its own code in the backend. A new model often failed in one feature and worked in another. Study Buddy could stream its answer only on Gemini. A change of model needed code changes in several places.
The shared layer has the same code for every provider. A new model is one row in a list. This PR moves the first three features to it.
Study Buddy, quiz import or demo chat asks for an answer.
One code path builds the request for the tenant's provider and model.
Gemini, Claude or OpenAI answers. Study Buddy gets the answer in pieces.
Each request writes one row with tokens and cost.
Three courses for demo chat. One quiz with 10 questions, as a Word file and as a PDF. One lesson transcript with 17 lines for Study Buddy.
Each feature ran as the real feature code on a real model. It ran one time on the old code and one time on the new shared layer.
Gemini with the default model name, Gemini 3.8 Flash, Claude Sonnet 5.5, Claude Haiku 5.5 and GPT-6 Luna. That is 37 runs.
The answer, the time, the usage row and its cost. For quiz import, the answers were compared with the 8 known correct answers.
"Old" is the code on main. "New" is the shared layer.
| Model | Path | Demo chat | Quiz import, Word and PDF | Study Buddy stream |
|---|---|---|---|---|
| Gemini, default model name | Old | Works | Works. 8 of 8 answers right | Streams. 13 text events |
| Gemini, default model name | New | Works | Works. 8 of 8 answers right | Streams. 16 text events |
| Gemini 3.8 Flash | Old | Works | Works. 8 of 8 answers right | Streams. 13 text events |
| Gemini 3.8 Flash | New | Works | Works. 8 of 8 answers right | Streams. 13 text events |
| Claude Sonnet 5.5 | Old | Fails (503) | Fails | No streaming (501) |
| Claude Sonnet 5.5 | New | Works | Works. 8 of 8 answers right | Streams. 13 text events |
| Claude Haiku 5.5 | Old | Fails (503) | Works. 8 of 8 answers right | No streaming (501) |
| Claude Haiku 5.5 | New | Works | Works. 8 of 8 answers right | Streams. 11 text events |
| GPT-6 Luna | Old | Fails (503) | Works. 8 of 8 answers right | No streaming (501) |
| GPT-6 Luna | New | Works | Works. 8 of 8 answers right | Streams. 15 text events |
Read every answer. Old and new answers, side by side. Khai Hoan must approve these before #762 can merge.
Cost is in USD, from the usage row. Reasoning tokens are the tokens the model used to think.
| Model | Feature | Path | Result | Time | Input | Output | Reasoning | Cost USD |
|---|---|---|---|---|---|---|---|---|
| Gemini, default model name | Demo chat | Old | Works. 282 characters | 6.1 s | 700 | 94 | 402 | 0.002385 |
| Gemini, default model name | Demo chat | New | Works. 332 characters | 9.8 s | 700 | 103 | 731 | 0.003652 |
| Gemini, default model name | Quiz import, Word file | Old | Works. 8 of 8 answers right, 11 items | 25.6 s | 2207 | 1626 | 2173 | 0.015558 |
| Gemini, default model name | Quiz import, Word file | New | Works. 8 of 8 answers right, 11 items | 11.4 s | 2207 | 1726 | 0 | 0.008128 |
| Gemini, default model name | Quiz import, PDF file | Old | Works. 8 of 8 answers right, 10 items | 28.0 s | 2855 | 1651 | 1736 | 0.013955 |
| Gemini, default model name | Quiz import, PDF file | New | Works. 8 of 8 answers right, 11 items | 13.3 s | 2855 | 1726 | 0 | 0.008614 |
| Gemini, default model name | Study Buddy, streamed | Old | Works. 13 text events, 9 citations | 9.1 s | 1088 | 381 | 475 | 0.004026 |
| Gemini, default model name | Study Buddy, streamed | New | Works. 16 text events, 11 citations | 7.7 s | 1088 | 429 | 468 | 0.004180 |
| Gemini 3.8 Flash | Demo chat | Old | Works. 344 characters | 8.5 s | 700 | 103 | 786 | 0.003859 |
| Gemini 3.8 Flash | Demo chat | New | Works. 360 characters | 6.5 s | 700 | 107 | 422 | 0.002509 |
| Gemini 3.8 Flash | Quiz import, Word file | Old | Works. 8 of 8 answers right, 11 items | 11.1 s | 2207 | 1394 | 0 | 0.006539 |
| Gemini 3.8 Flash | Quiz import, Word file | New | Works. 8 of 8 answers right, 11 items | 9.5 s | 2207 | 1394 | 0 | 0.006528 |
| Gemini 3.8 Flash | Quiz import, PDF file | Old | Works. 8 of 8 answers right, 11 items | 14.6 s | 2855 | 1103 | 0 | 0.005390 |
| Gemini 3.8 Flash | Quiz import, PDF file | New | Works. 8 of 8 answers right, 11 items | 11.8 s | 2855 | 1726 | 0 | 0.007719 |
| Gemini 3.8 Flash | Study Buddy, streamed | Old | Works. 13 text events, 9 citations | 8.0 s | 1088 | 393 | 597 | 0.004529 |
| Gemini 3.8 Flash | Study Buddy, streamed | New | Works. 13 text events, 9 citations | 8.2 s | 1088 | 410 | 479 | 0.004150 |
| Claude Sonnet 5.5 | Demo chat | Old | Fails (503) | 2.6 s | 0 | 0 | 0 | 0 |
| Claude Sonnet 5.5 | Demo chat | New | Works. 468 characters | 3.9 s | 1114 | 285 | 179 | 0.006868 |
| Claude Sonnet 5.5 | Quiz import, Word file | Old | Fails | 0.7 s | 0 | 0 | 0 | 0 |
| Claude Sonnet 5.5 | Quiz import, Word file | New | Works. 8 of 8 answers right, 11 items | 8.2 s | 3864 | 1540 | 0 | 0.023128 |
| Claude Sonnet 5.5 | Quiz import, PDF file | Old | Fails | 0.4 s | 0 | 0 | 0 | 0 |
| Claude Sonnet 5.5 | Quiz import, PDF file | New | Works. 8 of 8 answers right, 10 items | 7.1 s | 7032 | 1385 | 0 | 0.027914 |
| Claude Sonnet 5.5 | Study Buddy, streamed | Old | No streaming (501) | - | - | - | - | - |
| Claude Sonnet 5.5 | Study Buddy, streamed | New | Works. 13 text events, 11 citations | 4.0 s | 1954 | 595 | 0 | 0.009858 |
| Claude Haiku 5.5 | Demo chat | Old | Fails (503) | 0.0 s | 0 | 0 | 0 | 0 |
| Claude Haiku 5.5 | Demo chat | New | Works. 458 characters | 4.9 s | 1114 | 272 | 443 | no price yet |
| Claude Haiku 5.5 | Quiz import, Word file | Old | Works. 8 of 8 answers right, 10 items | 4.5 s | 4008 | 1409 | 0 | no price yet |
| Claude Haiku 5.5 | Quiz import, Word file | New | Works. 8 of 8 answers right, 10 items | 4.8 s | 3864 | 1385 | 0 | no price yet |
| Claude Haiku 5.5 | Quiz import, PDF file | Old | Works. 8 of 8 answers right, 10 items | 4.5 s | 7176 | 1409 | 0 | no price yet |
| Claude Haiku 5.5 | Quiz import, PDF file | New | Works. 8 of 8 answers right, 10 items | 5.0 s | 7032 | 1456 | 0 | no price yet |
| Claude Haiku 5.5 | Study Buddy, streamed | Old | No streaming (501) | - | - | - | - | - |
| Claude Haiku 5.5 | Study Buddy, streamed | New | Works. 11 text events, 11 citations | 4.9 s | 1954 | 635 | 532 | no price yet |
| GPT-6 Luna | Demo chat | Old | Fails (503) | 2.1 s | 0 | 0 | 0 | 0 |
| GPT-6 Luna | Demo chat | New | Works. 206 characters | 4.7 s | 668 | 69 | 124 | 0.000163 |
| GPT-6 Luna | Quiz import, Word file | Old | Works. 8 of 8 answers right, 10 items | 8.0 s | 2269 | 793 | 129 | 0.000484 |
| GPT-6 Luna | Quiz import, Word file | New | Works. 8 of 8 answers right, 10 items | 7.4 s | 2276 | 791 | 58 | 0.000709 |
| GPT-6 Luna | Quiz import, PDF file | Old | Works. 8 of 8 answers right, 10 items | 7.6 s | 2495 | 801 | 111 | 0.000481 |
| GPT-6 Luna | Quiz import, PDF file | New | Works. 8 of 8 answers right, 10 items | 7.9 s | 2311 | 847 | 0 | 0.000712 |
| GPT-6 Luna | Study Buddy, streamed | Old | No streaming (501) | - | - | - | - | - |
| GPT-6 Luna | Study Buddy, streamed | New | Works. 15 text events, 9 citations | 4.6 s | 1080 | 296 | 59 | 0.000189 |
Checked by hand. Claude Sonnet 5.5, demo chat: 1,114 input tokens at 2.00 USD for a million is 0.002228 USD. 464 output tokens (285 text and 179 reasoning) at 10.00 USD for a million is 0.004640 USD. The sum is 0.006868 USD, the same as the usage row.
Claude Haiku 5.5, a later run with the price: 1,114 input tokens at 0.10 USD for a million is 0.0001114 USD. 572 output tokens at 0.50 USD for a million is 0.000286 USD. The sum is 0.0003974 USD, the same as the usage row.
| Finding | Fix |
|---|---|
| The Study Buddy screen still asked for a streamed answer only on Gemini. Tenants on Claude or OpenAI got the answer in one piece. | The backend now reports streaming for every provider. A test covers it. |
| A gateway (a third-party endpoint for models) got a thinking setting that the old code never sent. Some gateways refuse it. | A gateway gets no thinking setting, as before. |
| In a list of several models, a gateway could get a PDF that it cannot read. | A gateway never gets a file. Quiz import sends the text of the PDF. |
| No test covered quiz import when a model cannot read the PDF. | A test now covers the change to the text of the PDF. |
| Earlier, on #760: gateways lost the change to PDF text. | Fixed in #762 before this review. |
| Decision | Choice |
|---|---|
| Who approves a change in AI answers? | Khai Hoan reads old and new answers side by side before the merge. |
| How do the two new Claude models start? | Marked beta in the list until their answers are reviewed. |
| What does a gateway get? | Text only, no file and no thinking setting, as with the old code. |
| When are the old provider adapters deleted? | In pull request 4, after this comparison is approved. |