This is the home of the new WAC3 Ai Benchmark! What is WAC3? It is the What About Chocolate Chip Cookies Ai Benchmark, yes that’s right, chocolate chip cookies. So in our mucking about with different Ai models we started asking a couple standard questions, “How many chocolate chips must a cookie contain to be a chocolate chip cookie?” followed by “Can you give me a simple chocolate chip cookie recipe?”. The results were interesting and we liked the questions because the first is somewhat abstract but would be simple for any child to answer, and the follow up (now informed by the first question) requires a level of precision and reasoning to be successful.
Baking, unlike cooking, requires ingredients to be within a certain threshold to be able to achieve the desired results. Not enough leavening agent and they wont rise, too much binding agent and they are a rock, and not to mention we get to compare how many chocolate chips end up in the cookie compared to how many the Ai suggested need to be in the cookie to be considered a “chocolate chip cookie”.
So Yv has taken on actually baking the cookie recipes, and for a control we use the standard Nestle Tollhouse Chocolate Chip Cookie Recipe. We then make a description and images of the bake and a somewhat subjective decision on taste, etc., and roll these results in to a comparison with the Ai answers to the number of chips in the cookie to how many the Ai said a cookie needs, ingredient ratios compared to our control, and more.
Mostly we are testing the Ai without access to the web, though we may include a couple with it, and many of these models are light enough to run on a home computer or even a phone. So now that you are on board, check out the current WAC3 Ai Benchmark below, or all the individual Ai responses, recipes, and cookie photos and ratings here.
| Ai Model | Mistral 7b | Qwen 2.5:8b | DeepSeek R1:8b | Gemma 2:2b |
| WAC3 Score (Quality of Finished Cookie, 0-10) | 7 | 7 | 2 | 6 |
| Ai Choc Chip Qty | 5-10 | 1+ | 1+ | 15-20 |
| Actual Choc Chip Qty | 12 | 8 | 17 | 11 |
| Batch Size | 28 | 56 | 24 | 38 |
| Chip Ratio to Control | 0.5 | 1.0 | 1.0 | 1.0 |
| Flour Ratio to Control | 0.7 | 1.3 | 0.4 | 1.0 |
| Egg Ratio to Control | 0.5 | 1.0 | 0.5 | 1.0 |
| Butter Ratio to Control | 0.5 | 1.0 | 0.5 | 1.0 |
| Baking Soda Ratio to Control | 0.5 | 1.0 | 0.0 | 1.0 |
| Granulated Sugar Ratio to Control | 0.7 | 1.0 | 1.0 | 1.0 |
| Brown Sugar Ratio to Control | 0.7 | 1.0 | 0.0 | 1.0 |
| Vanilla Ratio to Control | 1.0 | 2.0 | 1.0 | 2.0 |
| Salt Ratio to Control | 0.5 | 1.0 | 0.0 | 0.5 |
| Baking Time Ratio to Control | 0.9 | 1.1 | 1.1 | 1.0 |
| Baking Temperature Ratio to Control | 1.0 | 1.0 | 1.0 | 1.0 |
