(70% OFF) I Tested 10 AI Models on ONE Coding Task — The Price Gap Is INSANE (One API, Real Costs)

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: (70% OFF) I Tested 10 AI Models on ONE Coding Task — The Price Gap Is INSANE (One API, Real Costs) · Channel: Prompt Engineer 48 · 04:22 · Watch on YouTube

Topics: AI evaluation and benchmarks, Software engineering with AI, Markets and economy, AI models

Summary

The creator used Oxy why to test 10 AI models on the same Python regular-expression-engine task, with one API key and a script that calculated costs from token counts. Seven models passed all 80 hidden tests, while others failed on counted repeats, timing traps, or running out of output space; one model timed out and returned an error. The results showed large price and speed differences without better correctness from higher spending on this task. The creator recommends Haiku for small, well-defined jobs, while noting that this was only one run per model and a harder task could change the results.

Key points

The test used one API key and the same OpenAI-compatible chat format to run models through Oxy why.
The account’s model catalog listed 198 models with prices per million tokens, which the script used to estimate each answer’s cost.
The main task was to build a Python regular expression engine without using Python’s re module; it was graded with 80 hidden tests.
The test included 58 comparisons with Python’s re module, 17 invalid patterns, and five timing traps.
An easier calculator-parser warm-up was passed by nine of the 10 models; Gemini 3.7 Flash missed a test involving .5.
Seven models scored 80 out of 80 on the regex task; the reported list prices ranged from about 0.4 cents for Claude Haiku 5.5 to 28 cents for Opus 5.
DeepSeek V4 Flash scored 60 out of 80, while GLM 5.2 used much of its token budget thinking and did not finish its code block.
Grok 4.7 passed the warm-up but timed out twice on the regex task before the provider returned an error, so it received no score.
The creator’s launch promotion billed 40% of list price; the project cost about $1 across 59 requests, including tests, retries, and failed runs.
The creator suggests Haiku for small, well-defined jobs, Sonnet when more headroom is desired, and Opus for jobs where cheaper models fail; the results are limited to one run per model.

Chapters

Start	Chapter	Summary
00:01	One API key and a cost-tracking runner	The creator introduces the price gap between two equally scoring models and explains how the Oxy why runner uses one key and token counts to compare model costs.
00:33	Setting up the test	The video covers the runner, account limits, the regular-expression-engine task, and the 80-test grading setup.
01:39	Warm-up and live model run	After a calculator-parser warm-up, the creator runs three models against the regex task and compares their identical scores with their different costs and response times.
02:11	Full results and failures	The full results show seven perfect scores, differing speed and price, and failure cases involving incorrect repeats, unfinished output, and timeouts.
03:18	Actual billing and recommendations	The creator explains the promotional billing and run-to-run cost variation, recommends models for this kind of task, and cautions that one run is only an anecdote.

Quotes

That’s 66 times more for the exact same answer. — 00:01

Every model gets one try. No hints, no second chance. — 01:39

So, on this task, paying more bought no extra correctness. — 02:44

The whole project, every test, retry, and failed run, billed about $1 across 59 requests. — 03:50

One key, and I change a model name. — 04:22

Concepts and entities

Oxywhy, Claude Haiku 5.5, Claude Sonnet 5.5, Opus 5, Gemini 3.5 Flash, Gemini 3.7 Flash, GPT-5.6 Soul, GPT-5.6 Terra, DeepSeek V4 Flash, GLM-5.2, Grok 4.7, Expressão regular

Full transcript (original language)

00:01 Two AI models wrote a working regular expression engine from scratch. Same prompt, same 80 hidden tests, same perfect score. One cost under half a cent, the other cost 28 cents. That’s 66 times more for the exact same answer. I’ve lost count of how often I’ve picked a model on gut feeling then juggled a different key and a different bill for each one. So, I went looking for one key that reaches all of them, found Oxy why and used it to run 10 models on the same coding job. I wrote down what every answer really cost. Some passed, some failed in ways that surprised me. Oxy why gave me some credits to run these

00:33 tests and here’s how it fits into my actual workflow. This is the runner I wrote for these tests. It’s a short Python script and the only Oxy why parts are the base URL and the key. Everything else is the normal OpenAI chat format. First, it asks the models endpoint for the catalog. On my account, that’s 198 models each with its price per million tokens. So, my script can work out what every answer costs from the token counts in the response. After that, the loop is just a list of model names to test a different model. I change one string Claude, GPT, Gemini, Deep Seek, Grok. Same call, same key, same bill. To get a

01:06 key, you sign up OpenAI keys, name it and copy it because it’s only shown once. I won’t show mine. If you want to try Oxy why yourself, go to oxywhy.ai and sign up. The promo code is in the description. My account is on tier one. So, that’s 60 requests a minute and five at once. I’ll come back to that limit because I hit it. The job I picked, write a small regular expression engine in Python. Literals.star plus question mark counted repeat alternation, groups, and character classes and no Python re module allowed. Here’s the nasty part. Nested patterns on a 40 character string mustn’t blow up exponentially. I picked

01:39 it because a machine can grade it. 80 hidden tests, 58 checked against Python’s real re module, 17 bad patterns that must raise an error, and five timing traps. I wrote my own reference engine first and it scores 80 out of 80. So, I know the grader is fair. Every model gets one try. No hints, no second chance. Before that, an easier warm up. A calculator parser. Nine of the 10 models got every test. Gemini 3.7 flash missed one, a number written as .5. So, an easy task tells you almost nothing because everything passes. That’s why I moved on to the regex engine. Here’s a live run with three of the models

02:11 against the real API. Sonnet answers in about 30 seconds, Haiku in about 40, GPT 5.6 Soul takes 94. All three score 80 out of 80. Now, look at the cost column. My script works it out from the token counts in each response. Haiku is half a cent, Sonnet about 5 cents, Soul 12 cents. Same score, very different bill. Here’s the full table from the full run. Seven models got a perfect score. Cheapest is Claude Haiku 5.5 at 4/10 of a cent, then Gemini 3.5 flash, Sonnet 5.5, Gemini 3.7 flash, and GPT 5.6 Terra, all between 4 and 6 cents. Soul

02:44 is 9 cents, and Opus 5 is 28 cents. Speed splits them differently. Sonnet and Haiku finished in about 35 seconds. Gemini 3.5 flash took almost 3 minutes for a perfect answer. So, on this task, paying more bought no extra correctness. Sometimes it bought speed, sometimes nothing. But, this was one task, and a much harder one might change the order. Now, the two that failed, DeepSeek V4 flash got 60 out of 80. Look at which ones it missed. Most of the counted repeats, those came back wrong, and it never finished the five timing traps. It cost under a cent, so the price looks great right up until you read the

03:18 output. GLM 5.2 scored zero. It wasn’t wrong, exactly. It ran out of room, roughly 28,000 of its 32,000 tokens went on thinking, and the code block never closed. I call it a token budget problem, and I only capped it at 32,000. And Grok 4.7, on the easy round it passed, but it took 5 minutes. On the regex round it timed out twice, and then the provider returned an error, so there’s no score for it, and I won’t guess. Now, the real money. Everything I quoted is list price. My account had a launch promo, so I’m billed 40% of list, and the activity page confirms it down to the cent. The whole project, every

03:50 test, retry, and failed run, billed about $1 across 59 requests. One oddity, the activity page lists my Gemini 3.5 flash requests under 3.7 flash at the cheaper rate, so treat that pair with care. And those costs aren’t fixed. I ran Sonnet, Haiku, and Sol a second time. Sol used about a third more tokens and cost about a third more. Same prompt, same model. So what would I use for small, well-defined jobs like this one? Haiku first. It was the cheapest among the fastest, and it got every test right. Move up to Sonnet when you want more headroom, and keep Opus for jobs where cheaper models actually fail,

04:22 which this test didn’t show. Two caveats: one run per model is an anecdote, and a harder task could separate these models. That’s the runner I’ll keep using for these tests. One key, and I change a model name. Tell me in the comments which model you want me to run next.