OpenMobile-2 Is a CRAZY Open-Source Phone AI - I Tested It on a Fake Phone

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: OpenMobile-2 Is a CRAZY Open-Source Phone AI - I Tested It on a Fake Phone · Channel: Prompt Engineer 48 · 14:11 · Watch on YouTube

Topics: AI agents, Task and workflow automation, AI evaluation and benchmarks, Local and open-source models, Multimodal models

Summary

The video tests OpenMobile-2, a 9B phone-use model, on simulated apps and compares GUI-only interaction with hybrid mode, which adds app-native tools. The presenter explains the open-source stack, Mobile Gym++ environment, training approach, and the authors’ benchmark results, while noting that the larger model performs better and closed models still lead on harder real-device tests. In six personal tasks, GUI-only mode passed none, while hybrid mode passed two and used fewer steps overall, though it also made significant errors. The presenter concludes that the resettable, browser-based simulation environment may be more valuable than the headline score, but the model is not yet a dependable phone assistant.

Key points

OpenMobile-2 includes 9B and 27B Qwen fine-tunes, simulated apps, training data, a training recipe, and checkpoints released under Apache 2.0.
The agent operates in a screenshot-and-action loop, using taps, swipes, text entry, or completion responses.
Mobile Gym++ simulates 63 apps in a browser, with state that can be reset and benchmark tasks checked against app data.
The hybrid setup combines screen interaction with more than 300 app-native tools across more than 50 apps.
The training recipe includes supervised fine-tuning, hybrid training, and reinforcement learning on executable tasks.
The authors report 74.0 on Android World for the 9B model and 79.9 for the 27B, but the larger model trails closed models on harder tests cited in the video.
The presenter ran the simulator locally but served the 9B model remotely because its BF16 weights exceeded the laptop GPU’s memory.
Across six tested tasks, GUI-only mode passed zero; hybrid mode passed two, with total steps dropping from 199 to 82.
Hybrid mode was not consistently reliable: the model missed a major expense and both modes failed the most complex computer-and-TV task.
The presenter sees Mobile Gym++ as a valuable open research environment, while cautioning that the model is not ready as a reliable phone butler.

Chapters

Start	Chapter	Summary

Quotes

So, what they added is not a new architecture, it’s behavior, how to operate a phone. — 01:08

The paper calls this hybrid. — 04:12

60 steps and a fail versus six steps and a pass on the same task. — 09:34

So, no, this isn’t magic. — 12:08

the most valuable thing here isn’t the 79.9, it’s Mobile Gym plus plus — 13:40

Concepts and entities

OpenMobile-2, Mobile Gym++, Qwen, vLLM, RunPod, Playwright, Alipay, WeChat, Android World, GUI-only, modo híbrido

Full transcript (original language)

00:01 This is a 9-billion parameter model driving a fake phone on my laptop. Nobody is touching it. It looks at a screenshot, decides where to tap, and it just opened Alipay. On paper, its bigger sibling, Open Mobile 2 27B, scores 79.9 on Android world. That’s the top number in its own results table. So, I ran this smaller one myself on six tasks twice, and the results taught me more than the score does. >> [music]

00:37 >> Open Mobile 2 is a research release from about two dozen authors with Da Huanglin on the list. The paper is on rx rx and the GitHub repo, and the hugging face models are public. And it’s not just a model, it’s the whole stack, the simulated apps, the training data, the training recipe, and two checkpoints, a 9B and a 27B. Both are Apache 2.0, and both are fine-tunes of Qwen. The 9B starts from

01:08 Qwen 3.5 9B. The 27B starts from Qwen 3.6 27B. So, what they added is not a new architecture, it’s behavior, how to operate a phone. Here’s the loop. You give it a sentence like find my biggest expense in Alipay and write it in my notes. It gets a screenshot, and it answers with one action, tap here, swipe there, type this, or finish. Then it gets the next screenshot. That loop is the entire agent. Let me show you what that looks like in the real logs. The harness sends the

01:39 task and the last three screenshots. The model answers with a short paragraph of reasoning, then one tool call. Coordinates are normalized from 0 to 1,000, so the screen resolution doesn’t matter. Here’s step two from one of my runs. The model writes, “The user wants to find the largest expense in their Alipay transaction records. I’m on the home screen, I see the Alipay icon, so I need to click it. Then it clicks, that’s the whole cognition, and you can read every bit of it. Not every thought is that dry, by the way. In one run, the

02:09 very first line the model wrote was, “This stinks a bit, doesn’t it? We all want to keep our finances in check.” Then it swiped to find the Alipay icon at tab bar with opinions. Why is this hard to train? Because to teach a model to use a phone, you need a huge number of practice runs on real-looking apps, and real apps are a nightmare for that. You can’t easily reset a real shopping account, payments are risky, and a full Android emulator for every single run is heavy. So, the authors say, “Most open projects

02:39 practice on only a few dozen emulator apps, and that’s the bottleneck.” Their idea is to stop using the real apps, rebuild commercial apps as simulated clients in a browser where all the state can be reset cleanly. That’s Mobile Gym plus plus, and I can show you because I’m running it right now on this laptop. It’s a phone that lives in a web page. You clone the repo, you need Node 22, and one command starts a dev server on port 3000. That’s it. No Android SDK, no

03:10 emulator. The readme doesn’t even mention Docker. Here’s the home screen, 63 apps spread across a few pages. Alipay, eBay, Bilibili, WeChat, Meituan, Slack, Notes, Clock. 35 of those are new in this release, and notice the interface is mostly Chinese because many of these are Chinese apps. That’s going to matter later. They’re simulations, they’re not affiliated with the companies they imitate, and the repo says so in a disclaimer, but they have real behavior. This Alipay has a transaction history, a balance, transfers, tap, scroll, and it

03:42 all responds. And here’s the clever part. Every task is created by comparing the business state before and after. No vision model judging a screenshot. Did the note get created with the right number in it? Yes or no? The benchmark has 215 tasks and about half of them cross between apps. The second idea is app native tools. Besides tapping the screen, the model can call functions. The paper says over

04:12 300 tools across more than 50 apps. Things like search bills, summarize bills, create note, send message. Those tools write to the same state as the screen does. So, in one task, the agent can tap through one app, call a tool in another, and the grader still sees one consistent world. The paper calls this hybrid. Their finding here is subtle tool access barely helps a model before it’s trained for it. After hybrid training, the hybrid agent gains the most and finishes

04:43 in fewer steps. Keep that in mind because I tested exactly that. Training is a three-part recipe. First, supervised fine-tuning on about 11.6 thousand demonstration trajectories collected on those simulated apps. Second, a hybrid version mixing tabs and tool calls. Third, reinforcement learning on over 2,000 executable tasks, each with an automatic reward from that state checker. One more finding I liked. With a fixed

05:13 budget of trajectories, spreading them across more apps beats piling them onto fewer apps. It improves unseen apps without hurting the familiar ones. That’s the argument for building 35 apps instead of five. Now, the numbers. These come from the model cards, so they’re the authors’ own evaluations, not mine. On Android world, the 9B scores 74.0. Its base model Gwen 3.59B scores 54.3. Other 8B agents like

05:44 UI Venus, GUI OL, and MAI UI between 69 and 74. The 27B goes to 79.9. The next best in that table is the MAIUI model with 235 billion parameters at 76.7, and Gemini 3.1 Pro is listed at 70.7. But, here’s where I want to be careful because hype would skip this part on Mobile World, which is harder. The 27B scores 50.4. Gemini 3.1 Pro is listed at 58.1, and GPT 5.6 sold at 70.1 on SBA

06:17 bench, which uses real devices. The 27B gets 59.6 against 70.2 for Gemini and 68.1 for GPT. So, the honest reading is near frontier. The jump from the base model is huge. On SBA bench, the 27B goes from 31.9 to 59.6. But, on the hardest real device tests, the closed models are still ahead.

06:47 So, can I run it for the model on my laptop? No. The phone simulator runs fine locally, but the 9B model is about 18 GB of weights in BF16, and my RTX 4060 laptop GPU has 8 GB, and I didn’t find a quantized version on the Hugging Face page. So, I split it. The phone runs on my laptop. The model runs on a rented 48 GB A40 on RunPod served with vLLM, and my laptop calls it like any OpenAI style

07:19 API. It billed at 59 cents an hour, and I deleted it as soon as the runs finished. One practical note, my first run crashed with a Unicode error because the text text is Chinese, and my Windows console isn’t UTF-8. One environment variable, Python UTF-8 mode, fixed it. Worth knowing if you try this on Windows. If you want to try this yourself, here’s the shape of it. Clone the Mobile Gym Plus repo and start the phone. The

07:49 official script expects ZSH show on Windows. I just ran npm install and then npm run dev. Then in a second terminal, make a Python environment, install the bench requirements, and install Playwright’s Chromium. Serve the model with vllm on any machine with enough memory. Then point the benchmark runner at that URL, give it your model name, and pick a mode, GUI only or hybrid. It saves every step, the screenshot before and after, the model’s raw response, and the result file. That’s where all the footage in this

08:20 video comes from. My advice is to keep the official script’s settings as your baseline. Thinking turned on, the last three screenshots in context, coordinates normalized to 1,000, I changed one thing, XML tool call mode, so I didn’t have to restart the server with extra flags. Time to test. The task, “Look at Alipay, tell me whether January 2026 or December 2025 had higher spending, work out the difference, then create a note titled monthly spending comparison.” Same task,

08:50 two modes. First, GUI only. The model can only look and tap. Watch it work. It opens Alipay, finds the bills page, and opens a month picker, which is a spinning wheel, and it struggles. At one point, it swipes with a typo in its own coordinates. It does finally read December 405.6, but after 60 steps, it hits the step limit. Fail.

09:34 Now, the same task in hybrid mode. Step one, it opens Alipay. Step two, instead of tapping through a month picker, it calls a tool summarize bills for January. Step three, same tool for December. Then it does the subtraction in its head, opens notes and creates the note with a tool call. Six steps, the judge says pass. 60 steps and a fail versus six steps and a pass on the same task. That is exactly what the paper claims about tools and I watched it happen on my own machine.

10:04 Another one, find the three biggest expenses from the last 30 days in Alipay and send them to a contact on WeChat ending with a specific sentence. In hybrid mode, this pass in nine steps. And I liked how it got there. It’s first tool call guessed the wrong year, so the search came back with almost nothing. The model noticed that looked thin and fell back to tapping through the bills page by hand. Then it used a tool to find the contact and send the message.

10:36 Now the part you won’t see in a hype video. Here’s the first task start to finish. Home screen, swipe to find Alipay, the my tab, the bills page and a filter to expenses. Now look at step six, right after filtering, it declares the largest expense is 200. It never scrolled through the rest of the list. Then it goes to notes, types the title into the wrong field, notices, fixes it and saves. 17 steps and it reports success.

11:07 That run got partial credit and the judge report is specific. Recipient correct, amount wrong. The note says 200. The ground truth is 2500. It found the right person and missed the biggest transaction. And hybrid mode wasn’t a free win. On that same [clears throat] task, it took 25 steps and scored nothing. Across all six tasks, GUI only passed zero. Hybrid passed two and the total steps dropped from 199 to 82. Let me give you the other three quickly on eBay. Find the

11:38 cheapest new electric fan, check my Alipay balance, and note both GUI only took 25 steps and earned nothing. Hybrid took nine steps and earned 2/3 of the credit. Bilibili, give the top-ranked video the one-tap triple action, then post about it on WeChat Moments. GUI only took 37 steps for 60%. Hybrid took 13 steps for 80%. Close, but still not a pass. The hardest was the computer and TV task. Find the cheapest computer and the cheapest TV on eBay, work out what both would leave in my balance, and write all

12:08 of it in a note. Both modes scored zero. So, no, this isn’t magic. It’s a 9-billion parameter model, and it fails on long, multi-step arithmetic tasks. So, does this contradict the paper? Not on its face. I ran six tasks, not 215. I used the 9B, not the 27B. I used XML tool call mode, so I wouldn’t have to restart my server, and I can’t rule out that my setup differs from the authors

12:39 in ways that cost points. The authors’ own hybrid score for the 9B on this benchmark is 46.5%, so failing some is expected. What I can say, it plays the whole game. It navigates multi-app interfaces, reasons in plain steps, and stops itself. What I can’t say that I reproduced their number. There’s also a community space on Hugging Face linked from the model page that shows a single decision step. You upload a screenshot, type a task, and it predicts the next tap. Its own description says the full loop needs a

13:10 live emulator, so it can’t run the whole thing, but it’s a quick way to poke at the model without any setup. So, who is this for? If you research mobile agents, it’s a gift environments, data, recipes, and checkpoints. All Apache 2.0, including a phone you can run in a browser with one command. If you want a phone butler today, not yet. It’s research code. The code repo has 24 stars and no releases. The apps are simulations and the model cards show dozens of downloads, not thousands.

13:40 My take, the most valuable thing here isn’t the 79.9, it’s Mobile Gym plus plus, a resettable, scriptable, browser-based phone that you can point any model at, including your own. That is what was missing. Next, I want to try the 27B and run the full 215 tasks. If you want that, tell me in the comments and tell me which phone task you’d throw at it. If you’re looking for cheap and reliable GPUs, I use RunPod and the link is in the description. That’s open Mobile too. Thanks for

14:11 watching and I’ll see you in the next one.