跳转至

Experiment 6-11 AndroidWorld iteration report

  • Run ID: exp6-11-20260804T045559Z
  • Generated (UTC): 2026-08-04T09:13:41Z
  • Upstream commit: d9c569f764b3a5629321858de03ff653d0f24056
  • Device: sdk_gphone64_x86_64, API 33 (upstream tested reference: API 33)
  • Observation method: uiautomator_compact
  • Provider/model: local-vllm / qwen2.5-7b-instruct-local
  • Model source/runtime: local_gpu / vllm-0.19.0
  • Accelerator: NVIDIA_RTX_PRO_6000_Blackwell_96GB
  • Required apps: 24/24
  • Scope: 116 task(s), 5 trial(s), mode candidate_rerun
  • Full 116-task × 5-seed suite completed: false

The bundled ~88% baseline is historical input evidence. The manuscript's 88%→94% numbers are explicitly hypothetical and are not used as rerun results here.

1. Diagnose

  • The historical run evaluated 116 tasks once each and reports approximately 88% overall success.
  • Wi-Fi is a concentrated failure cluster: three of the four SystemWifiTurn* rows failed in the bundled per-task table.
  • The capability matrix links the cluster to weak complex_ui_understanding, information_retrieval, and requires_setup behavior.
  • The failed traces show navigation/state-verification loops; increasing the step cap alone would treat a symptom rather than the cause.

2. Hypothesis

The diagnosis produced explicit surface, middle, and deep hypotheses. Only one variable is changed in this run; the other hypotheses remain untested.

Layer / ID Proposed change Target Verification Status
surface / H1 Add Wi-Fi Settings navigation and final-state verification guidance. At least one net paired success across the four Wi-Fi tasks, with no regression. Paired upstream-prompt versus task-guideline ablation with matched seeds. tested in source phase 1
surface / H2 Add application-specific recognition rules for the non-standard Tasks UI. Improve at least two of the six historical Tasks failures with no regression. Paired Tasks-only prompt/tool-description ablation after app provisioning. not tested
middle / H3 Repair and validate the multimodal input path for transcription tasks. Raise transcription success above the historical 0% while bounding added tokens and latency. Paired screenshot-disabled versus screenshot-enabled transcription run. not tested
middle / H4 Conditionally enable deeper thinking for counting tasks. Improve math/counting success without applying the cost to unrelated tasks. Paired tag-routed thinking-mode ablation with latency and token guardrails. not tested
middle / H5 Replace the API-35-incompatible gRPC accessibility feed with upstream's UIAutomator observation path. At least one net paired Wi-Fi success with no regression and at most 1.5x latency/tokens. Paired a11y-forwarder versus UIAutomator run with the same upstream T3A prompt and matched seeds. tested in source phase 2
middle / H5C Filter non-semantic UIAutomator container nodes after H5 exposed excessive prompt-token cost. Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator. tested in this run
deep / H6 Combine screenshots with the structured UI tree and compare stronger vision-capable models. Improve complex-UI success enough to justify multimodal latency and token cost. Factorial UI-tree/screenshot/model ablation on the full tagged slice. not tested

Selected hypothesis: H5C - Change: Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements. - Expected measurable result: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. - Guardrails: Same model, seed, task parameters, emulator, checkout, and step budget; require at least four completed pairs, every compact-UIAutomator treatment pair successful, no paired regression, at most 1.5x mean latency, and at most 0.75x raw-UIAutomator mean tokens. Passing a paired gate permits only a full-suite candidate rerun; it is not deployment approval, and a subset must never be reported as full-suite success.

3. Controlled experiment

  • Phase: phase_2_cost_refinement — middle-layer input-pipeline cost refinement after the H5 success/cost result
  • Independent variable: raw versus semantic-filtered UIAutomator element list
  • Controls: same checkout, model, task parameters, generated seed, step budget, and emulator; arm order alternates by pair.
Arm Episodes Success Reward Steps Latency (s) LLM calls Mean tokens Input / output tokens Est. cost (USD)
candidate 116/116 0.060 0.147 9.681 115.062 19.052 171630.552 19773398 / 135746 0.000000

4. Data-driven decision

  • Outcome: candidate_subset_rerun_completed
  • Reason: A real modified candidate subset rerun completed, but it is not the 116-task × five-trial gate and cannot approve deployment.
  • Treatment/control mean latency ratio: n/a
  • Treatment/control mean token ratio: n/a
  • Treatment/control mean LLM-call ratio: n/a
  • Cost guardrails passed: false
  • Deployment approved: false

5. Rerun and next report

This run is a real controlled subset/smoke rerun, not the complete AndroidWorld benchmark. The next gate is a conditionally enabled candidate rerun over all 116 tasks with five seeds after provisioning the upstream API-33 app environment.

Observed residual failures: - candidate / AudioRecorderRecordAudio / trial 5: final evaluator state passed, but the agent never declared completion - candidate / AudioRecorderRecordAudioWithFileName / trial 5: evaluator reward / completion gate was not satisfied - candidate / BrowserDraw / trial 5: evaluator reward / completion gate was not satisfied - candidate / BrowserMaze / trial 5: evaluator reward / completion gate was not satisfied - candidate / BrowserMultiply / trial 5: evaluator reward / completion gate was not satisfied - candidate / CameraTakePhoto / trial 5: evaluator reward / completion gate was not satisfied - candidate / CameraTakeVideo / trial 5: evaluator reward / completion gate was not satisfied - candidate / ClockStopWatchPausedVerify / trial 5: final evaluator state passed, but the agent never declared completion - candidate / ClockStopWatchRunning / trial 5: evaluator reward / completion gate was not satisfied - candidate / ClockTimerEntry / trial 5: agent declared completion, but the real evaluator state failed - candidate / ContactsAddContact / trial 5: evaluator reward / completion gate was not satisfied - candidate / ExpenseAddMultiple / trial 5: evaluator reward / completion gate was not satisfied - candidate / ExpenseAddMultipleFromGallery / trial 5: evaluator reward / completion gate was not satisfied - candidate / ExpenseAddSingle / trial 5: evaluator reward / completion gate was not satisfied - candidate / ExpenseDeleteDuplicates / trial 5: final evaluator state passed, but the agent never declared completion - candidate / ExpenseDeleteDuplicates2 / trial 5: evaluator reward / completion gate was not satisfied - candidate / ExpenseDeleteMultiple / trial 5: final evaluator state passed, but the agent never declared completion - candidate / ExpenseDeleteMultiple2 / trial 5: evaluator reward / completion gate was not satisfied - candidate / ExpenseDeleteSingle / trial 5: final evaluator state passed, but the agent never declared completion - candidate / ContactsNewContactDraft / trial 5: evaluator reward / completion gate was not satisfied - candidate / ExpenseAddMultipleFromMarkor / trial 5: evaluator reward / completion gate was not satisfied - candidate / FilesDeleteFile / trial 5: evaluator reward / completion gate was not satisfied - candidate / FilesMoveFile / trial 5: evaluator reward / completion gate was not satisfied - candidate / MarkorAddNoteHeader / trial 5: evaluator reward / completion gate was not satisfied - candidate / MarkorChangeNoteContent / trial 5: evaluator reward / completion gate was not satisfied - candidate / MarkorCreateNote / trial 5: evaluator reward / completion gate was not satisfied - candidate / MarkorCreateNoteAndSms / trial 5: evaluator reward / completion gate was not satisfied - candidate / MarkorCreateNoteFromClipboard / trial 5: evaluator reward / completion gate was not satisfied - candidate / MarkorDeleteAllNotes / trial 5: evaluator reward / completion gate was not satisfied - candidate / MarkorDeleteNewestNote / trial 5: evaluator reward / completion gate was not satisfied - candidate / MarkorDeleteNote / trial 5: evaluator reward / completion gate was not satisfied - candidate / MarkorEditNote / trial 5: evaluator reward / completion gate was not satisfied - candidate / MarkorMergeNotes / trial 5: evaluator reward / completion gate was not satisfied - candidate / MarkorMoveNote / trial 5: evaluator reward / completion gate was not satisfied - candidate / MarkorTranscribeReceipt / trial 5: evaluator reward / completion gate was not satisfied - candidate / MarkorTranscribeVideo / trial 5: evaluator reward / completion gate was not satisfied - candidate / NotesTodoItemCount / trial 5: evaluator reward / completion gate was not satisfied - candidate / OpenAppTaskEval / trial 5: evaluator reward / completion gate was not satisfied - candidate / OsmAndFavorite / trial 5: evaluator reward / completion gate was not satisfied - candidate / OsmAndMarker / trial 5: agent declared completion, but the real evaluator state failed - candidate / OsmAndTrack / trial 5: evaluator reward / completion gate was not satisfied - candidate / RecipeAddMultipleRecipes / trial 5: evaluator reward / completion gate was not satisfied - candidate / RecipeAddMultipleRecipesFromImage / trial 5: evaluator reward / completion gate was not satisfied - candidate / RecipeAddMultipleRecipesFromMarkor / trial 5: evaluator reward / completion gate was not satisfied - candidate / RecipeAddMultipleRecipesFromMarkor2 / trial 5: evaluator reward / completion gate was not satisfied - candidate / RecipeAddSingleRecipe / trial 5: evaluator reward / completion gate was not satisfied - candidate / RecipeDeleteDuplicateRecipes / trial 5: evaluator reward / completion gate was not satisfied - candidate / RecipeDeleteDuplicateRecipes2 / trial 5: evaluator reward / completion gate was not satisfied - candidate / RecipeDeleteDuplicateRecipes3 / trial 5: evaluator reward / completion gate was not satisfied - candidate / RecipeDeleteMultipleRecipes / trial 5: evaluator reward / completion gate was not satisfied - candidate / RecipeDeleteMultipleRecipesWithNoise / trial 5: evaluator reward / completion gate was not satisfied - candidate / RecipeDeleteSingleWithRecipeWithNoise / trial 5: final evaluator state passed, but the agent never declared completion - candidate / RetroCreatePlaylist / trial 5: evaluator reward / completion gate was not satisfied - candidate / RetroPlaylistDuration / trial 5: evaluator reward / completion gate was not satisfied - candidate / RetroSavePlaylist / trial 5: evaluator reward / completion gate was not satisfied - candidate / SaveCopyOfReceiptTaskEval / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleCalendarAddOneEvent / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleCalendarAddOneEventInTwoWeeks / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleCalendarAddOneEventRelativeDay / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleCalendarAddOneEventTomorrow / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleCalendarAddRepeatingEvent / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleCalendarAnyEventsOnDate / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleCalendarDeleteEvents / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleCalendarDeleteEventsOnRelativeDay / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleCalendarDeleteOneEvent / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleCalendarEventOnDateAtTime / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleCalendarEventsInNextWeek / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleCalendarEventsInTimeRange / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleCalendarEventsOnDate / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleCalendarFirstEventAfterStartTime / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleCalendarLocationOfEvent / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleCalendarNextEvent / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleCalendarNextMeetingWithPerson / trial 5: final evaluator state passed, but the agent never declared completion - candidate / SimpleDrawProCreateDrawing / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleSmsReply / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleSmsResend / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleSmsSend / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleSmsSendClipboardContent / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleSmsSendReceivedAddress / trial 5: agent declared completion, but the real evaluator state failed - candidate / SportsTrackerActivitiesCountForWeek / trial 5: evaluator reward / completion gate was not satisfied - candidate / SportsTrackerActivitiesOnDate / trial 5: evaluator reward / completion gate was not satisfied - candidate / SportsTrackerActivityDuration / trial 5: evaluator reward / completion gate was not satisfied - candidate / SportsTrackerLongestDistanceActivity / trial 5: evaluator reward / completion gate was not satisfied - candidate / SportsTrackerTotalDistanceForCategoryOverInterval / trial 5: evaluator reward / completion gate was not satisfied - candidate / SportsTrackerTotalDurationForCategoryThisWeek / trial 5: evaluator reward / completion gate was not satisfied - candidate / SystemBluetoothTurnOff / trial 5: evaluator reward / completion gate was not satisfied - candidate / SystemBluetoothTurnOffVerify / trial 5: final evaluator state passed, but the agent never declared completion - candidate / SystemBluetoothTurnOn / trial 5: evaluator reward / completion gate was not satisfied - candidate / SystemBluetoothTurnOnVerify / trial 5: final evaluator state passed, but the agent never declared completion - candidate / SystemBrightnessMax / trial 5: evaluator reward / completion gate was not satisfied - candidate / SystemBrightnessMin / trial 5: evaluator reward / completion gate was not satisfied - candidate / SystemBrightnessMinVerify / trial 5: agent declared completion, but the real evaluator state failed - candidate / SystemCopyToClipboard / trial 5: evaluator reward / completion gate was not satisfied - candidate / SystemWifiTurnOff / trial 5: evaluator reward / completion gate was not satisfied - candidate / SystemWifiTurnOn / trial 5: agent declared completion, but the real evaluator state failed - candidate / TasksCompletedTasksForDate / trial 5: evaluator reward / completion gate was not satisfied - candidate / TasksDueNextWeek / trial 5: evaluator reward / completion gate was not satisfied - candidate / TasksDueOnDate / trial 5: evaluator reward / completion gate was not satisfied - candidate / TasksHighPriorityTasks / trial 5: evaluator reward / completion gate was not satisfied - candidate / TasksHighPriorityTasksDueOnDate / trial 5: evaluator reward / completion gate was not satisfied - candidate / TasksIncompleteTasksOnDate / trial 5: agent declared completion, but the real evaluator state failed - candidate / TurnOffWifiAndTurnOnBluetooth / trial 5: evaluator reward / completion gate was not satisfied - candidate / TurnOnWifiAndOpenApp / trial 5: evaluator reward / completion gate was not satisfied - candidate / VlcCreatePlaylist / trial 5: evaluator reward / completion gate was not satisfied - candidate / VlcCreateTwoPlaylists / trial 5: evaluator reward / completion gate was not satisfied - candidate / RetroPlayingQueue / trial 5: evaluator reward / completion gate was not satisfied - candidate / SimpleSmsReplyMostRecent / trial 5: evaluator reward / completion gate was not satisfied - candidate / NotesIsTodo / trial 5: final evaluator state passed, but the agent never declared completion - candidate / NotesMeetingAttendeeCount / trial 5: evaluator reward / completion gate was not satisfied

LLM analysis of this run

The following bounded interpretation was produced by the configured real LLM from the aggregate evidence (the JSON remains authoritative):

  • Summary: The candidate subset of 116 tasks was rerun, achieving a success rate of 6.03% with 7 successes out of 116 episodes. The mean latency was 115.06 seconds, and the mean total tokens were 171,630.55. The cost-benefit analysis indicates that while the candidate subset showed some success, the high latency and token usage suggest further optimization is needed.
  • Cost/benefit interpretation: The candidate subset showed some success but with high latency and token usage. Further optimization is needed to reduce these metrics while maintaining or improving success rates.
  • Residual pattern: Most tasks failed to achieve success, with only 7 out of 116 tasks completing successfully.
  • Residual pattern: The mean latency of 115.06 seconds is significantly high, indicating potential inefficiencies in the current implementation.
  • Residual pattern: The mean total tokens of 171,630.55 are also high, suggesting that the model is using more resources than necessary.
  • Next hypothesis H5C (middle): Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements. Target: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. Verification: Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator.

Environment boundaries

  • UIAutomator is an upstream AndroidWorld observation option selected by the companion runner; it preserves real UI actions/evaluators but is a compatibility path, not the upstream API-33 reference configuration.
  • Compact UIAutomator removes only non-semantic container nodes; observations, coordinates, Android actions, and AndroidWorld evaluators remain real.
  • The full-suite candidate uses model qwen2.5-7b-instruct-local, while the promoted paired H5C source used doubao-seed-1-6-250615. This local-GPU campaign evaluates the promoted observation treatment but is not a same-model extension of the paired result.
  • ContactsNewContactDraft's official success predicate was fed the upstream UIAutomator state.ui_elements because that observation mode does not populate state.forest; the predicate and requested contact fields were not changed.
  • Clipboard get/set retries once after the exact Clipper foreground-access runtime error; the operation, content, task, and evaluator are unchanged.
  • SimpleSmsReplyMostRecent polls the unchanged inbox query for up to five additional seconds because emulator-injected SMS delivery can lag past upstream's fixed wait. If the inbox remains empty, the exact last injected address/body is inserted into the same SMS database that upstream clears directly; task data and the evaluator are unchanged.
  • The pinned official Retro Music APK omits the playing_queue table, a known upstream runtime error. Only that exact missing-table condition was mapped to an empty observed queue so the unchanged exact queue predicate records an evaluator failure instead of losing the episode.
  • Runtime-error retries reuse the exact task parameters retained in the discarded error checkpoints; upstream parameter-generator drift cannot silently change the retried task.
  • If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes 8,192 characters only from the middle of the current-screen indexed UI section. Prompt prefix, goal, history, leading and trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
  • If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: at most 12,000 retained characters for action selection or 6,000 for each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.

The JSON beside this report is the authoritative evidence. It contains episode-level evaluator rewards, actions, timing, token counts, configuration, and explicit completion gates; credentials and raw prompts are not stored.