Agents operating in complex software environments benefit from reasoning about the consequences of their actions, as even a single incorrect user interface (UI) operation can derail long, artifact-preserving workflows. This challenge is particularly acute for computer-using scenarios, where real execution does not support counterfactual exploration, making large-scale trial-and-error learning and planning impractical despite the environment being fully digital and deterministic. We introduce the Computer-Using World Model (CUWM), a world model for desktop software that predicts the next UI state given the current state and a candidate action. CUWM adopts a two-stage factorization of UI dynamics: it first predicts a textual description of agent-relevant state changes, and then realizes these changes visually to synthesize the next screenshot. CUWM is trained on offline UI transitions collected from agents interacting with real Microsoft Office applications, and further refined with a lightweight reinforcement learning stage that aligns textual transition predictions with the structural requirements of computer-using environments. We evaluate CUWM via test-time action search, where a frozen agent uses the world model to simulate and compare candidate actions before execution. Across a range of Office tasks, world-model-guided test-time scaling improves decision quality and execution robustness.<\/p>\n","protected":false},"excerpt":{"rendered":"
Agents operating in complex software environments benefit from reasoning about the consequences of their actions, as even a single incorrect user interface (UI) operation can derail long, artifact-preserving workflows. This challenge is particularly acute for computer-using scenarios, where real execution does not support counterfactual exploration, making large-scale trial-and-error learning and planning impractical despite the environment […]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[{"type":"text","value":"Yiming Guan","user_id":0},{"type":"text","value":"Rui Yu","user_id":0},{"type":"text","value":"John Zhang","user_id":0},{"type":"edited_text","value":"Lu Wang","user_id":"44027"},{"type":"edited_text","value":"Chaoyun Zhang","user_id":"42387"},{"type":"text","value":"Liqun Li","user_id":0},{"type":"edited_text","value":"Bo Qiao","user_id":"37848"},{"type":"edited_text","value":"Si Qin","user_id":"42006"},{"type":"text","value":"He Huang","user_id":0},{"type":"edited_text","value":"Fangkai Yang","user_id":"41425"},{"type":"edited_text","value":"Pu Zhao","user_id":"38886"},{"type":"edited_text","value":"Lukas Wutschitz","user_id":"38775"},{"type":"edited_text","value":"Samuel Kessler","user_id":"43566"},{"type":"edited_text","value":"Huseyin Atahan Inan","user_id":"40426"},{"type":"edited_text","value":"Robert Sim","user_id":"36650"},{"type":"edited_text","value":"Saravan Rajmohan","user_id":"41039"},{"type":"text","value":"Qingwei Lin","user_id":0},{"type":"edited_text","value":"Dongmei Zhang","user_id":"31665"}],"msr_publishername":"","msr_publisher_other":"","msr_booktitle":"","msr_chapter":"","msr_edition":"","msr_editors":"","msr_how_published":"","msr_isbn":"","msr_issue":"","msr_journal":"","msr_number":"","msr_organization":"","msr_pages_string":"","msr_page_range_start":"","msr_page_range_end":"","msr_series":"","msr_volume":"","msr_copyright":"","msr_conference_name":"ICLR'26 Workshop on World Models","msr_doi":"10.48550\/arXiv.2602.17365","msr_arxiv_id":"2602.17365","msr_mag_id":"","msr_other_authors":"","msr_other_contributors":"","msr_speaker":"","msr_award":"","msr_affiliation":"","msr_institution":"","msr_host":"","msr_version":"","msr_duration":"","msr_release_tracker_id":"","msr_highlight_type":"","msr_date_display_format":"","msr_main_download_label":"","msr_external_link_label":"","msr_doi_label":"","msr_published_date":"2026-04-01","msr_startdate":"","msr_presentation_date":"","msr_highlight_text":"","msr_notes":"","msr_longbiography":"","msr_publicationurl":"","msr_external_url":"","msr_secondary_video_url":"","msr_conference_url":"","msr_journal_url":"","msr_year":2026,"msr_month":4,"msr_day":1,"msr_microsoftintellectualproperty":false,"msr_pub_id":"94af43d6fab77aeabb9eecab2e02754c6f23e504","msr_publication_uploader":[{"type":"url","title":"https:\/\/dblp.org\/rec\/journals\/corr\/abs-2602-17365.html","label_id":243109,"id":false,"viewUrl":false},{"type":"url","title":"https:\/\/arxiv.org\/abs\/2602.17365","label_id":243109,"id":false,"viewUrl":false}],"msr_related_uploader":[],"msr_original_fields_of_study":[],"msr_s2_paper_id":"","msr_s2_pdf_url":"","msr_citation_count_updated":"","msr_citation_count":0,"msr_influential_citations":0,"msr_reference_count":0,"msr_s2_open_access":false,"msr_s2_author_ids":[],"msr_pub_ids":[{"provider":"s2","id":"94af43d6fab77aeabb9eecab2e02754c6f23e504"},{"provider":"doi","id":"10.48550\/arXiv.2602.17365"},{"provider":"arxiv","id":"2602.17365"},{"provider":"dblp","id":"journals\/corr\/abs-2602-17365"},{"provider":"corpusid","id":"285787857"}],"msr_hide_image_in_river":0,"footnotes":""},"msr-research-highlight":[],"research-area":[13556,13563],"msr-publication-type":[193716],"msr-publisher":[],"msr-publication-cta":[],"msr-focus-area":[],"msr-locale":[268875],"msr-post-option":[],"msr-field-of-study":[246691],"msr-conference":[270401],"msr-journal":[268188],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-1176804","msr-research-item","type-msr-research-item","status-publish","hentry","msr-research-area-artificial-intelligence","msr-research-area-data-platform-analytics","msr-locale-en_us","msr-field-of-study-computer-science"],"msr_publishername":"","msr_edition":"","msr_affiliation":"","msr_published_date":"2026-04-01","msr_host":"","msr_duration":"","msr_version":"","msr_speaker":"","msr_other_contributors":"","msr_booktitle":"","msr_pages_string":"","msr_chapter":"","msr_isbn":"","msr_journal":"","msr_volume":"","msr_number":"","msr_editors":"","msr_series":"","msr_issue":"","msr_organization":"","msr_how_published":"","msr_notes":"","msr_highlight_text":"","msr_release_tracker_id":"","msr_original_fields_of_study":"","msr_download_urls":"","msr_external_url":"","msr_secondary_video_url":"","msr_longbiography":"","msr_microsoftintellectualproperty":0,"msr_main_download":"","msr_publicationurl":"","msr_doi":"10.48550\/arXiv.2602.17365","msr_publication_uploader":[{"type":"url","title":"https:\/\/dblp.org\/rec\/journals\/corr\/abs-2602-17365.html","label_id":243109,"id":false,"viewUrl":false},{"type":"url","title":"https:\/\/arxiv.org\/abs\/2602.17365","label_id":243109,"id":false,"viewUrl":false}],"msr_related_uploader":[],"msr_citation_count":0,"msr_citation_count_updated":"","msr_s2_paper_id":"","msr_influential_citations":0,"msr_reference_count":0,"msr_arxiv_id":"2602.17365","msr_s2_author_ids":[],"msr_s2_open_access":false,"msr_s2_pdf_url":null,"msr_attachments":[],"msr-author-ordering":[{"type":"text","value":"Yiming Guan","user_id":0,"rest_url":false},{"type":"text","value":"Rui Yu","user_id":0,"rest_url":false},{"type":"text","value":"John Zhang","user_id":0,"rest_url":false},{"type":"edited_text","value":"Lu Wang","user_id":44027,"rest_url":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Lu Wang"},{"type":"edited_text","value":"Chaoyun Zhang","user_id":42387,"rest_url":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Chaoyun Zhang"},{"type":"text","value":"Liqun Li","user_id":0,"rest_url":false},{"type":"edited_text","value":"Bo Qiao","user_id":37848,"rest_url":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Bo Qiao"},{"type":"edited_text","value":"Si Qin","user_id":42006,"rest_url":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Si Qin"},{"type":"text","value":"He Huang","user_id":0,"rest_url":false},{"type":"edited_text","value":"Fangkai Yang","user_id":41425,"rest_url":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Fangkai Yang"},{"type":"edited_text","value":"Pu Zhao","user_id":38886,"rest_url":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Pu Zhao"},{"type":"edited_text","value":"Lukas Wutschitz","user_id":38775,"rest_url":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Lukas Wutschitz"},{"type":"edited_text","value":"Samuel Kessler","user_id":43566,"rest_url":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Samuel Kessler"},{"type":"edited_text","value":"Huseyin Atahan Inan","user_id":40426,"rest_url":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Huseyin Atahan Inan"},{"type":"edited_text","value":"Robert Sim","user_id":36650,"rest_url":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Robert Sim"},{"type":"edited_text","value":"Saravan Rajmohan","user_id":41039,"rest_url":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Saravan Rajmohan"},{"type":"text","value":"Qingwei Lin","user_id":0,"rest_url":false},{"type":"edited_text","value":"Dongmei Zhang","user_id":31665,"rest_url":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Dongmei Zhang"}],"msr_impact_theme":[],"msr_research_lab":[],"msr_event":[],"msr_group":[1157919],"msr_project":[1155944],"publication":[],"video":[],"msr-tool":[],"msr_publication_type":"inproceedings","related_content":{"projects":[{"ID":1155944,"post_title":"Agents for Productivity","post_name":"agents-for-productivity","post_type":"msr-project","post_date":"2025-11-19 14:51:12","post_modified":"2025-11-21 10:20:04","post_status":"publish","permalink":"https:\/\/find.codeghost.online\/en-us\/research\/project\/agents-for-productivity\/","post_excerpt":"Agents for Productivity (A4P) is a M365 Research initiative to enable Microsoft to deliver reliable, highly capable, and scalable agentic solutions that drive measurable productivity impact. The strategy addresses two core challenges: technological gaps (tool integration\/selection, memory & context management, advanced reasoning) and operationalization barriers (realistic benchmarks, prod\u2011like environments, unified evaluation & tech transfer). The approach is composable and platform\u2011driven, pairing foundational components (orchestration, procedural memory, planning) with a research kit (benchmarks, environments, evaluation\/debug pipelines)…","_links":{"self":[{"href":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/wp\/v2\/msr-project\/1155944"}]}}]},"_links":{"self":[{"href":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/1176804","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item"}],"about":[{"href":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-research-item"}],"version-history":[{"count":3,"href":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/1176804\/revisions"}],"predecessor-version":[{"id":1176807,"href":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/1176804\/revisions\/1176807"}],"wp:attachment":[{"href":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1176804"}],"wp:term":[{"taxonomy":"msr-research-highlight","embeddable":true,"href":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/wp\/v2\/msr-research-highlight?post=1176804"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1176804"},{"taxonomy":"msr-publication-type","embeddable":true,"href":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/wp\/v2\/msr-publication-type?post=1176804"},{"taxonomy":"msr-publisher","embeddable":true,"href":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/wp\/v2\/msr-publisher?post=1176804"},{"taxonomy":"msr-publication-cta","embeddable":true,"href":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/wp\/v2\/msr-publication-cta?post=1176804"},{"taxonomy":"msr-focus-area","embeddable":true,"href":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/wp\/v2\/msr-focus-area?post=1176804"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1176804"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=1176804"},{"taxonomy":"msr-field-of-study","embeddable":true,"href":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/wp\/v2\/msr-field-of-study?post=1176804"},{"taxonomy":"msr-conference","embeddable":true,"href":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/wp\/v2\/msr-conference?post=1176804"},{"taxonomy":"msr-journal","embeddable":true,"href":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/wp\/v2\/msr-journal?post=1176804"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=1176804"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/find.codeghost.online\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=1176804"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}