ReSpace: Text-Driven 3D Indoor Scene Synthesis and Editing with Preference Alignment

How ReSpace Works

At the heart of our method lies a Structured Scene Representation (SSR) in JSON format, together with short object prompts for object addition in natural language.

For single object addition, an object prompt gets fed, together with the existing SSR, into SG-LLM, a specially trained model for spatial reasoning and object placement. We train SG-LLM via SFT+GRPO.

To go from text (SSR) to mesh-based scene, we employ a sampling engine for 3D assets that matches geometry and semantics for a queried object.

For object removal and full scene synthesis, we leverage a zero-shot LLM to edit the SSR directly in text space (removal) and to generate object prompt lists for addition (full scenes) that get passed autoregressively into SG-LLM.

Abstract

Scene synthesis and editing has emerged as a promising direction in computer graphics. Current trained approaches for 3D indoor scenes either oversimplify object semantics through one-hot class encodings (e.g., 'chair' or 'table'), require masked diffusion for editing, ignore room boundaries, or rely on floor plan renderings that fail to capture complex layouts. LLM-based methods enable richer semantics via natural language (e.g., `modern studio with light wood furniture'), but lack editing functionality, are limited to rectangular layouts, or rely on weak spatial reasoning from implicit world models. We introduce ReSpace, a generative framework for text-driven 3D indoor scene synthesis and editing using autoregressive language models. Our approach features a compact structured scene representation with explicit room boundaries that enables asset-agnostic deployment and frames scene editing as a next-token prediction task. We leverage a dual-stage training approach combining supervised fine-tuning and preference alignment, enabling a specially trained language model for object addition that accounts for user instructions, spatial geometry, object semantics, and scene-level composition. For scene editing, we employ a zero-shot LLM to handle object removal and prompts for addition. We further introduce a voxelization-based evaluation capturing fine-grained geometry beyond 3D bounding boxes. Experimental results surpass state-of-the-art on addition and achieve superior human-perceived quality on full scene synthesis.

Experiments: Qualitative Results

We conduct experiments with the 3D-FRONT and 3D-FUTURE datasets on both single object addition and removal and full scene synthesis. For both, we perform quantitative comparison with baselines: ATISS and MiDiffusion. More details are in the paper.

Instructions (object addition)

Full Scene Synthesis (BoN=1)

Prompt List (8): “plush cushions bed”, “gray minimalist nightstand”, “traditional three-door wardrobe”, “dual-tone table”, “decorative padded beige seat chair”, “white planter”, “wooden and metal bookcase”, “modern lamp”

Prompt List (11): “modern brown leather sofa”, “grey corner table with crossbar”, “low-profile floor lamp”, “simple white coffee table”, “blue velvet armchair”, “wooden wall art”, “classic pendant lamp”, “low storage cabinet”, “mid-century wooden side table”, “tv stand”, “hanging plant”

Full Scene Synthesis (BoN=8)

BibTeX

If using our dataset/model or if you found our work useful, please cite us as follows:

@article{bucher2025respace, title={ReSpace: Text-Driven 3D Indoor Scene Synthesis and Editing with Preference Alignment}, author={Bucher, Martin JJ and Armeni, Iro}, journal={arXiv preprint arXiv:2506.02459}, year={2025} }