Building Rugroom: from a room photo to an interactive 3D scene

A coffee table leg took more debugging than I expected. The rug looked convincing in the middle of the floor, but moving it underneath the table exposed a thin strip of incorrect pixels. That small failure ended up saying a lot about the project.

Rugroom lets someone place a rug inside a room image, move it around, rotate it and compare different placements. The visible interaction is simple. Underneath it are several systems that need to agree about the same photograph: image analysis, camera geometry, furniture visibility, lighting and a responsive browser renderer.

I built the project across Python and JavaScript. The analysis work turns pixels into scene information. The browser turns that information into something a person can interact with. Most of the difficult work happened at the boundary between those two responsibilities.

This is a development walkthrough of the decisions that mattered, the failures that helped me understand the system and the tools I used to make progress reproducible. The code samples are deliberately reduced examples of those engineering patterns. They leave out the product's model configuration, correction rules and deployment internals.

Keeping the photograph and the product intact

I wanted the selected rug to remain the selected rug. Its pattern, outline and proportions should survive the interaction. Generating a new room image on every change would make that harder to control.

I chose a hybrid scene. The room photograph provides the background. Three.js renders the rug as a mesh, with its actual product texture. Analysis supplies the information needed to place that mesh and hide the right parts of it behind furniture.

That decision made repeated interaction possible without running inference every time the user moved a rug. It also made the rendering mistakes easier to see. The photograph is already visually coherent. Anything added to it has to earn its place.

The scene is tied to the photograph's camera view. I do not reconstruct every hidden surface in the room. There is no complete model of the back of the sofa waiting outside the frame. The goal is a consistent placement experience from the view the photograph gives me.

The stack, with a job for each library

The visualizer itself uses JavaScript modules and Vite. React and Next.js belong to this portfolio, not the core scene runtime. Keeping that distinction clear matters when describing what I actually built.

The libraries handle useful pieces. My work is in how those pieces agree on dimensions, coordinate systems, ownership and lifecycle. A valid tensor and a valid texture can still produce an incorrect image when their assumptions differ.

Two paths through the architecture

A new photograph and a prepared room should not take the same route through the system. A new photo needs analysis. A prepared room should reuse the work already done.

The diagram shows the main responsibilities rather than every internal service. The distinction between the two paths is also how I think about performance. Model inference belongs to preparing a scene. Moving a rug belongs to interacting with that scene.

On the server, accepting a job, running it and delivering its result are separate events. In the browser, receiving that result and applying it to the currently selected room are separate events too. Treating all of them as one long request made failures much harder to reason about.

Defining the boundary between Python and the browser

A segmentation result is useful, but it is not yet a scene. The browser also needs to know which image the result belongs to, how large that image is and how its geometry should be interpreted.

I keep the photo and its analysis associated with the version that produced them. That lets me reopen an old result honestly. A change to the renderer should not make an earlier model run appear to have come from a newer pipeline.

This is a reduced type sketch of that boundary. The actual application uses JavaScript, and its production payload has additional fields. TypeScript is useful here because it makes the relationship between values visible.

type AnalyzedRoom = {
  photoHash: string
  analysisVersion: string
  imageSize: [width: number, height: number]
  masks: {
    floor: string
    furniture: string
  }
  camera: {
    focalPx: number
  }
  floorPlane: [a: number, b: number, c: number, d: number]
}

The image dimensions are particularly important. A mask at one resolution, a focal length expressed at another resolution and a texture uploaded with a different orientation can all be individually valid. Combining them without an explicit conversion gives you a scene that looks almost right and fails around the edges.

On the Python side, I use PyTorch for inference and array-based processing for the outputs. I inspect intermediate masks and geometry instead of treating the model call as an opaque success or failure. Model loading order and memory use also became part of the work, especially while running several large models locally.

The following shows the inference boundary in isolation. model, preprocess and image are supplied by the caller. It is not the project's model recipe.

import torch

def run_inference(model, preprocess, image, device):
    model.eval()
    inputs = preprocess(image).to(device)

    with torch.inference_mode():
        output = model(inputs)

    return output

The important application decision is what happens around that call: how inputs are normalized, which intermediate outputs are retained and how failures are reported. A successful inference call does not prove that the resulting room can support a convincing placement.

Why a floor mask was only the beginning

My first useful decomposition was to separate three questions. Which pixels look like floor? Where are the objects? How do the visible surfaces sit relative to the camera?

Those answers can disagree. A chair may have a correctly identified seat and a missing leg. A broad furniture region may cover the real floor visible through a metal frame. A dark area may be a shadow, a piece of fabric or an opening underneath a sofa.

For rug placement, both kinds of error are visible. Missing furniture pixels let the rug paint over an object. Extra furniture pixels remove pieces of the rug where nothing should be hiding it.

One room, three views

Source photoAnalysisPlacement

The source image. Look at the coffee table legs, sofa edges and visible floor openings.

I started describing failures by location and behavior. “The gap on the right of this table leg closes when the rug moves underneath it” was much more useful than “the mask needs improvement.” It gave me a specific observation to reproduce after a change.

For prepared rooms, I also reviewed selected boundaries against the source image. I kept those authored corrections separate from automatic model outputs. A carefully reviewed room is useful for a live demonstration, but it should not be counted as proof that every unseen image will work equally well.

The most useful quality comparisons kept the source photograph next to both results. In one review, I examined ten selected regions across three room images using two analysis flows. Some differences were concrete: a raised fireplace step remained in front of the rug in one result, while the other allowed the rug to cover it. Around a sofa and footstool, both flows preserved the main objects, and the differences were subtler.

I did not turn that small review into a global accuracy percentage. Nor did I use similarity to an older mask as the final definition of quality. The photograph was the reference I needed to explain the visible failure.

Getting from a pointer to the floor

Dragging feels simple because the cursor is on a flat screen. The rug, however, has a position on a plane in the scene. I need to translate between those spaces without quietly changing the rug's physical footprint.

A useful starting point is to normalize the pointer relative to the renderer's visible rectangle, construct a camera ray and intersect that ray with the floor plane. Three.js exposes the camera-ray step through Raycaster.setFromCamera.

import { Raycaster, Vector2, Vector3 } from 'three'

const raycaster = new Raycaster()
const pointer = new Vector2()
const hit = new Vector3()

function pointerOnFloor(event, canvas, camera, floorPlane) {
  const rect = canvas.getBoundingClientRect()
  if (!rect.width || !rect.height) return null

  pointer.set(
    ((event.clientX - rect.left) / rect.width) * 2 - 1,
    1 - ((event.clientY - rect.top) / rect.height) * 2
  )

  raycaster.setFromCamera(pointer, camera)
  return raycaster.ray.intersectPlane(floorPlane, hit)
}

This sample assumes the displayed image and camera viewport are already aligned. The application also deals with the photograph's fit, visible floor constraints and the offset between the pointer and the grabbed point. Those details are what keep a rug from jumping when a drag begins.

I use quarter turns as a practical geometry check. Rotating a rug should preserve its physical dimensions and floor contact. It should not change size because a screen-space bounding box changed shape.

I also had to check the same relationships after CSS zoom, viewport changes and different device pixel ratios. Pointer coordinates arrive in CSS pixels. The drawing buffer may have a different resolution. The photograph and masks have their own dimensions again. Much of the debugging was about making those conversions explicit.

Depth estimated from a single photograph is still an estimate. These checks establish consistency inside the scene. Without measured room references, they do not establish centimetre accuracy in the real room.

Lighting belongs to the room

Once the perspective and furniture boundaries looked better, the rug still looked too independent of its surroundings. The room had window light, warm surfaces and shadows. The added material looked as though it had arrived with its own lighting.

I built a lighting path that uses visible floor samples and contact information from the photograph. The difficult part is separating illumination from appearance. Wood grain and an existing rug pattern should not become a new pattern projected onto the selected product.

Inspect the lighting contribution

Lighting onLighting off

Room lighting enabled. Compare the rug around the coffee table and the lower edge of the sofa.

These two images come from the same development review. The source room and placement are held constant. Switching the lighting contribution makes the relationship around the coffee table and sofa easier to inspect.

The lighting field also needs to remain registered to the photograph. When a user drags the rug into a darker part of the room, the darker region should stay in the room. Attaching that field to the rug's own texture coordinates would move it with the product.

Single-view lighting is an approximation. I cannot recover every light source or hidden object surface. I can, however, keep the effect consistent with the parts of the photograph I can observe, and apply it through the same material path used by the main rug, comparisons and exports.

Moving image work off the UI thread

The lighting calculation runs in a browser worker. That keeps the computation away from the thread handling controls and scene interaction. The worker receives image samples and returns a typed array that the renderer can use.

The small messaging boundary is worth showing. This reduced version follows the worker pattern used in the project, with the estimator left out.

const worker = new Worker(
  new URL('./lighting.worker.js', import.meta.url),
  { type: 'module' }
)

// A temporary buffer owned by this calculation
worker.postMessage(
  { pixels, width, height },
  [pixels.buffer]
)

The transfer list changes ownership of the underlying buffer. The sending side cannot continue to use it as before. I use temporary arrays at this boundary so transferring them does not invalidate a buffer still needed by the scene. The ownership behavior is described in MDN's transferable objects documentation.

The other half of the problem is lifecycle. A result may arrive after the user has selected another room. Applying a perfectly valid lighting result to the wrong photograph is still a rendering bug.

let activeRoom = null

async function activateRoom(room) {
  activeRoom = room
  const result = await prepareLighting(room)

  if (activeRoom !== room) return
  applyLighting(result)
}

This captures the identity check. The full application also cancels pending work, applies timeouts, terminates workers and disposes of replaced GPU textures. The guard protects scene correctness. Cleanup protects the resources used to produce it. I needed both.

Making fabric movement behave under furniture

A rug that slides like a rigid board looks wrong. A rug that waves freely underneath a sofa also looks wrong. I wanted a small material response that stayed connected to the placement constraints.

I added bounded surface movement driven by dragging and rotation. It settles back to rest after interaction. Areas estimated to be under furniture remain constrained, and the woven edge follows the primary mesh.

I think of this as contact-aware visual behavior. It is not a full simulation of woven cloth and hidden furniture geometry. That scope helped me choose what to test: grounded contacts, bounded displacement, stable dimensions and a surface that actually returns to rest.

Frame rate also matters. A motion update that applies a fixed fraction per frame can feel different on different devices. Using elapsed time in the response is part of keeping the interaction consistent.

Here is the kind of invariant I care about, shown as a reduced Node test. The helper names are illustrative. In the project, these checks live alongside tests for motion, screen geometry and contact behavior.

import assert from 'node:assert/strict'

const result = simulateDragAndRelease({
  durationSeconds: 3,
  framesPerSecond: 60
})

assert(result.minimumHeight >= 0)
assert.equal(result.contactDisplacement, 0)
assert.equal(result.finalEnergy, 0)
assert.deepEqual(result.finalSize, result.initialSize)

A screenshot would not tell me whether the motion depends on frame rate or whether the rug eventually settles. Conversely, a passing numerical check would not tell me whether the edge looks detached. I use both kinds of evidence.

Saving analysis so I can debug the renderer

Running the entire analysis pipeline after every shader or interaction change made the feedback loop unnecessarily slow. It also introduced another moving part when I was trying to isolate a visual difference.

I built a replay workflow around saved photographs and their analysis outputs. I can reopen a room, place the same rug at a known location and inspect the change without starting new inference. This became one of the most useful development tools in the project.

For prepared rooms, I package the reviewed background, visibility information, lighting atlas and placement data ahead of time. Opening those rooms does not require segmentation, depth inference or a new lighting calculation.

An early September browser record measured the first prepared-room opening at about 2.2 seconds. The next three room switches were 309, 163 and 142 milliseconds. These are measurements from one local development run, not current production latency guarantees. They helped me verify the value of keeping scene preparation out of the repeated interaction path.

IndexedDB stores rooms locally so reopening a saved room can reuse its existing result. A saved result retains its actual analysis version. A newer renderer does not silently claim that an older photograph has been analyzed by a newer model.

Long-running jobs need more than a loading spinner

The server-side work needed similar discipline. An accepted analysis job should not disappear because a browser disconnected. A result delivery retry should not mean running the models again.

I separated durable job state from the lifetime of an individual request. The system records work that has been accepted, controls permission to execute it and handles repeated result delivery as the same logical operation.

There is an important tradeoff when a submission becomes ambiguous. If I cannot establish whether work already started, blindly retrying may execute it twice. Some uncertain jobs are deliberately left in an explicit unresolved state instead. That is a more honest outcome than pretending every network timeout means nothing happened.

I keep those concerns behind the analysis boundary. The browser needs understandable progress and a reusable result. It does not need the service credentials or the internal model layout.

The unglamorous browser work mattered too

Photo handling produced failures before analysis even began. A file picker can provide a usable image with an empty MIME type. HEIF support varies by decoding path. Orientation metadata can change which way the pixels need to face.

I consolidated photo preparation so the desktop and phone entry points use the same behavior. exifr reads the relevant camera metadata, and libheif-js supplies a locally bundled WASM fallback when native decoding is unavailable. Invalid files have a visible failure path rather than disappearing after selection.

The phone handoff also has several states: connecting, uploading and letting the desktop open the result. Generating a QR code only solves the address exchange. Recovery after an initial connection failure is a different part of the feature.

For the catalogue, I generate smaller previews with sharp instead of sending full scene textures to tiny cards. A product image used for rendering and the thumbnail used to choose it have different jobs. I also avoid fetching desktop hover images on a phone where that interaction is unavailable.

Exports have to use the same scene interpretation. I use jsPDF for document generation and qrcode for the handoff links, but the hard part is retaining the camera, lighting and visibility behavior in the exported result. A correct browser view is not enough if the downloaded image changes the placement.

Testing the failures I actually saw

I use Node assertions for deterministic geometry and motion checks, and Puppeteer for the parts that need a real browser. That includes pointer dragging, repeated rotation, changing rooms, mobile layouts and export behavior.

The visual checks are built around recorded problem areas. I keep the source photo, analysis revision, placement and comparison images together. When changing a boundary rule, I can return to the regions that previously worked as well as the one I am trying to fix.

One useful review pattern is to compare both versions through the current renderer. That reduces the chance of crediting a mask change for an unrelated lighting change. When two full analysis flows produce different geometry or lighting inputs, I also record that limitation rather than calling it a controlled model comparison.

The tests have caught issues outside image analysis too. Repeated quarter turns exposed geometry inconsistencies. Room switching exposed stale asynchronous work. Mobile photo fixtures exposed decoding and orientation problems. Export checks exposed assumptions that were only true in the visible canvas.

This is where the project became much more satisfying to work on. I could make a change, explain which behavior it was supposed to affect and produce evidence of what actually happened.

What I built, beyond the final screenshot

Rugroom brought together work that I usually encounter in separate projects: Python inference, image diagnostics, 3D geometry, shader integration, browser resource management and durable asynchronous jobs.

The part I am proudest of is being able to trace a visible failure through those layers. A bad edge may come from a mask. It may also come from a coordinate conversion, an alpha channel, an incorrectly registered lighting texture or a result arriving for a room that is no longer active.

I can now separate more of those causes, replay the relevant state and test the behavior that changed. The final screenshot matters, but being able to reproduce and explain it is the engineering result I value most.