Real-Time 3D Room Scanning & Semantic Geometry Capture with Apple's RoomPlan

Background

Capturing the measurements and geometry of space with a mobile device is becoming an increasingly common task. Knowing how rooms and spaces are organized with dimensions and what they contain is useful for everything from real-estate, design, architecture, construction, shopping, and even AR/VR entertainment.

Advancements in mobile phone depth sensors, like LiDAR, and photogrammetric techniques have made capturing the points in space of the room to create a point cloud or mesh a routine process in recent years. However, that 3D mesh is not structured data and the geometry has to be extracted for key dimensional elements and meta information on captured elements.

Mesh vs. Semantic Geometry

In the above examples, we can see that the mesh is not structured. On the other hand the Semantic Geometry has extracted key dimensional elements meta information on captured elements.

Semantic Geometry Capture

Typically to obtain the geometry and measurements of a space, a mesh or point cloud capture was performed first. Then the information was determined through various algorithms and ML techniques. A problem Laan Labs (and many many others) have been working on.

Why is 2D & 3D space geometry capture important?

• Virtual Spaces - Design, construction and other processes now rely on digital-doubles of spaces.
• Gaming / Entertainment - Bringing a real room into a vr/ar experience requires that the space is captured accurately. This way a table in a game is a table in a room.
• Real Estate - Showing floorplans and measures are not a de facto part of this business.
• Industrial Applications - Planning and manufacturing requires layout asset management.
• And Many More…

CREDIT: @GregMadison

How has this been done until now?

AR Tagging Approach

In an AR session a user manually tags corners along the floor-plane to create walls.

Left: Floor Plan App | Right: CREDIT @pixelplacement

Scan & Trace

Geometry is extracted semi-manually after a capture is complete by extracting a plan-view slice. The process of completing scans of individual rooms and then combining them in a single view with some manual and automated alignment (ICP). Then the complete scan is view from top to trace a floorplan on top of. Some geometries such as walls can be automatically extracted to aid in the process.

ML (Machine Learning) Techniques

Cloud processing for where ML Algorithms would automatically analyze the scanned mesh and extract floorplans.

Left: ML processed 3D Floorplan | Right: Room Floorplan outline from ML

Enter Real-Time 3D Semantic Geometry Capture, Apple's RoomPlan

Apple's new RoomPlan feature introduces real-time geometry capture for LiDAR devices in iOS 16. This "First Principal" approach targets creating a geometric representation of a space rather than capturing textures and meshes. Check out the RoomPlan Mode in 3D Scanner App.

The RoomPlan SDK makes it accessible to quickly add to apps. Some great demos:

Left: CREDIT Shopify (Russ Maschmeyer @StrangeNative) | Right: CREDIT @planner5d

RoomPlan Mode in 3D Scanner App

Laan Labs was super excited to add a RoomPlan mode to 3D Scanner App to complement our other capture modes of LiDAR, TrueDepth, and Photogrammetry.

New Features:

• RoomPlan + Texture - With 3D Scanner App, the geometries can be textured for a better representation of the space
• Automatic Dimensions - Measurement for a space can be automatically calculated.
• 360 Photos - With relocalization, once a RoomPlan capture is complete, you can add 360 Photos to a scan.

Still a long way to go

While RoomPlan is very impressive, It's (still?) very basic. Obviously this is the first version and Apple has large resources and talented developers at its disposal so can expect things to improve. The main issues as of September 2022:

• Accuracy - While very accurate, it can grab onto geometry that introduces inaccuracies as opposed to a traditional scan.
• Odd Shapes - RoomPlan works best on square spaces. Vaulted ceiling and odd shaped rooms can cause issues.
• Only General Info - not details of typical floorplan like doorswings, etc
• Object Hierarchy and connections - Walls are not currently connected in a way that typical floorplans work.

Right: credit vid2cad

Wrapup

Overall, real-time capture like RoomPlan are opening amazing opportunities for how users can interact with 3D data in various workflows. If you want to see how advancements in 3D Semantic Geometry Capture like Apple's RoomPlan can help with your use case, please don't hesitate to contact us.