A smartphone-based workflow mapped a welded joint onto a 3D model with a 4.2 mm root mean square error, giving robots a practical coarse target for finishing or inspection. The system combines SegFormer image segmentation, photogrammetry, and multi-view fusion so large workpieces can be screened quickly before a slower, higher-precision scan focuses on the weld.
What Did the Researchers Build?
Researchers built a vision pipeline that finds weld seams in ordinary camera images and places those seams onto a 3D reconstruction of the workpiece. The approach is designed as a fast first step before detailed scanning, robotic grinding, smoothing, finishing, or inspection.
The test setup used a standard smartphone camera, 68 images, and two sheets of AprilTag markers positioned around a welded steel component. AprilTags are printed visual markers with known dimensions. They provide scale and help align the camera views within one shared coordinate system.
The test workpiece consisted of two rectangular hollow steel tubes joined by a continuous circumferential butt weld. It measured 40 cm long, 10 cm wide, and 5 cm high. The camera captured all accessible sides at a resolution of 1,920 × 1,080 pixels.
The same images powered two processing branches. One branch used RealityScan to reconstruct the workpiece and estimate camera positions, camera calibration, and depth maps. The other branch used a SegFormer semantic segmentation model to label each image pixel as either weld or background.
The system then reduced each weld-shaped mask to a thin centerline and projected that centerline into the reconstructed 3D model. The result was a spatial weld map rather than a simple 2D image highlight.
What Were the Key Results?
The workflow produced a usable 3D representation of the test weld, but the results were intentionally presented as a proof of concept rather than a complete industrial accuracy benchmark. The reconstructed workpiece dimensions differed from the physical object by approximately 3 mm. The final projected weld centerline achieved a 4.2 mm root mean square error against a manually annotated reference line.
That accuracy is sufficient for coarse localization, especially when the map is used to direct a later high-precision scan toward a region of interest. It is not yet a replacement for the precise geometry needed to control a grinding tool directly against a weld surface.
| Measurement | Reported result | Practical meaning |
|---|---|---|
| Training images | 388 | Limited image base for the SegFormer model |
| Captured test images | 68 | Multi-view coverage of the workpiece |
| Image resolution | 1,920 × 1,080 pixels | Smartphone-quality input |
| Test workpiece size | 40 × 10 × 5 cm | Physical scale of the demonstration part |
| AprilTag marker spacing | 5.2 cm | Reference for scale and alignment |
| Reconstructed dimensional error | Approximately 3 mm | Overall 3D model accuracy |
| Weld-line localization error | 4.2 mm RMSE | Difference from the manually marked weld path |
The segmentation model identified the weld in most captured images. Missed detections mainly appeared under changing illumination, reflections, or difficult surface appearances. The model also sometimes labelled non-weld areas as welds, creating false spline points.
The source does not report a head-to-head comparison against a laser scanner, structured-light system, or low-resolution depth camera. Instead, the result establishes that smartphone imagery can support a practical coarse-localization stage.
How Does the Pipeline Work?
The process starts by placing AprilTag markers around the welded component. Because the marker dimensions and spacing are known, the tags establish a physical scale and provide reference points for matching views. The smartphone then captures overlapping images from multiple angles, allowing the reconstruction software to identify common visual features across the image set.
RealityScan processes those images into a 3D point cloud or surface representation. It also supplies the camera pose for each image, intrinsic camera parameters, and a depth map. The camera pose describes where the camera was located and oriented when each image was captured. The depth map estimates how far visible surfaces are from the camera.
In parallel, the SegFormer network performs binary semantic segmentation. Rather than detecting a weld as a single bounding box, it labels individual pixels as weld or background. This produces a mask that follows the visible shape of the seam.
The mask is cleaned by removing small noisy regions and separating connected components. Principal component analysis estimates the dominant direction of each component. The algorithm then slices the mask across that direction, calculates a center point in each slice, and connects those points into a spline-like 2D centerline.
Each centerline point is converted from image coordinates into camera coordinates using its depth value and the camera calibration. The corresponding camera pose then transforms the point into the global coordinate system used by the 3D reconstruction. Points from different images are merged, duplicates are removed, and outliers are filtered.
This fusion step is important because one weld can appear in many photographs. Instead of keeping separate, inconsistent 2D detections, the system combines them into one approximate 3D weld path on the reconstructed workpiece.
Why Does This Matter for Robotics?
Robotic post-processing depends on knowing where a weld is before a tool can follow it. A robot performing grinding, deburring, finishing, or inspection needs more than a photograph: it needs a path expressed in the workpiece’s 3D coordinate frame. This workflow connects visual recognition to that spatial representation.
The main operational benefit is selective measurement. A full high-resolution scan of a large component can consume time and generate substantial point-cloud data, much of which is irrelevant. A quick image-based pass can identify likely weld regions, after which a laser scanner, structured-light sensor, or other precision system can inspect only those areas.
That division of labour fits factories that already use industrial arms for finishing or inspection. Businesses assessing equipment for these cells can browse used industrial robots and match the robot’s reach, payload, and tool interface to a targeted scanning or grinding process. Smaller operations could also consider used cobots for sale where human workers and robots share a finishing station.
The approach also reduces dependence on a fixed fixture or preprogrammed weld coordinates. If the workpiece changes position, the photogrammetric model and AprilTag references provide a new coordinate frame. That flexibility is valuable for low-volume manufacturing, repair work, and large fabricated structures.
The current output is best viewed as a guidance map. It narrows the search area for a robot or precision sensor; it does not yet calculate the complete tool orientation, contact force, surface normal, or collision-free motion required for autonomous finishing.
What Are the Limitations and Open Questions?
The model was trained on only 388 images, so its performance depends heavily on the variety represented in that dataset. Lighting changes, glare, surface reflections, image blur, weld appearance, and background materials all caused segmentation errors. Photogrammetry also relies on sufficient texture and image quality; reflective or visually uniform metal can make reconstruction harder.
The experiment used one relatively simple workpiece and one continuous circumferential weld. It does not establish performance across fillet welds, interrupted welds, narrow seams, complex assemblies, or large production components. The reported 4.2 mm RMSE measures localization against a manually annotated line, not the final accuracy of a robot tool at the contact surface.
Future work needs broader training data, stronger augmentation, improved 3D path refinement, local surface-normal estimation, and tool-orientation planning. The key open question is whether the coarse map remains reliable enough to guide precision scanning and robot trajectory generation across varied industrial parts.
Frequently Asked Questions
Can a smartphone camera locate weld seams in 3D? Yes. The workflow combines smartphone images, photogrammetric depth, camera poses, and image-based weld segmentation to create an approximate 3D seam path.
How accurate was the weld localization? The projected weld centerline reached a 4.2 mm root mean square error against a manually annotated reference line.
Can the system directly control a robotic grinder? Not yet. It provides a coarse weld location, while direct tool control still requires refined geometry, surface normals, tool orientation, collision checking, and process-force control.
Why use photogrammetry instead of a full 3D scanner? Photogrammetry offers a lower-cost, wide-area screening step. It can identify regions that deserve detailed scanning without measuring every surface at high resolution.
Conclusion
The research demonstrates an integrated route from smartphone images to a 3D weld-seam map, combining SegFormer segmentation with photogrammetric reconstruction and multi-view fusion. With approximately 3 mm model error and 4.2 mm weld-line RMSE on a test component, the method provides a credible coarse-localization layer for future robotic finishing and inspection systems.