Post #20

Researchers at MIT, Stanford University, Intelligence Lab, and the Autodesk AI Lab developed AI that can figure out Lego Instructions Scientists collaborated to develop a learning-based framework that can travel 2D instructions to build 3D objects. This system called the Manual-to-Executable-Plan Network (MEPNet) was successfully tested on Lego sets and Minecraft-style building plans. So it will definitely help people who were driven mad with confusing Lego manuals. But the key idea is to integrate neural 2D keypoint detection modules and 2D-3D projection algorithms for high-precision prediction of unseen components. Interpreting 2D instructions could be tricky for AI. The key problems are identifying correspondence between 2D and 3D objects, and dealing with a lot of basic objects, which could be assembled into complex forms. «It requires inferring 3D poses of unseen components composed of seen primitives," the researchers said. At first, MEPNet analyses the current state of Lego set and creates 3D model of all components. Then the algorithm predicts a set of 2D keypoints and masks for each component. Once that's done, the 2D keypoints "are back-projected to 3D by finding possible connections between the base shape and the new components." The combination "maintains the efficiency of learning-based models, and generalizes better to unseen 3D components," the team wrote. The full paper of MEPNet is available via the link. And the algorithm’s code is also posted on GitHub. #AI#ML