Expanding Robot Sensing | MIT News

Robots have come a long way since Roomba. Nowadays, drones are starting to deliver goods, autonomous cars are driving some roads, robot dogs are helping first responders, and more robots are bounced back and helping on the floors of factories. Still, Luca Carlone believes the best is that it hasn’t arrived yet.

Carlone recently served as associate professor at AeroAstro for MIT Aeroastro, where he directed Spark Lab, where he and his students bridged the key gap between humans and robots: perception. The team conducted theoretical and experimental studies, all of which expanded robots’ understanding of the environment in a way that was close to human perception. As Carlone often says, perception is more than just detection.

Although robots have made great strides in their ability to detect and recognize objects around them, they still have a lot to learn when it comes to improving their sense of the environment. As humans, we have an intuitive sense of not only their shape and labels, but also their physical sense – how they are manipulated and moved, and how they relate to each other, the greater environment and ourselves.

This human-level view is how Carlone and his team hope to grant robots so that they can safely and seamlessly interact with people at home, workplaces, and other unstructured environments.

Since joining MIT faculty in 2017, Carlone has led his team to develop and apply understanding algorithms for perception and scenarios, including a variety of applications, including autonomous underground search and rescue vehicles, drones that can carry and manipulate objects, and autonomous driving cars. They may also be useful for domestic robots that follow natural language commands, and may even predict human needs based on advanced contextual cues.

“Perception is a big bottleneck that allows robots to help us in the real world,” Caron said. “If we could add cognitive and reasoning elements to robot perception, I believe they can do a lot.”

Expand your horizon

Born and raised near Salerno, Italy, Carlone is close to the scenic coast of Amalfi, and is the youngest of three boys. His mother, a retired elementary school teacher who teaches mathematics, and his father, a retired history professor and publisher, has been taking an analytical approach to his historical research. The brothers may unconsciously adopt the parental mindset, as the three continue to be engineers-the older two pursue electronic and mechanical engineering, while Carlone lands in robotics or electromechanical currency, as well as well known.

However, he did not arrive at the venue until his undergraduate studies were very late. Carlone attended the Polytechnic University in Turin, where he initially focused on theoretical work, especially control theory, in which he applied mathematics to develop algorithms that automatically control the behavior of physical systems, such as power grids, aircraft, cars and robots. Carlone then signed up for a course on robotics during his senior year, which explored advances in manipulation and how to program a robot to move and function.

“It’s love at first sight. Using algorithms and mathematics to develop a robot’s brain and move it and interact with the environment is one of the most satisfying experiences,” Caron said. “I immediately decided that this is what I wanted to do.”

He went on to earn his master’s degree in mechatronics and automation engineering in both the Polytechnical University of Turin and Polytechnical University of Milan, respectively. As part of the program, Carlone, also known as Alta Scuola Politecnica, also took a management course where he and students from a variety of academic backgrounds had to collaborate on conceptualization, establishing and developing marketing promotions for new product designs. Carlone’s team has developed a touchless desk lamp designed to follow user manual commands. The project prompted him to think about engineering from a different perspective.

“It’s like having to speak a different language,” he said. “It’s early access to the need to go beyond the engineering bubble and think about how to create technical jobs that could impact the real world.”

The next generation

Carlone stayed in Turin (Turin) to complete his PhD in Mechatronics. During this time, he was given the freedom to choose the subject of the paper, and he recalled “a little naive”.

“I’m exploring a topic that is considered to be mastered by many researchers who think there is nothing to say,” Caron said. “I underestimated the degree to which this topic was established and thought I could still make new contributions to it, and I was lucky enough to do so.”

The topic discussed is “localization and mapping at the same time” or SLAM – the issue of generating and updating maps of the robot’s environment while tracking the position of the robot in that environment. Carlone proposed a way to reframe the problem so that the algorithm can generate more precise maps without having to start with initial guesses, like most SLAM methods did at the time. His work helps to open up a field where most robots think people are doing better than existing algorithms.

“The Grand Slam is about figuring out the geometry of things and how robots move between them,” Caron said. “Now I’m part of a community, what are the next generation of Grand Slams?”

To find answers, he accepted a postdoctoral position at Georgia Tech, where he was upset about his vision of coding and computers—a field that may have been inspired by blind brushing: when he completed his PhD in Italy, he suffered severely affected medical complications.

“I was easily disappointed over the past year,” Caron said. “It made me think about the importance of vision and artificial vision.”

He was able to get good medical care and his condition was completely resolved so that he could continue to work. At Georgia Tech, his consultant Frank Dellaert showed him the way to code in a computer vision and developed elegant mathematical representations of complex three-dimensional problems. His consultant was also one of the first to develop an open source Grand Slam library, called GTSAM, and Carlone was quickly considered a valuable resource. More broadly, he saw great potential for advances in robotics to make software available to everyone.

“Historically, the Grand Slam progressed very slowly because people kept proprietary code and each group essentially had to start from scratch,” Caron said. “Then the open source pipeline started popping up, which changed the rules of the game, which largely drove the progress we’ve seen over the past decade.”

You have space

After Georgia Tech, Carlone joined MIT in 2015 as a postdoctoral fellow at the Laboratory of Information and Decision Systems (LIDS). During this time, he worked with Sertac Karaman, a professor of aviation and astronauts, to develop software to help palm-sized drones harness their surroundings using very few on-board power supplies. A year later, he was promoted to research scientist, and then in 2017, Carlone accepted a teacher position in Aeroastro.

“One of the things I fell in love with MIT is that all decisions are driven by the following questions: What is our mission? It’s not about low-level benefits. Motivation is really about how to improve society,” Caron said. “It’s very refreshing as a mindset.”

Today, Carlone’s team is developing methods that represent the robot’s surroundings instead of describing its geometry and semantics. It can be said that he is using deep learning and large language models to develop algorithms that enable robots to perceive environments through advanced lenses. Over the past six years, his lab has released more than 60 open source repositories, which are used by thousands of researchers and practitioners around the world. Much of his work fits in a larger emerging field known as “space AI.”

“Space AI is like slamming steroids,” Caron said. “In short, it’s useful to enable robots to think and understand the world like humans.”

This is a huge undertaking that can have a wide range of impacts in enabling more intuitive interactive robots to help at home, in the workplace, on the road, and in remote and potentially dangerous areas. There will be a lot of work to get closer to humanity’s view of the world, Caron said.

“I have twin daughters of 2 years old and I see them manipulating objects, carrying 10 different toys at a time, navigating easily across messy rooms and quickly adapting to new environments. The perception of the robot cannot match what a toddler can do,” Carlone said. “But we have new tools at Arsenal. The future is bright.”