Overview
We present semi-supervised deep learning approaches for traversability estimation from fisheye images. Our method, GONet, and the proposed extensions leverage Generative Adversarial Networks (GANs) to effectively predict whether the area seen in the input image(s) is safe for a robot to traverse. These methods are trained with many positive images of traversable places, but just a small set of negative images depicting blocked and unsafe areas. This makes the proposed methods practical. Positive examples can be collected easily by simply operating a robot through traversable spaces, while obtaining negative examples is time consuming, costly, and potentially dangerous. Through extensive experiments and several demonstrations, we show that the proposed traversability estimation approaches are robust and can generalize to unseen scenarios. Further, we demonstrate that our methods are memory efficient and fast, allowing for real-time operation on a mobile robot with single or stereo fisheye cameras. As part of our contributions, we open-source two new datasets for traversability estimation. These datasets are composed of approximately 24h of videos from more than 25 indoor environments. Our methods outperform baseline approaches for traversability estimation on these new datasets.
Our paper presented at IROS 2018 is selected as the Best Paper Award Finalist on Safety, Security and Rescue Robotics. And, our next project, VUNet, which can extend the traversability estimation of GONet, is shown in this web page
Visualization
GONet generates the image, corresponding to the input image, to extract the features. If the input image is positive image, which the robot can go through, the generated image can be similar to the input image. Because, the image is generated through the manifold of the positive images trained by GAN(Generative Adversarial Network) using huge positive images.
Input and Generated Images
Qualitative Results
GONet is tested on three applications as the typical atcual example.
Emergency Stops
GONet ran on a teleoperated TurtleBot2 to signal unsafe areas in front of the robot. When these areas were detected, GONet overrode teleoperation commands to force the robot to stop. Our traversability estimation approach was able to prevent the robot from falling down stairs, and it stopped the robot from colliding with glass walls, fences, doors, and people in previously unseen environments.
Accessibility Alarm
We implemented a navigational aid system that produces an audible warning when an obstacle or drop-off is detected by GONet, and placed the laptop with our system in a backpack. We then asked a person to carry this backpack and to hold the fisheye camera that was connected to the laptop so that it could gather images of the environment in front of the person. As this person walked blindfolded through unseen indoor and outdoor environments, GONet recognized unsafe areas, and our system correctly issued audible warnings. Note that we never trained GONet in outdoor environments, nor with images gathered by a pedestrian. Nonetheless, our traversability estimation approach showed great generalization potential under new conditions.
Cost Map
We added two more fisheye cameras on the robot to demonstrate the scalability of GONet at handling different views of the environment. Based on the traversability predictions from GONet, we automatically updated the robot's costmap as it navigated inside a building in the test set of our GS1 dataset. GONet identified blocked and unsafe areas that the lidar missed, such as parts of a fence and stairs. Because recognizing these critical areas is essential for autonomous robot operation
