UnrealStereo: Controlling Hazardous Factors to Analyze Stereo Vision

Abstract

Stereo algorithm is important for robotics applications, such as quadcopter and autonomous driving. It needs to be robust enough to handle images of challenging conditions, such as raining or strong lighting. Textureless and specular regions of these images make feature matching difficult and smoothness assumption invalid. It is important to understand whether an algorithm is robust to these hazardous regions. Many stereo benchmarks have been developed to evaluate the performance and track progress. But it is not easy to quantize the effect of these hazardous regions. In this paper, we develop a synthetic image generation tool and build a benchmark with synthetic images. First, we manually tweak hazardous factors in a virtual world, such as making objects more specular or transparent, to simulate corner cases to test the robustness of stereo algorithms. Second, we use ground truth information, such as object mask, material property, to automatically identify hazardous regions and evaluate the accuracy of these regions. Our tool is based on a popular game engine Unreal Engine 4 and will be open-source. Many publicly available realistic game contents can be used by our tool which can provide an enormous resource for algorithm development and evaluation.

Paper

UnrealStereo: Controlling Hazardous Factors to Analyze Stereo Vision
Yi Zhang, Weichao Qiu, Qi Chen, Xiaolin Hu, and Alan Yuille
3DV 2018 [pdf][GitHub]

Dataset

Six publicly available game scenes including a small indoor room, a large temple scene, three houses and one block of street. Different layouts in these houses such as living room, kitchen and bathroom.
The largest scene contains more than 1,000 objects while hundreds on average, including those with high reflectance such as mirrors, bathtubs and metal statues, transparent objects such as glass, glassdoors and windows. Subtle visual effects are included, e.g. leather sofa and the specular surface of a TV.
Rich groundtruth including disparity, instance segmentation, hazardous regions can be generated automatically using our tool.