Need autonomous driving training data? ›

Top Open-Source Lidar Driving Data Sets

Top Open-Source Lidar Driving Data Sets

Rapid advancements in lidar technology and lowering costs have driven most self-driving car teams to adopt lidar as a key component of their perception sensor stack. Yet despite this boom in lidar usage, availability of open-source lidar driving datasets lags significantly behind that of 2D image and video training datasets. But for those of you looking to get your hands on some lidar driving data, fear not! We’ve created a one-stop resource of commonly cited point cloud datasets that researchers can access today, and upcoming datasets to watch for in the future.


Datasets available today

  • Apollo Lidar Point Cloud Obstacle Detection & Classification Data Set: Baidu’s Apollo Lidar dataset provides 20,000 frames of 3D point cloud annotation data, including 10,000 frames of training data and 10,000 frames of test data. Approximately 475,000 obstacles are annotated across four object classes: pedestrians, vehicles, non-motor vehicles (cyclists), and others (dontCares). The annotations cover all the obstacles within 60m from the scene in a 360-degree view. The data was collected from a Velodyne HDL-64E S3.


  • KITTI Vision Benchmark Suite: KITTI provides benchmarks for the tasks of stereo, optical flow, visual odometry/SLAM, and 3D object detection and tracking, as well as all data in raw format. The 3D object detection, orientation, and tracking benchmarks consists of 7,481 training images and 7,518 test images as well as the corresponding point clouds, comprising a total of 80,256 labeled objects. The benchmarks provide accurate 3D information in the form of 3D bounding boxes for object classes such as cars, vans, trucks, pedestrians, and cyclists. Each bounding box is marked as visible, occluded, or truncated. In total, more than 200,000 3D object annotations are captured in cluttered scenarios, with up to 15 cars and 30 pedestrians visible per image. The datasets were collected by driving around the mid-size city of Karlsruhe, Germany, including rural areas and highways.


  • Ford Campus Vision and Lidar Data Set: This dataset from the University of Michigan contains time-registered point cloud, camera, and IMU data collected in downtown Dearborn, MI along several large- and small-scale loop closures, which is useful for testing various computer vision and simultaneous localization and mapping algorithms. The point cloud data is co-registered with imagery from an omnidirectional camera, such that the 3D point cloud is projected onto the corresponding camera images. The dataset is not annotated. Read the research paper here.


  • Stanford Track Collection: Stanford’s dataset contains 1.3 million labeled point clouds across roughly 14,000 tracks (individually labeled objects), extracted from one hour of 360-degree, 10Hz depth information captured from busy street scenes on Stanford’s campus. GPS and IMU data is also included. Labeled object classes include cars, pedestrians, bicyclists, and background. More details about the dataset in this paper.


  • Udacity CH3_002: As part of its effort to develop an open-source self-driving car, Udacity released over three hours of driving data collected along El Camino Real in California. The dataset contains lidar data from a Velodyne HDL-32E, along with camera, latitude, longitude, gear, brake, throttle, steering angles, and speed data. This smaller CHX_001 dataset was collected from a lap around the block at Udacity’s office in Mountain View. Neither dataset has been annotated.


Datasets to watch

As you might imagine, there are several groups planning to release lidar driving data in upcoming open-source datasets. Following is a list of future and existing large-scale driving datasets with public plans to add point cloud annotations in the future:


  • ApolloScape: Baidu has separately released ApolloScape, the world’s largest open-source dataset for self-driving cars. As of April 2018, the dataset contains 146,997 frames with corresponding pixel-level annotations and pose information as well as depth maps for static background. When completed, the dataset will include RGB videos with 200,000 high resolution images and per pixel annotation, survey-grade dense 3D points with semantic segmentation, stereoscopic video, and panoramic images. 


  • BDD100K: Berkeley Deep Drive’s dataset includes 100,000 HD video sequences of over 1,100 hours of driving, 100,000 annotated frames with 2D bounding boxes of 10 object classes, 10,000 diverse images with full-frame instance segmentation, and lane marking / drivable area annotations on 100,000 images across many different times of day, weather conditions, and driving scenarios. The dataset also contains GPS locations, IMU data, and timestamps, and the consortium plans to add point cloud data “in the near future.

image credit: Udacity