N-ImageNet: Towards Robust, Fine-Grained
Object Recognition with Event Cameras

We introduce N-ImageNet, a large-scale dataset targeted for robust, fine-grained object recognition with event cameras. The dataset is collected using programmable hardware in which an event camera consistently moves around a monitor displaying images from ImageNet. N-ImageNet serves as a challenging benchmark for event-based object recognition, due to its large number of classes and samples. We empirically show that pretraining on N-ImageNet improves the performance of event-based classifiers and helps them learn with few labeled data. In addition, we present several variants of N-ImageNet to test the robustness of event-based classifiers under diverse camera trajectories and severe lighting conditions, and propose a novel event representation to alleviate the performance degradation. To the best of our knowledge, we are the first to quantitatively investigate the consequences caused by various environmental conditions on event-based object recognition algorithms. N-ImageNet and its variants are expected to guide practical implementations for deploying event-based object recognition algorithms in the real world.

Event cameras are neuromorphic sensors that encode visual information as a sequence of events. In contrast to conventional frame-based cameras that output absolute brightness intensities, event cameras respond to brightness changes. The following figure shows a visual description of how event cameras function compared to conventional cameras. Notice how brightness changes are encoded as 'streams' in the spatio-temporal domain.

N-ImageNet is a large-scale event dataset that enables training and benchmarking object recongition algorithms using event camera input. The dataset surpasses all existing datasets in both size and label granularity.

To capture N-ImageNet, we design custom hardware to trigger perpetual camera motion. The device consists of two geared motors connected to a pair of perpendicularly adjacent gear racks where the upper and lower motors are responsible for vertical and horizontal motion, respectively. Each motor is further linked to a programmable Arduino board, which can control the camera movement.

Due to its size and label diversity, N-ImageNet can function as a challenging benchmark for event-based object recognition. The plot below shows the classification accuracy of existing event-based recognition algorithms on N-ImageNet. There exists a large performance gap with the ImageNet state-of-the-art, which suggests that mastering N-ImageNet is still a long way to go. We expect N-ImageNet to foster development of event classifiers that could readily function in the real world.