Neuromorphic iCub Infrastructure

The iCub is progressively updated to integrate ED technology. A modular infrastructure, supported by FPGA-based technology, serialisation , and YARP middleware, supports the integration of different ED sensors, neuromorphic computational platforms (SpiNNaker and DYNAP) and software modules for ED sensory processing for seamless integration on the robot. Amongst the latest developments, we implemented a new vision system integrating upgraded ED and frame-based sensors. The low spatial resolution, large field of view and motion sensitive ED sensors coupled with low temporal but high spatial resolution and small field of view frame-based sensors parallels the organisation of the primate's foveated vision. Coarse large field of view periphery can be used to detect salient regions in the scene, that guide sequential saccades that put the region of interest in the high acuity fovea for detailed stimulus processing. To explore ED tactile sensing, we are working on the emulation of ED encoding using the current capacitive sensors integrated on the iCub. Besides the improvement in communication bandwidth thanks to the sensor compression and use of the serial AER protocol, the final goal of this activity is to acquire asynchronous data from different types of sensors (vision and skin at first) and study the use of temporal correlations for multi-sensory integration.

HW infrastructure for sensing and computing

Amongst the latest developments, we implemented a new vision system integrating upgraded ED and frame-based sensors. The low spatial resolution, large field of view and motion sensitive ED sensors coupled with low temporal but high spatial resolution and small field of view frame-based sensors parallels the organisation of the primate's foveated vision. Coarse large field of view periphery can be used to detect salient regions in the scene, that guide sequential saccades that put the region of interest in the high acuity fovea for detailed stimulus processing. To explore ED tactile sensing, we are working on the emulation of ED encoding using the current capacitive sensors integrated on the iCub. Besides the improvement in communication bandwidth thanks to the sensor compression and use of the serial AER protocol, the final goal of this activity is to acquire asynchronous data from different types of sensors (vision and skin at first) and study the use of temporal correlations for multi-sensory integration.

 

YARP libraries for ED computation

The integration of event-driven systems into the iCub also includes the development of the full software pipeline to intelligently control the robot based on the event-driven perception. The robot infrastructure requires such a framework to be distributed across multiple computers, that don’t necessarily have to be mounted within the robot. The Event Driven Perception for Robotics group develops and maintains the code for embedded systems and operating system drivers, as well as the open-source libraries for event decoding, communication, multiple processing algorithms, visualization and robot control. The scientific research performed by the EDPR group is realised as a documented software packages as well as applications to demonstrate the results, live on the robot. These packages are made open-source to promote the replication of results for our research collaborators. The core libraries are developed in such a way as to promote fast communication between modules and ensure real-time operation for on-line and safe robotics applications. The libraries are fully compatible with the yet-another-robot-platform (YARP) that provides the software framework for all iCub robots.