Kinect sensor system work plan

PrimeSense technology is the foundation of Microsoft's Kinect sensor system designed to work with the Microsoft Xbox 360 console gaming system (Figure 1). Its working principle is very simple, but the implementation process is quite complicated. PS1080 System-on-Chip (SoC) is a product that perfectly supports PrimeSense technology.

This chip independently manages audio and video information that can be accessed via a USB connection. USB only supplies power to the PrimeSense unit, and Kinect also needs extra power to supply its servo system.

Most design engineers are at least very familiar with how Kinect works with Xbox. Kinect is mainly used to provide game programs with information about players. Players need to be in front of the TV and face Kinect, then interact with the game through movements and gestures. It is very interesting to know how Kinect gets this information.

Prior to the introduction of Kinect, such gesture recognition was implemented using LIDAR (laser-induced differential absorption radar) or lidar. Ultrasonic sensors do not provide sufficient accuracy. Another method is to use image analysis, but it is very complex and requires high computational requirements.

PrimeSense uses a very different approach. It projects an infrared dot pattern from the sensor and then uses a conventional CMOS image sensor with an infrared filter to detect it. The pattern changes according to the object that reflected the light. The size and location of those points will also vary with the distance of the object from the source of the emission.

PS1080 receives the result from the image sensor and then generates a depth scene by judging the difference. The resolution of the depth scene is 1024x758 (VGA), but the CMOS sensor has much higher resolution. The resolution of the image that can be captured by the hardware is actually 1600x1200, which is necessary to provide a depth scene, otherwise there is not enough resolution to detect the position and size of the projected infrared lattice.

The main job of the PS1080 chip is to identify infrared points and convert their state information to depth values. This is not a simple task, nor is it something that a typical microprocessor can handle. Fortunately, PS1080 can accomplish this task at a rate of 30 frames per second. Several points can generally be found within the area represented by one pixel.

The shortest distance is less than 1 meter (0.8 meters) and the maximum depth is about 3.5 meters. This range is just right for gamers in front of high-definition televisions. The visible area is a rectangular pyramid with 58 degrees and 45 degrees in the horizontal and vertical directions.

The resolution and detection quality depend on the position of the object relative to the sensor, but the above indicators are sufficient for playing games and robot object recognition and collision avoidance. At a distance of 2 meters, the depth resolution is 10 millimeters, while the horizontal and vertical resolutions are up to 3 millimeters.

Video sensors and deep CMOS sensors are placed next to each other to facilitate the combination of depth and color images. PS1080 correctly aligns color image (RGB) and depth (D) information by performing a registration process. RGBD information is the content that needs to be delivered to the host.

For a robot that needs to avoid an object, only depth information is fine. Sometimes enough in the game, depending on what action is being performed, how many players have to be tracked. The RGBD information can be further analyzed to allow the system to recognize objects and their relationships, such as the hand-arm-body relationship.

The host completes higher-level object and action recognition. The OpenNI (natural interaction) organization provides frameworks and application programming interfaces (APIs) for handling devices such as Kinect. OpenNI supports many devices including video and audio devices (Figure 3), and can perform object tracking and other functions through higher-end middleware.

Kinect integrates a pair of microphones in the system. The PS1080 can handle four external digital audio sources and provide visual video, depth and audio information in a synchronized manner via the USB interface.

Microsoft recognizes that in addition to working with Xbox, Kinect should be able to play a greater role. The Kinect interface has long existed, and today Microsoft has released a software development kit (SDK) for this interface. The commercial version may already be in use, and PrimeSense also has a corresponding software development kit.

PrimeSense technology is very extensible, so non-gaming and possibly non-robot applications are also interested in it. For example, accurate proximity detection may not require a visible video portion. PrimeSense technology will undoubtedly change our lifestyle.