Kinect 2 on Jetson AGX Xavier

Posted on Edited on In ,

Today, I wanna try out the old Kinect 2 on my old Jetson AGX Xavier.

1. My Working Environment

Jetson AGX Xavier 32G

For how I set up my old Jetson AGX Xavier, please refer to my previous blog NVidia Jetson Xavier AGX.

2. Kinect 2

2.1 Overview

Kinect 2 Overview

2.2 Hardware Specification

You can find specs of Kinect v2 in Evaluating the Accuracy of the Azure Kinect and Kinect v2.

3. lsusb and add udev rules

3.1 lsusb

1
2
3
4
$ lsusb | grep Microsoft
Bus 009 Device 015: ID 045e:02d9 Microsoft Corp. NuiSensor Adaptor      
Bus 010 Device 005: ID 045e:02d9 Microsoft Corp. NuiSensor Adaptor      
Bus 010 Device 006: ID 045e:02c4 Microsoft Corp. Xbox NUI Sensor

3.2 lsusb More Details

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
➜  ~ lsusb -v -d 045e:02c4

Bus 002 Device 015: ID 045e:02c4 Microsoft Corp. 
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               3.00
  bDeviceClass          239 Miscellaneous Device
  bDeviceSubClass         2 
  bDeviceProtocol         1 Interface Association
  bMaxPacketSize0         9
  idVendor           0x045e Microsoft Corp.
  idProduct          0x02c4 
  bcdDevice            1.00
  iManufacturer           1 Microsoft
  iProduct                2 Xbox NUI Sensor
  iSerial                 4 178602434347
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x00d6
    bNumInterfaces          4
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower              144mA
    Interface Association:
      bLength                 8
      bDescriptorType        11
      bFirstInterface         0
      bInterfaceCount         2
      bFunctionClass        255 Vendor Specific Class
      bFunctionSubClass     255 Vendor Specific Subclass
      bFunctionProtocol       0 
      iFunction               0 
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           4
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               0
        bMaxBurst               7
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               0
        bMaxBurst               7
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               0
        bMaxBurst               7
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes           19
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Feedback
        wMaxPacketSize     0x0040  1x 64 bytes
        bInterval              14
        bMaxBurst               0
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      bNumEndpoints           0
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 
      iInterface              0 
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       1
      bNumEndpoints           1
      bInterfaceClass       255 Vendor Specific Class
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x84  EP 4 IN
        bmAttributes            1
          Transfer Type            Isochronous
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0400  1x 1024 bytes
        bInterval               1
        bMaxBurst              10
        Mult                    2
    Interface Association:
      bLength                 8
      bDescriptorType        11
      bFirstInterface         2
      bInterfaceCount         2
      bFunctionClass          1 Audio
      bFunctionSubClass       2 Streaming
      bFunctionProtocol       0 
      iFunction               0 
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        2
      bAlternateSetting       0
      bNumEndpoints           0
      bInterfaceClass         1 Audio
      bInterfaceSubClass      1 Control Device
      bInterfaceProtocol      0 
      iInterface              0 
      AudioControl Interface Descriptor:
        bLength                 9
        bDescriptorType        36
        bDescriptorSubtype      1 (HEADER)
        bcdADC               1.00
        wTotalLength       0x001e
        bInCollection           1
        baInterfaceNr(0)        3
      AudioControl Interface Descriptor:
        bLength                12
        bDescriptorType        36
        bDescriptorSubtype      2 (INPUT_TERMINAL)
        bTerminalID             1
        wTerminalType      0x0205 Microphone Array
        bAssocTerminal          0
        bNrChannels             4
        wChannelConfig     0x0000
        iChannelNames           0 
        iTerminal               0 
      AudioControl Interface Descriptor:
        bLength                 9
        bDescriptorType        36
        bDescriptorSubtype      3 (OUTPUT_TERMINAL)
        bTerminalID             3
        wTerminalType      0x0101 USB Streaming
        bAssocTerminal          0
        bSourceID               1
        iTerminal               0 
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        3
      bAlternateSetting       0
      bNumEndpoints           0
      bInterfaceClass         1 Audio
      bInterfaceSubClass      2 Streaming
      bInterfaceProtocol      0 
      iInterface              0 
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        3
      bAlternateSetting       1
      bNumEndpoints           1
      bInterfaceClass         1 Audio
      bInterfaceSubClass      2 Streaming
      bInterfaceProtocol      0 
      iInterface              0 
      AudioStreaming Interface Descriptor:
        bLength                 7
        bDescriptorType        36
        bDescriptorSubtype      1 (AS_GENERAL)
        bTerminalLink           3
        bDelay                  1 frames
        wFormatTag         0x0001 PCM
      AudioStreaming Interface Descriptor:
        bLength                11
        bDescriptorType        36
        bDescriptorSubtype      2 (FORMAT_TYPE)
        bFormatType             1 (FORMAT_TYPE_I)
        bNrChannels             4
        bSubframeSize           4
        bBitResolution         32
        bSamFreqType            1 Discrete
        tSamFreq[ 0]        16000
      Endpoint Descriptor:
        bLength                 9
        bDescriptorType         5
        bEndpointAddress     0x85  EP 5 IN
        bmAttributes            1
          Transfer Type            Isochronous
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0100  1x 256 bytes
        bInterval               4
        bRefresh                0
        bSynchAddress           0
        bMaxBurst               0
        AudioStreaming Endpoint Descriptor:
          bLength                 7
          bDescriptorType        37
          bDescriptorSubtype      1 (EP_GENERAL)
          bmAttributes         0x00
          bLockDelayUnits         0 Undefined
          wLockDelay         0x0000
Binary Object Store Descriptor:
  bLength                 5
  bDescriptorType        15
  wTotalLength       0x0016
  bNumDeviceCaps          2
  SuperSpeed USB Device Capability:
    bLength                10
    bDescriptorType        16
    bDevCapabilityType      3
    bmAttributes         0x00
    wSpeedsSupported   0x000c
      Device can operate at High Speed (480Mbps)
      Device can operate at SuperSpeed (5Gbps)
    bFunctionalitySupport   2
      Lowest fully-functional device speed is High Speed (480Mbps)
    bU1DevExitLat          10 micro seconds
    bU2DevExitLat        2047 micro seconds
  USB 2.0 Extension Device Capability:
    bLength                 7
    bDescriptorType        16
    bDevCapabilityType      2
    bmAttributes   0x00000002
      HIRD Link Power Management (LPM) Supported
can't get debug descriptor: Resource temporarily unavailable
Device Status:     0x0001
  Self Powered
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
➜  ~ lsusb -v -d 045e:02d9 

Bus 002 Device 014: ID 045e:02d9 Microsoft Corp. NuiSensor Adaptor      
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               3.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 
  bDeviceProtocol         3 
  bMaxPacketSize0         9
  idVendor           0x045e Microsoft Corp.
  idProduct          0x02d9 
  bcdDevice            0.73
  iManufacturer           1 Microsoft Corporation  
  iProduct                2 NuiSensor Adaptor      
  iSerial                 0 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x001f
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes           19
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Feedback
        wMaxPacketSize     0x0002  1x 2 bytes
        bInterval               8
        bMaxBurst               0
Hub Descriptor:
  bLength              12
  bDescriptorType      42
  nNbrPorts             1
  wHubCharacteristic 0x000d
    Per-port power switching
    Compound device
    Per-port overcurrent protection
  bPwrOn2PwrGood      100 * 2 milli seconds
  bHubContrCurrent      0 milli Ampere
  bHubDecLat          0.4 micro seconds
  wHubDelay          4004 nano seconds
  DeviceRemovable    0x00
 Hub Port Status:
   Port 1: 0000.0203 5Gbps power U0 enable connect
Binary Object Store Descriptor:
  bLength                 5
  bDescriptorType        15
  wTotalLength       0x002a
  bNumDeviceCaps          3
  USB 2.0 Extension Device Capability:
    bLength                 7
    bDescriptorType        16
    bDevCapabilityType      2
    bmAttributes   0x00000002
      HIRD Link Power Management (LPM) Supported
  SuperSpeed USB Device Capability:
    bLength                10
    bDescriptorType        16
    bDevCapabilityType      3
    bmAttributes         0x00
    wSpeedsSupported   0x000e
      Device can operate at Full Speed (12Mbps)
      Device can operate at High Speed (480Mbps)
      Device can operate at SuperSpeed (5Gbps)
    bFunctionalitySupport   1
      Lowest fully-functional device speed is Full Speed (12Mbps)
    bU1DevExitLat           4 micro seconds
    bU2DevExitLat         231 micro seconds
  Container ID Device Capability:
    bLength                20
    bDescriptorType        16
    bDevCapabilityType      4
    bReserved               0
    ContainerID             {30eef35c-07d5-2549-b001-802d79434c30}
can't get debug descriptor: Resource temporarily unavailable
Device Status:     0x000d
  Self Powered
  U1 Enabled
  U2 Enabled

Bus 001 Device 008: ID 045e:02d9 Microsoft Corp. NuiSensor Adaptor      
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.10
  bDeviceClass            9 Hub
  bDeviceSubClass         0 
  bDeviceProtocol         1 Single TT
  bMaxPacketSize0        64
  idVendor           0x045e Microsoft Corp.
  idProduct          0x02d9 
  bcdDevice            0.70
  iManufacturer           1 Microsoft Corporation  
  iProduct                2 NuiSensor Adaptor      
  iSerial                 0 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0019
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0001  1x 1 bytes
        bInterval              12
Hub Descriptor:
  bLength               9
  bDescriptorType      41
  nNbrPorts             1
  wHubCharacteristic 0x00ed
    Per-port power switching
    Compound device
    Per-port overcurrent protection
    TT think time 32 FS bits
    Port indicators
  bPwrOn2PwrGood       50 * 2 milli seconds
  bHubContrCurrent    100 milli Ampere
  DeviceRemovable    0x00
  PortPwrCtrlMask    0xff
 Hub Port Status:
   Port 1: 0000.0100 power
Binary Object Store Descriptor:
  bLength                 5
  bDescriptorType        15
  wTotalLength       0x002a
  bNumDeviceCaps          3
  USB 2.0 Extension Device Capability:
    bLength                 7
    bDescriptorType        16
    bDevCapabilityType      2
    bmAttributes   0x00000002
      HIRD Link Power Management (LPM) Supported
  SuperSpeed USB Device Capability:
    bLength                10
    bDescriptorType        16
    bDevCapabilityType      3
    bmAttributes         0x00
    wSpeedsSupported   0x000e
      Device can operate at Full Speed (12Mbps)
      Device can operate at High Speed (480Mbps)
      Device can operate at SuperSpeed (5Gbps)
    bFunctionalitySupport   1
      Lowest fully-functional device speed is Full Speed (12Mbps)
    bU1DevExitLat           4 micro seconds
    bU2DevExitLat         231 micro seconds
  Container ID Device Capability:
    bLength                20
    bDescriptorType        16
    bDevCapabilityType      4
    bReserved               0
    ContainerID             {30eef35c-07d5-2549-b001-802d79434c30}
can't get debug descriptor: Resource temporarily unavailable
Device Status:     0x0001
  Self Powered

3.3 Add udev rules Under /etc/udev/rules.d

Take a look at OpenKinect libfreenect2 90-kinect2.rules:

1
2
3
4
5
6
7
8
9
10
$ cat platform/linux/udev/90-kinect2.rules            
───────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: platform/linux/udev/90-kinect2.rules
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ # this file belongs in /etc/udev/rules.d/
   2   │ # ATTR{product}=="Kinect2"
   3   │ SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02c4", MODE="0666"
   4   │ SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02d8", MODE="0666"
   5   │ SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02d9", MODE="0666"
───────┴───────────────────────────────────────────────────────────────────────────────────

We’ll do the following 2 steps:

  • step 1: 
    1
    $ sudo cp platform/linux/udev/90-kinect2.rules /etc/udev/rules.d
  • step 2: 
    1
    2
    $ sudo udevadm control --reload-rules
    $ sudo udevadm trigger

3.4 Don’t Use Default OpenNI2 from Default Repository

3.4.1 Incorrect Rules

Take a look at OpenNI2 Kinect2 primesense-usb.rules:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ cat Packaging/Linux/primesense-usb.rules               
───────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: Packaging/Linux/primesense-usb.rules
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ # Make primesense device mount with writing permissions (default is read only for unknown devices)
   2   │ SUBSYSTEM=="usb", ATTR{idProduct}=="0200", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
   3   │ SUBSYSTEM=="usb", ATTR{idProduct}=="0300", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
   4   │ SUBSYSTEM=="usb", ATTR{idProduct}=="0401", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
   5   │ SUBSYSTEM=="usb", ATTR{idProduct}=="0500", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
   6   │ SUBSYSTEM=="usb", ATTR{idProduct}=="0600", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
   7   │ SUBSYSTEM=="usb", ATTR{idProduct}=="0601", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
   8   │ SUBSYSTEM=="usb", ATTR{idProduct}=="0609", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
   9   │ SUBSYSTEM=="usb", ATTR{idProduct}=="1250", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
  10   │ SUBSYSTEM=="usb", ATTR{idProduct}=="1260", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
  11   │ SUBSYSTEM=="usb", ATTR{idProduct}=="1270", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
  12   │ SUBSYSTEM=="usb", ATTR{idProduct}=="1280", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
  13   │ SUBSYSTEM=="usb", ATTR{idProduct}=="1290", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
  14   │ SUBSYSTEM=="usb", ATTR{idProduct}=="f9db", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"

!Note: by installing libopenni2-dev and openni2-utils from default repo, you’re going to have the above rule primesense-usb.rules installed under /usr/lib/udev/rules.d, instead of /etc/udev/rules.d.

3.4.2 Incorrect NiViewer

Even if package openni2-utils installed an executable NiViewer2 under /usr/bin, it fails to even detect the device Kinect v2.

1
2
3
4
5
$ NiViewer2
openDevice failed:
	DeviceOpen using default: no devices found

Press any key to continue . . .

Due to the above Incorrect Rules, this NiViewer2, of course, should NOT detect the device.

Feel so disappointed …

😞

What made me feel worse, even desperate???

The above failure happens on both of my Jetson AGX Xavier running Ubuntu 20.04.6 LTS aarch64, and my desktop running Ubuntu 24.04.

4. For Windows

Please strictly follow Kinect for Windows SDK 2.0. For my Kinect 2, I got the following outputs on my laptop running Windows 11.

5. For Linux

5.1 Demonstration FIRST

OpenNI2 NiViewer
OpenNI2 NiViewer
libfreenect2 Protonect CPU
libfreenect2 Protonect CPU
libfreenect2 Protonect -GPU=0
libfreenect2 Protonect -GPU=0

5.2 Build OpenNI2 Github from Source

5.2.1 Key Components

Components Description
PSCommon The third-party library directly from PrimeSense
OniFile Must be dealing with OpenNI's .oni files
PS1080 Clearly related to PrimeSense
PSLink Clearly related to PrimeSense
Kinect I believe this is the driver for original Kinect v1, namely, Kinect for Xbox 360.
Kinect2 I believe this is the driver for Kinect2, based on Kinect for Windows SDK 2.0

To build Kinect or Kinect2, I think a Windows environment is a MUST, which is NOT for me.

5.2.2 Tool NiViewer

Driver Kinect2 in the above is a MUST for OpenNI2 Github's application NiViewer.

5.2.3 Any Driver to Drive Kinect2 and Can Work With OpenNI2 Github?

Now, stucked. Under Linux, how to build a driver working for my Kinect v2?

Solution: Build libfreenect2 from source, with BUILD_OPENNI2_DRIVER ON.

5.3 Build libfreenect2 from Source

Download OpenNI2 and save it, and built libfreenect2 with BUILD_OPENNI2_DRIVER set ON.

By doing so, you will have the following 2 variables set by default as:

1
2
OpenNI2_INCLUDE_DIRS             /opt/OpenNI/Include
OpenNI2_LIBRARY                  /opt/OpenNI/Redist/libOpenNI2.so

and you will have the following two libraries built out:

1
2
lib/libfreenect2-openni2.so
lib/libfreenect2.so

And, if needed, after the installation of libfreenect2, manually copy libfreenect2-openni2.so under folder /usr/local/lib/OpenNI2/Drivers. Now, ALL set. You’re good to go NiViewer.

5.4 Protonect or ProtonectSR from libfreenect2

5.4.1 ./bin/Protonect cuda

I got NO idea why ./bin/Protonect cuda failed to run ALL the time. Then, I created a simple Python script verify_cuda.py and ALWAYS meet the following error:

1
2
3
4
5
$ python verify_cuda.py   
[Info] [CudaDepthPacketProcessorImpl] device 0: NVIDIA GeForce RTX 3090 @ 1725MHz Memory 24219MB
[Info] [CudaDepthPacketProcessorImpl] selected device 0
[Error] [CudaDepthPacketProcessorImpl] cudaGetLastError(): the provided PTX was compiled with an unsupported toolchain.
CudaPacketPipeline is available.

Finally, I found the reason: Cuda version is NOT synced with NVidia driver version. More specifically, in my case:

To solve this problem, install NVidia driver during the installation of Cuda. Namely: when run cuda_12.5.1_555.42.06_linux.run, tick

1
2
- [X] Driver                                                                 │
│      [X] 555.42.06
  • You may have to reboot into text mode and have all previously-installed NVidia driver unloaded. For this issue, please refer to How to unload kernel module ‘nvidia-drm’?
  • The key: sudo systemctl isolate multi-user.target

Protonect cuda

5.4.2 ./bin/Protonect -gpu=0

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
$ ./bin/Protonect -gpu=0
Version: 0.2.0
Environment variables: LOGFILE=<protonect.log>
Usage: ./bin/Protonect [-gpu=<id>] [gl | cl | clkde | cuda | cudakde | cpu] [<device serial>]
        [-noviewer] [-norgb | -nodepth] [-help] [-version]
        [-frames <number of frames to process>]
To pause and unpause: pkill -USR1 Protonect
[Info] [Freenect2Impl] enumerating devices...
[Info] [Freenect2Impl] 23 usb devices connected
[Info] [Freenect2Impl] found valid Kinect v2 @10:5 with serial 178602434347
[Info] [Freenect2Impl] found 1 devices
libva info: VA-API version 1.20.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/nvidia_drv_video.so
libva info: va_openDriver() returns -1
[Error] [VaapiRgbPacketProcessorImpl] vaInitialize(display, &major_ver, &minor_ver): unknown libva error
[Info] [Freenect2DeviceImpl] opening...
[Info] [Freenect2DeviceImpl] transfer pool sizes rgb: 20*16384 ir: 60*8*33792
[Info] [Freenect2DeviceImpl] opened
[Info] [Freenect2DeviceImpl] starting...
[Debug] [Freenect2DeviceImpl] status 0x090000: 9855
[Debug] [Freenect2DeviceImpl] status 0x090000: 9855
[Info] [Freenect2DeviceImpl] submitting rgb transfers...
[Info] [Freenect2DeviceImpl] submitting depth transfers...
[Info] [Freenect2DeviceImpl] started
device serial: 178602434347
device firmware: 4.0.3916.0
[Debug] [DepthPacketStreamParser] not all subsequences received 0
[Debug] [DepthPacketStreamParser] not all subsequences received 1021
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Info] [DepthPacketStreamParser] 2 packets were lost
[Info] [DepthPacketStreamParser] 7 packets were lost
[Info] [OpenGLDepthPacketProcessor] avg. time: 5.12134ms -> ~195.261Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 4.06057ms -> ~246.271Hz
[Info] [TurboJpegRgbPacketProcessor] avg. time: 19.003ms -> ~52.6232Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 4.01312ms -> ~249.182Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 4.20707ms -> ~237.695Hz
[Info] [TurboJpegRgbPacketProcessor] avg. time: 17.376ms -> ~57.5507Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 3.95256ms -> ~253.001Hz
[2]  - 2620271 killed     ./bin/ProtonectSR cuda
[Info] [OpenGLDepthPacketProcessor] avg. time: 4.40171ms -> ~227.184Hz
[Info] [TurboJpegRgbPacketProcessor] avg. time: 17.8467ms -> ~56.0326Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 6.23296ms -> ~160.437Hz
[3]  + 2628690 killed     ./bin/Protonect
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Info] [OpenGLDepthPacketProcessor] avg. time: 9.88223ms -> ~101.192Hz
[Info] [DepthPacketStreamParser] 2 packets were lost
[Info] [TurboJpegRgbPacketProcessor] avg. time: 19.2099ms -> ~52.0565Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 5.8084ms -> ~172.164Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 8.7243ms -> ~114.622Hz
[Info] [TurboJpegRgbPacketProcessor] avg. time: 18.2581ms -> ~54.7703Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 9.24396ms -> ~108.179Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 10.1021ms -> ~98.9893Hz
[Info] [TurboJpegRgbPacketProcessor] avg. time: 18.3488ms -> ~54.4995Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 8.35109ms -> ~119.745Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 5.61051ms -> ~178.237Hz
[Info] [TurboJpegRgbPacketProcessor] avg. time: 17.9773ms -> ~55.6256Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 6.80606ms -> ~146.928Hz
[Info] [Freenect2DeviceImpl] stopping...
[Info] [Freenect2DeviceImpl] canceling rgb transfers...
[Info] [Freenect2DeviceImpl] canceling depth transfers...
[Info] [Freenect2DeviceImpl] stopped
[Info] [Freenect2DeviceImpl] closing...
[Info] [Freenect2DeviceImpl] releasing usb interfaces...
[Info] [Freenect2DeviceImpl] deallocating usb transfer pools...
[Info] [Freenect2DeviceImpl] closing usb device...
[Info] [Freenect2DeviceImpl] closed
[Info] [Freenect2DeviceImpl] closing...
[Info] [Freenect2DeviceImpl] already closed, doing nothing
[TurboJpegRgbPacketProcessor] 9.8382 10.0887 18.9733 20.2907 26.1092 mean=18.2873 std=2.86968 n=749
[OpenGLDepthPacketProcessor] 2.90518 3.87679 5.69045 12.5144 95.8873 mean=6.46404 std=4.29096 n=1528

5.4.3 ./bin/Protonect cpu

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
$ ./bin/Protonect cpu
Version: 0.2.0
Environment variables: LOGFILE=<protonect.log>
Usage: ./bin/Protonect [-gpu=<id>] [gl | cl | clkde | cuda | cudakde | cpu] [<device serial>]
        [-noviewer] [-norgb | -nodepth] [-help] [-version]
        [-frames <number of frames to process>]
To pause and unpause: pkill -USR1 Protonect
libva info: VA-API version 1.20.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/nvidia_drv_video.so
libva info: va_openDriver() returns -1
[Error] [VaapiRgbPacketProcessorImpl] vaInitialize(display, &major_ver, &minor_ver): unknown libva error
[Info] [Freenect2Impl] enumerating devices...
[Info] [Freenect2Impl] 23 usb devices connected
[Info] [Freenect2Impl] found valid Kinect v2 @10:5 with serial 178602434347
[Info] [Freenect2Impl] found 1 devices
[Info] [Freenect2DeviceImpl] opening...
[Info] [Freenect2DeviceImpl] transfer pool sizes rgb: 20*16384 ir: 60*8*33792
[Info] [Freenect2DeviceImpl] opened
[Info] [Freenect2DeviceImpl] starting...
[Debug] [Freenect2DeviceImpl] status 0x090000: 9729
[Debug] [Freenect2DeviceImpl] status 0x090000: 9731
[Info] [Freenect2DeviceImpl] submitting rgb transfers...
[Info] [Freenect2DeviceImpl] submitting depth transfers...
[Info] [Freenect2DeviceImpl] started
device serial: 178602434347
device firmware: 4.0.3916.0
[Debug] [DepthPacketStreamParser] not all subsequences received 0
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Info] [DepthPacketStreamParser] 30 packets were lost
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Info] [DepthPacketStreamParser] 30 packets were lost
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Info] [Freenect2DeviceImpl] stopping...
[Info] [Freenect2DeviceImpl] canceling rgb transfers...
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Info] [Freenect2DeviceImpl] canceling depth transfers...
[Info] [Freenect2DeviceImpl] stopped
[Info] [Freenect2DeviceImpl] closing...
[Info] [Freenect2DeviceImpl] releasing usb interfaces...
[Info] [Freenect2DeviceImpl] deallocating usb transfer pools...
[Info] [Freenect2DeviceImpl] closing usb device...
[Info] [Freenect2DeviceImpl] closed
[Info] [Freenect2DeviceImpl] closing...
[Info] [Freenect2DeviceImpl] already closed, doing nothing
[TurboJpegRgbPacketProcessor] 10.0384 10.1629 18.9524 20.9346 27.5923 mean=18.5922 std=3.23163 n=44
[CpuDepthPacketProcessor] 101.371 101.485 102.303 109.367 111.332 mean=102.829 std=2.25233 n=27

5.4.4 Conclusive Comparisons

Let’s do a simple comparison of the speed using 3 different parameters: cuda, -cpu=0, cpu.

Commands RGB Mean Time (ms) Depth Mean Time (ms)
./bin/Protonect cuda 18.8951 0.521472
./bin/Protonect -gpu=0 18.2873 6.46404
./bin/Protonect cpu 18.5922 102.829

The conclusion is quite obvious:

  • for depth: CudaDepthPacketProcessor > OpenGLDepthPacketProcessor > CpuDepthPacketProcessor.
  • for RGB: Always TurboJpegRgbPacketProcessor

5.5 pylibfreenect2 - libfreenect2's Python Wrapper

So far, in my personal point of view, pylibfreenect2 is the BEST Python wrapper of libfreenect2. But, according to the official documentation of libfreenect2, it’s clearly written:

Audio. Raw audio is accessible via Linux USB audio. There is no support for the calibrated directional audio.

5.6 Kinect2's Audio pyalsaaudio

5.6.1 lsusb Again

1
2
3
4
➜  ~ lsusb | grep Microsoft 
Bus 002 Device 003: ID 045e:02c4 Microsoft Corp. 
Bus 002 Device 002: ID 045e:02d9 Microsoft Corp. NuiSensor Adaptor      
Bus 001 Device 002: ID 045e:02d9 Microsoft Corp. NuiSensor Adaptor

5.6.2 arecord -l

1
2
➜  ~ arecord -l | grep NUI             
card 1: Sensor [Xbox NUI Sensor], device 0: USB Audio [USB Audio]
  • Card number: 1
  • Device number: 0

5.6.3 List Details of hw:1,0

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
➜  arecord -D hw:1,0 --dump-hw-params
Recording WAVE 'stdin' : Unsigned 8 bit, Rate 8000 Hz, Mono
HW Params of device "hw:1,0":
--------------------
ACCESS:  MMAP_INTERLEAVED RW_INTERLEAVED
FORMAT:  S32_LE
SUBFORMAT:  STD
SAMPLE_BITS: 32
FRAME_BITS: 128
CHANNELS: 4
RATE: 16000
PERIOD_TIME: [1000 2048000]
PERIOD_SIZE: [16 32768]
PERIOD_BYTES: [256 524288]
PERIODS: [2 1024]
BUFFER_TIME: [2000 4096000]
BUFFER_SIZE: [32 65536]
BUFFER_BYTES: [512 1048576]
TICK_TIME: ALL
--------------------
arecord: set_params:1368: Sample format non available
Available formats:
- S32_LE

A couple of concerns:

  • I personally think FRAME_BITS is a typo. It should be FRAME_SIZE.
  • What the sample rate is 16000, but NOT 44100? 😀

5.6.4 Demonstration

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
import alsaaudio
import wave
import time

# Open the audio device
inp = alsaaudio.PCM(alsaaudio.PCM_CAPTURE, alsaaudio.PCM_NORMAL, "plughw:Sensor")

# Set audio parameters
inp.setformat(alsaaudio.PCM_FORMAT_S32_LE)
inp.setchannels(4)
inp.setrate(16000)
inp.setperiodsize(128)

# Prepare a file to write the audio data
with wave.open("test.wav", "wb") as f:
    f.setnchannels(4)
    f.setsampwidth(4)
    f.setframerate(16000)

    # Record audio for 10 seconds
    timeout = 10  # seconds
    start_time = time.time()
    while time.time() - start_time < timeout:
        num_bytes, buffer = inp.read()
        f.writeframes(buffer)

# Close the audio device
inp.close()

print("Recording finished")

6. For Jetson AGX Xavier

Now, we’re done on my desktop. Let’s try Kinect2 on Jetson AGX Xavier.

6.1 libfreenect2 - Do NOT Enable ENABLE_TEGRAJPEG

By default, libfreenect2 has ENABLE_TEGRAJPEG set ON, which builds fine. However, the followig issue has been reported for over 8 years.

Solution for Jetson AGX Xavier:

libfreenect2 Protonect Cuda
libfreenect2 Protonect Cuda
libfreenect2 Protonect Cuda Speed
libfreenect2 Protonect Cuda

From the above output, we can easily come up with the following table:

Attribute RGB Depth
Resolution 1920x1080 512x424
Number of Channels 3 (Red, Green, Blue) 1 (Depth)
Bytes per Pixel 3 bytes 2 bytes
Average Processing Time ~22.4048 ms ~7.18678 ms
FPS ~44.6163 FPS ~110.134 FPS
Data per Frame (Bytes) 1920 * 1080 * 3 = 6,220,800 bytes 512 * 424 * 2 = 434,176 bytes
Data per Frame (MB) ~6.22 MB ~0.414 MB
Data per Second (MB/s) 6.22 MB * 44.6163 ≈ 277.57 MB/s 0.414 MB * 110.134 ≈ 45.60 MB/s

Total Data per Second: 277.57 MB + 45.60 MB ≈ 323.17 MB

6.2 Integrate pylibfreenect2 and pyalsaaudio with GStreamer for 3 Streams from Kinect2: depth, rgb, and audio

Sorry my friends. I’m NOT going to disclose my code, in Python.

6.2.1 Server Side

Kinect2 RTSP Server
Kinect2 RTSP Server
Kinect2 RTSP Server Stream Responses
Kinect2 RTSP Server Stream Responses

6.2.2 Client Side

RGB Stream
RGB Stream
Depth Stream
Depth Stream
Audio Stream
Audio Stream