Kinect 2 on Jetson AGX Xavier

Today, I wanna try out the old Kinect 2 on my old Jetson AGX Xavier.

0. My Working Environment

Jetson AGX Xavier 32G

For how I set up my old Jetson AGX Xavier, please refer to my previous blog NVidia Jetson Xavier AGX.

1. Kinect 2

1.1 Overview

Kinect 2 Overview

1.2 Hardware Specification

You can find specs of Kinect v2 in Evaluating the Accuracy of the Azure Kinect and Kinect v2.

2. lsusb and add udev rules

2.1 lsusb

1
2
3
4
$ lsusb | grep Microsoft
Bus 009 Device 015: ID 045e:02d9 Microsoft Corp. NuiSensor Adaptor
Bus 010 Device 005: ID 045e:02d9 Microsoft Corp. NuiSensor Adaptor
Bus 010 Device 006: ID 045e:02c4 Microsoft Corp. Xbox NUI Sensor

2.2 lsusb More Details

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
➜  ~ lsusb -v -d 045e:02c4

Bus 002 Device 015: ID 045e:02c4 Microsoft Corp.
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 3.00
bDeviceClass 239 Miscellaneous Device
bDeviceSubClass 2
bDeviceProtocol 1 Interface Association
bMaxPacketSize0 9
idVendor 0x045e Microsoft Corp.
idProduct 0x02c4
bcdDevice 1.00
iManufacturer 1 Microsoft
iProduct 2 Xbox NUI Sensor
iSerial 4 178602434347
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x00d6
bNumInterfaces 4
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xe0
Self Powered
Remote Wakeup
MaxPower 144mA
Interface Association:
bLength 8
bDescriptorType 11
bFirstInterface 0
bInterfaceCount 2
bFunctionClass 255 Vendor Specific Class
bFunctionSubClass 255 Vendor Specific Subclass
bFunctionProtocol 0
iFunction 0
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 4
bInterfaceClass 255 Vendor Specific Class
bInterfaceSubClass 0
bInterfaceProtocol 0
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0400 1x 1024 bytes
bInterval 0
bMaxBurst 7
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x02 EP 2 OUT
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0400 1x 1024 bytes
bInterval 0
bMaxBurst 7
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x83 EP 3 IN
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0400 1x 1024 bytes
bInterval 0
bMaxBurst 7
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x82 EP 2 IN
bmAttributes 19
Transfer Type Interrupt
Synch Type None
Usage Type Feedback
wMaxPacketSize 0x0040 1x 64 bytes
bInterval 14
bMaxBurst 0
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 1
bAlternateSetting 0
bNumEndpoints 0
bInterfaceClass 255 Vendor Specific Class
bInterfaceSubClass 0
bInterfaceProtocol 0
iInterface 0
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 1
bAlternateSetting 1
bNumEndpoints 1
bInterfaceClass 255 Vendor Specific Class
bInterfaceSubClass 0
bInterfaceProtocol 0
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x84 EP 4 IN
bmAttributes 1
Transfer Type Isochronous
Synch Type None
Usage Type Data
wMaxPacketSize 0x0400 1x 1024 bytes
bInterval 1
bMaxBurst 10
Mult 2
Interface Association:
bLength 8
bDescriptorType 11
bFirstInterface 2
bInterfaceCount 2
bFunctionClass 1 Audio
bFunctionSubClass 2 Streaming
bFunctionProtocol 0
iFunction 0
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 2
bAlternateSetting 0
bNumEndpoints 0
bInterfaceClass 1 Audio
bInterfaceSubClass 1 Control Device
bInterfaceProtocol 0
iInterface 0
AudioControl Interface Descriptor:
bLength 9
bDescriptorType 36
bDescriptorSubtype 1 (HEADER)
bcdADC 1.00
wTotalLength 0x001e
bInCollection 1
baInterfaceNr(0) 3
AudioControl Interface Descriptor:
bLength 12
bDescriptorType 36
bDescriptorSubtype 2 (INPUT_TERMINAL)
bTerminalID 1
wTerminalType 0x0205 Microphone Array
bAssocTerminal 0
bNrChannels 4
wChannelConfig 0x0000
iChannelNames 0
iTerminal 0
AudioControl Interface Descriptor:
bLength 9
bDescriptorType 36
bDescriptorSubtype 3 (OUTPUT_TERMINAL)
bTerminalID 3
wTerminalType 0x0101 USB Streaming
bAssocTerminal 0
bSourceID 1
iTerminal 0
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 3
bAlternateSetting 0
bNumEndpoints 0
bInterfaceClass 1 Audio
bInterfaceSubClass 2 Streaming
bInterfaceProtocol 0
iInterface 0
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 3
bAlternateSetting 1
bNumEndpoints 1
bInterfaceClass 1 Audio
bInterfaceSubClass 2 Streaming
bInterfaceProtocol 0
iInterface 0
AudioStreaming Interface Descriptor:
bLength 7
bDescriptorType 36
bDescriptorSubtype 1 (AS_GENERAL)
bTerminalLink 3
bDelay 1 frames
wFormatTag 0x0001 PCM
AudioStreaming Interface Descriptor:
bLength 11
bDescriptorType 36
bDescriptorSubtype 2 (FORMAT_TYPE)
bFormatType 1 (FORMAT_TYPE_I)
bNrChannels 4
bSubframeSize 4
bBitResolution 32
bSamFreqType 1 Discrete
tSamFreq[ 0] 16000
Endpoint Descriptor:
bLength 9
bDescriptorType 5
bEndpointAddress 0x85 EP 5 IN
bmAttributes 1
Transfer Type Isochronous
Synch Type None
Usage Type Data
wMaxPacketSize 0x0100 1x 256 bytes
bInterval 4
bRefresh 0
bSynchAddress 0
bMaxBurst 0
AudioStreaming Endpoint Descriptor:
bLength 7
bDescriptorType 37
bDescriptorSubtype 1 (EP_GENERAL)
bmAttributes 0x00
bLockDelayUnits 0 Undefined
wLockDelay 0x0000
Binary Object Store Descriptor:
bLength 5
bDescriptorType 15
wTotalLength 0x0016
bNumDeviceCaps 2
SuperSpeed USB Device Capability:
bLength 10
bDescriptorType 16
bDevCapabilityType 3
bmAttributes 0x00
wSpeedsSupported 0x000c
Device can operate at High Speed (480Mbps)
Device can operate at SuperSpeed (5Gbps)
bFunctionalitySupport 2
Lowest fully-functional device speed is High Speed (480Mbps)
bU1DevExitLat 10 micro seconds
bU2DevExitLat 2047 micro seconds
USB 2.0 Extension Device Capability:
bLength 7
bDescriptorType 16
bDevCapabilityType 2
bmAttributes 0x00000002
HIRD Link Power Management (LPM) Supported
can't get debug descriptor: Resource temporarily unavailable
Device Status: 0x0001
Self Powered
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
➜  ~ lsusb -v -d 045e:02d9 

Bus 002 Device 014: ID 045e:02d9 Microsoft Corp. NuiSensor Adaptor
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 3.00
bDeviceClass 9 Hub
bDeviceSubClass 0
bDeviceProtocol 3
bMaxPacketSize0 9
idVendor 0x045e Microsoft Corp.
idProduct 0x02d9
bcdDevice 0.73
iManufacturer 1 Microsoft Corporation
iProduct 2 NuiSensor Adaptor
iSerial 0
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x001f
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xe0
Self Powered
Remote Wakeup
MaxPower 0mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 1
bInterfaceClass 9 Hub
bInterfaceSubClass 0
bInterfaceProtocol 0 Full speed (or root) hub
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 19
Transfer Type Interrupt
Synch Type None
Usage Type Feedback
wMaxPacketSize 0x0002 1x 2 bytes
bInterval 8
bMaxBurst 0
Hub Descriptor:
bLength 12
bDescriptorType 42
nNbrPorts 1
wHubCharacteristic 0x000d
Per-port power switching
Compound device
Per-port overcurrent protection
bPwrOn2PwrGood 100 * 2 milli seconds
bHubContrCurrent 0 milli Ampere
bHubDecLat 0.4 micro seconds
wHubDelay 4004 nano seconds
DeviceRemovable 0x00
Hub Port Status:
Port 1: 0000.0203 5Gbps power U0 enable connect
Binary Object Store Descriptor:
bLength 5
bDescriptorType 15
wTotalLength 0x002a
bNumDeviceCaps 3
USB 2.0 Extension Device Capability:
bLength 7
bDescriptorType 16
bDevCapabilityType 2
bmAttributes 0x00000002
HIRD Link Power Management (LPM) Supported
SuperSpeed USB Device Capability:
bLength 10
bDescriptorType 16
bDevCapabilityType 3
bmAttributes 0x00
wSpeedsSupported 0x000e
Device can operate at Full Speed (12Mbps)
Device can operate at High Speed (480Mbps)
Device can operate at SuperSpeed (5Gbps)
bFunctionalitySupport 1
Lowest fully-functional device speed is Full Speed (12Mbps)
bU1DevExitLat 4 micro seconds
bU2DevExitLat 231 micro seconds
Container ID Device Capability:
bLength 20
bDescriptorType 16
bDevCapabilityType 4
bReserved 0
ContainerID {30eef35c-07d5-2549-b001-802d79434c30}
can't get debug descriptor: Resource temporarily unavailable
Device Status: 0x000d
Self Powered
U1 Enabled
U2 Enabled

Bus 001 Device 008: ID 045e:02d9 Microsoft Corp. NuiSensor Adaptor
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.10
bDeviceClass 9 Hub
bDeviceSubClass 0
bDeviceProtocol 1 Single TT
bMaxPacketSize0 64
idVendor 0x045e Microsoft Corp.
idProduct 0x02d9
bcdDevice 0.70
iManufacturer 1 Microsoft Corporation
iProduct 2 NuiSensor Adaptor
iSerial 0
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0019
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xe0
Self Powered
Remote Wakeup
MaxPower 0mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 1
bInterfaceClass 9 Hub
bInterfaceSubClass 0
bInterfaceProtocol 0 Full speed (or root) hub
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0001 1x 1 bytes
bInterval 12
Hub Descriptor:
bLength 9
bDescriptorType 41
nNbrPorts 1
wHubCharacteristic 0x00ed
Per-port power switching
Compound device
Per-port overcurrent protection
TT think time 32 FS bits
Port indicators
bPwrOn2PwrGood 50 * 2 milli seconds
bHubContrCurrent 100 milli Ampere
DeviceRemovable 0x00
PortPwrCtrlMask 0xff
Hub Port Status:
Port 1: 0000.0100 power
Binary Object Store Descriptor:
bLength 5
bDescriptorType 15
wTotalLength 0x002a
bNumDeviceCaps 3
USB 2.0 Extension Device Capability:
bLength 7
bDescriptorType 16
bDevCapabilityType 2
bmAttributes 0x00000002
HIRD Link Power Management (LPM) Supported
SuperSpeed USB Device Capability:
bLength 10
bDescriptorType 16
bDevCapabilityType 3
bmAttributes 0x00
wSpeedsSupported 0x000e
Device can operate at Full Speed (12Mbps)
Device can operate at High Speed (480Mbps)
Device can operate at SuperSpeed (5Gbps)
bFunctionalitySupport 1
Lowest fully-functional device speed is Full Speed (12Mbps)
bU1DevExitLat 4 micro seconds
bU2DevExitLat 231 micro seconds
Container ID Device Capability:
bLength 20
bDescriptorType 16
bDevCapabilityType 4
bReserved 0
ContainerID {30eef35c-07d5-2549-b001-802d79434c30}
can't get debug descriptor: Resource temporarily unavailable
Device Status: 0x0001
Self Powered

2.3 Add udev rules Under /etc/udev/rules.d

Take a look at OpenKinect libfreenect2 90-kinect2.rules:

1
2
3
4
5
6
7
8
9
10
$ cat platform/linux/udev/90-kinect2.rules            
───────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│ File: platform/linux/udev/90-kinect2.rules
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
1 │ # this file belongs in /etc/udev/rules.d/
2 │ # ATTR{product}=="Kinect2"
3 │ SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02c4", MODE="0666"
4 │ SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02d8", MODE="0666"
5 │ SUBSYSTEM=="usb", ATTR{idVendor}=="045e", ATTR{idProduct}=="02d9", MODE="0666"
───────┴───────────────────────────────────────────────────────────────────────────────────

We’ll do the following 2 steps:

  • step 1:
    1
    $ sudo cp platform/linux/udev/90-kinect2.rules /etc/udev/rules.d
  • step 2:
    1
    2
    $ sudo udevadm control --reload-rules
    $ sudo udevadm trigger

2.4 Don’t Use Default OpenNI2 from Default Repository

2.4.1 Incorrect Rules

Take a look at OpenNI2 Kinect2 primesense-usb.rules:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ cat Packaging/Linux/primesense-usb.rules               
───────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│ File: Packaging/Linux/primesense-usb.rules
───────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
1 │ # Make primesense device mount with writing permissions (default is read only for unknown devices)
2 │ SUBSYSTEM=="usb", ATTR{idProduct}=="0200", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
3 │ SUBSYSTEM=="usb", ATTR{idProduct}=="0300", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
4 │ SUBSYSTEM=="usb", ATTR{idProduct}=="0401", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
5 │ SUBSYSTEM=="usb", ATTR{idProduct}=="0500", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
6 │ SUBSYSTEM=="usb", ATTR{idProduct}=="0600", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
7 │ SUBSYSTEM=="usb", ATTR{idProduct}=="0601", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
8 │ SUBSYSTEM=="usb", ATTR{idProduct}=="0609", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
9 │ SUBSYSTEM=="usb", ATTR{idProduct}=="1250", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
10 │ SUBSYSTEM=="usb", ATTR{idProduct}=="1260", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
11 │ SUBSYSTEM=="usb", ATTR{idProduct}=="1270", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
12 │ SUBSYSTEM=="usb", ATTR{idProduct}=="1280", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
13 │ SUBSYSTEM=="usb", ATTR{idProduct}=="1290", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"
14 │ SUBSYSTEM=="usb", ATTR{idProduct}=="f9db", ATTR{idVendor}=="1d27", MODE:="0666", OWNER:="root", GROUP:="video"

!Note: by installing libopenni2-dev and openni2-utils from default repo, you’re going to have the above rule primesense-usb.rules installed under /usr/lib/udev/rules.d, instead of /etc/udev/rules.d.

2.4.2 Incorrect NiViewer

Even if package openni2-utils installed an executable NiViewer2 under /usr/bin, it fails to even detect the device Kinect v2.

1
2
3
4
5
$ NiViewer2
openDevice failed:
DeviceOpen using default: no devices found

Press any key to continue . . .

Due to the above Incorrect Rules, this NiViewer2, of course, should NOT detect the device.

Feel so disappointed …

😞

What made me feel worse, even desperate???

The above failure happens on both of my Jetson AGX Xavier running Ubuntu 20.04.6 LTS aarch64, and my desktop running Ubuntu 24.04.

3. For Windows

Please strictly follow Kinect for Windows SDK 2.0. For my Kinect 2, I got the following outputs on my laptop running Windows 11.

4. For Linux

4.1 Demonstration FIRST

OpenNI2 NiViewer
OpenNI2 NiViewer
libfreenect2 Protonect CPU
libfreenect2 Protonect CPU
libfreenect2 Protonect -GPU=0
libfreenect2 Protonect -GPU=0

4.2 Build OpenNI2 Github from Source

4.2.1 Key Components

Components Description
PSCommon The third-party library directly from PrimeSense
OniFile Must be dealing with OpenNI's .oni files
PS1080 Clearly related to PrimeSense
PSLink Clearly related to PrimeSense
Kinect I believe this is the driver for original Kinect v1, namely, Kinect for Xbox 360.
Kinect2 I believe this is the driver for Kinect2, based on Kinect for Windows SDK 2.0

To build Kinect or Kinect2, I think a Windows environment is a MUST, which is NOT for me.

4.2.2 Tool NiViewer

Driver Kinect2 in the above is a MUST for OpenNI2 Github's application NiViewer.

4.2.3 Any Driver to Drive Kinect2 and Can Work With OpenNI2 Github?

Now, stucked. Under Linux, how to build a driver working for my Kinect v2?

Solution: Build libfreenect2 from source, with BUILD_OPENNI2_DRIVER ON.

4.3 Build libfreenect2 from Source

Download OpenNI2 and save it, and built libfreenect2 with BUILD_OPENNI2_DRIVER set ON.

By doing so, you will have the following 2 variables set by default as:

1
2
OpenNI2_INCLUDE_DIRS             /opt/OpenNI/Include
OpenNI2_LIBRARY /opt/OpenNI/Redist/libOpenNI2.so

and you will have the following two libraries built out:

1
2
lib/libfreenect2-openni2.so
lib/libfreenect2.so

And, if needed, after the installation of libfreenect2, manually copy libfreenect2-openni2.so under folder /usr/local/lib/OpenNI2/Drivers. Now, ALL set. You’re good to go NiViewer.

4.4 Protonect or ProtonectSR from libfreenect2

4.4.1 ./bin/Protonect cuda

I got NO idea why ./bin/Protonect cuda failed to run ALL the time. Then, I created a simple Python script verify_cuda.py and ALWAYS meet the following error:

1
2
3
4
5
$ python verify_cuda.py   
[Info] [CudaDepthPacketProcessorImpl] device 0: NVIDIA GeForce RTX 3090 @ 1725MHz Memory 24219MB
[Info] [CudaDepthPacketProcessorImpl] selected device 0
[Error] [CudaDepthPacketProcessorImpl] cudaGetLastError(): the provided PTX was compiled with an unsupported toolchain.
CudaPacketPipeline is available.

Finally, I found the reason: Cuda version is NOT synced with NVidia driver version. More specifically, in my case:

To solve this problem, install NVidia driver during the installation of Cuda. Namely: when run cuda_12.5.1_555.42.06_linux.run, tick

1
2
- [X] Driver                                                                 │
│ [X] 555.42.06
  • You may have to reboot into text mode and have all previously-installed NVidia driver unloaded. For this issue, please refer to How to unload kernel module ‘nvidia-drm’?
  • The key: sudo systemctl isolate multi-user.target

Protonect cuda

4.4.2 ./bin/Protonect -gpu=0

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
$ ./bin/Protonect -gpu=0
Version: 0.2.0
Environment variables: LOGFILE=<protonect.log>
Usage: ./bin/Protonect [-gpu=<id>] [gl | cl | clkde | cuda | cudakde | cpu] [<device serial>]
[-noviewer] [-norgb | -nodepth] [-help] [-version]
[-frames <number of frames to process>]
To pause and unpause: pkill -USR1 Protonect
[Info] [Freenect2Impl] enumerating devices...
[Info] [Freenect2Impl] 23 usb devices connected
[Info] [Freenect2Impl] found valid Kinect v2 @10:5 with serial 178602434347
[Info] [Freenect2Impl] found 1 devices
libva info: VA-API version 1.20.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/nvidia_drv_video.so
libva info: va_openDriver() returns -1
[Error] [VaapiRgbPacketProcessorImpl] vaInitialize(display, &major_ver, &minor_ver): unknown libva error
[Info] [Freenect2DeviceImpl] opening...
[Info] [Freenect2DeviceImpl] transfer pool sizes rgb: 20*16384 ir: 60*8*33792
[Info] [Freenect2DeviceImpl] opened
[Info] [Freenect2DeviceImpl] starting...
[Debug] [Freenect2DeviceImpl] status 0x090000: 9855
[Debug] [Freenect2DeviceImpl] status 0x090000: 9855
[Info] [Freenect2DeviceImpl] submitting rgb transfers...
[Info] [Freenect2DeviceImpl] submitting depth transfers...
[Info] [Freenect2DeviceImpl] started
device serial: 178602434347
device firmware: 4.0.3916.0
[Debug] [DepthPacketStreamParser] not all subsequences received 0
[Debug] [DepthPacketStreamParser] not all subsequences received 1021
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Info] [DepthPacketStreamParser] 2 packets were lost
[Info] [DepthPacketStreamParser] 7 packets were lost
[Info] [OpenGLDepthPacketProcessor] avg. time: 5.12134ms -> ~195.261Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 4.06057ms -> ~246.271Hz
[Info] [TurboJpegRgbPacketProcessor] avg. time: 19.003ms -> ~52.6232Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 4.01312ms -> ~249.182Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 4.20707ms -> ~237.695Hz
[Info] [TurboJpegRgbPacketProcessor] avg. time: 17.376ms -> ~57.5507Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 3.95256ms -> ~253.001Hz
[2] - 2620271 killed ./bin/ProtonectSR cuda
[Info] [OpenGLDepthPacketProcessor] avg. time: 4.40171ms -> ~227.184Hz
[Info] [TurboJpegRgbPacketProcessor] avg. time: 17.8467ms -> ~56.0326Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 6.23296ms -> ~160.437Hz
[3] + 2628690 killed ./bin/Protonect
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Info] [OpenGLDepthPacketProcessor] avg. time: 9.88223ms -> ~101.192Hz
[Info] [DepthPacketStreamParser] 2 packets were lost
[Info] [TurboJpegRgbPacketProcessor] avg. time: 19.2099ms -> ~52.0565Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 5.8084ms -> ~172.164Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 8.7243ms -> ~114.622Hz
[Info] [TurboJpegRgbPacketProcessor] avg. time: 18.2581ms -> ~54.7703Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 9.24396ms -> ~108.179Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 10.1021ms -> ~98.9893Hz
[Info] [TurboJpegRgbPacketProcessor] avg. time: 18.3488ms -> ~54.4995Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 8.35109ms -> ~119.745Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 5.61051ms -> ~178.237Hz
[Info] [TurboJpegRgbPacketProcessor] avg. time: 17.9773ms -> ~55.6256Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 6.80606ms -> ~146.928Hz
[Info] [Freenect2DeviceImpl] stopping...
[Info] [Freenect2DeviceImpl] canceling rgb transfers...
[Info] [Freenect2DeviceImpl] canceling depth transfers...
[Info] [Freenect2DeviceImpl] stopped
[Info] [Freenect2DeviceImpl] closing...
[Info] [Freenect2DeviceImpl] releasing usb interfaces...
[Info] [Freenect2DeviceImpl] deallocating usb transfer pools...
[Info] [Freenect2DeviceImpl] closing usb device...
[Info] [Freenect2DeviceImpl] closed
[Info] [Freenect2DeviceImpl] closing...
[Info] [Freenect2DeviceImpl] already closed, doing nothing
[TurboJpegRgbPacketProcessor] 9.8382 10.0887 18.9733 20.2907 26.1092 mean=18.2873 std=2.86968 n=749
[OpenGLDepthPacketProcessor] 2.90518 3.87679 5.69045 12.5144 95.8873 mean=6.46404 std=4.29096 n=1528

4.4.3 ./bin/Protonect cpu

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
$ ./bin/Protonect cpu
Version: 0.2.0
Environment variables: LOGFILE=<protonect.log>
Usage: ./bin/Protonect [-gpu=<id>] [gl | cl | clkde | cuda | cudakde | cpu] [<device serial>]
[-noviewer] [-norgb | -nodepth] [-help] [-version]
[-frames <number of frames to process>]
To pause and unpause: pkill -USR1 Protonect
libva info: VA-API version 1.20.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/nvidia_drv_video.so
libva info: va_openDriver() returns -1
[Error] [VaapiRgbPacketProcessorImpl] vaInitialize(display, &major_ver, &minor_ver): unknown libva error
[Info] [Freenect2Impl] enumerating devices...
[Info] [Freenect2Impl] 23 usb devices connected
[Info] [Freenect2Impl] found valid Kinect v2 @10:5 with serial 178602434347
[Info] [Freenect2Impl] found 1 devices
[Info] [Freenect2DeviceImpl] opening...
[Info] [Freenect2DeviceImpl] transfer pool sizes rgb: 20*16384 ir: 60*8*33792
[Info] [Freenect2DeviceImpl] opened
[Info] [Freenect2DeviceImpl] starting...
[Debug] [Freenect2DeviceImpl] status 0x090000: 9729
[Debug] [Freenect2DeviceImpl] status 0x090000: 9731
[Info] [Freenect2DeviceImpl] submitting rgb transfers...
[Info] [Freenect2DeviceImpl] submitting depth transfers...
[Info] [Freenect2DeviceImpl] started
device serial: 178602434347
device firmware: 4.0.3916.0
[Debug] [DepthPacketStreamParser] not all subsequences received 0
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Info] [DepthPacketStreamParser] 30 packets were lost
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Info] [DepthPacketStreamParser] 30 packets were lost
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Info] [Freenect2DeviceImpl] stopping...
[Info] [Freenect2DeviceImpl] canceling rgb transfers...
[Debug] [DepthPacketStreamParser] skipping depth packet
[Debug] [DepthPacketStreamParser] skipping depth packet
[Info] [Freenect2DeviceImpl] canceling depth transfers...
[Info] [Freenect2DeviceImpl] stopped
[Info] [Freenect2DeviceImpl] closing...
[Info] [Freenect2DeviceImpl] releasing usb interfaces...
[Info] [Freenect2DeviceImpl] deallocating usb transfer pools...
[Info] [Freenect2DeviceImpl] closing usb device...
[Info] [Freenect2DeviceImpl] closed
[Info] [Freenect2DeviceImpl] closing...
[Info] [Freenect2DeviceImpl] already closed, doing nothing
[TurboJpegRgbPacketProcessor] 10.0384 10.1629 18.9524 20.9346 27.5923 mean=18.5922 std=3.23163 n=44
[CpuDepthPacketProcessor] 101.371 101.485 102.303 109.367 111.332 mean=102.829 std=2.25233 n=27

4.4.4 Conclusive Comparisons

Let’s do a simple comparison of the speed using 3 different parameters: cuda, -cpu=0, cpu.

Commands RGB Mean Time (ms) Depth Mean Time (ms)
./bin/Protonect cuda 18.8951 0.521472
./bin/Protonect -gpu=0 18.2873 6.46404
./bin/Protonect cpu 18.5922 102.829

The conclusion is quite obvious:

  • for depth: CudaDepthPacketProcessor > OpenGLDepthPacketProcessor > CpuDepthPacketProcessor.
  • for RGB: Always TurboJpegRgbPacketProcessor

4.5 pylibfreenect2 - libfreenect2's Python Wrapper

So far, in my personal point of view, pylibfreenect2 is the BEST Python wrapper of libfreenect2. But, according to the official documentation of libfreenect2, it’s clearly written:

Audio. Raw audio is accessible via Linux USB audio. There is no support for the calibrated directional audio.

4.6 Kinect2's Audio pyalsaaudio

4.6.1 lsusb Again

1
2
3
4
➜  ~ lsusb | grep Microsoft 
Bus 002 Device 003: ID 045e:02c4 Microsoft Corp.
Bus 002 Device 002: ID 045e:02d9 Microsoft Corp. NuiSensor Adaptor
Bus 001 Device 002: ID 045e:02d9 Microsoft Corp. NuiSensor Adaptor

4.6.2 arecord -l

1
2
➜  ~ arecord -l | grep NUI             
card 1: Sensor [Xbox NUI Sensor], device 0: USB Audio [USB Audio]
  • Card number: 1
  • Device number: 0

4.6.3 List Details of hw:1,0

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
➜  arecord -D hw:1,0 --dump-hw-params
Recording WAVE 'stdin' : Unsigned 8 bit, Rate 8000 Hz, Mono
HW Params of device "hw:1,0":
--------------------
ACCESS: MMAP_INTERLEAVED RW_INTERLEAVED
FORMAT: S32_LE
SUBFORMAT: STD
SAMPLE_BITS: 32
FRAME_BITS: 128
CHANNELS: 4
RATE: 16000
PERIOD_TIME: [1000 2048000]
PERIOD_SIZE: [16 32768]
PERIOD_BYTES: [256 524288]
PERIODS: [2 1024]
BUFFER_TIME: [2000 4096000]
BUFFER_SIZE: [32 65536]
BUFFER_BYTES: [512 1048576]
TICK_TIME: ALL
--------------------
arecord: set_params:1368: Sample format non available
Available formats:
- S32_LE

A couple of concerns:

  • I personally think FRAME_BITS is a typo. It should be FRAME_SIZE.
  • What the sample rate is 16000, but NOT 44100? 😀

4.6.4 Demonstration

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
import alsaaudio
import wave
import time

# Open the audio device
inp = alsaaudio.PCM(alsaaudio.PCM_CAPTURE, alsaaudio.PCM_NORMAL, "plughw:Sensor")

# Set audio parameters
inp.setformat(alsaaudio.PCM_FORMAT_S32_LE)
inp.setchannels(4)
inp.setrate(16000)
inp.setperiodsize(128)

# Prepare a file to write the audio data
with wave.open("test.wav", "wb") as f:
f.setnchannels(4)
f.setsampwidth(4)
f.setframerate(16000)

# Record audio for 10 seconds
timeout = 10 # seconds
start_time = time.time()
while time.time() - start_time < timeout:
num_bytes, buffer = inp.read()
f.writeframes(buffer)

# Close the audio device
inp.close()

print("Recording finished")

5. For Jetson AGX Xavier

Now, we’re done on my desktop. Let’s try Kinect2 on Jetson AGX Xavier.

5.1 libfreenect2 - Do NOT Enable ENABLE_TEGRAJPEG

By default, libfreenect2 has ENABLE_TEGRAJPEG set ON, which builds fine. However, the followig issue has been reported for over 8 years.

Solution for Jetson AGX Xavier:

libfreenect2 Protonect Cuda
libfreenect2 Protonect Cuda
libfreenect2 Protonect Cuda Speed
libfreenect2 Protonect Cuda

From the above output, we can easily come up with the following table:

Attribute RGB Depth
Resolution 1920x1080 512x424
Number of Channels 3 (Red, Green, Blue) 1 (Depth)
Bytes per Pixel 3 bytes 2 bytes
Average Processing Time ~22.4048 ms ~7.18678 ms
FPS ~44.6163 FPS ~110.134 FPS
Data per Frame (Bytes) 1920 * 1080 * 3 = 6,220,800 bytes 512 * 424 * 2 = 434,176 bytes
Data per Frame (MB) ~6.22 MB ~0.414 MB
Data per Second (MB/s) 6.22 MB * 44.6163 ≈ 277.57 MB/s 0.414 MB * 110.134 ≈ 45.60 MB/s

Total Data per Second: 277.57 MB + 45.60 MB ≈ 323.17 MB

5.2 Integrate pylibfreenect2 and pyalsaaudio with GStreamer for 3 Streams from Kinect2: depth, rgb, and audio

Sorry my friends. I’m NOT going to disclose my code, in Python.

5.2.1 Server Side

Kinect2 RTSP Server
Kinect2 RTSP Server
Kinect2 RTSP Server Stream Responses
Kinect2 RTSP Server Stream Responses

5.2.2 Client Side

RGB Stream
RGB Stream
Depth Stream
Depth Stream
Audio Stream
Audio Stream