Bash Skills

Posted on Edited on In ,

Back to Vancouver, Canada now. Start writing something again. Just wanna write down some very useful bash scripts.

Skill 1 - How to upgrade all Python packages with pip?

Solution

Please refer to https://stackoverflow.com/questions/2720014/how-to-upgrade-all-python-packages-with-pip. Bash script is:

1
2
3
nice python3 -m pip list --outdated --format=json | \
      jq -r '.[] | "\(.name)==\(.latest_version)"' | \
      xargs --no-run-if-empty -n1 pip3 install -U --user

Test

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
➜  ~ pip list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1  | xargs -n1 pip install -U --user
Collecting distro-info
Installing collected packages: distro-info
  Found existing installation: distro-info 0.0.0
    Uninstalling distro-info-0.0.0:
      Successfully uninstalled distro-info-0.0.0
Successfully installed distro-info-0.0.0
Collecting Markdown
  Downloading https://files.pythonhosted.org/packages/c0/4e/fd492e91abdc2d2fcb70ef453064d980688762079397f779758e055f6575/Markdown-3.1.1-py2.py3-none-any.whl (87kB)
     |████████████████████████████████| 92kB 4.5MB/s 
Requirement already satisfied, skipping upgrade: setuptools>=36 in ./.local/lib/python3.7/site-packages (from Markdown) (41.0.1)
Installing collected packages: Markdown
  WARNING: The script markdown_py is installed in '/home/lvision/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed Markdown-3.1.1
Collecting pycups
  Using cached https://files.pythonhosted.org/packages/00/ce/836a0deb8b24bcd5f850f8fb97f99fb4abd7374e078b9e6df5a0838f8eb5/pycups-1.9.74.tar.bz2
Building wheels for collected packages: pycups
  Building wheel for pycups (setup.py) ... error
  ERROR: Complete output from command /usr/bin/python3 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-9jc5iw0k/pycups/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-k2dtuswe --python-tag cp37:
  ERROR: running bdist_wheel
  running build
  running build_ext
  building 'cups' extension
  creating build
  creating build/temp.linux-x86_64-3.7
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DVERSION="1.9.74" -I/usr/include/python3.7m -c cupsmodule.c -o build/temp.linux-x86_64-3.7/cupsmodule.o
  cupsmodule.c:23:10: fatal error: cups/cups.h: No such file or directory
   #include <cups/cups.h>
            ^~~~~~~~~~~~~
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for pycups
  Running setup.py clean for pycups
Failed to build pycups
Installing collected packages: pycups
  Running setup.py install for pycups ... error
    ERROR: Complete output from command /usr/bin/python3 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-9jc5iw0k/pycups/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-imbt0ikc/install-record.txt --single-version-externally-managed --compile --user --prefix=:
    ERROR: running install
    running build
    running build_ext
    building 'cups' extension
    creating build
    creating build/temp.linux-x86_64-3.7
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DVERSION="1.9.74" -I/usr/include/python3.7m -c cupsmodule.c -o build/temp.linux-x86_64-3.7/cupsmodule.o
    cupsmodule.c:23:10: fatal error: cups/cups.h: No such file or directory
     #include <cups/cups.h>
              ^~~~~~~~~~~~~
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command "/usr/bin/python3 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-9jc5iw0k/pycups/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-imbt0ikc/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-install-9jc5iw0k/pycups/
Collecting Pygments
  Downloading https://files.pythonhosted.org/packages/5c/73/1dfa428150e3ccb0fa3e68db406e5be48698f2a979ccbcec795f28f44048/Pygments-2.4.2-py2.py3-none-any.whl (883kB)
     |████████████████████████████████| 890kB 3.8MB/s 
Installing collected packages: Pygments
  WARNING: The script pygmentize is installed in '/home/lvision/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed Pygments-2.4.2
Collecting PyGObject
  Using cached https://files.pythonhosted.org/packages/c1/36/c31a6ce9411b22a8ac0759c9b5ffda09368d5e0643450bb8ed83c271588c/PyGObject-3.32.1.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied, skipping upgrade: pycairo>=1.11.1 in ./.local/lib/python3.7/site-packages (from PyGObject) (1.18.1)
Building wheels for collected packages: PyGObject
  Building wheel for PyGObject (PEP 517) ... done
  Stored in directory: /home/lvision/.cache/pip/wheels/30/7a/00/cdefec8644e21ce0d5a438f96ce0ba65b51eb648c4e4614d62
Successfully built PyGObject
Installing collected packages: PyGObject
Successfully installed PyGObject-3.32.1
➜  ~ pip list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1  | xargs -n1 pip install -U --user
Collecting distro-info
Installing collected packages: distro-info
  Found existing installation: distro-info 0.0.0
    Uninstalling distro-info-0.0.0:
      Successfully uninstalled distro-info-0.0.0
Successfully installed distro-info-0.0.0
Collecting pycups
  Using cached https://files.pythonhosted.org/packages/00/ce/836a0deb8b24bcd5f850f8fb97f99fb4abd7374e078b9e6df5a0838f8eb5/pycups-1.9.74.tar.bz2
Building wheels for collected packages: pycups
  Building wheel for pycups (setup.py) ... done
  Stored in directory: /home/lvision/.cache/pip/wheels/bb/4a/db/d9ea6edeead55ff1b909f3e7aa4384db28ff6ce3ecef0c94f7
Successfully built pycups
Installing collected packages: pycups
Successfully installed pycups-1.9.74
➜  ~ pip list --outdated
Package     Version Latest Type 
----------- ------- ------ -----
distro-info 0.0.0   0.10   sdist
➜  ~

In the above test, clearly, we ran the following bash script twice:

1
➜  ~ pip list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1  | xargs -n1 pip install -U --user

For the 1st time, we successfully had distro-info, Markdown, Pygments, PyGObject installed, but failed to install pycups. For the 2nd time, after libcups2-dev had been installed from Ubuntu repository, we had pycups installed successfully.

However, we still have one LAST issue: why distro-info is still oudated? ^_^ Can anybody give me an explanation?

Skill 2 - How to list all auto-removable packages?

1
apt-get --dry-run autoremove | grep -Po '^Remv \K[^ ]+'

Skill 3 - How to deal with the bug "Error: ENOSPC: System limit for number of file watchers reached"?

1
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

Skill 4 - How to find all connected devices with nmap?

Solution

Please refer to https://vitux.com/find-devices-connected-to-your-network-with-nmap/. Bash script is:

1
nmap -sP 192.168.1.254/24

Test

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
➜  ~ nmap -sP 192.168.1.254/24
Starting Nmap 7.70 ( https://nmap.org ) at 2019-06-03 08:03 PDT
Nmap scan report for 192.168.1.64
Host is up (0.082s latency).
Nmap scan report for 192.168.1.65
Host is up (0.0092s latency).
Nmap scan report for 192.168.1.66
Host is up (0.010s latency).
Nmap scan report for 192.168.1.67
Host is up (0.037s latency).
Nmap scan report for 192.168.1.71
Host is up (0.032s latency).
Nmap scan report for 192.168.1.73
Host is up (0.032s latency).
Nmap scan report for 192.168.1.74
Host is up (0.061s latency).
Nmap scan report for 192.168.1.98
Host is up (0.044s latency).
Nmap scan report for 192.168.1.103
Host is up (0.15s latency).
Nmap scan report for lvision-GT72-6QE (192.168.1.200)
Host is up (0.00047s latency).
Nmap scan report for _gateway (192.168.1.254)
Host is up (0.015s latency).
Nmap done: 256 IP addresses (11 hosts up) scanned in 8.21 seconds
➜  ~

Skill 5 - How to install OS img with dd?

Solution

Please refer to Installing operating system images on Linux for Raspberry Pi . Bash script is:

1
sudo dd bs=4M if=Armbian_5.83_Nanopineo_Ubuntu_bionic_next_4.19.38.img of=/dev/mmcblk0 conv=fsync

Test

1
2
3
4
5
➜  ~ sudo dd bs=4M if=Armbian_5.83_Nanopineo_Ubuntu_bionic_next_4.19.38.img of=/dev/mmcblk0 conv=fsync
[sudo] password for lvision: 
223+0 records in
223+0 records out
935329792 bytes (935 MB, 892 MiB) copied, 89.9065 s, 10.4 MB/s

Skill 6 - How to add NVidia GPG Public Key?

Talked NVidia Issues:

1
➜  ~ wget -qO - https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub | sudo apt-key add -

Skill 7 - How to change username and it’s home directory?

Now, let’s try to change from longervision to lvision. Two commands after login with root:

1
2
➜  ~ sudo usermod -l lvision longervision
➜  ~ sudo usermod -d /home/lvision -m lvision

Skill 8 - How to fuse video and audio using FFmpeg?

1
❯ ffmpeg -i video.mp4 -i audio.mp3 -c:v copy -c:a aac -map 0✌0 -map 1🅰0 -shortest output.mp4

Skill 9 - How to install Python packages from source?

  • Traditional Way
1
2
$ python setup.py build
$ python setup.py install --user
  • Preferred Way
1
$ pip install --no-deps --prefix=~/.local/ .

Skill 10 - Obtain System Info in Graphs.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
   ✔  neofetch
            .-/+oossssoo+/-.               longervision@longervision-GT72-6QE 
        `:+ssssssssssssssssss+:`           ---------------------------------- 
      -+ssssssssssssssssssyyssss+-         OS: Ubuntu 18.04.4 LTS x86_64 
    .ossssssssssssssssssdMMMNysssso.       Host: GT72 6QE REV:1.0 
   /ssssssssssshdmmNNmmyNMMMMhssssss/      Kernel: 5.3.0-53-generic 
  +ssssssssshmydMMMMMMMNddddyssssssss+     Uptime: 20 hours, 45 mins 
 /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/    Packages: 4115 
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Shell: fish 3.1.2-630-gad020e84d 
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Resolution: 1920x1080 
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   DE: GNOME 3.28.4 
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   WM: GNOME Shell 
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   WM Theme: Adwaita 
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Theme: Ambiance [GTK2/3] 
 /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/    Icons: Ubuntu-mono-dark [GTK2/3] 
  +sssssssssdmydMMMMMMMMddddyssssssss+     Terminal: gnome-terminal 
   /ssssssssssshdmNNNNmyNMMMMhssssss/      CPU: Intel i7-6700HQ (8) @ 3.500GHz 
    .ossssssssssssssssssdMMMNysssso.       GPU: NVIDIA GeForce GTX 980M 
      -+sssssssssssssssssyyyssss+-         Memory: 8543MiB / 48025MiB 
        `:+ssssssssssssssssss+:` 
            .-/+oossssoo+/-.                                       
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
36  ✘  screenfetch                                                                                                                                                                                  1s  ~ 
                         ./+o+-       longervision@longervision-GT72-6QE
                 yyyyy- -yyyyyy+      OS: Ubuntu 18.04 bionic
              ://+//////-yyyyyyo      Kernel: x86_64 Linux 5.3.0-53-generic
          .++ .:/++++++/-.+sss/`      Uptime: 20h 56m
        .:++o:  /++++++++/:--:/-      Packages: 4116
       o:+o+:++.`..```.-/oo+++++/     Shell: fish 3.1.2-630-gad020e84d
      .:+o:+o/.          `+sssoo+/    Resolution: 1920x1080
 .++/+:+oo+o:`             /sssooo.   DE: GNOME 
/+++//+:`oo+o               /::--:.   WM: GNOME Shell
\+/+o+++`o++o               ++////.   WM Theme: Adwaita
 .++.o+++oo+:`             /dddhhh.   GTK Theme: Ambiance [GTK2/3]
      .+.o+oo:.          `oddhhhh+    Icon Theme: ubuntu-mono-dark
       \+.++o+o``-````.:ohdhhhhh+     Font: Ubuntu 11
        `:o+++ `ohhhhhhhhyo++os:      CPU: Intel Core i7-6700HQ @ 8x 3.5GHz
          .o:`.syhhhhhhh/.oo++o`      GPU: GeForce GTX 980M
              /osyyyyyyo++ooo+++/     RAM: 7910MiB / 48025MiB
                  ````` +oo+++o\:    
                         `oo++.

Skill 11 - How to SSH Before Logon?

Solution: Connect to network before user login

1
2
3
$ cd /etc/NetworkManager/system-connections
xxxxx@yyyyy:/etc/NetworkManager/system-connections$ ls
ZZZZZZZZZZ

Then, modify file ZZZZZZZZZZ, remove everything after permission=.

Skill 12 - How to SSH-KEYGEN?

All you need is on the Internet: ssh-keygen - Generate a New SSH Key?

Skill 13 - New Installed Kernel Does Not Show In Menu While Rebooting?

Solution: Not rebooting to the lastest kernel after update to grub 2

1
2
$ sudo grub-install /dev/sdb
$ sudo update-grub

Reason: grub is NOT installed under the RIGHT hard drive.

Skill 14 - Meson Conflicts Against pathlib?

Solution: 

1
$ pip uninstall pathlib

More details can be found https://github.com/Kozea/WeasyPrint/issues/807.

1
2
3
4
5
6
7
8
9
10
 12  ✔  python
  4s  ~/.local/lib/python3.6/site-packages 
Python 3.6.9 (default, Jul 17 2020, 12:50:27) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pathlib import Path
>>> print(Path().read_text)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'PosixPath' object has no attribute 'read_text'

Skill 15 - Run gdb with user root

  • Create a script called gdb in e.g. my home directory, containing: pkexec /usr/bin/gdb "$@"
  • make it executable
  • modify the launch.json in VSCode to call the script (obviously change username accordingly) by adding miDebuggerPath: 
    1
    2
    3
    4
    5
    ...
      "externalConsole": false,
      "miDebuggerPath": "/home/<username>/gdb",
      "MIMode": "gdb",
    ...
  • whilst debugging, use top or such like to verify the process is running as root.

Solution: How to debug programs with “sudo” in VSCODE.

Skill 16 - Command Line to Check Multiple Versions of A Package

pkg-config --debug --modversion glib-2.0 really helps. Refer to https://askubuntu.com/questions/488187/pkg-config-modversion-glib-2-0-reports-older-version.

Skill 17 - Pip Workaround

Basically, this is just an pip issue.

1
2
3
4
➜  pip list --outdated
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.

Therefore, python -m pip is used instead:

1
2
3
➜  bin python -m pip list --outdated
➜  bin python -m pip --version
pip 21.3.1 from /home/jetbot/.local/lib/python3.6/site-packages/pip (python 3.6)

Skill 18 - HOWTO – Resize partitions in raw disk (dd) image files (.img)

Refer to HOWTO – Resize partitions in raw disk (dd) image files (.img)

Skill 19 - error: failed to push some refs to 'XXX.XXX.XXX.XXX:~/myWebsite.git'

The Issue

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
➜  myWebsite git:(master) git push        
Enumerating objects: 1498, done.
Counting objects: 100% (1498/1498), done.
Delta compression using up to 48 threads
Compressing objects: 100% (1227/1227), done.
Writing objects: 100% (1498/1498), 1.20 MiB | 10.40 MiB/s, done.
Total 1498 (delta 182), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (182/182), done.
remote: error: refusing to update checked out branch: refs/heads/master
remote: error: By default, updating the current branch in a non-bare repository
remote: is denied, because it will make the index and work tree inconsistent
remote: with what you pushed, and will require 'git reset --hard' to match
remote: the work tree to HEAD.
remote: 
remote: You can set the 'receive.denyCurrentBranch' configuration variable
remote: to 'ignore' or 'warn' in the remote repository to allow pushing into
remote: its current branch; however, this is not recommended unless you
remote: arranged to update its work tree to match what you pushed in some
remote: other way.
remote: 
remote: To squelch this message and still keep the default behaviour, set
remote: 'receive.denyCurrentBranch' configuration variable to 'refuse'.
To XXX.XXX.XXX.XXX:~/myWebsite.git
 ! [remote rejected] master -> master (branch is currently checked out)
error: failed to push some refs to 'XXX.XXX.XXX.XXX:~/myWebsite.git'

Solution

1
git config receive.denyCurrentBranch ignore

Skill 20 - Find ALL Broken Symbolic Links

Refer to How can I find broken symlinks

1
find . -type l -exec sh -c 'file -b "$1" | grep -q "^broken"' sh {} \; -print

Skill 21 - VLC Hangs At checking for protobuf-lite.... How?

Just add --disable-chromecast.

1
./configure --enable-thumbnailer --enable-preparser --disable-chromecast

Skill 25 - 10 Cool Command Line Tools For Your Linux Terminal

Refer to 10 Cool Command Line Tools For Your Linux Terminal

Skill 26 - nginx service failed.

1
2
3
4
5
6
7
8
⋊> ~ systemctl status nginx.service                                                                                                                                                                                                          17:12:04
× nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2023-06-21 17:12:04 PDT; 13s ago
       Docs: man:nginx(8)
    Process: 13424 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
    Process: 13427 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
        CPU: 188ms

Solution:

sudo service nginx restart

after

1
2
sudo fuser -k 80/tcp
sudo fuser -k 443/tcp