Back to Vancouver, Canada now. Start writing something again. Just wanna write down some very useful bash scripts.
Skill 1 - How to upgrade all Python packages with pip? Solution Please refer to https://stackoverflow.com/questions/2720014/how-to-upgrade-all-python-packages-with-pip . Bash script is:
1 pip --disable-pip-version-check list --outdated --format=json | python -c "import json, sys; print('\n'.join([x['name'] for x in json.load(sys.stdin)]))" | xargs -n1 pip install -U
Test 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 ➜ ~ pip list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip install -U --user Collecting distro-info Installing collected packages: distro-info Found existing installation: distro-info 0.0.0 Uninstalling distro-info-0.0.0: Successfully uninstalled distro-info-0.0.0 Successfully installed distro-info-0.0.0 Collecting Markdown Downloading https://files.pythonhosted.org/packages/c0/4e/fd492e91abdc2d2fcb70ef453064d980688762079397f779758e055f6575/Markdown-3.1.1-py2.py3-none-any.whl (87kB) |████████████████████████████████| 92kB 4.5MB/s Requirement already satisfied, skipping upgrade: setuptools>=36 in ./.local/lib/python3.7/site-packages (from Markdown) (41.0.1) Installing collected packages: Markdown WARNING: The script markdown_py is installed in '/home/lvision/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. Successfully installed Markdown-3.1.1 Collecting pycups Using cached https://files.pythonhosted.org/packages/00/ce/836a0deb8b24bcd5f850f8fb97f99fb4abd7374e078b9e6df5a0838f8eb5/pycups-1.9.74.tar.bz2 Building wheels for collected packages: pycups Building wheel for pycups (setup.py) ... error ERROR: Complete output from command /usr/bin/python3 -u -c 'import setuptools, tokenize;__file__=' "'" '/tmp/pip-install-9jc5iw0k/pycups/setup.py' "'" ';f=getattr(tokenize, ' "'" 'open' "'" ', open)(__file__);code=f.read().replace(' "'" '\r\n' "'" ', ' "'" '\n' "'" ');f.close();exec(compile(code, __file__, ' "'" 'exec' "'" '))' bdist_wheel -d /tmp/pip-wheel-k2dtuswe --python-tag cp37: ERROR: running bdist_wheel running build running build_ext building 'cups' extension creating build creating build/temp.linux-x86_64-3.7 x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DVERSION="1.9.74" -I/usr/include/python3.7m -c cupsmodule.c -o build/temp.linux-x86_64-3.7/cupsmodule.o cupsmodule.c:23:10: fatal error: cups/cups.h: No such file or directory ^~~~~~~~~~~~~ compilation terminated. error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 ---------------------------------------- ERROR: Failed building wheel for pycups Running setup.py clean for pycups Failed to build pycups Installing collected packages: pycups Running setup.py install for pycups ... error ERROR: Complete output from command /usr/bin/python3 -u -c 'import setuptools, tokenize;__file__=' "'" '/tmp/pip-install-9jc5iw0k/pycups/setup.py' "'" ';f=getattr(tokenize, ' "'" 'open' "'" ', open)(__file__);code=f.read().replace(' "'" '\r\n' "'" ', ' "'" '\n' "'" ');f.close();exec(compile(code, __file__, ' "'" 'exec' "'" '))' install --record /tmp/pip-record-imbt0ikc/install-record.txt --single-version-externally-managed --compile --user --prefix=: ERROR: running install running build running build_ext building 'cups' extension creating build creating build/temp.linux-x86_64-3.7 x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DVERSION="1.9.74" -I/usr/include/python3.7m -c cupsmodule.c -o build/temp.linux-x86_64-3.7/cupsmodule.o cupsmodule.c:23:10: fatal error: cups/cups.h: No such file or directory ^~~~~~~~~~~~~ compilation terminated. error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 ---------------------------------------- ERROR: Command "/usr/bin/python3 -u -c 'import setuptools, tokenize;__file__='" '"' /tmp/pip-install-9jc5iw0k/pycups/setup.py'"' "';f=getattr(tokenize, '" '"' open'"' "', open)(__file__);code=f.read().replace('" '"' \r\n'"' "', '" '"' \n'"' "');f.close();exec(compile(code, __file__, '" '"' exec '"' "'))' install --record /tmp/pip-record-imbt0ikc/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-install-9jc5iw0k/pycups/ Collecting Pygments Downloading https://files.pythonhosted.org/packages/5c/73/1dfa428150e3ccb0fa3e68db406e5be48698f2a979ccbcec795f28f44048/Pygments-2.4.2-py2.py3-none-any.whl (883kB) |████████████████████████████████| 890kB 3.8MB/s Installing collected packages: Pygments WARNING: The script pygmentize is installed in '/home/lvision/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. Successfully installed Pygments-2.4.2 Collecting PyGObject Using cached https://files.pythonhosted.org/packages/c1/36/c31a6ce9411b22a8ac0759c9b5ffda09368d5e0643450bb8ed83c271588c/PyGObject-3.32.1.tar.gz Installing build dependencies ... done Getting requirements to build wheel ... done Preparing wheel metadata ... done Requirement already satisfied, skipping upgrade: pycairo>=1.11.1 in ./.local/lib/python3.7/site-packages (from PyGObject) (1.18.1) Building wheels for collected packages: PyGObject Building wheel for PyGObject (PEP 517) ... done Stored in directory: /home/lvision/.cache/pip/wheels/30/7a/00/cdefec8644e21ce0d5a438f96ce0ba65b51eb648c4e4614d62 Successfully built PyGObject Installing collected packages: PyGObject Successfully installed PyGObject-3.32.1 ➜ ~ pip list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip install -U --user Collecting distro-info Installing collected packages: distro-info Found existing installation: distro-info 0.0.0 Uninstalling distro-info-0.0.0: Successfully uninstalled distro-info-0.0.0 Successfully installed distro-info-0.0.0 Collecting pycups Using cached https://files.pythonhosted.org/packages/00/ce/836a0deb8b24bcd5f850f8fb97f99fb4abd7374e078b9e6df5a0838f8eb5/pycups-1.9.74.tar.bz2 Building wheels for collected packages: pycups Building wheel for pycups (setup.py) ... done Stored in directory: /home/lvision/.cache/pip/wheels/bb/4a/db/d9ea6edeead55ff1b909f3e7aa4384db28ff6ce3ecef0c94f7 Successfully built pycups Installing collected packages: pycups Successfully installed pycups-1.9.74 ➜ ~ pip list --outdated Package Version Latest Type ----------- ------- ------ ----- distro-info 0.0.0 0.10 sdist ➜ ~
In the above test, clearly, we ran the following bash script twice:
1 ➜ ~ pip list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip install -U --user
For the 1st time, we successfully had distro-info, Markdown, Pygments, PyGObject installed, but failed to install pycups .libcups2-dev had been installed from Ubuntu repository, we had pycups installed successfully.
However, we still have one LAST issue : why distro-info is still oudated? ^_^ Can anybody give me an explanation?
1 pip --disable-pip-version-check list --outdated --format=json | python -c "import json, sys; print('\n'.join([x['name'] for x in json.load(sys.stdin)]))" | xargs -n1 pip install -U
Skill 2 - How to list all auto-removable packages? 1 apt-get --dry-run autoremove | grep -Po '^Remv \K[^ ]+'
Skill 3 - How to deal with the bug “Error: ENOSPC: System limit for number of file watchers reached”? 1 echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
Skill 4 - How to find all connected devices with nmap? Solution Please refer to https://vitux.com/find-devices-connected-to-your-network-with-nmap/ . Bash script is:
1 nmap -sP 192.168.1.254/24
Test 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 ➜ ~ nmap -sP 192.168.1.254/24 Starting Nmap 7.70 ( https://nmap.org ) at 2019-06-03 08:03 PDT Nmap scan report for 192.168.1.64 Host is up (0.082s latency). Nmap scan report for 192.168.1.65 Host is up (0.0092s latency). Nmap scan report for 192.168.1.66 Host is up (0.010s latency). Nmap scan report for 192.168.1.67 Host is up (0.037s latency). Nmap scan report for 192.168.1.71 Host is up (0.032s latency). Nmap scan report for 192.168.1.73 Host is up (0.032s latency). Nmap scan report for 192.168.1.74 Host is up (0.061s latency). Nmap scan report for 192.168.1.98 Host is up (0.044s latency). Nmap scan report for 192.168.1.103 Host is up (0.15s latency). Nmap scan report for lvision-GT72-6QE (192.168.1.200) Host is up (0.00047s latency). Nmap scan report for _gateway (192.168.1.254) Host is up (0.015s latency). Nmap done : 256 IP addresses (11 hosts up) scanned in 8.21 seconds ➜ ~
Skill 5 - How to install OS img with dd? Solution Please refer to Installing operating system images on Linux for Raspberry Pi . Bash script is:
1 sudo dd bs=4M if =Armbian_5.83_Nanopineo_Ubuntu_bionic_next_4.19.38.img of=/dev/mmcblk0 conv=fsync
Test 1 2 3 4 5 ➜ ~ sudo dd bs=4M if =Armbian_5.83_Nanopineo_Ubuntu_bionic_next_4.19.38.img of=/dev/mmcblk0 conv=fsync [sudo] password for lvision: 223+0 records in 223+0 records out 935329792 bytes (935 MB, 892 MiB) copied, 89.9065 s, 10.4 MB/s
Skill 6 - How to add NVidia GPG Public Key? Talked NVidia Issues :
1 ➜ ~ wget -qO - https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub | sudo apt-key add -
Skill 7 - How to change username and it’s home directory? Now, let’s try to change from longervision to lvision . Two commands after login with root :
1 2 ➜ ~ sudo usermod -l lvision longervision ➜ ~ sudo usermod -d /home/lvision -m lvision
Skill 8 - How to fuse video and audio using FFmpeg? 1 ❯ ffmpeg -i video.mp4 -i audio.mp3 -c:v copy -c:a aac -map 0✌ 0 -map 1🅰 0 -shortest output.mp4
Skill 9 - How to install Python packages from source?
1 2 $ python setup.py build $ python setup.py install --user
1 $ pip install --no-deps --prefix=~/.local/ .
Skill 10 - Obtain System Info in Graphs. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 ✔ neofetch .-/+oossssoo+/-. longervision@longervision-GT72-6QE `:+ssssssssssssssssss+:` ---------------------------------- -+ssssssssssssssssssyyssss+- OS: Ubuntu 18.04.4 LTS x86_64 .ossssssssssssssssssdMMMNysssso. Host: GT72 6QE REV:1.0 /ssssssssssshdmmNNmmyNMMMMhssssss/ Kernel: 5.3.0-53-generic +ssssssssshmydMMMMMMMNddddyssssssss+ Uptime: 20 hours, 45 mins /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/ Packages: 4115 .ssssssssdMMMNhsssssssssshNMMMdssssssss. Shell: fish 3.1.2-630-gad020e84d +sssshhhyNMMNyssssssssssssyNMMMysssssss+ Resolution: 1920x1080 ossyNMMMNyMMhsssssssssssssshmmmhssssssso DE: GNOME 3.28.4 ossyNMMMNyMMhsssssssssssssshmmmhssssssso WM: GNOME Shell +sssshhhyNMMNyssssssssssssyNMMMysssssss+ WM Theme: Adwaita .ssssssssdMMMNhsssssssssshNMMMdssssssss. Theme: Ambiance [GTK2/3] /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/ Icons: Ubuntu-mono-dark [GTK2/3] +sssssssssdmydMMMMMMMMddddyssssssss+ Terminal: gnome-terminal /ssssssssssshdmNNNNmyNMMMMhssssss/ CPU: Intel i7-6700HQ (8) @ 3.500GHz .ossssssssssssssssssdMMMNysssso. GPU: NVIDIA GeForce GTX 980M -+sssssssssssssssssyyyssss+- Memory: 8543MiB / 48025MiB `:+ssssssssssssssssss+:` .-/+oossssoo+/-.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 36 ✘ screenfetch 1s ~ ./+o+- longervision@longervision-GT72-6QE yyyyy- -yyyyyy+ OS: Ubuntu 18.04 bionic ://+//////-yyyyyyo Kernel: x86_64 Linux 5.3.0-53-generic .++ .:/++++++/-.+sss/` Uptime: 20h 56m .:++o: /++++++++/:--:/- Packages: 4116 o:+o+:++.`..```.-/oo+++++/ Shell: fish 3.1.2-630-gad020e84d .:+o:+o/. `+sssoo+/ Resolution: 1920x1080 .++/+:+oo+o:` /sssooo. DE: GNOME /+++//+:`oo+o /::--:. WM: GNOME Shell \+/+o+++`o++o ++////. WM Theme: Adwaita .++.o+++oo+:` /dddhhh. GTK Theme: Ambiance [GTK2/3] .+.o+oo:. `oddhhhh+ Icon Theme: ubuntu-mono-dark \+.++o+o``-````.:ohdhhhhh+ Font: Ubuntu 11 `:o+++ `ohhhhhhhhyo++os: CPU: Intel Core i7-6700HQ @ 8x 3.5GHz .o:`.syhhhhhhh/.oo++o` GPU: GeForce GTX 980M /osyyyyyyo++ooo+++/ RAM: 7910MiB / 48025MiB ````` +oo+++o\: `oo++.
Skill 11 - How to SSH Before Logon? Solution: Connect to network before user login
1 2 3 $ cd /etc/NetworkManager/system-connectionsxxxxx@yyyyy:/etc/NetworkManager/system-connections$ ls ZZZZZZZZZZ
Then, modify file ZZZZZZZZZZ , remove everything after permission=.
Skill 12 - How to SSH-KEYGEN? All you need is on the Internet: ssh-keygen - Generate a New SSH Key ?
Solution : Not rebooting to the lastest kernel after update to grub 2
1 2 $ sudo grub-install /dev/sdb $ sudo update-grub
Reason : grub is NOT installed under the RIGHT hard drive.
Skill 14 - Meson Conflicts Against pathlib? Solution :
More details can be found https://github.com/Kozea/WeasyPrint/issues/807 .
1 2 3 4 5 6 7 8 9 10 12 ✔ python 4s ~/.local/lib/python3.6/site-packages Python 3.6.9 (default, Jul 17 2020, 12:50:27) [GCC 8.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. > >> from pathlib import Path > >> print (Path().read_text) Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'PosixPath' object has no attribute 'read_text'
Skill 15 - Run gdb with user root
Create a script called gdb in e.g. my home directory, containing: pkexec /usr/bin/gdb "$@"
make it executable
modify the launch.json in VSCode to call the script (obviously change username accordingly) by adding miDebuggerPath:1 2 3 4 5 ... "externalConsole" : false , "miDebuggerPath" : "/home/<username>/gdb" , "MIMode" : "gdb" , ...
whilst debugging, use top or such like to verify the process is running as root .
Solution: How to debug programs with “sudo” in VSCODE .
Skill 16 - Command Line to Check Multiple Versions of A Package pkg-config --debug --modversion glib-2.0 really helps. Refer to https://askubuntu.com/questions/488187/pkg-config-modversion-glib-2-0-reports-older-version .
Skill 17 - Pip Workaround Basically, this is just an pip issue .
1 2 3 4 ➜ pip list --outdated WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip. Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue. To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
Therefore, python -m pip is used instead:
1 2 3 ➜ bin python -m pip list --outdated ➜ bin python -m pip --version pip 21.3.1 from /home/jetbot/.local/lib/python3.6/site-packages/pip (python 3.6)
Skill 18 - HOWTO – Resize partitions in raw disk (dd) image files (.img) Refer to HOWTO – Resize partitions in raw disk (dd) image files (.img)
Skill 19 - error: failed to push some refs to ‘XXX.XXX.XXX.XXX:~/myWebsite.git’ The Issue 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 ➜ myWebsite git:(master) git push Enumerating objects: 1498, done. Counting objects: 100% (1498/1498), done. Delta compression using up to 48 threads Compressing objects: 100% (1227/1227), done. Writing objects: 100% (1498/1498), 1.20 MiB | 10.40 MiB/s, done. Total 1498 (delta 182), reused 0 (delta 0), pack-reused 0 remote: Resolving deltas: 100% (182/182), done. remote: error: refusing to update checked out branch: refs/heads/master remote: error: By default, updating the current branch in a non-bare repository remote: is denied, because it will make the index and work tree inconsistent remote: with what you pushed, and will require 'git reset --hard' to match remote: the work tree to HEAD. remote: remote: You can set the 'receive.denyCurrentBranch' configuration variable remote: to 'ignore' or 'warn' in the remote repository to allow pushing into remote: its current branch; however, this is not recommended unless you remote: arranged to update its work tree to match what you pushed in some remote: other way. remote: remote: To squelch this message and still keep the default behaviour, set remote: 'receive.denyCurrentBranch' configuration variable to 'refuse'. To XXX.XXX.XXX.XXX:~/myWebsite.git ! [remote rejected] master -> master (branch is currently checked out) error: failed to push some refs to 'XXX.XXX.XXX.XXX:~/myWebsite.git'
Solution 1 git config receive.denyCurrentBranch ignore
Skill 20 - Find ALL Broken Symbolic Links Refer to How can I find broken symlinks
1 find . -type l -exec sh -c 'file -b "$1" | grep -q "^broken"' sh {} \; -print
Refer to 10 Cool Command Line Tools For Your Linux Terminal
Skill 26 - nginx service failed. 1 2 3 4 5 6 7 8 ⋊> ~ systemctl status nginx.service 17:12:04 × nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Wed 2023-06-21 17:12:04 PDT; 13s ago Docs: man:nginx(8) Process: 13424 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Process: 13427 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE) CPU: 188ms
Solution :
sudo service nginx restart
after
1 2 sudo fuser -k 80/tcp sudo fuser -k 443/tcp
Skill 27 - How to Evaluate a Package Has Been Installed? 1 $ pkg-config --exists PackageName && echo "PackageName is installed." || echo "PackageName is not installed."
1 $ cmake --find-package -DNAME=PackageName -DCOMPILER_ID=GNU -DLANGUAGE=CXX -DMODE=EXIST