Happy new year everyone. Such a long time I haven't taken a look at activation functions. It might be a good time for me to review and check.
1. Resource
Let's first take a look at activation functions of year 2022.
- Wikipedia
- Swish Vs Mish: Latest Activation Functions
- Smish: A Novel Activation Function for Deep Learning Methods, strongly recommend to read this article
- ACON - Activate or Not: Learning Customized Activation, CVPR 2021., same thing can be found at Summary of neural network activation function. 2022.01, strongly recommend to read this article
- Activation Functions for Neural Networks and their Implementation in Python
- 12 Types of Neural Network Activation Functions: How to Choose?
I would strongly recommend you to read the 2 articles mentioned above, particularly: Smish.