Activation Functions

Happy new year everyone. Such a long time I haven’t taken a look at activation functions. It might be a good time for me to review and check.

1. Resource

Let’s first take a look at activation functions of year 2022.

I would strongly recommend you to read the 2 articles mentioned above, particularly: Smish.