When ReLU’s extrapolation capabilities aren’t sufficient
Neural networks are identified to be nice approximators for any perform — at the very least at any time when we don’t transfer too distant from our dataset. Allow us to see what meaning. Right here is a few information:
It doesn’t solely appear like a sine wave, it truly is, with some noise added. We will now practice a standard feed-forward neural community having 1 hidden layer with 1000 neurons and ReLU activation. We get the next match:
It seems fairly respectable, other than the perimeters. We may repair this by including extra neurons to the hidden layer in line with Cybenko’s common approximation theorem. However I wish to level you one thing else:
We may argue now that this extrapolation habits is unhealthy if we assume the wave sample to proceed exterior of the noticed vary. But when there is no such thing as a area information or extra information we will resort to, it could simply be this: an assumption.
Nevertheless, within the the rest of this text, we will assume that any periodic sample we will decide up inside the information continues exterior as effectively. This can be a frequent assumption when doing time sequence modeling, the place we naturally wish to extrapolate into the longer term. We assume that any noticed seasonality within the coaching information will simply proceed like that, as a result of what else can we are saying with none further info? On this article, I wish to present you the way utilizing sine-based activation features helps bake this assumption into the mannequin.
However earlier than we go there, allow us to shortly dive deeper into how ReLU-based neural networks extrapolate typically, and why we should always not use them for time sequence forecasting as is.
