Convolutional neural networks: aren't central neurons over-represented at the output?

[This question has also been asked on Cross Validated]

Briefly question

I am studying Convolutional Neural Networks and I find that these networks do not process every input neuron (pixel / parameter) equivalently. Imagine we have a deep mesh (many layers) that applies convolution on some input image. The neurons in the “middle” of the image have many unique pathways for many of the deeper layer neurons, which means that a small change in the middle neurons has a strong effect on output. However, the neurons at the edge of the image have only one way (or, depending on the exact implementation, order 1) of the ways in which their information travels through the graph. They appear to be "underrepresented".

This worries me as this edge neuron discrimination scales exponentially with the depth (number of layers) of the network. Even adding a layer with the maximum pool will not stop the exponential increase, only a full connection brings all neurons on an equal footing. I'm not sure if my reasoning is correct, so my questions are:

• Is it correct that this effect takes place in deep convolutional networks?
• Is there any theory about this ever mentioned in the literature?
• Are there ways to overcome this effect?

Since I'm not sure if this provides enough information, I'll go into a little more detail about the problem, and why I believe it is a concern.

More detailed explanation

Imagine we have a deep neural network that takes an image as input. Suppose we apply a 64x64 pixel convolutional filter on the image, where each time we switch the convolution window by 4 pixels. This means that each neuron at the input sends it an activation in 16x16 = 265 neurons in layer 2. Each of these neurons can send its activation to another 265, so that our topmost neuron is represented in 265 ^ 2 output neurons, etc. This, however, is not true for neurons at the edges: they can only be represented in a small number of convolution windows, which makes them activate (of the order of) only 1 neuron in the next layer. Using tricks like edge mirroring won't help: the second level neurons projected so far are still on the edges, which meansthat minor neurons will be underrepresented (thus limiting the importance of our marginal neurons). As you can see, this discrepancy grows exponentially with the number of layers.

I created an image to visualize the problem, which can be found here (I am not allowed to include images in the post). This network has a convolution window of size 3. The numbers next to the neurons indicate the number of paths down to the deepest neuron. The image resembles Pascal's Triangle .

https://www.dropbox.com/s/7rbwv7z14j4h0jr/deep_conv_problem_stackxchange.png?dl=0

Why is this a problem?

This effect does not seem to be a problem at first glance: in principle, the weights should be automatically adjusted so that the network works. In addition, the edges of the image are not that important in image recognition. This effect may not be noticeable in everyday image recognition tests, but it still concerns me for two reasons: 1) generalization to other applications and 2) problems that arise in the case of very deep networks. 1) There may be other applications, such as speech or sound recognition, where it is not true that the most important neurons are the most important. Convolution is often done in this area, but I couldn't find a single document that mentions the effect I'm experiencing. 2) Very deep networks will notice the exponentially poor discrimination effect of borderline neurons, which meansthat central neurons can be overrepresented by several orders of magnitude (imagine we have 10 layers, so the above example will give 265 ^ 10 ways central neurons can project their information). As the number of layers increases, one of them must fall into the limit where the weights cannot really compensate for this effect. Now imagine that we are immune to all neurons by a small amount. Central neurons will result in a larger change in output by several orders of magnitude compared to marginal neurons. I believe that for general applications and for very deep networks it is necessary to find ways around my problem?so the above example will give 265 ^ 10 ways the central neurons can project their information). As the number of layers increases, one of them must fall into the limit where the weights cannot really compensate for this effect. Now imagine that we are immune to all neurons by a small amount. Central neurons will result in a larger change in output by several orders of magnitude compared to marginal neurons. I believe that for general applications and for very deep networks it is necessary to find ways around my problem?so the above example will give 265 ^ 10 paths central neurons can project their information). As the number of layers increases, one of them must fall into the limit where the weights cannot really compensate for this effect. Now imagine that we are immune to all neurons by a small amount. Central neurons will result in a larger change in output by several orders of magnitude compared to marginal neurons. I believe that for general applications and for very deep networks it is necessary to find ways around my problem?Central neurons will result in a larger change in output by several orders of magnitude compared to marginal neurons. I believe that for general applications and for very deep networks it is necessary to find ways around my problem?Central neurons will result in a larger change in output by several orders of magnitude compared to marginal neurons. I believe that for general applications and for very deep networks it is necessary to find ways around my problem?