Learnings:

1. Network depth with small convolutional filters:

By increasing the depth of the network (19 layers) and using very small 3x3 convolutional layers the network demonstrated improved performance on the ImageNet dataset.

Why so?

1. Receptive Field Equivlence:

• A single 5x5 convolution has a receptive field of 5x5.
• Two stacked 3x3 convolutions also have an effective receptive field of 5x5. Here's why:
  • The first 3x3 convolution covers a 3x3 region of the input.
  • The second 3x3 convolution covers a 3x3 region of the first layer's output, which corresponds to a 5x5 region of the input (because each output pixel of the first layer depends on a 3x3 region of the input).

Similarly:

• Three stacked 3x3 convolutions have an effective receptive field of 7x7.

Maths:

1. The first 3x3 convolution covers a 3x3 region of the input.
2. The second 3x3 convolution covers a 3x3 region of the first layer's output.
   • Each pixel in this 3x3 region corresponds to a 3x3 region of the input.
   • The total region covered by the second convolution is the union of these overlapping 3x3 regions.

The formula for the effective receptive field (RFRF) after two stacked 3x3 convolutions is:

$$ RF=RF_1+(k−1)×s $$

where:

• $RF_1$ is the receptive field of the first layer (3x3).
• k is the kernel size of the second layer (3x3).
• s is the stride of the second layer (1 in this case).