Skip to main content

Inside a CNN: Testing Similar and Dissimilar Images

· 6 min read
Ross Bulat
Ross Bulat
Full Stack Engineer

I used CNN Explainer to follow two images through a small convolutional neural network: an espresso, which matches one of the model's trained classes, and a sunflower, which does not. The contrast made the main limitation of image classifiers very clear: a model can only choose from the labels and examples represented in its dataset.

The Activity​

CNN Explainer visualises a pretrained Tiny VGG network in the browser. It accepts a 64 × 64 RGB image and shows the activation maps produced by convolution, ReLU and max-pooling layers before flattening the learned features and applying softmax across ten output classes.

The complete CNN Explainer network with the built-in espresso example

The full network. Each column is a layer and each small square is an activation map.

I first explored the supplied examples to understand the interface. I then used two public-domain photographs from Wikimedia Commons:

  • Espresso 01.jpg, photographed by Calvin Ballantine and released into the public domain.
  • Sunflower.JPG, photographed by Adam Monkhouse and released into the public domain.

The espresso is a similar image because espresso is one of the model's output classes. The sunflower is deliberately dissimilar: there is no sunflower, flower or general plant class among the ten possible answers.

What Changed Across the Layers?​

The early convolutional maps still resembled the input. On the espresso image, different filters emphasised the rim of the cup, the saucer, shadows and changes in brightness. These are local features rather than a complete concept of “coffee”.

The ReLU layer applied max(0, x) element by element. Positive responses remained visible, while negative responses were replaced with zero. In the selected example below, an input activation of -0.2 therefore produced an output of 0. This sparsity helps the network retain useful responses and introduces the non-linearity needed to learn more complex patterns.

CNN Explainer's ReLU formula view for the uploaded espresso

An early ReLU activation: negative values are suppressed while positive filter responses are retained.

As the image travelled deeper into the network, the activation maps became smaller and less recognisable. Max-pooling reduced their spatial size by keeping the strongest response in each local region. Later convolutional layers combined simple edges and colour changes into more abstract arrangements. The final pooling layer produced ten 13 × 13 maps, which were flattened into a vector of 1,690 values for classification.

Similar Image: Espresso​

The uploaded espresso produced a strong and coherent path through the network. Several early maps clearly highlighted the circular cup and saucer, and later maps responded to more selective combinations of these features.

Full CNN activation maps for the uploaded public-domain espresso

The uploaded espresso moving through convolution, ReLU and pooling layers.

The final softmax output assigned 93.49% to espresso. The next-highest score was only about 3.58% for ladybug. This is a successful result: the object belongs to a trained class and shares useful visual characteristics with the training examples.

Softmax view showing a 0.9349 espresso score

Flattened features feed the softmax output, which scores the uploaded image as espresso at 0.9349.

Dissimilar Image: Sunflower​

The sunflower created strong circular and radial activation patterns, especially around the flower head. Those patterns survived into deeper layers even though the network had never been given a sunflower label.

Full CNN activation maps for the uploaded public-domain sunflower

The sunflower also produces meaningful activations, but the correct semantic class is unavailable.

Softmax still had to distribute all probability across the ten known classes. It returned 72.82% for bell pepper, approximately 14.36% for orange, and 12.74% for ladybug. The radial shape, warm colour and central region may resemble features the network learned for those classes, but bell pepper is not a meaningful description of the image.

Softmax view showing a 0.7282 bell pepper score for the sunflower

The dissimilar image is confidently forced into the nearest available class: bell pepper at 0.7282.

Dataset Applicability and Challenges​

This experiment demonstrates why dataset design matters as much as model architecture. A compact ten-class dataset makes the network fast enough to inspect interactively and is useful for learning how layers transform an image. It is not representative of open-world image recognition, where an input may contain any object—or several objects at once.

The espresso result shows that CNNs can generalise beyond an exact training photograph when the new image is similar to the training distribution. The sunflower result shows the opposite problem: softmax confidence is conditional on the available labels. It answers “which of these ten classes is most likely?”, not “is this actually one of these ten classes?”. A high score can therefore be confidently wrong when the dataset omits the true class.

Other challenges include limited variation in viewpoint, scale, background and lighting; class imbalance; low-resolution inputs; and hidden bias in how training images were collected. A stronger evaluation would use many images per condition, include out-of-distribution examples, report top-k predictions and calibration, and allow the system to reject an image as unknown when no class is suitable.

Reflection​

Looking inside the intermediate layers made the classification process less mysterious. Early layers detected local visual patterns, ReLU removed negative responses, pooling compressed the representation, and later layers assembled increasingly abstract evidence. However, none of those operations can recover a label that is missing from the training task. The model's usefulness is therefore bounded by both the quality of its learned features and the scope of its dataset.

References​