Introduction
Convolutional Neural Networks (CNNs) have been the default architecture for computer vision for quite some time. More recently, Vision Transformers (ViTs) have become a strong alternative, borrowing the Transformer architecture originally developed for natural language processing.
At a high level, both approaches learn representations from images and use those representations to solve tasks such as image classification. The interesting part is how they process the image and, more specifically, what assumptions they make about the structure of visual data.
I will cover the main architectural differences between CNNs and ViTs, look at some published benchmark results and then connect these findings to Richard Sutton’s The Bitter Lesson.
Two different approaches
Convolutional Neural Networks
A CNN processes an image using small filters (also called kernels) that slide across the image. These filters learn to detect local patterns such as edges, corners and textures. The same filters are reused at every location, which is known as parameter sharing.
Multiple convolutional layers are typically stacked together. Early layers detect relatively simple patterns, while later layers combine those patterns into more complex structures. For example, edges can form contours, contours can form object parts, and object parts can contribute to recognising an object.
This architecture comes with a few useful assumptions, or inductive biases:
- Locality: nearby pixels are likely to be related.
- Translation equivariance: the same visual pattern can appear in different locations, so the same detector can be reused across the image.
- Hierarchical composition: complex patterns can be built from simpler ones.
These assumptions are particularly useful when the amount of training data is limited, because the model does not need to learn all of this structure from scratch.


Vision Transformers
A Vision Transformer takes a different route. It divides an image into a grid of fixed-size patches, converts each patch into a vector (a token) and feeds the resulting sequence into a Transformer.
Since a Transformer does not inherently know where a token came from, positional information is added to the patch embeddings. The tokens are then processed by repeated Transformer blocks containing self-attention and feed-forward layers.
The key mechanism is self-attention. Each patch can use information from other patches when updating its representation. In other words, a patch representing an object’s face can directly interact with a patch representing its tail, even if they are far apart in the image.
This is different from a standard CNN, where information initially flows through local neighbourhoods and larger receptive fields emerge as layers are stacked.

The trade-off is that ViTs have fewer built-in assumptions about images. They can learn relationships between distant regions directly, but they also need to learn more of the underlying visual structure from data. In the original ViT setup, self-attention also has a computational cost that grows quadratically with the number of patches.
A useful way to summarise the difference is that CNNs bake some assumptions about images into the architecture, whereas ViTs give the model more flexibility to learn those relationships from examples.
What do the benchmarks tell us?
The original Vision Transformer paper, An Image is Worth 16×16 Words (Dosovitskiy et al., 2021), provides a particularly interesting experiment. The authors compared ViTs and CNNs while increasing the size of the pretraining dataset.
The broad result was:
- With ImageNet-1K (roughly 1.3 million images), the original ViT struggled to match strong CNNs.
- With ImageNet-21K (roughly 14 million images), the gap narrowed considerably.
- With JFT-300M (roughly 300 million images), ViTs matched or outperformed the comparable CNN baseline in their experiments.
For example, after pretraining on JFT-300M, ViT-L/16 achieved 87.76% top-1 accuracy on ImageNet, compared with 87.54% for BiT-L, a ResNet-based CNN. A larger ViT-H/14 reached 88.55%.
This is the most direct evidence for the effect of training-data scale, although the model sizes and training setups are not identical in every comparison.

There are a couple of important qualifications.
First, the original ViT was not the end of the story. The DeiT paper (Touvron et al., 2021) showed that better training techniques, augmentation and knowledge distillation could make Transformers much more data-efficient. Its DeiT-B model reached 81.8% top-1 accuracy on ImageNet-1K, while a distilled version reached 85.2%.
Second, CNNs did not simply become obsolete. The ConvNeXt paper (Liu et al., 2022) revisited CNN design using several ideas popularised by Transformers, while keeping the architecture convolutional. On ImageNet-1K, ConvNeXt-T reached 82.1%, compared with 81.3% for the similarly sized Swin-T Transformer. At larger model sizes, ConvNeXt continued to be competitive.
The numbers above should not be treated as one perfectly controlled leaderboard: the papers use different training recipes, data augmentation, model sizes and sometimes pretraining datasets. The most useful comparisons are those made within the same paper, under the same evaluation setup.
Overall, the evidence suggests that ViTs can benefit substantially from scale, but it does not show that Transformers are inherently better than CNNs in every setting.
How does this relate to the Bitter Lesson?
In his essay The Bitter Lesson, Richard Sutton argues that approaches which leverage computation and general methods for learning tend to improve more reliably as computational resources increase than approaches that rely heavily on handcrafted, domain-specific knowledge.
The evolution from CNNs to ViTs is an interesting example of this idea, although it is not a perfectly clean demonstration.
CNNs already represent a major step away from hand-engineered computer vision. Instead of manually defining features such as edges, corners or textures, we provide a general learning architecture and let the model discover useful filters from data. However, CNNs still encode some assumptions about visual data directly into their architecture, such as locality and translation equivariance.
ViTs take this a step further. They start with image patches and use a relatively general mechanism, self-attention, to learn relationships between them. They do not hard-code the same local processing strategy as standard CNNs. The original ViT experiments showed that, when provided with enough data and compute, this less specialised architecture could learn visual representations that matched or exceeded strong CNNs.
That said, the ConvNeXt results are a useful reminder not to overinterpret this. A well-designed CNN can also scale very effectively, and architectural inductive biases are not necessarily a disadvantage. They can make learning more efficient and improve performance when data or compute is constrained.
I think the more nuanced takeaway is that we should be careful about assuming that the structure we impose based on our understanding of a problem is necessarily the best structure for a learning system. Useful inductive biases can help, especially at smaller scales, but they should be justified by their empirical value rather than by intuition alone.
As data and compute increase, it becomes increasingly important to test whether a more general learning mechanism can discover representations that are at least as good, without relying on as many assumptions designed by us.
References
- Dosovitskiy et al., An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale, ICLR 2021.
- Touvron et al., Training Data-Efficient Image Transformers & Distillation Through Attention, ICML 2021.
- Liu et al., A ConvNet for the 2020s, CVPR 2022.
- Sutton, The Bitter Lesson, 2019.
- Google Research, Transformers for Image Recognition at Scale, 2020.