How well can DenseNet classify difficult CIFAR-10 classes such as cat and dog images?
I am writing a deep learning algorithm that operates with two classes. For this purpose, I chose to modify DenseNet and test its performance by choosing the first two classes of CIFAR-10, i.e.
[plane, car]. Well, my neural net did pretty well and I was happy until I tried it on real data. It took me a while to realize that it is really easy to classify planes and cars since most plane images are dominated by a blue background. That’s when I moved to
[cat, dog] classes. I was consistently getting low classification accuracies. But I wasn’t sure if the issue was with my implementation or cats and dogs are really difficult to classify. That’s when I became curious to know how much more difficult it is to classify cats and dogs from other images using the state of the art neural nets (at the time of this post). I wrote a simple Python/PyTorch code for separating CIFAR classes which you can find as GitHub gist (also displayed at the end of this post).
I wrote a simple PyTorch code to separate CIFAR classes which you can find as a GitHub gist
(also displayed at the end of this post). Then, I trained this DenseNet implementation from GitHub
on all 10 classes of CIFAR and obtained an accuracy of %95.0, which is very close to what they get on the DenseNet paper
. After ensuring that everything is working as it should, I trained the same network on cats and dogs and I was able to get an accuracy of 90.0%.
So, in general, we can roughly
[cat, dog] accuracy
is 0.95 of the average CIFAR-10 accuracy
. Since the highest accuracy on CIFAR-10 (that I know of) is 96.54% (see figure below from the same paper), an accuracy higher than %91.5 on difficult subclasses (
) should not be expected. (For reference purposes, my lowest (cross-entropy) loss for [cat, dog] was 0.394.)
Since I couldn’t find the confusion matrix of DenseNet and CIFAR-10 anywhere, I thought other people might find this post useful.