Mastering the Classics: Training, Validation, and Test Datasets

Welcome to today’s data science interview challenge! Here it goes:

Let’s say if you are building a model for a Kaggle competition, and you tried a lot of experiments and ended up with 180 models.

You checked your validation accuracy for each one of these models. You found one model performed exceptionally well.

However, when you checked the accuracy in your test dataset that you locked away during the model development process, you found that the performance was terrible.

So what do you do now?

Source: Hold-out Method for Training Machine Learning Models

Here are some tips for readers’ reference:

Unfortunately you’ll have to go back to square one. There really isn’t any choice other than starting again. Your stellar results shown on the validation set could potentially be due to pure chance, just a coincidence because you probably overfit using the validation set. Kaggle has two test sets, one for giving you feedback on the leaderboard during the competition and a second one to show you…