What is “in-context learning”?

Welcome to today’s data science interview challenge! Today’s challenge is inspired by Natural Language Understanding (NLU) lecture given by Professor and Chair, Christopher Potts from Department of Linguistics at Stanford. Here it goes:

Question 1: What is “in-context learning” in the context of Large Language Models (LLMs)?

Question 2: Can you contrast standard supervision learning vs few-shot in-context learning?

Source: A Survey on In-context Learning (2023)

Here are some tips for readers’ reference:

Question 1:

In-context learning (ICL) is a paradigm that allows large language models (LLMs) to learn tasks given only a few examples in the form of demonstration. This is in contrast to traditional machine learning, where models are trained on large datasets of labeled data.

The key idea behind ICL is that LLMs can learn to generalize from a few examples by identifying the underlying relationships between the examples. This is done by using the model’s ability to understand the context of the examples.

For example, an LLM could be given the following examples to learn the task of summarizing news articles:

• “The president gave a speech about the economy.”
• “The stock market went up after the speech.”
• “The president’s approval ratings went up after the speech.

The LLM could then use the context of these examples to learn that the president’s speech caused the stock market to go up and his approval ratings to go up. This knowledge could then be used to summarize new articles about the president’s speeches.

Question 2:

The key idea of in-context learning is to learn from analogy.

• Standard supervised learning is a traditional machine learning paradigm where the model is trained on a large dataset of labeled data. The labels provide the model with information about the correct output for each input. The model then learns to map inputs to outputs by minimizing a loss function.