KGA is a method for unlearning in machine learning models, particularly large language models (LLMs). The goal is to make the model forget specific data (the forget dataset $D_f$) while retaining its performance on the remaining data (the retain dataset $D_r$).

Key Idea

• The unlearned model $A^*$ should behave differently on the forget data $D_f$ (as if it never saw it) and similarly on the retain data $D_r$
• To achieve this, KGA aligns the knowledge gap—the difference in predictions between models trained on different datasets.

Inputs to KGA:

The inputs are divided into two parts : Data and the Models

Data:

1. Previous Data: The original dataset that is used for training the model.
2. Forget Data: The subset of the original dataset that needs to be forgotten.
3. Extra/External Data: A small dataset that is not part of the original dataset.

Models:

1. Original Model: The model trained on the entire original dataset.
2. Unlearned Model: The model that we want to get out as the output which forgets $D_f$ but retains all of the information about the retain dataset $D_r$.

What do we want to achieve? Let us say we have 2 models - a parent model and a child model. The parent model is trained on the original dataset D while the child model needs to forget some part of the dataset while learning everything else.

The child should act as if it has never seen the data while behaving like the parent when it comes to the retain data. The KGA method ensures that the predictions made by the child on the forget data are similar to the predictions made on the extra data (data it has not seen).

While the predictions made on the retain data by the child should be same as that of the parent’s predictions.

The MATHSSSSSSSSSSSSSSSSSSS: