Assignment introduction

The assignment builds on the Data preprocessing exercise using the data in the data folder as well as your own. You will start with the data of test_subject_0 . You will then try out the model using the data from the data collection session.

The assignment is structured in the following way:

• Train gaze estimation models using data from test_subject_0 training set.
  • Test the models using test_subject_0 test set.
  • Test the models using test_subject_3 .
• Train gaze estimation models using data from test_subject_3 , grid .
  • Test the models using test_subject_3 .
• Train gaze estimtion models using your own data set.

Gaze mapping function

The goal of this exercise is to estimate gaze from image sequences using a regression model. Define $f_{\mathbf{w}}(x, y)$ as the gaze model which maps pupil positions $(x, y)$ to screen coordinates $(x', y')$. The model parameters $\mathbf{w}$ are learned from a training set containing paired pupil and screen positions.

Gaze estimation test subject 0

The first step is to train a gaze estimation model using the dataset of test_person_0 . You will:

• Train the models using the data pupils_n_training and screen_coordinates_training .
• Test the performance of the model using the pupils_n_testing and screen_coordinates_testing .

Data and visualization

import os
import numpy as np
import matplotlib.pyplot as plt
import iml_util

D = iml_util.gen_data_subject_0()
for i in range(4):
    dataset = D[i]
    grid = iml_util.create_image_grid_viz(dataset)
    iml_util.image_and_scatter(grid, dataset)

Implement a gaze estimation model

The true nature of the function that transforms pupil positions to screen positions is not easy to model. Even when the head is stationary, the relationship between pupil position in the image and gaze is non-linear due to the pupil's movement along a spherical curve. For now, we'll approximate the gaze mapping by using a linear function.

Since the goal is to predict a $2D$ screen coordinate from a $2D$ pupil coordinate, it's more straightforward to train two separate models for each coordinate by following the pattern of previous exercises. The linear gaze model can be expressed as:struct

$$ \begin{aligned} x' &= ax + by + c\\ y' &= dx + ey + f. \end{aligned} $$

Gaze mappings is demonstrated in Figure 3 to the right. Here, the $x$ coordinate of the pupil maps to the $x$ coordinate on the screen. In the real model, we use both $x$ and $y$ as inputs to both models estimating the $x$ position and $y$ position on the screen.

Hint

Use the previous exercise as a guideline.

In the upcoming tasks, you will be guided through the process of identifying essential entities necessary for establishing the linear equations required to learn the model parameters of the given model. The gaze model is divided into two distinct regression models, denoted as $x' = f_{w_1}(x, y)$ and $y' = f_{w_2}(x, y)$, each associated with its unique set of model parameters $w_1$ and $w_2$. Essentially, one model predicts the value of $x'$, and the other model predicts the value of $y'$. Follow the steps outlined below for the model $x' = f_{w_1}(x, y) = ax + by + c$, and subsequently, replicate the same steps for $y'$.

1. Identify model parameters and inputs to the model. We recommended that you use Least Squares from the numpy library linalg.lstsq , it is recommended due to numerical stability.
2. Isolate the unknown model parameters and the known into the design matrix.
3. Setup the linear equations $Aw=b$, where $A$ is the design matrix, $w$ the unknown model parameters and $b$ contains labels.

#Write your reflection here...

def get_design_matrix(pupil_positions):
    """
    Constructs a design matrix.
    Parameters:
    pupil_positions (N x 2 numpy array): n samples, 2 features (px, py).
    
    Returns:
    design_matrix (N x 3 numpy array) : n_samples, 2 features + 1
    """
    # Write your implementation

def linear(param, x):
    return param[0] + param[1]*x[0] + param[2]*x[1]

class GazeModel:
    def calibrate(self, dataset):
        ...

    def predict(self, pupil):
        ...

    def predict_many(self, pupils):
        return np.array([self.predict(pupil) for pupil in pupils])


class LinearGaze(GazeModel):
    def calibrate(self, dataset, pupil_training_set, screen_training_set):
        """
        Calibrates the LinearGaze model by fitting two linear regression models for x and y coordinates based 
        on the training dataset.

        Parameters:
            dataset (dict): The dictionary D containing the training data with two keys:
                - "pupils_train" (N x 2 numpy array): n samples, 2 features (px, py) representing pupil 
                  positions.
                - "positions_train" (N x 2 numpy array): n samples, 2 features (screen coordinate x, screen coordinates y) representing
                  the true gaze positions being the screen coordinates.

        Returns:
            None: Updates the model's parameters (self.model_x, self.model_y) based on 
            the least squares solution for x and y positions.
        """
        # Write your implementation


    def predict(self, pupil):
        """
        Predicts the gaze position (x, y) based on the given pupil position using the calibrated model.

        Parameters:
            pupil (1 x 2 numpy array): 2 features (px, py) representing the current pupil position.

        Returns:
            x, y (float, float): Predicted gaze position (x, y) based on the linear model.
        """
        # Write your implementation

def results_for_model(model, dataset_train, dataset_test, pupil_training_set, screen_training_set, pupil_test_set, screen_test_set, d_max):
    """
    Evaluates the performance of the model across multiple datasets and computes prediction errors.

    Parameters:
        model (object): A gaze model object.
        d_max (int, optional): Maximum number of patterns of a dataset to evaluate. Default is 4.

    Returns:
        results (dict): A dictionary where each key corresponds to a dataset index (0 to d_max-1), and each value is a dictionary containing:
            - "predicted" (1 x N numpy array): The predicted values
            - "ground_truth" (1 x N numpy array): The corresponding ground truth values to the predicted
            - "model_x" (list): List of model paramters for model x 
            - "model_y" (list): List of model paramters for model y 
            - "errors" (N x 2 numpy array): Absolute errors between predicted and ground truth positions (x, y).
            - "rmse" (float): Root Mean Squared Error (RMSE) for the predictions across the dataset.
            - "dist" (float): Mean Euclidean distance between the predicted and actual positions.
            - "mae" (1 x 2 numpy array): Mean absolute error in x and y dimensions.
    """


    results = {}
    for d in range(d_max):
        train = dataset_train[d]
        test = dataset_test[d]
        model_x, model_y = model.calibrate(train, pupil_training_set, screen_training_set)
        predicted = model.predict_many(test[pupil_test_set])
        ground_truth = test[screen_test_set]
    # Write your solution here
    return results
return results
    

 
results_linear = results_for_model(LinearGaze(), D, D,  'pupils_n_train', 'screen_coordinates_train', 'pupils_n_test', 'screen_coordinates_test', 4)

Below you find a cell containing a function for visualizing the performance of the model.

iml_util.plot_results_grid(results_linear, 4)

#Write your reflection here...

You have now went through the first part of the assignment processing the session of test_subject_0 . The remaining part of the assignment use the same approach as you have just gone through with the purpose of you working with and reflecting on the use of your own data collected.

Test on test subject 3

In the data processing exercise Data preprocessing the pupil corrdinates mean_pupil_coordinates.csv and the corresponding screen coordinates screen_coordinates.csv were saved in the in the folder with the test subjects and patterns. Recall the four calibration patterns used for the data collection being grid, circle, line and random.

In this exercise you will use the calibration (training data) of test_subject_0 on the data from test_subject_3 and visualized in Figure 4.

D3 = iml_util.gen_data_subject('test_subject_3')

results_linear_ts3 = results_for_model(LinearGaze(), D, D3, 'pupils_n_train', 'screen_coordinates_train', 'pupils_test', 'screen_coordinates_test', 4)

iml_util.plot_results_grid(results_linear_ts3, 4)

#Write your reflections here...

Gaze estimation test subject 3

In this the gaze estimation model is trained using the grid pattern of test_subject_3 , and test it on the patterns.

# Write your code here

#Write your reflections here...

You may by now you have observed that the model is struggling with properly predicting the y coordinates. The cell below visualizes the model predictions separating the x and y coordinate.

for i in range(len(results_linear_ts3)):
    iml_util.plot_from_results_dict(results_linear_ts3[i], D3[i])

To further elaborate on the prediction of the y coordinate, the cell below visualizes the Sum of Squared Error (SSE) loss function, in parameter space. The cell below creates two plots, left is the loss when changing parameters a and b while keeping c fixed. The right plot shows loss when varying c while keeping a and b fixed.

# Example usage with sample data
x = D3[0]['pupils_train'][:,0]
y = D3[0]['pupils_train'][:,1]
z = D3[0]['screen_coordinates_train'][:,1]  # Actual targets

# Set fixed values for a, b, and c (based on an example linear model)
a_fixed = results_linear_ts3[0]['model_y'][2]
b_fixed = results_linear_ts3[0]['model_y'][1]
c_fixed = results_linear_ts3[0]['model_y'][0]

# Plot the error surfaces
iml_util.plot_error_surfaces(a_fixed, b_fixed, c_fixed, x, y, z)

#Write your reflections here...

#Write your reflections here...

Until now the models have been trained using the nine calibration points from the grid pattern. You will now increase the amount of training data by incoorporating all patterns except circle . The pattern circle will be used for testing. In the cell below a dictionary D3_mod contains the training and test data similar to the structure of D and D3 . .

D3_mod = {}

ts3_p_train = D3[0]['pupils_test']
ts3_s_train = D3[0]['screen_coordinates_test'] 

ts3_p_test = D3[1]['pupils_test']
ts3_s_test = D3[1]['screen_coordinates_test']

for i in range(2,4):
    ts3_p_train = np.vstack((ts3_p_train,D3[i]['pupils_test']))
    ts3_s_train = np.vstack((ts3_s_train,D3[i]['screen_coordinates_test']))
    
D3_mod['pupils_train'] = ts3_p_train
D3_mod['screen_coordinates_train'] = ts3_s_train
D3_mod['pupils_test'] = ts3_p_test
D3_mod['screen_coordinates_test'] = ts3_s_test

D3_mod = [D3_mod]

# Write your code here

#Write your reflections here...

Individual dataset

The pipeline of collecting data, training a model, testing, and evaluating it has been introduced. In this task, the data of your eyes will be used to train a model. Use the evaluation tools introduced in the assignment.

#Note the values here
# Threshold = 
# x_margin = 
# y_margin = 
# side =

# update the parameter and uncomment the line below
#iml_util.gen_data_subject('group-xx-xxxx')

#Use this cell to train and evaluate your model...

# Write your reflections here...

Optional improvements

You may experience that the models behave differently on your dataset than on test_subject_0 and test_subject_3 . This can be due to several factors, some of which you may have already reflected on in Task 8 in exercise Data preprocessing . If you experience interesting behaviour in the predictions of your model you are of course welcome to explore it further such as:

• Different combinations of training set (use can use the creation of D3_mod for inspiration as to how to manipulate the combination of data for training and testing).
  • If you have multiple data sets in your group, you can incoorporate those or use the ones provided, test_subject_3 or test_subject_1 .
• Other evaluation metrics
• Take inspiration from the optional exercises below, to explore your dataset further.

Scale the data

In this task, you will improve the model using normalization. In Task 13, you reflected on the effect of having a large scale difference between input and output. Use the data of test_subject_3 in dictionary D3 .

from sklearn.preprocessing import MinMaxScaler
def normalize(data_set):
    """
    Normalizes the pupil and screen coordinate data in the dataset using MinMaxScaler.
    Parameters:
        data_set (list of dict): A list of dictionaries where each dictionary contains training and testing data 
                                 for pupils and screen coordinates. Keys include:
                                 - 'pupils_train': N x 2 array of pupil coordinates for training.
                                 - 'screen_coordinates_train': N x 2 array of screen coordinates for training.
                                 - 'pupils_test': N x 2 array of pupil coordinates for testing.
                                 - 'screen_coordinates_test': N x 2 array of screen coordinates for testing. 
    Returns:
        d (list of dict): A list of dictionaries with normalized pupil and screen coordinate data.
                          Keys include:
                          - 'pupils_train': Normalized N x 2 array of pupil coordinates for training.
                          - 'screen_coordinates_train': Normalized N x 2 array of screen coordinates for training.
                          - 'pupils_test': Normalized N x 2 array of pupil coordinates for testing.
                          - 'screen_coordinates_test': Normalized N x 2 array of screen coordinates for testing.
        scaler (MinMaxScaler object): The fitted MinMaxScaler object used for normalization.
    """
    scaler = MinMaxScaler()
    # Write your code here

def denormalize(dataset, scaler):
    """
    Denormalizes predicted and ground truth coordinates and calculates errors and performance metrics.
    Parameters:
        dataset (list of dict): A list of dictionaries where each dictionary contains prediction results.
                                Keys include:
                                - 'predicted': N x 2 array of normalized predicted coordinates.
                                - 'ground_truth': N x 2 array of normalized ground truth coordinates.
        scaler (MinMaxScaler object): The MinMaxScaler object used for normalization, applied here to inverse-transform data.
    Returns:
        d (dict): A dictionary where each key corresponds to a dataset index, and each value contains the following:
                  - 'predicted': Denormalized N x 2 array of predicted coordinates.
                  - 'ground_truth': Denormalized N x 2 array of ground truth coordinates.
                  - 'errors': N x 2 array of absolute errors between predicted and ground truth coordinates.
                  - 'mse' (1 x 2 numpy array): Mean squared error for the x and y coordinates.
                  - 'rmse' (float): Root mean squared error (RMSE) across the dataset.
                  - 'dist' (float): Mean Euclidean distance between predicted and ground truth positions.
                  - 'mae' (1 x 2 numpy array): Mean absolute error for the x and y coordinates.
    """
    # Write your code here


# Write your code here for training, testing and visualizing results

#Write your reflections here...

Improve the model

This final part of the exercise requires you to modify the gaze model $f_{\mathbf{w}}(x,y)$ into a quadratic model. As in Task 5, you will create a model for each output coordinate, i.e. $x', y'$.

Since the model is two-dimensional, the quadratic polynomial has more model parameters than for one dimension model. The equation for each axis is:

The design matrices then have the following form:

def get_design_matrix_quad(pupil_positions):
    # Write your implementation

def quad(param, x):
    return param[0] + param[1]*x[0] + param[2]*x[1] + param[3]*x[0]*x[1] + param[4]*x[0]**2 + param[5]*x[1]**2

class QuadGaze(GazeModel):
    def calibrate(self, dataset, pupil_training_set, screen_training_set):
        """
        Calibrates the LinearGaze model by fitting two linear regression models for x and y coordinates based 
        on the training dataset.

        Parameters:
            dataset (dict): The dictionary D containing the training data with two keys:
                - "pupils_train" (N x 2 numpy array): n samples, 2 features (px, py) representing pupil 
                  positions.
                - "positions_train" (N x 2 numpy array): n samples, 2 features (screen coordinate x, screen coordinates y) representing
                  the true gaze positions being the screen coordinates.

        Returns:
            None: Updates the model's parameters (self.model_x, self.model_y) based on 
            the least squares solution for x and y positions.
        """
        # Write your implementation


    def predict(self, pupil):
        # Write your implementation


results_quad = results_for_model(QuadGaze(), D, D, 'pupils_n_train', 'screen_coordinates_train', 'pupils_n_test', 'screen_coordinates_test', 4)




# Write your implementation here