Training a model is an iterative process: at each iteration, the model estimates the output, computes the error in its estimate (loss<\/em>), collects the derivatives of the error with respect to its parameters, and optimizes<\/strong> these parameters<\/strong> using gradient descent<\/strong>.<\/p>\n\n Gradient descent is an algorithm<\/strong> used to find<\/strong> the minimum<\/strong> of a function<\/strong>.<\/p>\n\n Intuitive approach:<\/strong><\/p>\n\n Mathematical approach:<\/strong><\/p>\n\n Combined approaches:<\/strong><\/p>\n\n To do so, we start by taking an initial x (x0<\/strong>) at random. x0 = -1<\/em> -> f(x0) = 6.08)<\/em><\/p>\n\n We then compute the slope value at this point (the derivative f'(x0)<\/strong>). f'(-1) = -2sin(-1)-4cos(-1)-5 \u2248 -5.47827<\/em><\/p>\n\n We “move” in the direction opposite to the slope: x1 = x0 \u2212 \u03b1f'(x0)<\/strong> (with \u03b1 = learning_rate = 0.05). x1 \u2248 -0.72609<\/em><\/p>\n\n We then repeat the operation until we find xmin<\/strong>. xmin = 3.8<\/em><\/p>\n\n This optimization process can be directly influenced by the developer through hyperparameters<\/strong>. These hyperparameters are adjustable parameters that impact model training and its convergence rate. The convergence rate corresponds to how many iterations the model needs to obtain an optimal result. <\/p>\n\n To train the model, we have 3 hyperparameters:<\/p>\n\n More concretely, we initially have 2 things:<\/p>\n\n We start by splitting the dataset<\/em> into subsets<\/em>, all of the same size. These subsets<\/em> are called batches<\/strong>, each of size batch_size<\/strong>. <\/p>\n\n Next, each batch<\/strong> is propagated through the network (passed as input to it). When all batches<\/strong> have been passed through the model, we have completed 1 epoch<\/strong>. The process is then repeated epochs_number<\/strong> times. <\/p>\n\n These steps can be visualized in the following image.<\/p>\n\n First of all, the hyperparameters must be defined. For example: <\/p>\n\n Once the hyperparameters have been defined, the model training and optimization loop can begin. As a reminder, each iteration of this loop is therefore an epoch<\/strong>. <\/p>\n\n More precisely, each epoch<\/strong> consists of 2 phases:<\/p>\n\n In the first loop, since our network has not yet been trained, it is very unlikely to produce a good answer\/result. We then need to measure the distance<\/strong> between the obtained result<\/strong> and the expected result<\/strong>. This distance is computed using the loss function (loss_function<\/strong>). <\/p>\n\n The goal is therefore, as the loop iterations progress, to minimize<\/strong> this loss_function<\/strong> in order to obtain a result as close as possible to the expected result. We note here the usefulness of gradient descent<\/strong> which, as explained earlier, is an algorithm used to find the minimum of a function\u2014here, the loss_function<\/strong>. <\/p>\n\n Optimization therefore consists in updating the parameters at each loop to minimize this loss function. This optimization is encapsulated in an optimizer<\/strong> object that is applied to the model. The latter therefore takes the model hyperparameters as input. In the training loop, optimization is more specifically carried out in 3 steps: <\/p>\n\nGradient descent (reminder)<\/h2>\n\n
\n
![\"\"](\"https:\/\/neogeo.fr\/wp-content\/uploads\/2024\/10\/DG_1.png\")
<\/figure>\n\n\n
![\"\"](\"https:\/\/neogeo.fr\/wp-content\/uploads\/2024\/10\/DG_2.png\")
<\/figure>\n\n\n
![\"\"](\"https:\/\/neogeo.fr\/wp-content\/uploads\/2024\/10\/DG_3.png\")
<\/figure>\n\n\n
![\"\"](\"https:\/\/neogeo.fr\/wp-content\/uploads\/2024\/10\/DG_4.png\")
<\/figure>\n\n![\"\"](\"https:\/\/neogeo.fr\/wp-content\/uploads\/2024\/10\/DG_5.png\")
<\/figure>\n\n![\"\"](\"https:\/\/neogeo.fr\/wp-content\/uploads\/2024\/10\/DG_6.png\")
<\/figure>\n\nHyperparameters<\/h2>\n\n
\n
Explanation of hyperparameters<\/h3>\n\n
\n
![\"\"](\"https:\/\/neogeo.fr\/wp-content\/uploads\/2024\/10\/hyperparameters_sketch.png\")
<\/figure>\n\nModel training loop<\/h2>\n\n
\n
\n