PyTorch_Tutorial_Optimizing Model parameters 파이토치 공식사이트

Optimizing Model Parameters 모델 매개변수 최적화하기

Now that we have a model and data it's time to train, validate and test our model by optimizing its parameters on our data.

Training a model is an iterative process; in each iteration the model makes a guess about the output, calculates the error in its guess (loss), collcects the. derivatives of the error with respect to its parameters , and optimizes these parameters using gradient descent.

각 반복 단계에서 모델은 출력을 추측하고 추측과 정답 사이의 오류(손실(loss))를 계산하고, 매개변수에 대한 오류의 도함수(derivative)를 수집한 뒤, 경사하강법을 사용하여 최적화(optimize)한다.

 

Prerequisite Code

import torch
from torch import nn
from torch.utils.data import dataloader
from torchvision import datasets
from torchvision.transforms import ToTensor

training_data = datasets.FashionMNIST(
	root="data",
    train=True,
    download=True,
    transform=ToTensor()
)

test_data = datasets.FashionMNIST(
	root="data",
    train=False,
    download=True,
    transform=ToTensor()
)

train_dataloader = DataLoader(training_data, batch_size=64)
test_datalaoder = DataLoader(test_data, batch_size=64)

class NeuralNetwork(nn.Module):
	def __init__(self):
    	super(neuralNetwork, self).__init()__
        self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
        	nn.Linear(28*28, 512)
            nn.ReLU()
            nn.Linear(512, 512)
            nn.ReLU()
            nn.Linear(512, 10)
        )
	def forawrd(self, x):
    x=self.flatten(x)
    logits=self.linear_relu_stack(x)
    return logits
    
model = NeuralNetwork()

Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz to data/FashionMNIST/raw/train-images-idx3-ubyte.gz

  0%|          | 0/26421880 [00:00<?, ?it/s]
  0%|          | 32768/26421880 [00:00<01:25, 307111.05it/s]
  0%|          | 65536/26421880 [00:00<01:27, 302330.83it/s]
  0%|          | 131072/26421880 [00:00<01:00, 438040.20it/s]
  1%|          | 229376/26421880 [00:00<00:42, 619920.25it/s]
  2%|1         | 491520/26421880 [00:00<00:20, 1258867.01it/s]
  4%|3         | 950272/26421880 [00:00<00:11, 2255445.08it/s]
  7%|7         | 1933312/26421880 [00:00<00:05, 4453727.26it/s]
 15%|#4        | 3833856/26421880 [00:00<00:02, 8567539.61it/s]
 26%|##6       | 6946816/26421880 [00:00<00:01, 14772223.31it/s]
 38%|###8      | 10092544/26421880 [00:01<00:00, 19037585.45it/s]
 49%|####8     | 12943360/26421880 [00:01<00:00, 21109286.70it/s]
 61%|######    | 16056320/26421880 [00:01<00:00, 23314724.14it/s]
 73%|#######2  | 19169280/26421880 [00:01<00:00, 24842002.43it/s]
 83%|########2 | 21921792/26421880 [00:01<00:00, 24890329.80it/s]
 95%|#########4| 25001984/26421880 [00:01<00:00, 25857568.15it/s]
100%|##########| 26421880/26421880 [00:01<00:00, 16022803.07it/s]
Extracting data/FashionMNIST/raw/train-images-idx3-ubyte.gz to data/FashionMNIST/raw

Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz to data/FashionMNIST/raw/train-labels-idx1-ubyte.gz

  0%|          | 0/29515 [00:00<?, ?it/s]
100%|##########| 29515/29515 [00:00<00:00, 270546.15it/s]
100%|##########| 29515/29515 [00:00<00:00, 269336.54it/s]
Extracting data/FashionMNIST/raw/train-labels-idx1-ubyte.gz to data/FashionMNIST/raw

Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz to data/FashionMNIST/raw/t10k-images-idx3-ubyte.gz

  0%|          | 0/4422102 [00:00<?, ?it/s]
  1%|          | 32768/4422102 [00:00<00:14, 302756.98it/s]
  1%|1         | 65536/4422102 [00:00<00:14, 301904.93it/s]
  3%|2         | 131072/4422102 [00:00<00:09, 439237.31it/s]
  5%|5         | 229376/4422102 [00:00<00:06, 623323.56it/s]
 11%|#1        | 491520/4422102 [00:00<00:03, 1267826.21it/s]
 21%|##1       | 950272/4422102 [00:00<00:01, 2270605.41it/s]
 44%|####3     | 1933312/4422102 [00:00<00:00, 4481734.59it/s]
 87%|########6 | 3833856/4422102 [00:00<00:00, 8623883.58it/s]
100%|##########| 4422102/4422102 [00:00<00:00, 5073760.83it/s]
Extracting data/FashionMNIST/raw/t10k-images-idx3-ubyte.gz to data/FashionMNIST/raw

Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz to data/FashionMNIST/raw/t10k-labels-idx1-ubyte.gz

  0%|          | 0/5148 [00:00<?, ?it/s]
100%|##########| 5148/5148 [00:00<00:00, 25165824.00it/s]
Extracting data/FashionMNIST/raw/t10k-labels-idx1-ubyte.gz to data/FashionMNIST/raw

Hyperparameters

Hyperparameters are adjustable parameters that let you control the model optimization process.

Different hyperparameter values can impact model training and covergence rates.

하이퍼파라미터는 모델 최적화 과정을 제어할 수 있는 매개변수이다.

 

We define the following hyperparameters for training:

● Number of Epochs - the number times to iterate over the dataset

● Batch Size - the number of data samples propagated through the network before the parameters are updated

● Learning Rate - how much to update models parameters at each batch/epoch. Smaller values yield slow learning speed, while large values may result in unpredictable bahvior during training.

learning_rate = 1e-3
batch_size = 64
epochs = 5

Optimization Loop

Once we set our hyperparameters, we can then train and optimize our model with an optimization loop. Each iteration of the optimization loop is called an epoch.

 

Each epoch consists of two main parts:

● The Train Loop - iterate over the training dataset and try to coverage to optimal parameters.

● The Validation/Test Loop - iterate over the test dataset to check if model performance is imporving.

 

Let's beiefly familiarize ourselves with some of the concept used in the training loop.

 

Loss Function

When presented with some training data, our untrained network is likely not to give the correct answer. Loss function measures the degree of dissimilarity of obtained result to the target value, and it is the loss function that we want to minimize during training. To calculate the loss we make a prediction using the inputs of our given data sample and compare it against the true data label value.

 

Common loss functions include nn.MSELoss (Means Square Error) for regreesion tasks, and nn.NLLLoss(Negative Log Likelihood) for classfication. nn.CrossEntropyLoss combines nn.LogSoftmax and nn.NLLLoss.

 

We pass our model's output logits to nn.CrossEntropyLoss, which will normalize the logits and compute the prediction error.

# Initialize the loss function
loss_fn = nn.CrossEntropyLoss()

 

Optimizer

Optimization is the process of adjusting model parameters to reduce model error in each training step.

Optimization algorithms define how this process is performed(in this example we use Stochastic Gradient Descent).

All optimization logic is encapsulated in the optimizer object. Here, we use the SGD optimizer; additionally, there are many different optimizers available in PyTorch such as ADAM and RMSProp, that work better for different kinds of models and data.

 

We initialize the optimizer by registering the model's parameters that need to be trained, and passing in the learning rate hyperparameter.

optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

 

Inside the training loop, optimization happens in three steps:

● Call optimizer.zero_grad() to reset the gradients of model parameters. Gradients by default add up; to prevent double-counting, we explicitly zero them at each iteration.

● Backpropagate the prediction loss with a call to loss.backward(). PyTorch deposits the gradients of the loss w.r.t each parameter.

● Once we have our gradients, we call optimizer.step() to adjust the parameters by the gradients collcected in the backward pass.

optimizer.zero_grad()(모델 매개변수의 변화도를 재설정) -> loss.backwards()(예측 손실을 역전파) -> optimizer.step()(역전파 단계에서 수집된 변화도로 매개변수를 조정)

 

Full Implementation

We define train_loop that loops over our optimization code, and test_loop that evaluates the model's performance against our test data.

def train_loop(dataloader, model, loss_fn, optimizer):
	size = len(dataloader.dataset)
    for batch, (X, y) in enumerate(dataloader):
    # Compute prediction and loss
    pred = model(X)
    loss = loss_fn(pred, y)
    
    # Backpropagation
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    
    if batch%100 == 0:
    	loss, current = loss.item(), batch*len(X)
        print(f"loss: {loss:>7f} [{current:>5d}/{size:>5d}")
        
def test_loot(dataloader, model, loss_fn):
	size = len(dataloader.dataset)
    num_batches = len(dataloader)
    test_loss, correct = 0,0
    
    with torch.no_grad():
    	for X, y in dataloader:
        	pred = model(X)
            test_loss += loss_fn(pred, y).item()
            correct += (pred.argmax(1)==y.type(torch.float).sum().item()
            
	test_loss /= num_batches
    correct /= size
    print(f"Test Error: \n Accuracy: {(100*correct):>0.1f}%, Avg loss:{test_loss:>8f|\n")

We initialize the loss function and optimizer, and pass it to train_loop and test_loop..

Feel free to increase the number of epochs to track the model's improving performance.

loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

epochs = 10
for t in range(epochs):
	print(f"Epch {t+1}\n-------------------------")
    train_loop(train_dataloader, model, loss_fn, optimizer)
    test_loop(test_dataloader, model, loss_fn)
print("Done!")

Epoch 1
-------------------------------
loss: 2.313860  [    0/60000]
loss: 2.296742  [ 6400/60000]
loss: 2.280525  [12800/60000]
loss: 2.274902  [19200/60000]
loss: 2.247680  [25600/60000]
loss: 2.241043  [32000/60000]
loss: 2.242453  [38400/60000]
loss: 2.211962  [44800/60000]
loss: 2.205440  [51200/60000]
loss: 2.172555  [57600/60000]
Test Error:
 Accuracy: 37.8%, Avg loss: 2.166341

Epoch 2
-------------------------------
loss: 2.177963  [    0/60000]
loss: 2.170609  [ 6400/60000]
loss: 2.117870  [12800/60000]
loss: 2.135853  [19200/60000]
loss: 2.075072  [25600/60000]
loss: 2.033468  [32000/60000]
loss: 2.060745  [38400/60000]
loss: 1.979945  [44800/60000]
loss: 1.986767  [51200/60000]
loss: 1.923508  [57600/60000]
Test Error:
 Accuracy: 47.0%, Avg loss: 1.916104

Epoch 3
-------------------------------
loss: 1.942822  [    0/60000]
loss: 1.925890  [ 6400/60000]
loss: 1.812407  [12800/60000]
loss: 1.860552  [19200/60000]
loss: 1.733851  [25600/60000]
loss: 1.693230  [32000/60000]
loss: 1.719835  [38400/60000]
loss: 1.605775  [44800/60000]
loss: 1.636995  [51200/60000]
loss: 1.541491  [57600/60000]
Test Error:
 Accuracy: 58.5%, Avg loss: 1.548259

Epoch 4
-------------------------------
loss: 1.604988  [    0/60000]
loss: 1.580056  [ 6400/60000]
loss: 1.427251  [12800/60000]
loss: 1.511601  [19200/60000]
loss: 1.371354  [25600/60000]
loss: 1.374740  [32000/60000]
loss: 1.392799  [38400/60000]
loss: 1.300494  [44800/60000]
loss: 1.342699  [51200/60000]
loss: 1.251812  [57600/60000]
Test Error:
 Accuracy: 63.1%, Avg loss: 1.269188

Epoch 5
-------------------------------
loss: 1.340759  [    0/60000]
loss: 1.328273  [ 6400/60000]
loss: 1.161626  [12800/60000]
loss: 1.274336  [19200/60000]
loss: 1.137650  [25600/60000]
loss: 1.170882  [32000/60000]
loss: 1.192125  [38400/60000]
loss: 1.115200  [44800/60000]
loss: 1.158397  [51200/60000]
loss: 1.085218  [57600/60000]
Test Error:
 Accuracy: 64.6%, Avg loss: 1.097636

Epoch 6
-------------------------------
loss: 1.164975  [    0/60000]
loss: 1.172097  [ 6400/60000]
loss: 0.990208  [12800/60000]
loss: 1.126935  [19200/60000]
loss: 0.993334  [25600/60000]
loss: 1.033727  [32000/60000]
loss: 1.068909  [38400/60000]
loss: 0.998777  [44800/60000]
loss: 1.040032  [51200/60000]
loss: 0.982320  [57600/60000]
Test Error:
 Accuracy: 65.4%, Avg loss: 0.987760

Epoch 7
-------------------------------
loss: 1.042835  [    0/60000]
loss: 1.071654  [ 6400/60000]
loss: 0.873834  [12800/60000]
loss: 1.029063  [19200/60000]
loss: 0.901954  [25600/60000]
loss: 0.936855  [32000/60000]
loss: 0.988096  [38400/60000]
loss: 0.923568  [44800/60000]
loss: 0.959101  [51200/60000]
loss: 0.914047  [57600/60000]
Test Error:
 Accuracy: 66.6%, Avg loss: 0.913421

Epoch 8
-------------------------------
loss: 0.953351  [    0/60000]
loss: 1.002631  [ 6400/60000]
loss: 0.790970  [12800/60000]
loss: 0.960368  [19200/60000]
loss: 0.840599  [25600/60000]
loss: 0.866329  [32000/60000]
loss: 0.931050  [38400/60000]
loss: 0.873486  [44800/60000]
loss: 0.901637  [51200/60000]
loss: 0.865278  [57600/60000]
Test Error:
 Accuracy: 67.6%, Avg loss: 0.860329

Epoch 9
-------------------------------
loss: 0.884891  [    0/60000]
loss: 0.951307  [ 6400/60000]
loss: 0.729339  [12800/60000]
loss: 0.909877  [19200/60000]
loss: 0.796889  [25600/60000]
loss: 0.813612  [32000/60000]
loss: 0.887888  [38400/60000]
loss: 0.838673  [44800/60000]
loss: 0.858991  [51200/60000]
loss: 0.828259  [57600/60000]
Test Error:
 Accuracy: 68.9%, Avg loss: 0.820456

Epoch 10
-------------------------------
loss: 0.830392  [    0/60000]
loss: 0.910644  [ 6400/60000]
loss: 0.681585  [12800/60000]
loss: 0.871315  [19200/60000]
loss: 0.763827  [25600/60000]
loss: 0.773537  [32000/60000]
loss: 0.852972  [38400/60000]
loss: 0.813022  [44800/60000]
loss: 0.825912  [51200/60000]
loss: 0.798537  [57600/60000]
Test Error:
 Accuracy: 70.3%, Avg loss: 0.788864

Done!