PyTorch_Tutorial_Datasets & Dataloaders

https://pytorch.org/tutorials/beginner/basics/data_tutorial.html

Datasets & DataLoaders — PyTorch Tutorials 1.13.1+cu117 documentation

Datasets & DataLoaders

Code for processing data samples can get messy and hard to maintain; we ideally want our dataset code to be decoupled from our model training code for better readability and modularity. 

PyTorch provides two data primitives: torch.utils.data.DataLoader and torch.utils.data.Dataset that allows you to. se pre-loaded datasets as well as your won data.

Dataset stores the samples and their corresponding labels,

and DataLoader wraps an iterable around the Dataset to enable easy access to the samples.

 

Loading a Dataset

Here is an example of how to load the Fashion-MNIST dataset from TorchVision. Fashion-MNIST is a dataset of Zalando's article images consisting of 60,000 training examples and 10,000 test examples. Each example comprises a 28 x 28 grayscale image and an associated label from one of 10 classes.

 

We load the FashionMNIST Dataset with the following parameters:

※ root is the path where the train/test. data is stored,

※ train specifies training or test dataset,

※ download=True downloads the data from the internet if it's not available at root

※ transform and target_transform specify the feature and label transformations

import torch
from otrch.utils.data import Dataset
from torchvision import datasets
from torchvision.transforms import ToTensor
import matplotlib.pyplot as plt

training_data = datasets.FashionMNIST(
	root="data",
    train=True,
    download=True,
    transform=ToTensor()
)

test_data = datasets.FashionMNIST(
	root="data",
    train=False,
    download=True,
    transform=ToTensor()
)

Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-images-idx3-ubyte.gz to data/FashionMNIST/raw/train-images-idx3-ubyte.gz

  0%|          | 0/26421880 [00:00<?, ?it/s]
  0%|          | 32768/26421880 [00:00<01:27, 301653.92it/s]
  0%|          | 65536/26421880 [00:00<01:27, 300113.73it/s]
  0%|          | 131072/26421880 [00:00<01:00, 436247.75it/s]
  1%|          | 229376/26421880 [00:00<00:42, 618231.19it/s]
  2%|1         | 491520/26421880 [00:00<00:20, 1254014.99it/s]
  4%|3         | 950272/26421880 [00:00<00:11, 2249507.79it/s]
  7%|7         | 1933312/26421880 [00:00<00:05, 4438255.38it/s]
 15%|#4        | 3833856/26421880 [00:00<00:02, 8591848.90it/s]
 26%|##5       | 6815744/26421880 [00:00<00:01, 14401815.20it/s]
 38%|###7      | 9928704/26421880 [00:01<00:00, 18778349.54it/s]
 49%|####9     | 13008896/26421880 [00:01<00:00, 21644226.35it/s]
 60%|#####9    | 15761408/26421880 [00:01<00:00, 22689361.62it/s]
 71%|#######1  | 18841600/26421880 [00:01<00:00, 24271084.56it/s]
 83%|########3 | 21954560/26421880 [00:01<00:00, 25475216.11it/s]
 95%|#########4| 25034752/26421880 [00:01<00:00, 26194971.72it/s]
100%|##########| 26421880/26421880 [00:01<00:00, 16013467.93it/s]
Extracting data/FashionMNIST/raw/train-images-idx3-ubyte.gz to data/FashionMNIST/raw

Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/train-labels-idx1-ubyte.gz to data/FashionMNIST/raw/train-labels-idx1-ubyte.gz

  0%|          | 0/29515 [00:00<?, ?it/s]
100%|##########| 29515/29515 [00:00<00:00, 266129.40it/s]
100%|##########| 29515/29515 [00:00<00:00, 264798.49it/s]
Extracting data/FashionMNIST/raw/train-labels-idx1-ubyte.gz to data/FashionMNIST/raw

Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-images-idx3-ubyte.gz to data/FashionMNIST/raw/t10k-images-idx3-ubyte.gz

  0%|          | 0/4422102 [00:00<?, ?it/s]
  1%|          | 32768/4422102 [00:00<00:14, 304561.71it/s]
  1%|1         | 65536/4422102 [00:00<00:14, 303210.38it/s]
  3%|2         | 131072/4422102 [00:00<00:09, 440861.07it/s]
  5%|5         | 229376/4422102 [00:00<00:06, 624569.34it/s]
 11%|#1        | 491520/4422102 [00:00<00:03, 1272931.70it/s]
 21%|##1       | 950272/4422102 [00:00<00:01, 2279897.26it/s]
 44%|####3     | 1933312/4422102 [00:00<00:00, 4494161.18it/s]
 87%|########6 | 3833856/4422102 [00:00<00:00, 8663760.42it/s]
100%|##########| 4422102/4422102 [00:00<00:00, 5086968.08it/s]
Extracting data/FashionMNIST/raw/t10k-images-idx3-ubyte.gz to data/FashionMNIST/raw

Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz
Downloading http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/t10k-labels-idx1-ubyte.gz to data/FashionMNIST/raw/t10k-labels-idx1-ubyte.gz

  0%|          | 0/5148 [00:00<?, ?it/s]
100%|##########| 5148/5148 [00:00<00:00, 23938222.83it/s]
Extracting data/FashionMNIST/raw/t10k-labels-idx1-ubyte.gz to data/FashionMNIST/raw

Iterating and Visualizing the Dataset (데이터넷을 순회하고 시각화하기)

We can index Datasets manually like a list: training_data[index]. We use matplotlib to visualize some samples in our training data.

labels_map = {
	0: "T-Shirt",
    1: "Trouser",
    2: "Pullover",
    3: "Dress",
    4: "Coat",
    5: "Sandal",
    6: "Shirt",
    7: "Sneaker",
    8: "Bag",
    9: "Ankle Boot",
}
figure = plt.figure(figsize=(8,8))
cols, rows = 3, 3
for i in range(1, cols*rows+1):
	sample_dix = torch.randint(len(training_data), size=(1,)).item()
    img, label = training_data[sample_idx[
    figure.add_subplot(rows, cols, i)
    plt.title(labels_map[label])
    plt.axis("off")
    plt.imshow(img.squeeze(), cmap="gray")
plt.show()

Creating a Custom Dataset for your files (파일에서 사용자 정의 데이터셋 만들기)

A custom Dataset class must implement three fuctions: __init__, __len__, and __getitem__.

Take a look at this implementations; the FashionMNIST images are stored in a directory img_dir, and their labels are stored separately in CSV file annotations_file.

import os
import pandas as pd
from torchvision.io import read_image

class CustomImageDataset(Dataset):
	def __init__(self, annotations_file, img_dir, transform=None, target_transform=None):
		self.img_labels = pd.read_csv(annotations_file)
		self.img_dir = img_dir
		self.transform = transform
		self.target_transform = target_transform
        
	def __len__(self):
		return len(self.img_labels)
    
    def __getitems__(self, idx):
    	img_path = os.path.join(self.img_dir, self.img_labels.iloc[idx,0])
        image = read_image(img_path)
        label = self.img_labels.iloc[idx, 1]
        if self.transform:
        	image = self.transform(image)
        if self.target_transform:
        	label = self.target_transform(label)
		return image, label

__init__

The __init__fuctions is run once when instantiating the Dataset object. 

We initialize the directory containing the images, the annotations file, and both transforms.

 

the labels.csv file looks like:

tshirt1.jpg, 0
tshirt2.jpg, 0
......
ankleboot999.jpg, 9

def __init__(self, anntations_file, img_dir, transform=None, target_transform=None):
	self.img_labels = pd.read_csv(annotations_file)
    self.img_dir = img_dir
    self.transform = transform
    self.target_transform = target_transform

__len__

The __len__function returns the number of samples in our dateset. (샘플 개수를 반환)

 

def __len__(self):
	return len(self.img_labels)

__getitem__

The __getitem__fuction loads and returns a sample from the dataset at the given index idx. Based on the index, it identifies the image's location on disk, converts that to tensor using read_image, retrieves the corresponding label from the csv data in self.img_labels, calls the transform fuctions on them (if applicable), and returns the tensor image and corresponding label in a tuple.

주어진 인덱스에 해당하는 샘플을 데이터셋에서 불러오고 반환한다.

def __getitem__(self, idx):
	img_path = os.path.join(self.img_dir, self.img_labels.iloc([idx,0])
    image = read_images(img_path)
    label = self.img_labels.iloc[idx, 1]
    if self.transform:
    	image = self.transform(image)
    if self.target transform:
    	label = self.target_transform(label)
    return image, label

Preparing your data training with DataLoaders (DataLoader로 학습용 데이터 준비하기)

The Dataset retrieves our dataset's features and labels one sample at a time. While training a model, we typically want to pass smaples in "minibatches", reshuffle the data at every epoch to reduce model overfitting, and use Python's multiprocessing to speed up data retrieval.

DataLoader is an iterable that abstracts this complexity for us in an easy API.

from torch.utils.data import DataLoader

train_dataLoader = DataLoader(training_data, batch_size=64, shuffle=True)
test_dataloader = DataLoader(test_data, batch_size=64, shuffle=True)

Iterable through the DataLoader

We have loaded that dataset into the DataLoader and can iterate through the dataset as needed.

Each iteration below returns a batch of train_features and train_labels(containing batch_size=64 features and labels respectively). Because we specified shuffle=True, after we iterate over all batches the data is shuffled

# Display image and label.
train_features, train_labels = next(iter(train_dataloader))
print(f"Feature batch shape: {train_features.size()}")
print(f"Labels batch shape: {train_labels.size()}")
img = train_features[0].squeeze()
label = train_labels[0]
plt.imshow(img, cmap="gary")
plt.show()
print(f"Lbael: {label}")

Feature batch shape: torch.Size([64, 1, 28, 28])
Labels batch shape: torch.Size([64])
Label: 3