AI Snake Game - Reinforcement Learning with PyTorch[1] by Patrick Loeber

Reinforcement Learning

Reinforcement Learning (RL) is an area of machine learning concerned with how software agents outgh to take actions in an environment in order to maximize the notion of cumulative reward.

Or:

RL is teaching a software agent how to behave in an environment by telling it how good it's doing.

 

Rought overview

part 1.

Game(Pygame) 파이게임을 활용하여 비주얼 게임을 만든다. 게임루프

- play_step(action) 각 스텝에 따라 보상 및 게임 점수 등을 리턴한다.

-> reward, gam_over, score

 

part 2.

Agend

- game

- model

Training:

- state = get_state(game) state를 계산

-action = get_move(state): state에 따른 다음 동작을 계산

- model.predict()

- reward, game_over, score = game.play_step(action)

- new_state = get_state(game)

- remember

-model.train()

 

part 3.

Model(PyTorch)

Linear_QNet(DQN) 뉴럴네트워크 

- model.predict(state)

-> action

 

Reward Setting

- eat food: +10

- game over: -10

- else: 0

 

Action 다음 동작을 결정하는 요소, (위, 아래, 오른쪽, 왼쪽 4가지 방향을 설정 할 수 있으나 180도 반대 방향의 입력이 들어가는 경우 게임이 바로 끝나버리는 오류를 예방하고자 3개의 동작만을 설정)

[1,0,0] -> straing

[0,1,0] -> right turn

[0,0,1] -> left turn

 

State (11 values) Agent에게 설정할 수 있는 다음 환경을 알려주는 용도

[danger s traing, danger right, danger left, direction left, direction right, direction up, direction down, food left, food right, food up, food down]

 

Setting Model

 

(Deep) Q Learning  

Q Value = Quality of action, Q의 의미

 

0. Init Q Value (= init model) 모델을 랜덤 파라미터로 초기화함

1. Choose action (model.predict(state)) 다음 액션을 선택(혹은 아직 학습이 안된 경우, 랜덤으로 선택)

2. Perform action

3. Measure reward

4. Update Q value ( + train model)

repeat the steps 1 ~ 4

 

Bellman Equation:

Q Update Rule Simplified:

Q = model.predict(state0)

Qnew = R + r * max(Q(state1))

 

Loss function:

loss = (Qnew - Q)^2 -> Means Square Value