---
title: "Neural Networks - MNIST - Convolution NN"
subtitle: "TMA4268 Statistical Learning V2018"
author: "Thiago G. Martins, Department of Mathematical Sciences, NTNU"
date: "Spring 2018"
output: #3rd letter intentation hierarchy
#  beamer_presentation:
###    incremental: true # or >* for one at a time
#  slidy_presentation:
#    font_adjustment: +1  
   prettydoc::html_pretty:
    theme: cayman
    highlight: github
    toc: true
    toc_depth: 4
   # prettydoc::html_pretty:
   #  theme: architect
   #  highlight: github
#  pdf_document:
  #  toc: true
  #  toc_depth: 2

---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

```{r, include=FALSE}
library(keras)
```

## Convolutional Neural Networks (Convnets or CNNs)

A typical CNN sketch:

<center>
<figure>
  <img src="imgs/convolution_model_illustrative.jpg" height="70%" width="70%" />
  <figcaption>Source: [Mathworks page about CNN](https://fr.mathworks.com/discovery/convolutional-neural-network.html)</figcaption>
</figure>
</center>

Feature Learning part of the CNN defined in Keras:

```{r, message=FALSE, warning=FALSE}
model <- keras_model_sequential() %>%
  layer_conv_2d(filters = 32, kernel_size = c(3, 3), activation = "relu", input_shape = c(28, 28, 1)) %>%
  layer_max_pooling_2d(pool_size = c(2, 2)) %>%
  layer_conv_2d(filters = 64, kernel_size = c(3, 3), activation = "relu") %>%
  layer_max_pooling_2d(pool_size = c(2, 2)) %>%
  layer_conv_2d(filters = 64, kernel_size = c(3, 3), activation = "relu")
```

Classification part of the CNN defined in Keras:

```{r, message=FALSE, warning=FALSE}
model <- model %>%
  layer_flatten() %>%
  layer_dense(units = 64, activation = "relu") %>%
  layer_dense(units = 10, activation = "softmax")
```

### Convnet layers


* Dense layers learn global patterns in their input space
* Convolutional layers learn local patterns 
* The patterns they learn are translation invariant
    * If they learn a pattern in the lower-right of an image, they would also recognise the pattern on the upper-left
    * Dense layers would have to learn the pattern anew in a different position
    * This makes convnets data efficient, needing less data to learn
* They can learn spatial hierarchies of patterns
    * The first layers learn small patterns such as edges
    * Later layers learn larger patterns based on the small patterns

<center>
<img src="imgs/convolution_hierarchical.png" height="50%" width="50%" />
</center>

A CNN model alternate between convolutional and max-pooling layers.

#### 2D convolutional layer

A 2D convolutional layer is defined by `layer_conv_2d` in Keras. The two main arguments for the layer are `filters` and `kernel_size`.

On the GIF below, we see one filter being produced by kernel size `(3, 3)` from an image with dimension `(5,5,1)`.
This particular filter has dimension `(3,3)`.

<center>
<img src="imgs/convolution_schematic.gif" height="70%" width="70%" />
</center>

The following layer will create 32 filters such as the one above by applying a convolution with kernel size `(3, 3)` into a image with dimension `(28, 28, 1)`. Each of the 32 filters would have dimension `(26, 26)`

```{r, message=FALSE, warning=FALSE, eval=FALSE}
layer_conv_2d(filters = 32, kernel_size = c(3, 3), activation = "relu", input_shape = c(28, 28, 1)) %>%
```

The dimension of the resulting filter is `(26, 26)` and not `(28, 28)` due to border effects.

Each filter will go through the following transformation:

```
filter = convolution(input) 
output = activation(filter + bias)
```

The total number of parameters defined in `layer_conv_2d` is given by `kernel_height * kernel_width * input_channel * filters` (convolution operation) + `filters` (one bias per filter).

Averaging adjacent pixels blur the image:

<center>
<img src="imgs/convolution_average.png" height="50%" width="50%" />
</center>

Adjacent pixels are very different in the direction perpendicular to the edge:

<center>
<img src="imgs/convolution_edge.png" height="50%" width="50%" />
</center>

#### 2D max pooling layer

The max pooling layer is conceptually similar to the convolutional layer, with two main differences.

* Instead of transforming local patchs via a learned linear transformation, they are transformed via a hard-coded max operation.
* The window size is usually `(2,2)` and the stride is equal to 2 (instead of 1 for the convolutional layer), downsampling the filters.

<center>
<img src="imgs/convolution_max_pooling.png" height="70%" width="70%" />
</center>

The reasons to downsample are:

* to reduce the number of coefficients to process
* to induce spatial-filter hierarchies by making successive convolution layers look at increasingly large windows (in terms of the fraction of the original input they cover).

## MNIST dataset

The objective here is to classify the digit contained in a image using a convnet model.

### Convnet model

Convolutional Neural Network in Keras:

```{r, message=FALSE, warning=FALSE}
model <- keras_model_sequential() %>%
  layer_conv_2d(filters = 32, kernel_size = c(3, 3), activation = "relu", input_shape = c(28, 28, 1)) %>%
  layer_max_pooling_2d(pool_size = c(2, 2)) %>%
  layer_conv_2d(filters = 64, kernel_size = c(3, 3), activation = "relu") %>%
  layer_max_pooling_2d(pool_size = c(2, 2)) %>%
  layer_conv_2d(filters = 64, kernel_size = c(3, 3), activation = "relu")
```

Adding a classifier on top of the convnet:

```{r, message=FALSE, warning=FALSE}
model <- model %>%
  layer_flatten() %>%
  layer_dense(units = 64, activation = "relu") %>%
  layer_dense(units = 10, activation = "softmax")
```

Inspect the model:

```{r, message=FALSE, warning=FALSE}
model
```

Training the convnet on MNIST images:

```{r, message=FALSE, warning=FALSE, cache=TRUE}
mnist <- dataset_mnist()
c(c(train_images, train_labels), c(test_images, test_labels)) %<-% mnist
train_images <- array_reshape(train_images, c(60000, 28, 28, 1))
train_images <- train_images / 255
test_images <- array_reshape(test_images, c(10000, 28, 28, 1))
test_images <- test_images / 255
train_labels <- to_categorical(train_labels)
test_labels <- to_categorical(test_labels)
```

```{r, message=FALSE, warning=FALSE, eval=TRUE}
model %>% compile(
  optimizer = "rmsprop",
  loss = "categorical_crossentropy",
  metrics = c("accuracy")
)
model %>% fit(
  train_images, train_labels,
  epochs = 5, batch_size=64
)
```

Evaluate the model on the test data:

```{r, message=FALSE, warning=FALSE, eval=TRUE}
results <- model %>% evaluate(test_images, test_labels)
results
```