---
title: "Neural Networks - IMDB - Recurrent NN"
subtitle: "TMA4268 Statistical Learning V2018"
author: "Thiago G. Martins, Department of Mathematical Sciences, NTNU"
date: "Spring 2018"
output: #3rd letter intentation hierarchy
#  beamer_presentation:
###    incremental: true # or >* for one at a time
#  slidy_presentation:
#    font_adjustment: +1  
   prettydoc::html_pretty:
    theme: cayman
    highlight: github
    toc: true
    toc_depth: 4
   # prettydoc::html_pretty:
   #  theme: architect
   #  highlight: github
#  pdf_document:
  #  toc: true
  #  toc_depth: 2

---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

```{r, include=FALSE}
library(keras)
```

## Recurrent Neural Networks (RNNs)

A RNN process sequences by iterating through the sequence elements and maintaining a state containing information relative to what it has seen so far.

<center>
<img src="imgs/recurrent_unroled.png" height="70%" width="70%" />
</center>

* The state of the RNN is reset between processing two different, independent sequences (such as two different IMDB reviews).

Transition equation for a simple RNN:

```
output_t <- tanh(as.numeric((W %*% input_t) + (U %*% state_t) + b))
```

Following is R pseudo-code for a simple RNN layer:

```{r, message=FALSE, warning=FALSE, eval=FALSE}
state_t <- 0
for (input_t in input_sequence) {
  output_t <- activation(dot(W, input_t) + dot(U, state_t) + b)
  state_t <- output_t
}
```

## Embedding layer

Previously, we have encoded text data using integers. 

* The order of the words were not preserved
* The representation did not captured any text semantics

Another approach to feed text to our models is to use an embedding layer:

* low-dimensional floating-point vectors learned from data.
* geometric relationships between word vectors should reflect the semantic relationships between these words. 
* Take a 2D input tensor of integers of shape `(samples, sequence_length)`
* Return a 3D floating-point tensor of shape `(samples, sequence_length, embedding_dimensionality)`
* Such a 3D tensor can be processed by an RNN layer.
* The vector are randomly initialized prior to training.

## The IMDB dataset

The objective here is to classify a movie review as either positive or negative.

### Recurring neural networks with an embedding layer

* Data preparation parameters

```{r, message=FALSE, warning=FALSE}
max_features <- 10000 # Number of most frequent words
maxlen <- 500         # Padding the sequence of words to be of equal length
batch_size <- 32      # Batch size used for training
```

* Downloading the data

```{r, message=FALSE, warning=FALSE, cache=TRUE}
imdb <- dataset_imdb(num_words = max_features)
c(c(input_train, y_train), c(input_test, y_test)) %<-% imdb
cat(length(input_train), "train sequences\n")
cat(length(input_test), "test sequences")
```

* Padding the sequences

```{r, message=FALSE, warning=FALSE}
input_train <- pad_sequences(input_train, maxlen = maxlen)
input_test <- pad_sequences(input_test, maxlen = maxlen)
cat("input_train shape:", dim(input_train), "\n")
cat("input_test shape:", dim(input_test), "\n")
```

* Defining the model with embedding and simple RNN layers:

```{r, message=FALSE, warning=FALSE}
model <- keras_model_sequential() %>%
  layer_embedding(input_dim = max_features, output_dim = 32) %>%
  layer_simple_rnn(units = 32) %>%
  layer_dense(units = 1, activation = "sigmoid")
```

```{r, message=FALSE, warning=FALSE}
model
```

* Compiling the model

```{r, message=FALSE, warning=FALSE, eval=TRUE}
model %>% compile(
  optimizer = "rmsprop",
  loss = "binary_crossentropy",
  metrics = c("acc")
)
```

* Training and validation

```{r, message=FALSE, warning=FALSE, eval=TRUE}
history <- model %>% fit(
  input_train, y_train,
  epochs = 10,
  batch_size = 128,
  validation_split = 0.2
)
```

```{r, message=FALSE, warning=FALSE, eval=TRUE}
plot(history)
```

### LSTM layer

The simple RNN layer should theoretically be able to retain at time `t` information about inputs seen many timesteps before.

* But in practice, such long-term dependencies are very hard to learn.

The LSTM (long-short term memory) layer was designed to address this issue.

* It allow past information to be reinjected at a later time.

Using the LSTM layer in Keras:

```{r, message=FALSE, warning=FALSE, eval=TRUE}
model <- keras_model_sequential() %>%
  layer_embedding(input_dim = max_features, output_dim = 32) %>%
  layer_lstm(units = 32) %>%
  layer_dense(units = 1, activation = "sigmoid")
model %>% compile(
  optimizer = "rmsprop",
  loss = "binary_crossentropy",
  metrics = c("acc")
)
history <- model %>% fit(
  input_train, y_train,
  epochs = 10,
  batch_size = 128,
  validation_split = 0.2
)
```

```{r, message=FALSE, warning=FALSE, eval=TRUE}
plot(history)
```
