---
title: "Neural Networks Module - Basic components"
subtitle: "TMA4268 Statistical Learning V2018"
author: "Thiago G. Martins, Department of Mathematical Sciences, NTNU"
date: "Spring 2018"
bibliography: references.bib
output: #3rd letter intentation hierarchy
#  beamer_presentation:
###    incremental: true # or >* for one at a time
#  slidy_presentation:
#    font_adjustment: +1  
   prettydoc::html_pretty:
    theme: cayman
    highlight: github
    toc: true
    toc_depth: 4
   # prettydoc::html_pretty:
   #  theme: architect
   #  highlight: github
#   html_document:
#     theme: yeti
#     toc: true
#     toc_float: true
#  pdf_document:
  #  toc: true
  #  toc_depth: 2

---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

```{r, include=FALSE}
library(keras)
```

## Tensor: Data representation for neural networks

Currently, most (if not all) current ML systems use tensors as their basic data structure.

Tensors are a generalisation of vectors and matrices to an arbitrary number of dimensions.

- In the context of tensor, a dimension is often called an axis or rank

A tensor is defined by 3 key attributes:

- Number of axis (rank or dimension)
- Shape (how many dimensions along each axis)
- Data type

### Abstract examples

#### Scalars (0D tensors)

- R doesn't have a data type to represent scalars. All numeric objects are vectors, matrices, or arrays.

- But an R vector that is always length 1 is conceptually similar to a scalar.

```{r, message=FALSE, warning=FALSE}
# scalar
tensor <- as.array(1)
```

```{r, message=FALSE, warning=FALSE}
# rank
length(dim(tensor))
# shape
dim(tensor)
# data type
typeof(tensor)
```

#### Vectors (1D tensors)

* Don’t confuse a 5D vector with a 5D tensor!

```{r, message=FALSE, warning=FALSE}
# vector
tensor <- as.array(c(12, 3, 6, 14, 10))
```

```{r, message=FALSE, warning=FALSE}
# rank
length(dim(tensor))
# shape
dim(tensor)
# data type
typeof(tensor)
```

#### Matrices (2D tensors)

* A matrix has two axes (often referred to as rows and columns)

```{r, message=FALSE, warning=FALSE}
tensor <- matrix(rep(0, 3*5), nrow = 3, ncol = 5)
```

```{r, message=FALSE, warning=FALSE}
# rank
length(dim(tensor))
# shape
dim(tensor)
# data type
typeof(tensor)
```

#### 3D tensors and higher-dimensional tensors

```{r, message=FALSE, warning=FALSE}
tensor <- array(rep(0, 2*3*2), dim = c(2,3,2))
```

```{r, message=FALSE, warning=FALSE}
# rank
length(dim(tensor))
# shape
dim(tensor)
# data type
typeof(tensor)
```

### Concrete data examples

#### Vector data

- 2D tensor of shape `(samples, features)`

- Data structure most commonly used by statistical packages

- Usually, this is the expected data structure when fitting fully connected neural networks

- Example: The Boston housing price dataset

```{r, message=FALSE, warning=FALSE, cache=TRUE, echo=FALSE}
boston_dataset <- dataset_boston_housing()
```

```{r, message=FALSE, warning=FALSE, echo=FALSE}
c(c(boston_train_data, boston_train_targets), c(boston_test_data, boston_test_targets)) %<-% boston_dataset
```

```{r, message=FALSE, warning=FALSE}
str(boston_train_data)
```

- More complex data types can be mapped to vector data format.
    - Image data: Usually represented in 3D tensors can be flatened into a 2D tensor

#### Timeseries or sequence data

- 3D tensor of shape `(samples, timesteps, features)`

<center>
<img src="imgs/3d_tensor_timeseries.png" height="50%" width="50%" />
</center>

- Stock price dataset
    - Each minute we record the current price, last minute lowest price and highest price
    - A trading day has 390 minutes and a trading year has 250 days
    - One year of data can be store in a 3D tensor `(250, 390, 3)`
- A statistician would usually store in a 2D tensor with a date column.

#### Image data

- 4D tensor of shape `(samples, height, width, channels)`

<center>
<img src="imgs/4d_tensor_image.png" height="40%" width="40%" />
</center>

- Grayscale images (like our MNIST digits) have only a single color channel, but are still stored in 4D tensors by convention.

**Note**: Unlike TensorFlow, Theano uses the convention `(samples, channels, height, width)`

#### Video data
        
- 5D tensor of shape `(samples, frames, height, width, channels)`

- A video can be understood as a sequence of frames, each frame being a color image.

## Tensor operations

- All transformations learned by deep neural networks can be reduced to a handful of tensor operations applied to tensors of numeric data

- Example: Dense layer

```
output = relu(dot(W, input) + b)
```

- Element-wise operation: `relu(x) = max(x,0)`

- Addition between a 2D tensor and a 1D tensor

- Dot product between `W` and `input`

### Element-wise operations

- Operations that are applied independently to each entry in the tensors being considered

- Massively parallel implementations available (vectorised implementations)

### Operations involving tensors of different dimensions

- The R `sweep()` function enables you to perform operations between higher-dimension tensors and lower-dimension tensors

```
# Add vector y for each column of matrix x
sweep(x, 2, y, `+`)
```

### Tensor dot

- Most common and useful tensor operation

- An element-wise product is done with the `*` operator in R, whereas dot products use the `%*%` operator

- Mathematically, what does the dot operation do?

```{r, message=FALSE, warning=FALSE, eval=FALSE}
# dot product between two vector of the same length
naive_vector_dot <- function(x, y) {          
  z <- 0
  for (i in 1:length(x))
    z <- z + x[[i]] * y[[i]]
  z
}
```

```{r, message=FALSE, warning=FALSE, eval=FALSE}
# dot product between a matrix x and a vector y
naive_matrix_vector_dot <- function(x, y) {
  z <- rep(0, nrow(x))
  for (i in 1:nrow(x))
    row_x <- x[i,]    
    z[[i]] <- naive_vector_dot(row_x, y)
  z
}
```

```{r, message=FALSE, warning=FALSE, eval=FALSE}
# dot product between a matrix x and a matrix y
naive_matrix_dot <- function(x, y) {                   1
  z <- matrix(0, nrow = nrow(x), ncol = ncol(y))
  for (i in 1:nrow(x))
    for (j in 1:ncol(y)) {
      row_x <- x[i,]
      column_y <- y[,j]
      z[i, j] <- naive_vector_dot(row_x, column_y)
    }
  z
}
```

<center>
<img src="imgs/dot_product_matrix_matrix.png" height="50%" width="50%" />
</center>

- The dot product generalizes for higher dimensions, given the apropriate shape compatibility observed before:
    * (a, b, c, d) . (d) -> (a, b, c) 
    * (a, b, c, d) . (d, e) -> (a, b, c, e) 
    * And so on.

### Tensor reshaping
  
- Always use the `array_reshape()` function when reshaping R arrays that will be passed to Keras.
    - Uses row-major semantics (as opposed to R's default column-major semantics)
    - Compatible with the way the numerical libraries called by Keras (NumPy, TensorFlow, and so on) interpret array dimensions.

```{r, message=FALSE, warning=FALSE}
x <- matrix(c(0, 1,
              2, 3,
              4, 5),
            nrow = 3, ncol = 2, byrow = TRUE)
x

# R uses column-major semantics by default
dim(x) <- c(6,1)
x
```

```{r, message=FALSE, warning=FALSE}
x <- matrix(c(0, 1,
              2, 3,
              4, 5),
            nrow = 3, ncol = 2, byrow = TRUE)
x

# array_reshape uses row-major semantics
x <- array_reshape(x, dim = c(6, 1))
x
```

## Reference material

This lecture note is based on [@chollet2018deep].

## References
