Update README.md

Experiments/Tensorflow/RNN/belling_the_cat.txt

@@ -1 +1 @@
-long ago , the mice had a general council to consider what measures they could take to outwit their common enemy , the cat . some said this , and some said that but at last a young mouse got up and said he had a proposal to make , which he thought would meet the case . you will all agree , said he , that our chief danger consists in the sly and treacherous manner in which the enemy approaches us . now , if we could receive some signal of her approach , we could easily escape from her . i venture , therefore , to propose that a small bell be procured , and attached by a ribbon round the neck of the cat . by this means we should always know when she was about , and could easily retire while she was in the neighbourhood . this proposal met with general applause , until an old mouse got up and said  that is all very well , but who is to bell the cat ? the mice looked at one another and nobody spoke . then the old mouse said it is easy to propose impossible remedies . 
+long ago , the mice had a general council to consider what measures they could take to outwit their common enemy , the cat . some said this , and some said that but at last a young mouse got up and said he had a proposal to make , which he thought would meet the case . you will all agree , said he , that our chief danger consists in the sly and treacherous manner in which the enemy approaches us . now , if we could receive some signal of her approach , we could easily escape from her . i venture , therefore , to propose that a small bell be procured , and attached by a ribbon round the neck of the cat . by this means we should always know when she was about , and could easily retire while she was in the neighbourhood . this proposal met with general applause , until an old mouse got up and said  that is all very well , but who is to bell the cat ? the mice looked at one another and nobody spoke . then the old mouse said it is easy to propose impossible remedies .

Experiments/Tensorflow/RNN/rnn_words.py

@@ -37,9 +37,8 @@ def read_data(fname):
     with open(fname) as f:
         content = f.readlines()
     content = [x.strip() for x in content]
-    content = [content[i].split() for i in range(len(content))]
+    content = [word for i in range(len(content)) for word in content[i].split()]
     content = np.array(content)
-    content = np.reshape(content, [-1, ])
     return content
 
 training_data = read_data(training_file)

README.md

@@ -1,6 +1,6 @@
-# Deep Learning: Theory and Experiments
-
-## Notes
+# Deep Learning Lecture Notes and Experiments
+### Code samples have links to other repo that I maintain ([Advanced Deep Learning with Keras](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras) book) or contribute ([Keras](https://github.com/keras-team/keras))
+## Lecture Notes
 0. Course Roadmap
   - [Deep Learning](https://docs.google.com/presentation/d/1JSmMV3SICXJ3SQOSj5WgJipssOpBYZKNZm1YnU5HO3g/edit?usp=sharing)
   - [Limits of Deep Learning](https://docs.google.com/presentation/d/13nsjiEjpiUpidxThT6hoCg19o0TMG-ho4tzgvuPkWV8/edit?usp=sharing)
@@ -16,13 +16,76 @@
 3. Deep Neural Networks
   - [Deep Feedforward Neural Networks, Cost, Output, Hidden Units](https://docs.google.com/presentation/d/1woHBsNgnwzjJndMcXXznaBKlLvWywuA6T7BFi0K7Yhg/edit?usp=sharing)
   - [Back Propagation](https://docs.google.com/presentation/d/1XD0tA6oxOETfFn1DTGJByhhyH3MF586OCN06WvAP22E/edit?usp=sharing)
+- PyTorch Sample Code
+  - [Backprop on a single unit MLP](backprop/backprop.ipynb)
+- Keras Sample Code
+  - [Overview](https://docs.google.com/presentation/d/15Y1snbE73g8vw16RN6uehVHyDFxAK_b0iKcmId1j5qM/edit?usp=sharing)
+  - [MLP on Linear Model](keras/mlp/linear.ipynb)
+  - [MNIST Sampler](keras/mlp/mnist-sampler.ipynb)
+  - [MLP on MNIST](keras/mlp/mlp-mnist.ipynb)
 4. [Regularization](https://docs.google.com/presentation/d/1lg4oxRDvfUIEtzMJ7E-Lqv1cDNiwoNeT1r5T-XnFIQI/edit?usp=sharing)
+- Keras Sample Code
+  - [MLP on MNIST no Regularizer](keras/regularization/mlp-mnist-noreg.ipynb)
+  - [MLP on MNIST with L2](keras/regularization/mlp-mnist-l2.ipynb)
+  - [MLP on MNIST with Dropout](keras/regularization/mlp-mnist-dropout.ipynb)
+  - [MLP on MNIST with Data Augmentation](keras/regularization/mlp-mnist-data_augment.ipynb) 
+
 5. [Optimization](https://docs.google.com/presentation/d/1wt53ds5dywq3WUm-jkdKFUjiHayBAV6-CSFAJg76Clg/edit?usp=sharing)
-6. [Convolutional Neural Networks](https://docs.google.com/presentation/d/1vxCMwjbssYKisIWt2UYiuOFMsJaFv-5-I6mYvtJ6Hr8/edit?usp=sharing)
-7. [Embeddings](https://docs.google.com/presentation/d/1YtKWA53T2NqXoL0vnk8jWl1WCBWCLhbh5OWz_1JrGdU/edit?usp=sharing) 
-8. [Recurrent Neural Networks, LSTM, GRU](https://docs.google.com/presentation/d/1qjQkUwnr2V--7JPz0H_wkzRyTYX3UtJsYrB3MQPGKLE/edit?usp=sharing)
+
+6. [Convolutional Neural Networks (CNN)](https://docs.google.com/presentation/d/1vxCMwjbssYKisIWt2UYiuOFMsJaFv-5-I6mYvtJ6Hr8/edit?usp=sharing)
+- [Various Types of CNN](https://towardsdatascience.com/various-types-of-convolutional-neural-network-8b00c9a08a1b)
+- Keras Sample Code
+  - [CNN on MNIST](keras/cnn/cnn-mnist.ipynb)
+  - [CNN on MNIST using Functional API](keras/cnn/cnn-functional.ipynb)
+  - [CNN on MNIST Siamese Network](keras/cnn/cnn-siamese.ipynb)
+
+7. [Deep Networks](https://docs.google.com/presentation/d/14aFawAa4zNqvPRkhmS5YATVxSlS01fig7qnstecRgG0/edit?usp=sharing)
+- Keras Sample Code
+  - [DenseNet](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter2-deep-networks/densenet-cifar10-2.4.1.py)  
+  - [ResNet](https://github.com/keras-team/keras/blob/master/examples/cifar10_resnet.py)
+8. [Embeddings](https://docs.google.com/presentation/d/1YtKWA53T2NqXoL0vnk8jWl1WCBWCLhbh5OWz_1JrGdU/edit?usp=sharing) 
+- Keras Sample Code
+  - [Simple Embedding and Sentiment Analysis](keras/embedding/sentiment_analysis.ipynb)
+  - [Glove Embedding](keras/embedding/glove_embedding.ipynb)
+9. [Recurrent Neural Networks, LSTM, GRU](https://docs.google.com/presentation/d/1qjQkUwnr2V--7JPz0H_wkzRyTYX3UtJsYrB3MQPGKLE/edit?usp=sharing)
+- Keras Sample Code
+  - [SimpleRNN on MNIST](keras/rnn/simple-rnn-mnist.ipynb)
+  - [CuDNNLSTM on MNIST](keras/rnn/cudnnlstm-mnist.ipynb)
+10. [AutoEncoders](https://docs.google.com/presentation/d/1gXWl0luuDe1qoQLSKdOUrzoq51WhhNLeq7x3PxQXYkA/edit?usp=sharing)
+- Keras Sample Code
+  - [AutoEncoder](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter3-autoencoders/autoencoder-mnist-3.2.1.py)
+  - [Denoising AutoEncoder](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter3-autoencoders/denoising-autoencoder-mnist-3.3.1.py)
+  - [Colorization AutoEncoder](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter3-autoencoders/colorization-autoencoder-cifar10-3.4.1.py)
+11. [Generative Adversarial Networks (GAN)](https://docs.google.com/presentation/d/13fiFibqjl9ps_CktJzMNAvoZXOlzHQDu8eRSb3a227g/edit?usp=sharing)
+- Keras Sample Code
+  - [DCGAN](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter4-gan/dcgan-mnist-4.2.1.py)
+  - [CGAN](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter4-gan/cgan-mnist-4.3.1.py)
+
+11a. [Improved GANs](https://docs.google.com/presentation/d/1tATpY1gzJo8x6Ziceln-KjcgEeqX-fnapL6IFdc9Wbk/edit?usp=sharing)
+- Keras Sample Code
+  - [WGAN](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter5-improved-gan/wgan-mnist-5.1.2.py)
+  - [LSGAN](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter5-improved-gan/lsgan-mnist-5.2.1.py)
+  - [ACGAN](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter5-improved-gan/acgan-mnist-5.3.1.py)
+
+11b. [Disentangled GAN](https://docs.google.com/presentation/d/1XboUGxLB1wYqJppsYiToq120JLhAOiuxLMUNbSfy_dk/edit?usp=sharing)
+  - [InfoGAN](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter6-disentangled-gan/infogan-mnist-6.1.1.py)
+
+11c. [Cross-Domain GAN](https://docs.google.com/presentation/d/17lizm6BGtDB7OIR1XQHGH-VzcoM5N0vBnjOhIlcAwZs/edit?usp=sharing)
+  - [CycleGAN](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter7-cross-domain-gan/cyclegan-7.1.1.py)
+
+12. [Variational Autoencoder (VAE)](https://docs.google.com/presentation/d/1ORVwhh5PgWEehcUQYL9t8nBCk9cj4TgXmUJIl6WMkpo/edit?usp=sharing)
+- Keras Sample Code
+  - [VAE MLP](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter8-vae/vae-mlp-mnist-8.1.1.py)
+  - [VAE CNN](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter8-vae/vae-cnn-mnist-8.1.2.py)
+  - [CVAE](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter8-vae/cvae-cnn-mnist-8.2.1.py)
+13. [Deep Reinforcement Learning (DRL)](https://docs.google.com/presentation/d/1oZC5qGofbx-dlPcnr00_fPBK0GEtt6FnMdpsnA6XYX8/edit?usp=sharing)
+- Keras Sample Code
+  - [Q-Learning](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter9-drl/q-learning-9.3.1.py)
+  - [Q-Learning on FrozenLake-v0](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter9-drl/q-frozenlake-9.5.1.py)
+  - [DQN and DDQN on CartPole-v0](https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/master/chapter9-drl/dqn-cartpole-9.6.1.py)
 
-## [Tensorflow](https://www.tensorflow.org/) (tf) Experiments
+### Warning: The following are old experiments that are no longer updated and maintained
+### [Tensorflow](https://www.tensorflow.org/) Experiments
 1. [Hello World!](https://github.com/roatienza/Deep-Learning-Experiments/blob/master/Experiments/Tensorflow/Intro/hello.py) 
 2. [Linear Algebra](https://github.com/roatienza/Deep-Learning-Experiments/blob/master/Experiments/Tensorflow/Math/linear_algebra.py)
 3. [Matrix Decomposition](https://github.com/roatienza/Deep-Learning-Experiments/blob/master/Experiments/Tensorflow/Math/decomposition.py)

backprop/backprop.ipynb

@@ -0,0 +1,188 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Backprop demo on a single unit linear MLP\n",
+    "This is a demonstration of how backpropagation works on a single unit MLP"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 96,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import torch\n",
+    "import torch.nn as nn\n",
+    "import torch.nn.functional as F\n",
+    "import torch.optim as optim"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "A single neuron/unit MLP with linear activation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 97,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class Net(nn.Module):\n",
+    "    def __init__(self):\n",
+    "        super(Net, self).__init__()\n",
+    "        self.fc1 = nn.Linear(1, 1)\n",
+    "    \n",
+    "    def forward(self, x):\n",
+    "        x = self.fc1(x)\n",
+    "        return x"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Build a network. Model a function: y = 2*x + 1\n",
+    "\n",
+    "Perform supervised learning using the following dataset:\n",
+    "\n",
+    "\n",
+    "| Step |  x   |  y  |\n",
+    "| :-- | ---: | ---:|\n",
+    "|  0   | 0.   |  1. |\n",
+    "|  1   | 1.   |  3. |\n",
+    "\n",
+    "At each step, perform backprop and print the gradients."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 107,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "---------------------0-----------------------\n",
+      "0. Bias grad:  None\n",
+      "0. Weights grad:  None\n",
+      "0. Bias:  tensor(0., grad_fn=<SelectBackward>)\n",
+      "0. Weights:  tensor([1.], grad_fn=<SelectBackward>)\n",
+      "0. Input:  tensor([0.])\n",
+      "0. Predicted Output:  tensor([0.], grad_fn=<AddBackward0>)\n",
+      "0. Loss:  tensor(1., grad_fn=<MseLossBackward>)\n",
+      "\n",
+      "---------------------1-----------------------\n",
+      "1. Bias grad:  tensor([-2.])\n",
+      "1. Weights grad:  tensor([[0.]])\n",
+      "1. Bias:  tensor(0.2000, grad_fn=<SelectBackward>)\n",
+      "1. Weights:  tensor([1.], grad_fn=<SelectBackward>)\n",
+      "1. Input:  tensor([1.])\n",
+      "1. Predicted Output:  tensor([1.2000], grad_fn=<AddBackward0>)\n",
+      "1. Loss:  tensor(3.2400, grad_fn=<MseLossBackward>)\n",
+      "\n",
+      "---------------------2-----------------------\n",
+      "2. Bias grad:  tensor([-3.6000])\n",
+      "2. Weights grad:  tensor([[-3.6000]])\n",
+      "2. Bias:  tensor(0.5600, grad_fn=<SelectBackward>)\n",
+      "2. Weights:  tensor([1.3600], grad_fn=<SelectBackward>)\n"
+     ]
+    }
+   ],
+   "source": [
+    "net = Net()\n",
+    "\n",
+    "# for ease of backprop demo, we set weight=1.0 and bias=0.0\n",
+    "net.fc1.bias = torch.nn.Parameter(torch.tensor([0.]))\n",
+    "net.fc1.weight = torch.nn.Parameter(torch.tensor([[1.]]))\n",
+    "print(\"---------------------0-----------------------\")\n",
+    "print(\"0. Bias grad: \", net.fc1.bias.grad)\n",
+    "print(\"0. Weights grad: \", net.fc1.weight.grad)\n",
+    "print(\"0. Bias: \", net.fc1.bias[0])\n",
+    "print(\"0. Weights: \", net.fc1.weight[0])\n",
+    "\n",
+    "# x=0, y=1.\n",
+    "input = torch.tensor([0.])\n",
+    "print(\"0. Input: \", input)\n",
+    "input.unsqueeze(0)\n",
+    "output = net(input)\n",
+    "print(\"0. Predicted Output: \", output)\n",
+    "target = torch.tensor([1.])\n",
+    "target = target.view(1, -1)\n",
+    "\n",
+    "# Use MSE Loss\n",
+    "criterion = nn.MSELoss()\n",
+    "loss = criterion(output, target)\n",
+    "print(\"0. Loss: \", loss)\n",
+    "\n",
+    "# Use SGD optimizer with learning rate of 0.1\n",
+    "optimizer = optim.SGD(net.parameters(), lr=0.1)\n",
+    "# Clear optimizer gradient\n",
+    "optimizer.zero_grad()\n",
+    "# Perform backprop\n",
+    "loss.backward(retain_graph=True)\n",
+    "# Update weight and bias\n",
+    "optimizer.step()\n",
+    "print(\"\\n---------------------1-----------------------\")\n",
+    "print(\"1. Bias grad: \", net.fc1.bias.grad)\n",
+    "print(\"1. Weights grad: \",net.fc1.weight.grad)\n",
+    "print(\"1. Bias: \", net.fc1.bias[0])\n",
+    "print(\"1. Weights: \",net.fc1.weight[0])\n",
+    "\n",
+    "# x=1.0, y=3.0\n",
+    "input = torch.tensor([1.])\n",
+    "print(\"1. Input: \", input)\n",
+    "input.unsqueeze(0)\n",
+    "output = net(input)\n",
+    "print(\"1. Predicted Output: \", output)\n",
+    "target = torch.tensor([3.])\n",
+    "target = target.view(1, -1)\n",
+    "\n",
+    "optimizer.zero_grad()\n",
+    "loss = criterion(output, target)\n",
+    "print(\"1. Loss: \", loss)\n",
+    "loss.backward()\n",
+    "optimizer.step()\n",
+    "print(\"\\n---------------------2-----------------------\")\n",
+    "print(\"2. Bias grad: \", net.fc1.bias.grad)\n",
+    "print(\"2. Weights grad: \",net.fc1.weight.grad)\n",
+    "print(\"2. Bias: \", net.fc1.bias[0])\n",
+    "print(\"2. Weights: \",net.fc1.weight[0])"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

keras/chapter1/cnn-mnist-1.4.1.py

@@ -0,0 +1,92 @@
+''' CNN MNIST digits classification
+
+3-layer CNN for MNIST digits classification 
+First 2 layers - Conv2D-ReLU-MaxPool
+3rd layer - Conv2D-ReLU-Dropout
+4th layer - Dense(10)
+Output Activation - softmax
+Optimizer - Adam
+
+99.4% test accuracy in 10epochs
+
+https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras
+'''
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import numpy as np
+from tensorflow.keras.models import Sequential
+from tensorflow.keras.layers import Activation, Dense, Dropout
+from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten
+from tensorflow.keras.utils import to_categorical
+# from tensorflow.keras.utils import plot_model
+from tensorflow.keras.datasets import mnist
+
+import datetime
+
+# load mnist dataset
+(x_train, y_train), (x_test, y_test) = mnist.load_data()
+
+# compute the number of labels
+num_labels = len(np.unique(y_train))
+
+# convert to one-hot vector
+y_train = to_categorical(y_train)
+y_test = to_categorical(y_test)
+
+# input image dimensions
+image_size = x_train.shape[1]
+# resize and normalize
+x_train = np.reshape(x_train,[-1, image_size, image_size, 1])
+x_test = np.reshape(x_test,[-1, image_size, image_size, 1])
+x_train = x_train.astype('float32') / 255
+x_test = x_test.astype('float32') / 255
+
+# network parameters
+# image is processed as is (square grayscale)
+input_shape = (image_size, image_size, 1)
+batch_size = 128
+kernel_size = 3
+pool_size = 2
+filters = 64
+dropout = 0.2
+
+# model is a stack of CNN-ReLU-MaxPooling
+model = Sequential()
+model.add(Conv2D(filters=filters,
+                 kernel_size=kernel_size,
+                 activation='relu',
+                 input_shape=input_shape))
+model.add(MaxPooling2D(pool_size))
+model.add(Conv2D(filters=filters,
+                 kernel_size=kernel_size,
+                 activation='relu'))
+model.add(MaxPooling2D(pool_size))
+model.add(Conv2D(filters=filters,
+                 kernel_size=kernel_size,
+                 activation='relu'))
+model.add(Flatten())
+# dropout added as regularizer
+model.add(Dropout(dropout))
+# output layer is 10-dim one-hot vector
+model.add(Dense(num_labels))
+model.add(Activation('softmax'))
+model.summary()
+# plot_model(model, to_file='cnn-mnist.png', show_shapes=True)
+
+# loss function for one-hot vector
+# use of adam optimizer
+# accuracy is good metric for classification tasks
+model.compile(loss='categorical_crossentropy',
+              optimizer='adam',
+              metrics=['accuracy'])
+# train the network
+start_time = datetime.datetime.now()
+model.fit(x_train, y_train, epochs=10, batch_size=batch_size)
+elapsed_time = datetime.datetime.now() - start_time
+print("Elapsed time (train): %s" % elapsed_time)
+
+loss, acc = model.evaluate(x_test, y_test, batch_size=batch_size)
+print("\nTest accuracy: %.2f%%" % (100.0 * acc))