Add files via upload

Author: Arsenii Ashukha

deepbayes-school/2018/description.md

@@ -0,0 +1,27 @@
+<h2 align="center">Qualification task DeepBayes 2018</h2> 
+
+The strong knowledge of math, programming, and the ability to do research have become the most important skills in machine learning. 
+In this task, we address unsupervised learning, a very fundamental AI problem. 
+More specifically, we suggest you to implement Fully Connected Autoencoder using PyTorch for dimensionality reduction, and optionally to do some research with this model. 
+
+We define Autoencoder as two parametric differentiable functions, namely the Encoder (E) and the Decoder(D). Parameters of these two functions are adjusted by minimization of the following loss function:
+<center>
+  <div>
+    <img src="pics/eq.png" align="middle" width="640"> 
+  </div>
+</center>
+where the left part is L2 reconstruction loss, the right part is L1 regularizer, and λ is a scalar regularizer weight.  
+
+You are requested to implement the AutoEncoder class. 
+Use [ ```task.py```]() as a template of the solution; all technical details are described there. 
+Your solution has to be implemented in _python 3.6_.
+Changing prototypes of the functions and using additional libraries (except included) is prohibited, otherwise, the solution will be automatically rejected.
+Please copy to the form only the implementation of the AutoEncoder class.
+
+In the optional research assignment, we ask you to study:
+
+- Which recontraction loss and regularizer lead to more natural images?
+- How different parameters of your architecture influence the solution? For example, consider a different number of layers and a number of neurons. Is it beneficial to use noisy layers like Dropout?
+
+We suggest you to check the solution via ```test_work``` function. 
+Please, note that ```test_work``` checks only interfaces, but does not check whether the solution is correct.

deepbayes-school/2018/pics/eq.png

@@ -0,0 +1,129 @@
+import torch
+import argparse
+import numpy as np
+import torch.utils.data
+
+from torch import nn, optim
+from torch.autograd import Variable
+from torchvision import datasets, transforms
+from torchvision.utils import save_image
+
+
+class AutoEncoder(nn.Module):
+    def __init__(self, inp_size, hid_size):
+        super(AutoEncoder, self).__init__()
+        """
+        Here you should define layers of your autoencoder
+        Please note, if a layer has trainable parameters, it should be nn.Linear. 
+        ## !! CONVOLUTIONAL LAYERS MUST NOT BE HERE !! ##
+        However, you can use any noise inducing layers, e.g. Dropout.
+
+        Your network must not have more than six layers with trainable parameters.
+        :param inp_size: integer, dimension of the input object
+        :param hid_size: integer, dimension of the hidden representation
+        """
+        pass
+
+    def encode(self, x):
+        """
+        Encodes objects to hidden representations (E: R^inp_size -> R^hid_size)
+
+        :param x: inputs, Variable of shape (batch_size, inp_size)
+        :return:  hidden represenation of the objects, Variable of shape (batch_size, hid_size)
+        """
+        pass
+
+    def decode(self, h):
+        """
+        Decodes objects from hidden representations (D: R^hid_size -> R^inp_size)
+
+        :param h: hidden represenatations, Variable of shape (batch_size, hid_size)
+        :return:  reconstructed objects, Variable of shape (batch_size, inp_size)
+        """
+        pass
+
+    def forward(self, x):
+        """
+        Encodes inputs to hidden representations and decodes back.
+
+        x: inputs, Variable of shape (batch_size, inp_size)
+        return: reconstructed objects, Variable of shape (batch_size, inp_size)
+        """
+        return self.decode(self.encode(x))
+
+    @staticmethod
+    def loss_function(recon_x, x):
+        """
+        Calculates the loss function.
+
+        :params recon_x: reconstructed object, Variable of shape (batch_size, inp_size)
+        :params x: original object, Variable of shape (batch_size, inp_size)
+        :return: loss
+        """
+        pass
+
+
+def train(model, optimizer, train_loader, test_loader):
+    for epoch in range(10):
+        model.train()
+        train_loss, test_loss = 0, 0
+        for data, _ in train_loader:
+            data = Variable(data).view(-1, 784)
+            x_rec = model(data)
+            loss = model.loss_function(x_rec, data)
+
+            optimizer.zero_grad()
+            loss.backward()
+            optimizer.step()
+            train_loss += loss.data[0]
+        print('=> Epoch: %s Average loss: %.3f' % (epoch, train_loss / len(train_loader.dataset)))
+
+        model.eval()
+        for data, _ in test_loader:
+            data = Variable(data, volatile=True).view(-1, 784)
+            x_rec = model(data)
+            test_loss += model.loss_function(x_rec, data).data[0]
+
+        test_loss /= len(test_loader.dataset)
+        print('=> Test set loss: %.3f' % test_loss)
+
+        n = min(data.size(0), 8)
+        comparison = torch.cat([data.view(-1, 1, 28, 28)[:n], x_rec.view(-1, 1, 28, 28)[:n]])
+        save_image(comparison.data.cpu(), 'pics/reconstruction_' + str(epoch) + '.png', nrow=n)
+    return model
+
+
+def test_work():
+    print('Start test')
+    get_loader = lambda train: torch.utils.data.DataLoader(
+        datasets.MNIST('../data', train=train, download=True, transform=transforms.ToTensor()),
+        batch_size=50, shuffle=True)
+    train_loader, test_loader = get_loader(True), get_loader(False)
+    
+    try:
+        model = AutoEncoder(inp_size=784, hid_size=20)
+        optimizer = optim.Adam(model.parameters(), lr=1e-3)
+    except Exception:
+        assert False, 'Error during model creation'
+        return
+
+    try:
+        model = train(model, optimizer, train_loader, test_loader)
+    except Exception:
+        assert False, 'Error during training'
+        return
+
+    test_x = Variable(torch.randn(1, 784))    
+    rec_x, hid_x = model(test_x), model.encode(test_x)
+    submodules = dict(model.named_children())
+    layers_with_params = np.unique(['.'.join(n.split('.')[:-1]) for n, _ in model.named_parameters()])
+    
+    assert (hid_x.dim() == 2) and (hid_x.size(1) == 20),  'Hidden representation size must be equal to 20'
+    assert (rec_x.dim() == 2) and (rec_x.size(1) == 784), 'Reconstruction size must be equal to 784'
+    assert len(layers_with_params) <= 6, 'The model must contain not more than 6 layers'
+    assert np.all(np.concatenate([list(p.shape) for p in model.parameters()]) <= 800), 'All hidden sizes must be less than 800'
+    assert np.all([isinstance(submodules[name], nn.Linear) for name in layers_with_params]), 'All layers with parameters must be nn.Linear'
+    print('Success!🎉')
+
+
+test_work()