Skip to content
Snippets Groups Projects
Commit c7c48614 authored by GILSON Matthieu's avatar GILSON Matthieu
Browse files

add autodiff

parent e8d5a11b
No related branches found
No related tags found
No related merge requests found
# Installing pytorch
Using anaconda (or miniconda):
- create a new environment named 'pytorch': `conda create -n pytorch`
- get into new environment: `conda activate pytorch`
- install pytorch for cpu: `conda install pytorch torchvision torchaudio cpuonly -c pytorch`
Check [https://pytorch.org/get-started/locally/](https://pytorch.org/get-started/locally/) for details using pip or other means to install pytorch, also with GPU/CUDA option.
%% Cell type:markdown id:7f27c493-5ce5-4eec-a4e2-68d0a7c8a267 tags:
# Auto-differentiation: first example building a multilayer perceptron from scratch
@author: gilsonmatthieu
%% Cell type:code id:b7879792-1e21-401e-b037-1471ddf2cb91 tags:
``` python
import os
import numpy as np
import matplotlib.pyplot as pp
# working directory
work_dir = 'tmp/'
if not os.path.exists(work_dir):
print('create directory:', work_dir)
os.makedirs(work_dir)
grph_fmt = 'png'
# colors for plotting
cols_gr = []
for ii in [2,1,0]:
cols_gr += [[ii*0.3,ii*0.3,ii*0.3]]
```
%% Cell type:markdown id:6d8fafc1-835d-4b53-b070-a5077f561412 tags:
We first load the MNIST dataset made of handwritten digits.
%% Cell type:code id:57dc9276-317c-4bb4-95e4-69dde16c379c tags:
``` python
# load MNIST dataset
locals().update(np.load('../data/MNIST/mnist.npz'))
# number of samples, pixels, features (when vectorized)
n_train = train_data.shape[0]
n_test = test_data.shape[0]
n_px = train_data.shape[1]
n_feat = n_px**2
# number of categories / classes
n_cat = np.unique(train_labels).size
print('number of classes:', n_cat)
print('classes:', np.unique(train_labels, return_counts=True))
```
%% Cell type:markdown id:9fc468e6-7917-4889-84fc-b0f7df20c998 tags:
We want to build a network of 2 layers of neurons with a given nonlinear function (e.g. tanh), whose weights are optimized to reduce a loss function (error between output and a corresponding target for each sample).
%% Cell type:code id:b2cb5fa6-b1ad-47c9-9062-f1bca177e52f tags:
``` python
# layer of artificial neurons, batch calculations
class layer:
# forward function
def f(self, x):
# tanh
ex1 = np.exp(x)
ex2 = np.exp(-x)
return (ex1 - ex2)/ (ex1 + ex2)
# forward pass for x of shape (input dim)
def fwd(self, x, W, return_x1=False):
# augmented vector by one extra element (for bias)
x1 = np.ones([x.shape[0]+1])
x1[:-1] = x
# calculate output after weight multiplication and function f
y = self.f(np.einsum('ij, j -> i', W, x1))
# return y or y and x1
return y
```
%% Cell type:code id:b860d280-efa3-45dc-bd59-785ecb6e4f4e tags:
``` python
# let's test the forward pass
M = 2 # dimensionality of input x
N = 1 # dimensionality of output y
W = np.array([[1.0, -1.0, 0.5]]) # weights of shape (N,M+1) with bias
# generate random input x
x = np.random.normal(loc=0.0, size=[M])
l = layer()
y = l.fwd(x, W)
print('sample: x={} mapped to y={}'.format(x,y))
```
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment