My Experiment with Machine Learning: Reverse Neural Network

Trying to stimulate Reverse Neuron Activation

In our SEE ML class of Neural Network in 5th Exercise Prof Andrew Ng has taught us a wonderful Algorithm, Neural Network, and its detail implementation was required for the Exercise. The Given Problem  in exercise was to recognize the Hand-Written NUMERAL (0-9) from the dataset of 5000 samples collected from Post-offices in US. As expected after the end of the exercise, our implemented NN Algo did a wonderful job with accuracy of almost 98%.

As we know Concept of Neural Network is originated from the simulation of Human Brain. And when we learn and start recognizing things by referring them with some name, Our brain develop a capability of Imaging the same stuff, even wothout seeing that. For example further after recoggnizing digit Zero ‘0’, Whenever i will hear word Zero, i can draw a pic of Zero in Brain (Well in very plain way, otherwise brain has capability to imagine linked stuff too, and i’m sure that would be some where in Advance Machine Learning). So i planned to do similar stuff with our data, twisting the problem to draw a 20x20 matrix of pixels to draw a gray-scale image of a given digit.

With given dataset, i thought of two approach, i’m not sure if any work

1. Train the data to Classify the “i x j” pixel to either one or zero, that might give me a Black&White Image or a almost similar way but instead of classifying with sigmoid just let the value of Prediction as it is and consider it as Pixel value of Gray-Scale Image
2. Do a Reverse Neural Networking ( i just coined it) , i.e. by making Y data as feature and X data as Output layer having Classification into 400 different Category.

So far, i’m working with 2nd method because that seesm to be easy and in very sync with HW-4 so just some small modification in code to go with hypothesis.
I converted Y data (that was 5000x1) Vector into a Matrices of (5000x10) with each row representing a vector of 1X10, with corresponding ‘y’ as 1 keeping other zero (e.g. y = 5 becomes [0 0 0 0 1 0 0 0 0 0]. And Output Layer as 400x1 Vector, changed the input and out put size. And ready to go my home made Rev-Neural Net.

I trained it over 50 iteration and not to be so very amazement , i dint fail, it failed BRUTLY.

After training, when i fed it y = 1, i.e.([1 0 0 ….])

h1 = sigmoid([1 y]*Theta1’)
h2 = sigmoid([1 h1]*Theta2)
displayData(h2)



Aah... what its given was just not presentable, i still dint loose my heart and just increased my Iter to 400, and it’s tarining is on in background.

i’ll update with result.


UPDATE:
That too Failed Badly, After that try few other steps like making bias term 0, though i have yet to check it with Learning Curve, i'll be doing in some times. Mean while if some one does so can update...


Also please discuss what is wrong in this approach or why this Approach is at all wrong?

Getting Started : Pre- Requisite

Off course the first requirement is having Python on your machine. Good if you get pip installed on your machine. Additionally you may have virtualenv to create a separate work-space all together without messing your previous configuration.
For our purpose, we need numerical extension of Python which are used for linear algebra, data visualization and other Scientific and Numerical stuff. We need to install following packges for that.

•  SciPy : 
•  NumPy
•  Matplotlib

Additionally you may like to add, iPython for interactive data plotting with matplotlib

Installing them is a easy task. i used Ubuntu,

$ sudo pip install numpy 

$ sudo pip install scipy 

$ sudo pip install matplotlib 

$ sudo pip install ipython

Do check the order, SciPy depends on NumPy. Also Scipy didn't go that easy for me, and i faced trouble in installing by pip command so finally i took the source-code and built from it. Now check your installation. Open Terminal,

$ ipython --pylab 

it'll open iPython interactive shell with matplotlib ready option (i.e. you need not to import it)

]: import numpy as np
]: import scipy as sp
]: plot(np.arange(10))

This will plot a Line Like this in Picture.

One very important think to remember, all arrays and list and all sequences in Python are '0'-based unlike Octave where everything is 1-based.



Note: I'm not an expert in using these tools, so you can always help me with your experience.

 

Welcome

Welcome to this Blog. As you all know Stanford University started an experiment of providing there various Computer science classes online worldwide, and started with first 3 classes AI, ML, DB from last October.

One of the most popular class among them is Machine Learning Class by Prof Andrew Ng. He is undoubtedly doing wonderful job, Excellent way of teaching, so easy to grasp lectures by him made us learn ML so delightful. First of all Many Congratulation and Heartfelt Thank you Sir.

Now, ML is subject involves lot of data churning and Linear Algebra plays a key-role in it to identify the optimum and the best solution for given data-set. Also visualization of same is very necessary to understand it completely that frequently require us to plot the data in various forms. Where Numerical Computing tools like  MATLAB comes very handy, but at the same times it requires a pretty good learning-curve and it costs as well. So, Professor suggested us to use Octave as a good alternative, its a very fast processing and easy to learn tool, comes handy with not much heavy load. Also it looks to be a good tool for any proto-typing as Ng suggested. And so far by experience I can attest that.

Well, As a python user, I have got this natural curiosity to test Python Numerical ability to do the similar stuff with its very efficient extensions Scipy & Friends. And i tried to work with that in parallel. And find a good amount of similarity in between both. And in this blog, i'll be posting the Python Conversion of Octave Codes that we'll get as programming exercise (Off-course only after due date).

Well, I'm also a very new user of both so my solution might not be very efficient or the best one expected, but then you are always welcome with your expertise.

One more thing if you are  total unaware of Scipy as of now, than i won't suggest to learn Octave and Scipy simultaneously, you may end-up in messy analogy.

So if you are also a enthusiastic python fan or just curious newbie of ML, Welcome to the Blog.