org.joone.structure
Class Nakayama

java.lang.Object
  extended by org.joone.structure.Nakayama
All Implemented Interfaces:
java.io.Serializable, java.util.EventListener, ConvergenceListener, NeuralNetListener, NeuralValidationListener

public class Nakayama
extends java.lang.Object
implements NeuralNetListener, NeuralValidationListener, ConvergenceListener, java.io.Serializable

This class performs the method of optimizing activation functions as described in:

K.Nakayama and Y.Kimura, "Optimization of activation functions in multilayer neural network applied to pattern classification", Proc. IEEE ICNN'94 Florida, pp.431-436, June 1994.

This techniques probably fails whenever the NeuralNet.join() method is called because this optimization technique stops the network to perform the optimization, use a NeuralNetListener instead.

Author:
Boris Jansen
See Also:
Serialized Form

Constructor Summary
Nakayama(NeuralNet aNet)
          Creates a new instance of Nakayama.
 
Method Summary
 void addLayer(Layer aLayer)
          Adds layers to this optimizer.
 void addLayers(NeuralNet aNeuralNet)
          Adds all the hidden layers to this optimizer.
 void cicleTerminated(NeuralNetEvent e)
           
protected  void cleanUp()
          This method is called after optimization, e.g.
protected  void doOptimize()
          Optimizes the activation functions of the network.
 void errorChanged(NeuralNetEvent e)
           
protected  void evaluateNeurons()
          Evaluates neurons, that is, this function calculates information related to the contribution of the activation functions, based on the following three criteria:
Information from neurons to its output layers.
protected  int[] findCorrelation(java.util.List aCorrelations, int aLayer, int aNeuron)
          Finds a correlation from a given list of correlations.
protected  int findIndex(java.util.List aStatuses, int aLayer, int aNeuron)
          Finds the index of a neuron taking into account the deletion of previous neurons.
protected  Layer findInputLayer(Synapse aSynapse)
          Finds the input layer of a synapse.
protected  Layer findOutputLayer(Synapse aSynapse)
          Finds the output layer of a synapse.
 double getEpsilon()
          Gets epsilon, the threshold to decide if a neuron should be deleted or not.
protected  double getGamma(int aLayer1, int aNeuron1, int aLayer2, int aNeuron2)
          Gets gammajj' (is equal to gammaj'j).
protected  double[] getMinCorrelation(int aLayer, int aNeuron)
          Gets the minimum correlation for a certain neuron j.
protected  double getSumAbsoluteWeights(Layer aLayer, int aNeuron)
          Sums up all the absolute values of the output weights of a neuron within a layer.
protected  double[] getSumOutputs(int aLayer, int aNeuron)
          Sums up the (normal and absolute) values of the outputs of a neuron over all patterns.
 void netConverged(ConvergenceEvent anEvent, ConvergenceObserver anObserver)
          This method is called whenever the network has converged according to some ConvergenceObserver
 void netStarted(NeuralNetEvent e)
           
 void netStopped(NeuralNetEvent e)
           
 void netStoppedError(NeuralNetEvent e, java.lang.String error)
           
 void netValidated(NeuralValidationEvent event)
           
 boolean optimize()
          Optimizes the activation functions of the neural network.
(package private)  void patternFinished()
          This method is called after every pattern, so we can retrieve information from the network that is related to the pattern that was just forwarded through the network.
protected  void removeAllListeners()
          Removes all the listeners from the neural network (temporarely).
protected  void removeNeuron(int aLayer, int aNeuron)
          Removes a neuron.
protected  void restoreAllListeners()
          Restore all the listeners to the neural network.
protected  void runValidation()
          Runs the network with a validator, this way we are able to collect information related to the different patterns.
protected  void selectNeurons()
          Selects neurons to be deleted based on the information calculated by evaluateNeurons().
 void setEpsilon(double anEpsilon)
          Sets epsilon, the threshold to decide if a neuron should be deleted or not.
protected  void weightsUpdateCorrelation(java.util.List aStatuses, java.util.List aCorrelations, int aLayer, int aNeuron)
          Updates weights before a neuron is removed (because of its similar correlation).
protected  void weightsUpdateVariance(int aLayer, int aNeuronOriginal, int aNeuron)
          Updates weights before a neuron is removed (because of its low variance).
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

Nakayama

public Nakayama(NeuralNet aNet)
Creates a new instance of Nakayama.

Parameters:
aNet - the network to be optimized.
Method Detail

addLayer

public void addLayer(Layer aLayer)
Adds layers to this optimizer. The layers will be optimized. Layers should be on the same (layer) level, otherwise the optimization does not make sense.

Parameters:
aLayer - the layer to be added.

addLayers

public void addLayers(NeuralNet aNeuralNet)
Adds all the hidden layers to this optimizer. The layers will be optimized. The neuron network should consist of only one hidden layer, i.e. the hidden layers should be on the same level else this method doesn't make any sense. If the hidden layers are not all on the same level then the layers should be added individually my using , adding only the layers that are on the same hidden level.

Parameters:
aNeuralNet - the network holding the hidden layers.

optimize

public boolean optimize()
Optimizes the activation functions of the neural network.

Returns:
whether the network was optimized or not, i.e. if neurons where deleted or not.

runValidation

protected void runValidation()
Runs the network with a validator, this way we are able to collect information related to the different patterns. Everytime a pattern is forwarded to the network we collect certain info patternFinished().


doOptimize

protected void doOptimize()
Optimizes the activation functions of the network.


cleanUp

protected void cleanUp()
This method is called after optimization, e.g. to restore the listeners.


selectNeurons

protected void selectNeurons()
Selects neurons to be deleted based on the information calculated by evaluateNeurons(). It selects neurons to be deleted and performs the deletion.


weightsUpdateCorrelation

protected void weightsUpdateCorrelation(java.util.List aStatuses,
                                        java.util.List aCorrelations,
                                        int aLayer,
                                        int aNeuron)
Updates weights before a neuron is removed (because of its similar correlation).

Parameters:
aStatuses - the status of the neurons (used to find the correct neuron taking into account any deletions of neurons).
aCorrelations - a list holding all the correlations (neurons to be removed and the correlated neuron (which will take over the weights)).
aLayer - the layer of the neuron to be removed.
aNeuron - the neuron to be removed.

getGamma

protected double getGamma(int aLayer1,
                          int aNeuron1,
                          int aLayer2,
                          int aNeuron2)
Gets gammajj' (is equal to gammaj'j).

Parameters:
aLayer1 - the layer of neuron j.
aNeuron1 - the neuron j.
aLayer2 - the layer of neuron j'.
aNeuron2 - the neuron j'. return gammajj'.

findIndex

protected int findIndex(java.util.List aStatuses,
                        int aLayer,
                        int aNeuron)
Finds the index of a neuron taking into account the deletion of previous neurons.

Parameters:
aStatuses - the status of neurons.
aLayer - the layer of the neuron.
aNeuron - the index of the neuron, not considering any deletions.
Returns:
the index of the neuron taking into account any previous deletions.

findCorrelation

protected int[] findCorrelation(java.util.List aCorrelations,
                                int aLayer,
                                int aNeuron)
Finds a correlation from a given list of correlations.

Parameters:
aCorrelations - a list holding correlations.
aLayer - the layer of the neuron to find the correlation for.
aNeuron - the neuron to find the correlation for.
Returns:
the index of the neuron of the correlation with (aLayer, aNeuron), return null in case the correlation is not found.

weightsUpdateVariance

protected void weightsUpdateVariance(int aLayer,
                                     int aNeuronOriginal,
                                     int aNeuron)
Updates weights before a neuron is removed (because of its low variance).

Parameters:
aLayer - the index of the layer of the neuron.
aNeuronOriginal - the index of the neuron to be removed (NOT taking into account previous deletions).
aNeuron - the index of the neuron to be removed (taking into account previous deletions).

removeNeuron

protected void removeNeuron(int aLayer,
                            int aNeuron)
Removes a neuron.

Parameters:
aLayer - the index of the layer in which we should remove the neuron.
aNeuron - the index of the neuron to be removed (taking into account previous deletions).

findInputLayer

protected Layer findInputLayer(Synapse aSynapse)
Finds the input layer of a synapse.

Parameters:
aSynapse - the synapse to find the input layer for.

findOutputLayer

protected Layer findOutputLayer(Synapse aSynapse)
Finds the output layer of a synapse.

Parameters:
aSynapse - the synapse to find the output layer for.

evaluateNeurons

protected void evaluateNeurons()
Evaluates neurons, that is, this function calculates information related to the contribution of the activation functions, based on the following three criteria:
This information will be used in a next stage to select neurons to delete.


getMinCorrelation

protected double[] getMinCorrelation(int aLayer,
                                     int aNeuron)
Gets the minimum correlation for a certain neuron j. A correlation between neuron j and j' is defined as Rjj' = 1 - |Gammajj'|. Rjmin (which this function calculates is defined as Rjmin = min-j{Rjj'}. It also returns the index of the neuron j that is the argument to this function as well as j' that is the minimum.

Parameters:
anLayer - the index of the layer of the neuron aNeuron.
aNeuron - the neuron within the layer (j).
Returns:
the minimum correlation Rjmin, together with the index of the neuron as an argument and the neuron of the minimum (layer and neuron). The lower index of the two neurons is at position 1, 2 and the higher index is at position 3, 4. The minimum itself is at position 0. Finally at position 5 we will indicate if the argument was the lower index neuron (<0, so it is now at position 1, 2) or if the argument was the higher index neuron (>0, so now it is at position 3, 4).

getSumOutputs

protected double[] getSumOutputs(int aLayer,
                                 int aNeuron)
Sums up the (normal and absolute) values of the outputs of a neuron over all patterns.

Parameters:
aLayer - an index of the layer to retrieve the outputs of that layer.
aNeuron - the neuron in the layer.
Returns:
the sum of (index 0: normal, index 1: absolute) outputs of neuron aNeuron.

getSumAbsoluteWeights

protected double getSumAbsoluteWeights(Layer aLayer,
                                       int aNeuron)
Sums up all the absolute values of the output weights of a neuron within a layer.

Parameters:
aLayer - the layer holding neuron aNeuron.
aNeuron - the neuron in the layer.
Returns:
the sum of absolute values of the output weights of neuron aNeuron within layer aLayer.

cicleTerminated

public void cicleTerminated(NeuralNetEvent e)
Specified by:
cicleTerminated in interface NeuralNetListener

errorChanged

public void errorChanged(NeuralNetEvent e)
Specified by:
errorChanged in interface NeuralNetListener

netStarted

public void netStarted(NeuralNetEvent e)
Specified by:
netStarted in interface NeuralNetListener

netStopped

public void netStopped(NeuralNetEvent e)
Specified by:
netStopped in interface NeuralNetListener

netStoppedError

public void netStoppedError(NeuralNetEvent e,
                            java.lang.String error)
Specified by:
netStoppedError in interface NeuralNetListener

netValidated

public void netValidated(NeuralValidationEvent event)
Specified by:
netValidated in interface NeuralValidationListener

removeAllListeners

protected void removeAllListeners()
Removes all the listeners from the neural network (temporarely). They will be added again after optimization and the network is restarted.


restoreAllListeners

protected void restoreAllListeners()
Restore all the listeners to the neural network.


patternFinished

void patternFinished()
This method is called after every pattern, so we can retrieve information from the network that is related to the pattern that was just forwarded through the network.


getEpsilon

public double getEpsilon()
Gets epsilon, the threshold to decide if a neuron should be deleted or not.

Returns:
the threshold epsilon.

setEpsilon

public void setEpsilon(double anEpsilon)
Sets epsilon, the threshold to decide if a neuron should be deleted or not.

Parameters:
anEpsilon - the new epsilon.

netConverged

public void netConverged(ConvergenceEvent anEvent,
                         ConvergenceObserver anObserver)
Description copied from interface: ConvergenceListener
This method is called whenever the network has converged according to some ConvergenceObserver

Specified by:
netConverged in interface ConvergenceListener
Parameters:
anEvent - the event that is generated.
anObserver - the observer that generated the event.


Submit Feedback to pmarrone@users.sourceforge.net