An Introduction to Neural Network Rule Extraction Algorithms By - - PowerPoint PPT Presentation
An Introduction to Neural Network Rule Extraction Algorithms By Sarah Jackson Can we trust magic? Neural Networks Machine learning black boxes Magical, unexplainable results Problems People won't trust Neural Networks since
✗ Neural Networks
✗ Machine learning black boxes ✗ Magical, unexplainable results
✗ Problems
✗ People won't trust Neural Networks since it
✗ End result isn't always the only thing we
✗ Unacceptable risk for certain scenarios
✗ Neural Networks have been shown to
✗ Neural Networks are capable of learning
✗ Rules help to bridge the gap between
✗ Rule extraction from Neural Networks will
✗ Rules will also improve usefulness of data
✗ Validation
✗ We can tell something has been learned
✗ Integration
✗ Can be used with symbolic systems
✗ Theory discovery
✗ May not have been seen otherwise
✗ Explanation ability
✗ Allows exploration of knowledge in network
✗ Accuracy
✗ Correctly classify unseen examples
✗ Fidelity
✗ Same behavior as Neural Network
✗ Consistency
✗ Classify unseen examples the same
✗ Comprehensibility
✗ Size of rule set and number of clauses per
✗ Knowledge in Neural Networks represented
✗ Extraction algorithms attempt to directly or
✗ Neural Network behavior is explained
✗ Knowledge is extracted from each node in the
✗ Each node's rules are based on previous layers ✗ Usually simply described and accurate ✗ Require threshold approximation for each node ✗ Restricted generalization and scalability
✗ Special training procedure ✗ Special network architecture
✗ Require sigmoidal transfer functions for hidden
✗ Describe output nodes as functions of input
✗ Internal structure of network is not
✗ Represent networks as decision trees ✗ Extract rules from constructed decision
✗ May not be efficient as complexity of
✗ Uses aspects of decompositional and
✗ Network architecture and value of weights
✗ Attempts to gain advantages of each
✗ Trees Parroting Networks ✗ Global method
✗ Represents network knowledge through a
✗ Uses same construction as C4.5 and
✗ Uses breadth-first search to construct the
✗ Classes used for decision tree are those
✗ List of leaf nodes kept with related data
✗ Subset of training data ✗ Set of complementary data ✗ Set of constraints
✗ Data sets used to determine if node should
✗ Data sets meet constraints
✗ Nodes are removed from list when split or
✗ Never added to list again ✗ Children are added to list
✗ Decision function determines type of decision
✗ M-of-N – Each node represents an m-of-n test ✗ 1-of-N – Each node represents a 1-of-n test ✗ Simple – Each node represents a test for one
✗ Comparison on UCI Tic-Tac-Toe Data
✗ 27 inputs, 20 hidden nodes, 2 outputs
✗ Typically, shortest tree is easiest to understand ✗ M-of-N has fewest nodes, but is very difficult to
✗ TREPAN provides higher quality information
✗ Only uses training data to construct decision tree
✗ TREPAN uses training data and may use
✗ Uses CN2 and C4.5 algorithms
✗ Bound Decomposition Tree
✗ Decomposition Algorithm
✗ Designed with goals of no retraining, high
✗ Algorithm works for Multi-Layer
✗ Maximum upper bounds on any neuron
✗ All inputs that have positive weight have a
✗ Inputs with negative weight have a value of
✗ Minimum lower bounds on any neuron
✗ Only inputs that have negative weight have
✗ Inputs with positive weight have a value of
✗ Each neuron has its own minimum and maximum
✗ Minimum is found by adding the bias plus all
✗ Maximum is found by adding the bias plus all
Weight Min Bound Max Bound I1
I2 0.65 0.65 I3
I4 0.72 0.72 Bias (-1) 1
0.37
✗ Each neuron (cube) is divided into two subcubes
✗ One subcube assumes 0 as the value and the
✗ Remaining inputs are used to construct the input
✗ Bounds are calculated for each subcube
✗ Positive subcube – lower bound is positive ✗ Negative subcube – upper bound is negative ✗ Uncertain subcube – lower bound is negative and
✗ Positive subcubes will always fire
✗ Represents a rule for the neuron
✗ Negative subcubes will never fire ✗ Uncertain subcubes must be further
✗ Rules for a neuron are the set of all input
✗ Can have a Δ over 0 to prune the neuron
Milare, R., De Carvalho, A., & Monard, M. (2002). An Approach to Explain Neural Networks Using Symbolic Algorithms. International Journal of Computational Intelligence and Applications. 2(4), 365-376. Heh, J. S., Chen, J. C., & Chang, M. (2008). Designing a decompositional rule extraction algorithm for neural networks with bound decomposition tree. Neural Computing and Applications. 17, 297- 309. Nobre, C., Martinelle, E., Braga, A., De Carvalho, A., Rezende, S., Braga, J. L. & Ludermir,