norsys.netica
Class NetTester

java.lang.Object
  |
  +--norsys.netica.NetTester

public class NetTester

extends java.lang.Object

A tool for grading a Bayes net by using a set of real cases to see how well the predictions or diagnosis of the net match the actual cases. It is not for decision networks.

The following outlines how you typically use a NetTester object.

1. Choose the nodes that you wish the Net to predict and get rated on, and then place them in a NodeList. These nodes are called the "test" nodes. Their values in the case file are treated as though they were hidden from the Bayes net.

2. Choose any additional nodes you do not wish the network to know the value of during its inference. For example, if the network is for medical diagnosis, you might select the disease nodes and nodes representing other unobservable internal states. These nodes are called the "unobserved" nodes. It is okay if this list is empty or null when passed to the NetTester constructor.

3. Construct the NetTester. E.g.: NetTester tester = new NetTester (testNodes, unObsNodes, -1);

4. Run the method testWithFile(), supplying the case file to use for testing. Netica will read the case file, processing the cases one-by-one. When Netica reads a case, it will ignore any findings for the unobserved nodes. It then does belief updating to generate beliefs for each of the unobserved nodes. It goes back and checks the true value for those nodes as supplied by the case file (if they are supplied for that case), and compares them with the beliefs it generated. It accumulates all the comparisons into summary statistics.

5. Retrieve any of the summary statistics for the nodes you are interested in. Call any of getConfusion, getErrorRate, getLogLoss, or getQuadraticLoss, as desired.

6. Repeat steps 4 and 5 as often as desired, on possibly new case files, thus accumulating the cases and observing how the statistics change.

7. Optionally, cleanup up all the resources allocated for testing by calling the finalize method.

A coding example of the above is given in the example for NetTester.

Since:

2.08

Version:

2.21 - May 7, 2002

Constructor Summary

NetTester(NodeList testNodes, NodeList unObsNodes, int tests)

Creates a NetTester object which is a tool for grading a Bayes net (it is not for decision networks) using a set of real cases to see how well the predictions or diagnosis of the net match the actual cases.

Method Summary

void	finalize() Informs the native NetTester that it is no longer required, so that it may free-up its resources.
double	getConfusion(Node node, int predictedState, int actualState) Returns the number of times the Net predicted predictedState for node, but the case file actually held actualState as the value of that node.
double	getErrorRate(Node node) Returns the cumulative "error rate" of node under the tests previously performed.
double	getLogLoss(Node node) Returns the "logarithmic loss" of node under the tests previously performed.
double	getQuadraticLoss(Node node) Returns the "quadratic loss" of node under the tests previously performed.
void	testWithFile(Streamer inStream) Perform the tests on the Bayes net using the case data in inStream.

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

NetTester

public NetTester(NodeList testNodes,
                 NodeList unObsNodes,
                 int tests)
          throws NeticaException

Creates a NetTester object which is a tool for grading a Bayes net (it is not for decision networks) using a set of real cases to see how well the predictions or diagnosis of the net match the actual cases.

testNodes are nodes that the Bayes net will predict and get rated on. Their values in the case file are all hidden from the Bayes net (ie, unobserved) whenever a case is read. For each such case, the Bayes net does a prediction and compares that prediction with the true value from the case file, cumulating statistics as it goes.

If unObsNodes is non-null, then the nodes it contains will also be unobserved.

Pass -1 for tests.

Parameters:

NodeList		testNodes		the nodes that the Bayes net will predict and get rated on
NodeList		unObsNodes		a list of nodes that will will also be unobserved. If there are no such nodes, you may pass null.
int		tests		a parameter for future use; pass -1 for now.

Version:

Versions 2.08 and later have this method.

See Also:

testWithFile		Accumulate case data into the test
getConfusion		Get elements of the confusion matrix
getErrorRate		Get fraction of test cases where prediction failed
getLogLoss		Get the "logarithmic loss" score of the test
getQuadraticLoss		Get the "quadratic loss" score of the test
finalize		Free up tester and all its resources

Example:

  Net net = new Net (new Streamer ("BookBags.dne"));
  net.compile();
  NodeList testNodes = new NodeList (net);
  Node testNode = net.getNode ("Book_Bag");
  testNodes.add (testNode);
  NetTester tester = new NetTester (testNodes, null, -1);
  tester.testWithFile (new Streamer ("BookBags100.cas"));
  printConfusionMatrix (tester, testNode); // defined in example for  getConfusion
  System.out.println ("Error rate for " + node.getName() + " = " + tester.getErrorRate(node));
  tester.finalize();

Method Detail

finalize

public void finalize()
              throws NeticaException

Informs the native NetTester that it is no longer required, so that it may free-up its resources.

If you override this method, be sure to call the base class method (super.finalize();).

Version:

Versions 2.08 and later have this method.
In the C Version of the API, this function is named DeleteNetTester_bn.

See Also:

NetTester

Construct the NetTester object

Overrides:

finalize in class java.lang.Object

testWithFile

public void testWithFile(Streamer inStream)
                  throws NeticaException

Perform the tests on the Bayes net using the case data in inStream.

The net must be compiled (see compile) before calling this.

This method can be called multiple times with different files to accumulate the results of all the cases.

Calls to this method can be intermingled with calls to getConfusion, getErrorRate, getLogLoss, and getQuadraticLoss.

Parameters:

Streamer

inStream

Version:

Versions 2.08 and later have this method.
In the C Version of the API, this function is named TestWithFile_bn.

See Also:

NetTester

Construct the NetTester object

getConfusion

public double getConfusion(Node node,
                           int predictedState,
                           int actualState)
                    throws NeticaException

Returns the number of times the Net predicted predictedState for node, but the case file actually held actualState as the value of that node. These are the entries of a table traditionally called the "confusion matrix".

The prediction is formed by reading all of the observed node values from the case file, updating beliefs, and then picking the state of this node which has the highest resultant (posterior) probability.

node must have been in the testNodes list originally passed to NetTester.

Parameters:

Node		node		The node being examined.
int		predictedState		The state predicted by this net, as compared to the actualState in the case file.
int		actualState		the actual state in the case file, as compared to the predictedState

Version:

Versions 2.08 and later have this method.
In the C Version of the API, this function is named GetTestConfusion_bn.

See Also:

NetTester		Construct the NetTester object
getErrorRate		Get fraction of test cases where prediction failed
getLogLoss		Get the "logarithmic loss" score of the test
getQuadraticLoss		Get the "quadratic loss" score of the test

Example:

  /*
   * Print a confusion matrix table.
   */
  public static void printConfusionMatrix (NetTester nt, Node node) throws NeticaException {
      int numStates = node.getNumStates();
      System.out.println("Confusion matrix for " + node.getName() + ":");
  
      for (int i=0;  i < numStates;  ++i){
          System.out.print( "\t" + node.state(i).getName() );
      }
      System.out.println( "\t" + "Actual" );
  
      for (int a=0;  a < numStates;  ++a){
          for (int p=0;  p < numStates;  ++p){
              System.out.print( "\t" + (int) (nt.getConfusion(node, p, a )) );
          }
          System.out.println( "\t" + node.state(a).getName() );
      }
  }
  
  Sample output from running the above method might be:
  
      Confusion matrix for Book_Bag:
              bag_1   bag_2   Actual
              36      7       bag_1
              7       50      bag_2
  

getErrorRate

public double getErrorRate(Node node)
                    throws NeticaException

Returns the cumulative "error rate" of node under the tests previously performed. This is the fraction of times the Net predicted or diagnosed states incorrectly for node, out of all the cases which provided a value for node.

The prediction is formed by reading all of the observed node values from the case file, updating beliefs, and then picking the state of this node which has the highest resultant (posterior) probability.

A result of 0.0 means no prediction errors, whereas a result of 1.0 means all predictions were in error.

node must have been in the testNodes list originally passed to NetTester.

Parameters:

Node

node

The node being examined.

Version:

Versions 2.08 and later have this method.
In the C Version of the API, this function is named GetTestErrorRate_bn.

See Also:

NetTester		Construct the NetTester object
getConfusion		Get elements of the confusion matrix
getLogLoss		Get the "logarithmic loss" score of the test
getQuadraticLoss		Get the "quadratic loss" score of the test

getLogLoss

public double getLogLoss(Node node)
                  throws NeticaException

Returns the "logarithmic loss" of node under the tests previously performed.

The "logarithmic loss" is defined as: MOAC [- log (Pc)],

where MOAC stands for the mean (average) over all cases (i.e., all cases
for which the case file provides a value for the node in question), and

where log(Pc) is the natural logarithm of the probability predicted for the state that turns out to be correct.

node must have been in the testNodes list originally passed to NetTester.

Parameters:

Node

node

The node being examined.

Version:

Versions 2.08 and later have this method.
In the C Version of the API, this function is named GetTestLogLoss_bn.

See Also:

NetTester		Construct the NetTester object
getConfusion		Get elements of the confusion matrix
getErrorRate		Get fraction of test cases where prediction failed
getQuadraticLoss		Get the "quadratic loss" score of the test
Sensitivity.getMutualInfo		Use a Sensitivity measurer to find the mutual info (entropy reduction) between two nodes

getQuadraticLoss

public double getQuadraticLoss(Node node)
                        throws NeticaException

Returns the "quadratic loss" of node under the tests previously performed.

The "quadratic loss" is defined as: MOAC [1 - 2 * Pc + sum[j=1 to n] (Pj ^ 2)],

where MOAC stands for the mean (average) over all cases (i.e., all cases
for which the case file provides a value for the node in question),

where Pc is the probability predicted for the state that turns out to be correct,

where Pj is the probability predicted for state j, and

where n is the number of states of the node.

node must have been in the testNodes list originally passed to NetTester.

Parameters:

Node

node

The node being examined.

Version:

Versions 2.08 and later have this method.
In the C Version of the API, this function is named GetTestQuadraticLoss_bn.

See Also:

NetTester		Construct the NetTester object
getConfusion		Get elements of the confusion matrix
getErrorRate		Get fraction of test cases where prediction failed
getLogLoss		Get the "logarithmic loss" score of the test