org.antlr.analysis
Class DecisionProbe

java.lang.Object
  extended by org.antlr.analysis.DecisionProbe

public class DecisionProbe
extends java.lang.Object

Collection of information about what is wrong with a decision as discovered while building the DFA predictor. The information is collected during NFA->DFA conversion and, while some of this is available elsewhere, it is nice to have it all tracked in one spot so a great error message can be easily had. I also like the fact that this object tracks it all for later perusing to make an excellent error message instead of lots of imprecise on-the-fly warnings (during conversion). A decision normally only has one problem; e.g., some input sequence can be matched by multiple alternatives. Unfortunately, some decisions such as a : ( A | B ) | ( A | B ) | A ; have multiple problems. So in general, you should approach a decision as having multiple flaws each one uniquely identified by a DFAState. For example, statesWithSyntacticallyAmbiguousAltsSet tracks the set of all DFAStates where ANTLR has discovered a problem. Recall that a decision is represented internall with a DFA comprised of multiple states, each of which could potentially have problems. Because of this, you need to iterate over this list of DFA states. You'll note that most of the informational methods like getSampleNonDeterministicInputSequence() require a DFAState. This state will be one of the iterated states from stateToSyntacticallyAmbiguousAltsSet. This class is not thread safe due to shared use of visited maps etc... Only one thread should really need to access one DecisionProbe anyway.


Field Summary
protected  java.util.Set<java.lang.Integer> altsWithProblem
          The overall list of alts within the decision that have at least one conflicting input sequence.
protected  java.util.Set<DFAState> danglingStates
          The set of states w/o emanating edges and w/o resolving sem preds.
 DFA dfa
           
protected  boolean nonLLStarDecision
          If decision with > 1 alt has recursion in > 1 alt, it's nonregular lookahead.
static java.lang.Integer REACHABLE_BUSY
           
static java.lang.Integer REACHABLE_NO
           
static java.lang.Integer REACHABLE_YES
           
protected  java.util.Map<java.lang.Integer,java.lang.Integer> stateReachable
          Used to find paths through syntactically ambiguous DFA.
protected  java.util.Set<DFAState> statesResolvedWithSemanticPredicatesSet
          Was a syntactic ambiguity resolved with predicates? Any DFA state that predicts more than one alternative, must be resolved with predicates or it should be reported to the user.
protected  java.util.Set<java.lang.String> statesVisitedAtInputDepth
          Used while finding a path through an NFA whose edge labels match an input sequence.
protected  java.util.Set<java.lang.Integer> statesVisitedDuringSampleSequence
           
protected  java.util.Set<DFAState> statesWithSyntacticallyAmbiguousAltsSet
          Track all DFA states with nondeterministic alternatives.
protected  java.util.Map<DFAState,java.util.Map<java.lang.Integer,SemanticContext>> stateToAltSetWithSemanticPredicatesMap
          Track the predicates for each alt per DFA state; more than one DFA state might have syntactically ambig alt prediction.
protected  java.util.Map<DFAState,java.util.Map<java.lang.Integer,java.util.Set<antlr.Token>>> stateToIncompletelyCoveredAltsMap
          Tracks alts insufficiently covered.
protected  MultiMap<java.lang.Integer,NFAConfiguration> stateToRecursionOverflowConfigurationsMap
          Recursion is limited to a particular depth.
protected  java.util.Map<DFAState,java.util.Set<java.lang.Integer>> stateToSyntacticallyAmbiguousTokensRuleAltsMap
          Track just like stateToSyntacticallyAmbiguousAltsMap, but only for nondeterminisms that arise in the Tokens rule such as keyword vs ID rule.
protected  boolean timedOut
          Did ANTLR have to terminate early on the analysis of this decision?
static boolean verbose
           
 
Constructor Summary
DecisionProbe(DFA dfa)
           
 
Method Summary
 boolean analysisOverflowed()
          Took too long to analyze a DFA
 boolean analysisTimedOut()
          Did the analysis complete it's work?
 java.util.Set getDanglingStates()
          return set of states w/o emanating edges and w/o resolving sem preds.
 java.lang.String getDescription()
          Return a string like "3:22: ( A {;} | B )" that describes this decision.
protected  java.util.Set getDFAPathStatesToTarget(DFAState targetState)
           
 java.util.Set getDFAStatesWithSyntacticallyAmbiguousAlts()
          Return all DFA states in this DFA that have NFA configurations that conflict.
 java.util.Set getDisabledAlternatives(DFAState d)
          Which alts were specifically turned off to resolve nondeterminisms? This is different than the unreachable alts.
 java.util.Map<java.lang.Integer,java.util.Set<antlr.Token>> getIncompletelyCoveredAlts(DFAState d)
          Return a list of alts whose predicate context was insufficient to resolve a nondeterminism for state d.
 java.lang.String getInputSequenceDisplay(java.util.List labels)
          Given List
protected  boolean getNFAPath(NFAState s, int labelIndex, java.util.List labels, java.util.List path)
          Given a sample input sequence, you usually would like to know the path taken through the NFA.
 java.util.List getNFAPathStatesForAlt(int firstAlt, int alt, java.util.List labels)
          Given an alternative associated with a nondeterministic DFA state, find the path of NFA states associated with the labels sequence.
 java.util.Set getNonDeterministicAlts()
           
 java.util.List getNonDeterministicAltsForState(DFAState targetState)
          Return the sorted list of alts that conflict within a single state.
 java.util.Set getNondeterministicStatesResolvedWithSemanticPredicate()
           
 int getNumberOfStates()
          How many states does the DFA predictor have?
protected  void getSampleInputSequenceUsingStateSet(State startState, State targetState, java.util.Set states, java.util.List<Label> labels)
          Given a start state and a final state, find a list of edge labels between the two ignoring epsilon.
 java.util.List<Label> getSampleNonDeterministicInputSequence(DFAState targetState)
          Return a List
 SemanticContext getSemanticContextForAlt(DFAState d, int alt)
          Each state in the DFA represents a different input sequence for an alt of the decision.
protected  java.lang.String getStateLabelIndexKey(int s, int i)
           
 java.lang.String getTokenNameForTokensRuleAlt(int alt)
          From an alt number associated with artificial Tokens rule, return the name of the token that is associated with that alt.
 java.util.List<java.lang.Integer> getUnreachableAlts()
          Get a list of all unreachable alternatives for this decision.
 boolean hasPredicate()
          At least one alt refs a sem or syn pred
 boolean isCyclic()
           
 boolean isDeterministic()
          If no states are dead-ends, no alts are unreachable, there are no nondeterminisms unresolved by syn preds, all is ok with decision.
 boolean isNonLLStarDecision()
          Found recursion in > 1 alt
 boolean isReduced()
           
protected  void issueRecursionWarnings()
           
 void issueWarnings()
           
protected  boolean reachesState(DFAState startState, DFAState targetState, java.util.Set states)
          Given a start state and a target state, return true if start can reach target state.
 void removeRecursiveOverflowState(DFAState d)
          If a recursion overflow is resolve with predicates, then we need to shut off the warning that would be generated.
 void reportAltPredicateContext(DFAState d, java.util.Map altPredicateContext)
          Report the list of predicates found for each alternative; copy the list because this set gets altered later by the method tryToResolveWithSemanticPredicates() while flagging NFA configurations in d as resolved.
 void reportAnalysisTimeout()
           
 void reportDanglingState(DFAState d)
          Report the fact that DFA state d is not a state resolved with predicates and yet it has no emanating edges.
 void reportIncompletelyCoveredAlts(DFAState d, java.util.Map<java.lang.Integer,java.util.Set<antlr.Token>> altToLocationsReachableWithoutPredicate)
           
 void reportLexerRuleNondeterminism(DFAState d, java.util.Set<java.lang.Integer> nondeterministicAlts)
          Currently the analysis reports issues between token definitions, but we don't print out warnings in favor of just picking the first token definition found in the grammar ala lex/flex.
 void reportNondeterminism(DFAState d, java.util.Set<java.lang.Integer> nondeterministicAlts)
           
 void reportNondeterminismResolvedWithSemanticPredicate(DFAState d)
           
 void reportNonLLStarDecision(DFA dfa)
          Report that at least 2 alts have recursive constructs.
 void reportRecursionOverflow(DFAState d, NFAConfiguration recursionNFAConfiguration)
           
 void reset()
           
protected  void stripWildCardAlts(java.util.Set disabledAlts)
          Get the last disabled alt number and check in the grammar to see if that alt is a simple wildcard.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

dfa

public DFA dfa

statesWithSyntacticallyAmbiguousAltsSet

protected java.util.Set<DFAState> statesWithSyntacticallyAmbiguousAltsSet
Track all DFA states with nondeterministic alternatives. By reaching the same DFA state, a path through the NFA for some input is able to reach the same NFA state by starting at more than one alternative's left edge. Though, later, we may find that predicates resolve the issue, but track info anyway. Note that from the DFA state, you can ask for which alts are nondeterministic.


stateToSyntacticallyAmbiguousTokensRuleAltsMap

protected java.util.Map<DFAState,java.util.Set<java.lang.Integer>> stateToSyntacticallyAmbiguousTokensRuleAltsMap
Track just like stateToSyntacticallyAmbiguousAltsMap, but only for nondeterminisms that arise in the Tokens rule such as keyword vs ID rule. The state maps to the list of Tokens rule alts that are in conflict.


statesResolvedWithSemanticPredicatesSet

protected java.util.Set<DFAState> statesResolvedWithSemanticPredicatesSet
Was a syntactic ambiguity resolved with predicates? Any DFA state that predicts more than one alternative, must be resolved with predicates or it should be reported to the user.


stateToAltSetWithSemanticPredicatesMap

protected java.util.Map<DFAState,java.util.Map<java.lang.Integer,SemanticContext>> stateToAltSetWithSemanticPredicatesMap
Track the predicates for each alt per DFA state; more than one DFA state might have syntactically ambig alt prediction. Maps DFA state to another map, mapping alt number to a SemanticContext (pred(s) to execute to resolve syntactic ambiguity).


stateToIncompletelyCoveredAltsMap

protected java.util.Map<DFAState,java.util.Map<java.lang.Integer,java.util.Set<antlr.Token>>> stateToIncompletelyCoveredAltsMap
Tracks alts insufficiently covered. For example, p1||true gets reduced to true and so leaves whole alt uncovered. This maps DFA state to the set of alts


danglingStates

protected java.util.Set<DFAState> danglingStates
The set of states w/o emanating edges and w/o resolving sem preds.


altsWithProblem

protected java.util.Set<java.lang.Integer> altsWithProblem
The overall list of alts within the decision that have at least one conflicting input sequence.


nonLLStarDecision

protected boolean nonLLStarDecision
If decision with > 1 alt has recursion in > 1 alt, it's nonregular lookahead. The decision cannot be made with a DFA. the alts are stored in altsWithProblem.


stateToRecursionOverflowConfigurationsMap

protected MultiMap<java.lang.Integer,NFAConfiguration> stateToRecursionOverflowConfigurationsMap
Recursion is limited to a particular depth. If that limit is exceeded the proposed new NFAConfiguration is recorded for the associated DFA state.


timedOut

protected boolean timedOut
Did ANTLR have to terminate early on the analysis of this decision?


stateReachable

protected java.util.Map<java.lang.Integer,java.lang.Integer> stateReachable
Used to find paths through syntactically ambiguous DFA. If we've seen statement number before, what did we learn?


REACHABLE_BUSY

public static final java.lang.Integer REACHABLE_BUSY

REACHABLE_NO

public static final java.lang.Integer REACHABLE_NO

REACHABLE_YES

public static final java.lang.Integer REACHABLE_YES

statesVisitedAtInputDepth

protected java.util.Set<java.lang.String> statesVisitedAtInputDepth
Used while finding a path through an NFA whose edge labels match an input sequence. Tracks the input position we were at the last time at this node. If same input position, then we'd have reached same state without consuming input...probably an infinite loop. Stop. Set. The strings look like stateNumber_labelIndex.


statesVisitedDuringSampleSequence

protected java.util.Set<java.lang.Integer> statesVisitedDuringSampleSequence

verbose

public static boolean verbose
Constructor Detail

DecisionProbe

public DecisionProbe(DFA dfa)
Method Detail

getDescription

public java.lang.String getDescription()
Return a string like "3:22: ( A {;} | B )" that describes this decision.


isReduced

public boolean isReduced()

isCyclic

public boolean isCyclic()

isDeterministic

public boolean isDeterministic()
If no states are dead-ends, no alts are unreachable, there are no nondeterminisms unresolved by syn preds, all is ok with decision.


analysisTimedOut

public boolean analysisTimedOut()
Did the analysis complete it's work?


analysisOverflowed

public boolean analysisOverflowed()
Took too long to analyze a DFA


isNonLLStarDecision

public boolean isNonLLStarDecision()
Found recursion in > 1 alt


getNumberOfStates

public int getNumberOfStates()
How many states does the DFA predictor have?


getUnreachableAlts

public java.util.List<java.lang.Integer> getUnreachableAlts()
Get a list of all unreachable alternatives for this decision. There may be multiple alternatives with ambiguous input sequences, but this is the overall list of unreachable alternatives (either due to conflict resolution or alts w/o accept states).


getDanglingStates

public java.util.Set getDanglingStates()
return set of states w/o emanating edges and w/o resolving sem preds. These states come about because the analysis algorithm had to terminate early to avoid infinite recursion for example (due to left recursion perhaps).


getNonDeterministicAlts

public java.util.Set getNonDeterministicAlts()

getNonDeterministicAltsForState

public java.util.List getNonDeterministicAltsForState(DFAState targetState)
Return the sorted list of alts that conflict within a single state. Note that predicates may resolve the conflict.


getDFAStatesWithSyntacticallyAmbiguousAlts

public java.util.Set getDFAStatesWithSyntacticallyAmbiguousAlts()
Return all DFA states in this DFA that have NFA configurations that conflict. You must report a problem for each state in this set because each state represents a different input sequence.


getDisabledAlternatives

public java.util.Set getDisabledAlternatives(DFAState d)
Which alts were specifically turned off to resolve nondeterminisms? This is different than the unreachable alts. Disabled doesn't mean that the alternative is totally unreachable necessarily, it just means that for this DFA state, that alt is disabled. There may be other accept states for that alt that make an alt reachable.


removeRecursiveOverflowState

public void removeRecursiveOverflowState(DFAState d)
If a recursion overflow is resolve with predicates, then we need to shut off the warning that would be generated.


getSampleNonDeterministicInputSequence

public java.util.List<Label> getSampleNonDeterministicInputSequence(DFAState targetState)
Return a List

getInputSequenceDisplay

public java.lang.String getInputSequenceDisplay(java.util.List labels)
Given List

getNFAPathStatesForAlt

public java.util.List getNFAPathStatesForAlt(int firstAlt,
                                             int alt,
                                             java.util.List labels)
Given an alternative associated with a nondeterministic DFA state, find the path of NFA states associated with the labels sequence. Useful tracing where in the NFA, a single input sequence can be matched. For different alts, you should get different NFA paths. The first NFA state for all NFA paths will be the same: the starting NFA state of the first nondeterministic alt. Imagine (A|B|A|A): 5->9-A->o | 6->10-B->o | 7->11-A->o | 8->12-A->o There are 3 nondeterministic alts. The paths should be: 5 9 ... 5 6 7 11 ... 5 6 7 8 12 ... The NFA path matching the sample input sequence (labels) is computed using states 9, 11, and 12 rather than 5, 7, 8 because state 5, for example can get to all ambig paths. Must isolate for each alt (hence, the extra state beginning each alt in my NFA structures). Here, firstAlt=1.


getSemanticContextForAlt

public SemanticContext getSemanticContextForAlt(DFAState d,
                                                int alt)
Each state in the DFA represents a different input sequence for an alt of the decision. Given a DFA state, what is the semantic predicate context for a particular alt.


hasPredicate

public boolean hasPredicate()
At least one alt refs a sem or syn pred


getNondeterministicStatesResolvedWithSemanticPredicate

public java.util.Set getNondeterministicStatesResolvedWithSemanticPredicate()

getIncompletelyCoveredAlts

public java.util.Map<java.lang.Integer,java.util.Set<antlr.Token>> getIncompletelyCoveredAlts(DFAState d)
Return a list of alts whose predicate context was insufficient to resolve a nondeterminism for state d.


issueWarnings

public void issueWarnings()

stripWildCardAlts

protected void stripWildCardAlts(java.util.Set disabledAlts)
Get the last disabled alt number and check in the grammar to see if that alt is a simple wildcard. If so, treat like an else clause and don't emit the error. Strip out the last alt if it's wildcard.


issueRecursionWarnings

protected void issueRecursionWarnings()

reportDanglingState

public void reportDanglingState(DFAState d)
Report the fact that DFA state d is not a state resolved with predicates and yet it has no emanating edges. Usually this is a result of the closure/reach operations being unable to proceed


reportAnalysisTimeout

public void reportAnalysisTimeout()

reportNonLLStarDecision

public void reportNonLLStarDecision(DFA dfa)
Report that at least 2 alts have recursive constructs. There is no way to build a DFA so we terminated.


reportRecursionOverflow

public void reportRecursionOverflow(DFAState d,
                                    NFAConfiguration recursionNFAConfiguration)

reportNondeterminism

public void reportNondeterminism(DFAState d,
                                 java.util.Set<java.lang.Integer> nondeterministicAlts)

reportLexerRuleNondeterminism

public void reportLexerRuleNondeterminism(DFAState d,
                                          java.util.Set<java.lang.Integer> nondeterministicAlts)
Currently the analysis reports issues between token definitions, but we don't print out warnings in favor of just picking the first token definition found in the grammar ala lex/flex.


reportNondeterminismResolvedWithSemanticPredicate

public void reportNondeterminismResolvedWithSemanticPredicate(DFAState d)

reportAltPredicateContext

public void reportAltPredicateContext(DFAState d,
                                      java.util.Map altPredicateContext)
Report the list of predicates found for each alternative; copy the list because this set gets altered later by the method tryToResolveWithSemanticPredicates() while flagging NFA configurations in d as resolved.


reportIncompletelyCoveredAlts

public void reportIncompletelyCoveredAlts(DFAState d,
                                          java.util.Map<java.lang.Integer,java.util.Set<antlr.Token>> altToLocationsReachableWithoutPredicate)

reachesState

protected boolean reachesState(DFAState startState,
                               DFAState targetState,
                               java.util.Set states)
Given a start state and a target state, return true if start can reach target state. Also, compute the set of DFA states that are on a path from start to target; return in states parameter.


getDFAPathStatesToTarget

protected java.util.Set getDFAPathStatesToTarget(DFAState targetState)

getSampleInputSequenceUsingStateSet

protected void getSampleInputSequenceUsingStateSet(State startState,
                                                   State targetState,
                                                   java.util.Set states,
                                                   java.util.List<Label> labels)
Given a start state and a final state, find a list of edge labels between the two ignoring epsilon. Limit your scan to a set of states passed in. This is used to show a sample input sequence that is nondeterministic with respect to this decision. Return List

getNFAPath

protected boolean getNFAPath(NFAState s,
                             int labelIndex,
                             java.util.List labels,
                             java.util.List path)
Given a sample input sequence, you usually would like to know the path taken through the NFA. Return the list of NFA states visited while matching a list of labels. This cannot use the usual interpreter, which does a deterministic walk. We need to be able to take paths that are turned off during nondeterminism resolution. So, just do a depth-first walk traversing edges labeled with the current label. Return true if a path was found emanating from state s.


getStateLabelIndexKey

protected java.lang.String getStateLabelIndexKey(int s,
                                                 int i)

getTokenNameForTokensRuleAlt

public java.lang.String getTokenNameForTokensRuleAlt(int alt)
From an alt number associated with artificial Tokens rule, return the name of the token that is associated with that alt.


reset

public void reset()