Traversing tokens using ParserRuleContext in listener - ANTLR4
While retrying with tokens using the Listener, I would like to know how to use the ParserRuleContext to look in the next token or the next few tokens in the token stream?
In the code below, I am trying to view all tokens after the current token before EOF:
@Override
public void enterSemicolon(JavaParser.SemicolonContext ctx) {
Token tok, semiColon = ctx.getStart();
int currentIndex = semiColon.getStartIndex();
int reqInd = currentIndex+1;
TokenSource tokSrc= semiColon.getTokenSource();
CharStream srcStream = semiColon.getInputStream();
srcStream.seek(currentIndex);
while(true){
tok = tokSrc.nextToken() ;
System.out.println(tok);
if(tok.getText()=="<EOF>"){break;}
srcStream.seek(reqInd++);
}
}
But I get the result:
.
.
.
.
.
[@-1,131:130='',<-1>,13:0]
[@-1,132:131='',<-1>,13:0]
[@-1,133:132='',<-1>,13:0]
[@-1,134:133='',<-1>,13:0]
[@-1,135:134='',<-1>,13:0]
[@-1,136:135='',<-1>,13:0]
[@-1,137:136='',<-1>,13:0]
[@-1,138:137='',<-1>,13:0]
[@-1,139:138='',<-1>,13:0]
[@-1,140:139='',<-1>,13:0]
[@-1,141:140='',<-1>,13:0]
[@-1,142:141='',<-1>,13:0]
[@-1,143:142='',<-1>,13:0]
[@-1,144:143='',<-1>,13:0]
[@-1,145:144='',<-1>,13:0]
[@-1,146:145='',<-1>,13:0]
[@-1,147:146='',<-1>,13:0]
[@-1,148:147='',<-1>,13:0]
[@-1,149:148='',<-1>,13:0]
[@-1,150:149='',<-1>,13:0]
[@-1,151:150='',<-1>,13:0]
[@-1,152:151='',<-1>,13:0]
[@-1,153:152='',<-1>,13:0]
[@-1,154:153='',<-1>,13:0]
[@-1,155:154='',<-1>,13:0]
[@-1,156:155='',<-1>,13:0]
[@-1,157:156='',<-1>,13:0]
[@-1,158:157='',<-1>,13:0]
[@-1,159:158='',<-1>,13:0]
[@-1,160:159='',<-1>,13:0]
[@-1,161:160='<EOF>',<-1>,13:0]
[@-1,137:136='',<-1>,13:0]
[@-1,138:137='',<-1>,13:0]
[@-1,139:138='',<-1>,13:0]
[@-1,140:139='',<-1>,13:0]
[@-1,141:140='',<-1>,13:0]
[@-1,142:141='',<-1>,13:0]
[@-1,143:142='',<-1>,13:0]
[@-1,144:143='',<-1>,13:0]
[@-1,145:144='',<-1>,13:0]
[@-1,146:145='',<-1>,13:0]
[@-1,147:146='',<-1>,13:0]
[@-1,148:147='',<-1>,13:0]
[@-1,149:148='',<-1>,13:0]
[@-1,150:149='',<-1>,13:0]
[@-1,151:150='',<-1>,13:0]
[@-1,152:151='',<-1>,13:0]
[@-1,153:152='',<-1>,13:0]
[@-1,154:153='',<-1>,13:0]
[@-1,155:154='',<-1>,13:0]
[@-1,156:155='',<-1>,13:0]
[@-1,157:156='',<-1>,13:0]
[@-1,158:157='',<-1>,13:0]
[@-1,159:158='',<-1>,13:0]
[@-1,160:159='',<-1>,13:0]
[@-1,161:160='<EOF>',<-1>,13:0]
.
.
.
.
We can see that although I can go through all the tokens prior to EOF, I cannot get the actual content or type of tokens. I would like to know if there is a neat way to do this with listen listener.
source to share
It's hard to be sure but
tok = tokSrc.nextToken() ;
seems to repeat the lexer starting at the intended proper marker boundary, but without resetting the lexer. Lexer throw errors can explain the observed behavior.
However, the best approach would be to simply restore the existing Token stream:
public class Walker implements YourJavaListener {
CommonTokenStream tokens;
public Walker(JavaParser parser) {
tokens = (CommonTokenStream) parser.getTokenStream()
}
then enter the stream to get the markers you want:
@Override
public void enterSemicolon(JavaParser.SemicolonContext ctx) {
TerminalNode semi = ctx.semicolon(); // adjust as needed for your impl.
Token tok = semi.getSymbol();
int idx = tok.getTokenIndex();
while(tok.getType() != IntStream.EOF) {
System.out.println(tok);
tok = tokens.get(idx++);
}
}
A completely different approach that might serve your ultimate goal is to get a limited set of tokens directly from the parent context:
ParserRuleContext pctx = ctx.getParent();
List<TerminalNode> nodes = pctx.getTokens(pctx.getStart(), pctx.getStop());
source to share