Class TokenRewriteStream

java.lang.Object
org.antlr.runtime.BufferedTokenStream
org.antlr.runtime.CommonTokenStream
org.antlr.runtime.TokenRewriteStream
All Implemented Interfaces:
IntStream, TokenStream
public class TokenRewriteStream extends CommonTokenStream
Useful for dumping out the input stream after doing some augmentation or other manipulations. You can insert stuff, replace, and delete chunks. Note that the operations are done lazily--only if you convert the buffer to a String. This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the toString() method(s) check to see if there is an operation at the current index. If so, the operation is done and then normal String rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape. :) Since the operations are done lazily at toString-time, operations do not screw up the token index values. That is, an insert operation at token index i does not change the index values for tokens i+1..n-1. Because operations never actually alter the buffer, you may always get the original token stream back without undoing anything. Since the instructions are queued up, you can easily simulate transactions and roll back any changes if there is an error just by removing instructions. For example, CharStream input = new ANTLRFileStream("input"); TLexer lex = new TLexer(input); TokenRewriteStream tokens = new TokenRewriteStream(lex); T parser = new T(tokens); parser.startRule(); Then in the rules, you can execute Token t,u; ... input.insertAfter(t, "text to put after t");} input.insertAfter(u, "text after u");} System.out.println(tokens.toString()); Actually, you have to cast the 'input' to a TokenRewriteStream. :( You can also have multiple "instruction streams" and get multiple rewrites from a single pass over the input. Just name the instruction streams and use that name again when printing the buffer. This could be useful for generating a C file and also its header file--all from the same buffer: tokens.insertAfter("pass1", t, "text to put after t");} tokens.insertAfter("pass2", u, "text after u");} System.out.println(tokens.toString("pass1")); System.out.println(tokens.toString("pass2")); If you don't use named rewrite streams, a "default" stream is used as the first example shows.

  • Field Details

  • Constructor Details

    • TokenRewriteStream

      public TokenRewriteStream()

    • TokenRewriteStream

      public TokenRewriteStream(TokenSource tokenSource)

    • TokenRewriteStream

      public TokenRewriteStream(TokenSource tokenSource, int channel)

  • Method Details

    • init

      protected void init()

    • rollback

      public void rollback(int instructionIndex)

    • rollback

      public void rollback(String programName, int instructionIndex)
      Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream. UNTESTED!

    • deleteProgram

      public void deleteProgram()

    • deleteProgram

      public void deleteProgram(String programName)
      Reset the program so that no instructions exist

    • insertAfter

      public void insertAfter(Token t, Object text)

    • insertAfter

      public void insertAfter(int index, Object text)

    • insertAfter

      public void insertAfter(String programName, Token t, Object text)

    • insertAfter

      public void insertAfter(String programName, int index, Object text)

    • insertBefore

      public void insertBefore(Token t, Object text)

    • insertBefore

      public void insertBefore(int index, Object text)

    • insertBefore

      public void insertBefore(String programName, Token t, Object text)

    • insertBefore

      public void insertBefore(String programName, int index, Object text)

    • replace

      public void replace(int index, Object text)

    • replace

      public void replace(int from, int to, Object text)

    • replace

      public void replace(Token indexT, Object text)

    • replace

      public void replace(Token from, Token to, Object text)

    • replace

      public void replace(String programName, int from, int to, Object text)

    • replace

      public void replace(String programName, Token from, Token to, Object text)

    • delete

      public void delete(int index)

    • delete

      public void delete(int from, int to)

    • delete

      public void delete(Token indexT)

    • delete

      public void delete(Token from, Token to)

    • delete

      public void delete(String programName, int from, int to)

    • delete

      public void delete(String programName, Token from, Token to)

    • getLastRewriteTokenIndex

      public int getLastRewriteTokenIndex()

    • getLastRewriteTokenIndex

      protected int getLastRewriteTokenIndex(String programName)

    • setLastRewriteTokenIndex

      protected void setLastRewriteTokenIndex(String programName, int i)

    • getProgram

      protected List<TokenRewriteStream.RewriteOperation> getProgram(String name)

    • initializeProgram

      private List<TokenRewriteStream.RewriteOperation> initializeProgram(String name)

    • toOriginalString

      public String toOriginalString()

    • toOriginalString

      public String toOriginalString(int start, int end)

    • toString

      public String toString()
      Description copied from class: BufferedTokenStream
      Grab *all* tokens from stream and return string
      Overrides:
      toString in class BufferedTokenStream

    • toString

      public String toString(String programName)

    • toString

      public String toString(int start, int end)
      Description copied from interface: TokenStream
      Return the text of all tokens from start to stop, inclusive. If the stream does not buffer all the tokens then it can just return "" or null; Users should not access $ruleLabel.text in an action of course in that case.
      Specified by:
      toString in interface TokenStream
      Overrides:
      toString in class BufferedTokenStream

    • toString

      public String toString(String programName, int start, int end)

    • reduceToSingleOperationPerIndex

      protected Map<Integer, ? extends TokenRewriteStream.RewriteOperation> reduceToSingleOperationPerIndex(List<? extends TokenRewriteStream.RewriteOperation> rewrites)
      We need to combine operations and report invalid operations (like overlapping replaces that are not completed nested). Inserts to same index need to be combined etc... Here are the cases: I.i.u I.j.v leave alone, nonoverlapping I.i.u I.i.v combine: Iivu R.i-j.u R.x-y.v | i-j in x-y delete first R R.i-j.u R.i-j.v delete first R R.i-j.u R.x-y.v | x-y in i-j ERROR R.i-j.u R.x-y.v | boundaries overlap ERROR Delete special case of replace (text==null): D.i-j.u D.x-y.v | boundaries overlap combine to max(min)..max(right) I.i.u R.x-y.v | i in (x+1)-y delete I (since insert before we're not deleting i) I.i.u R.x-y.v | i not in (x+1)-y leave alone, nonoverlapping R.x-y.v I.i.u | i in x-y ERROR R.x-y.v I.x.u R.x-y.uv (combine, delete I) R.x-y.v I.i.u | i not in x-y leave alone, nonoverlapping I.i.u = insert u before op @ index i R.x-y.u = replace x-y indexed tokens with u First we need to examine replaces. For any replace op: 1. wipe out any insertions before op within that range. 2. Drop any replace op before that is contained completely within that range. 3. Throw exception upon boundary overlap with any previous replace. Then we can deal with inserts: 1. for any inserts to same index, combine even if not adjacent. 2. for any prior replace with same left boundary, combine this insert with replace and delete this replace. 3. throw exception if index in same range as previous replace Don't actually delete; make op null in list. Easier to walk list. Later we can throw as we add to index → op map. Note that I.2 R.2-2 will wipe out I.2 even though, technically, the inserted stuff would be before the replace range. But, if you add tokens in front of a method body '{' and then delete the method body, I think the stuff before the '{' you added should disappear too. Return a map from token index to operation.

    • catOpText

      protected String catOpText(Object a, Object b)

    • getKindOfOps

      protected <T extends TokenRewriteStream.RewriteOperation> List<? extends T> getKindOfOps(List<? extends TokenRewriteStream.RewriteOperation> rewrites, Class<T> kind)

    • getKindOfOps

      protected <T extends TokenRewriteStream.RewriteOperation> List<? extends T> getKindOfOps(List<? extends TokenRewriteStream.RewriteOperation> rewrites, Class<T> kind, int before)
      Get all operations before an index of a particular kind

    • toDebugString

      public String toDebugString()

    • toDebugString

      public String toDebugString(int start, int end)