Package pyre :: Package parsing :: Module Tokenizer :: Class Tokenizer
[show private | hide private]
[frames | no frames]

Type Tokenizer

object --+
         |
        Tokenizer


Method Summary
  __init__(self, file)
  fetch(self, scanner)
Return a token from the input stream
  locator(self)
  unfetch(self, token)
Put a token back into the token stream
  _newLine(self)
  _tokenize(self, scanner)
    Inherited from object
  __delattr__(...)
x.__delattr__('name') <==> del x.name
  __getattribute__(...)
x.__getattribute__('name') <==> x.name
  __hash__(x)
x.__hash__() <==> hash(x)
  __new__(T, S, ...)
T.__new__(S, ...) -> a new object with type S, a subtype of T
  __reduce__(...)
helper for pickle
  __reduce_ex__(...)
helper for pickle
  __repr__(x)
x.__repr__() <==> repr(x)
  __setattr__(...)
x.__setattr__('name', value) <==> x.name = value
  __str__(x)
x.__str__() <==> str(x)

Class Variable Summary
classobj EndOfFile = pyre.parsing.EndOfFile.EndOfFile
classobj TokenizationException = pyre.parsing.TokenizationException.TokenizationException
Diagnostic _info = <journal.Diagnostic.Diagnostic object at 0x2b7d5...

Method Details

fetch(self, scanner)

Return a token from the input stream

unfetch(self, token)

Put a token back into the token stream

Class Variable Details

_info

Type:
Diagnostic
Value:
<journal.Diagnostic.Diagnostic object at 0x2b7d50>                     

Generated by Epydoc 2.1 on Fri Apr 30 12:01:42 2004 http://epydoc.sf.net