StringScanner
The base class for all Scanners.
It is a subclass of Ruby’s great StringScanner, which makes it easy to access the scanning methods inside.
It is also Enumerable, so you can use it like an Array of Tokens:
require 'coderay' c_scanner = CodeRay::Scanners[:c].new "if (*p == '{') nest++;" for text, kind in c_scanner puts text if kind == :operator end # prints: (*==)++;
OK, this is a very simple example :) You can also use map, any?, find and even sort_by, if you want.
Raised if a Scanner fails while scanning
The default options for all scanner classes.
Define @default_options for subclasses.
# File lib/coderay/scanner.rb, line 85 85: def file_extension extension = nil 86: if extension 87: @file_extension = extension.to_s 88: else 89: @file_extension ||= plugin_id.to_s 90: end 91: end
Create a new Scanner.
code is the input String and is handled by the superclass StringScanner.
options is a Hash with Symbols as keys. It is merged with the default options of the class (you can overwrite default options here.)
block is the callback for streamed highlighting.
If you set :stream to true in the options, the Scanner uses a TokenStream with the block as callback to handle the tokens.
Else, a Tokens object is used.
# File lib/coderay/scanner.rb, line 119 119: def initialize code='', options = {}, &block 120: raise "I am only the basic Scanner class. I can't scan " "anything. :( Use my subclasses." if self.class == Scanner 121: 122: @options = self.class::DEFAULT_OPTIONS.merge options 123: 124: super Scanner.normify(code) 125: 126: @tokens = options[:tokens] 127: if @options[:stream] 128: warn "warning in CodeRay::Scanner.new: :stream is set, " "but no block was given" unless block_given? 129: raise NotStreamableError, self unless kind_of? Streamable 130: @tokens ||= TokenStream.new(&block) 131: else 132: warn "warning in CodeRay::Scanner.new: Block given, " "but :stream is #{@options[:stream]}" if block_given? 133: @tokens ||= Tokens.new 134: end 135: @tokens.scanner = self 136: 137: setup 138: end
# File lib/coderay/scanner.rb, line 69 69: def normify code 70: code = code.to_s 71: if code.respond_to? :force_encoding 72: debug, $DEBUG = $DEBUG, false 73: begin 74: code.force_encoding 'utf-8' 75: code[/\z/] # raises an ArgumentError when code contains a non-UTF-8 char 76: rescue ArgumentError 77: code.force_encoding 'binary' 78: ensure 79: $DEBUG = debug 80: end 81: end 82: code.to_unix 83: end
Returns if the Scanner can be used in streaming mode.
# File lib/coderay/scanner.rb, line 65 65: def streamable? 66: is_a? Streamable 67: end
# File lib/coderay/scanner.rb, line 202 202: def column pos = self.pos 203: return 0 if pos <= 0 204: string = string() 205: if string.respond_to?(:bytesize) && (defined?(@bin_string) || string.bytesize != string.size) 206: @bin_string ||= string.dup.force_encoding('binary') 207: string = @bin_string 208: end 209: pos - (string.rindex(\n\, pos) || 0) 210: end
Traverses the tokens.
# File lib/coderay/scanner.rb, line 187 187: def each &block 188: raise ArgumentError, 189: 'Cannot traverse TokenStream.' if @options[:stream] 190: tokens.each(&block) 191: end
Returns the Plugin ID for this scanner.
# File lib/coderay/scanner.rb, line 159 159: def lang 160: self.class.plugin_id 161: end
The current line position of the scanner.
Beware, this is implemented inefficiently. It should be used for debugging only.
# File lib/coderay/scanner.rb, line 198 198: def line 199: string[0..pos].count("\n") + 1 200: end
# File lib/coderay/scanner.rb, line 212 212: def marshal_dump 213: @options 214: end
# File lib/coderay/scanner.rb, line 216 216: def marshal_load options 217: @options = options 218: end
# File lib/coderay/scanner.rb, line 143 143: def reset 144: super 145: reset_instance 146: end
Whether the scanner is in streaming mode.
# File lib/coderay/scanner.rb, line 182 182: def streaming? 183: !!@options[:stream] 184: end
# File lib/coderay/scanner.rb, line 148 148: def string= code 149: code = Scanner.normify(code) 150: super code 151: reset_instance 152: end
Scans the code and returns all tokens in a Tokens object.
# File lib/coderay/scanner.rb, line 164 164: def tokenize new_string=nil, options = {} 165: options = @options.merge(options) 166: self.string = new_string if new_string 167: @cached_tokens = 168: if @options[:stream] # :stream must have been set already 169: reset unless new_string 170: scan_tokens @tokens, options 171: @tokens 172: else 173: scan_tokens @tokens, options 174: end 175: end
Scanner error with additional status information
# File lib/coderay/scanner.rb, line 247 247: def raise_inspect msg, tokens, state = 'No state given!', ambit = 30 248: raise ScanError, ***ERROR in %s: %s (after %d tokens)tokens:%scurrent line: %d column: %d pos: %dmatched: %p state: %pbol? = %p, eos? = %psurrounding code:%p ~~ %p***ERROR*** % [ 249: File.basename(caller[0]), 250: msg, 251: tokens.size, 252: tokens.last(10).map { |t| t.inspect }.join("\n"), 253: line, column, pos, 254: matched, state, bol?, eos?, 255: string[pos - ambit, ambit], 256: string[pos, ambit], 257: ] 258: end
# File lib/coderay/scanner.rb, line 240 240: def reset_instance 241: @tokens.clear unless @options[:keep_tokens] 242: @cached_tokens = nil 243: @bin_string = nil if defined? @bin_string 244: end
This is the central method, and commonly the only one a subclass implements.
Subclasses must implement this method; it must return tokens and must only use Tokens#<< for storing scanned tokens!
# File lib/coderay/scanner.rb, line 235 235: def scan_tokens tokens, options 236: raise NotImplementedError, 237: "#{self.class}#scan_tokens not implemented." 238: end
Disabled; run with --debug to generate this.
Generated with the Darkfish Rdoc Generator 1.1.6.