encoding: utf-8
encoders/term.rb By Rob Aldred (robaldred.co.uk) Based on idea by Nathan Weizenbaum (nex-3.com) MIT License (www.opensource.org/licenses/mit-license.php)
A CodeRay encoder that outputs code highlighted for a color terminal. Check out robaldred.co.uk
CodeRay is a Ruby library for syntax highlighting.
I try to make CodeRay easy to use and intuitive, but at the same time fully featured, complete, fast and efficient.
See README.
It consists mainly of
the main engine: CodeRay (Scanners::Scanner, Tokens/TokenStream, Encoders::Encoder), PluginHost
the scanners in CodeRay::Scanners
the encoders in CodeRay::Encoders
Here’s a fancy graphic to light up this gray docu:
See CodeRay, Encoders, Scanners, Tokens.
Remember you need RubyGems to use CodeRay, unless you have it in your load path. Run Ruby with -rubygems option if required.
require 'coderay' print CodeRay.scan('puts "Hello, world!"', :ruby).html # prints something like this: puts <span class="s">"Hello, world!"</span>
require 'coderay' print CodeRay.scan(File.read('ruby.h'), :c).div print CodeRay.scan_file('ruby.h').html.div
You can include this div in your page. The used CSS styles can be printed with
% coderay_stylesheet
If you are one of the hasty (or lazy, or extremely curious) people, just run this file:
% ruby -rubygems /path/to/coderay/coderay.rb > example.html
and look at the file it created in your browser.
The CodeRay module provides convenience methods for the engine.
The lang and format arguments select Scanner and Encoder to use. These are simply lower-case symbols, like :python or :html.
All methods take an optional hash as last parameter, options, that is send to the Encoder / Scanner.
Input and language are always sorted in this order: code, lang. (This is in alphabetical order, if you need a mnemonic ;)
You should be able to highlight everything you want just using these methods; so there is no need to dive into CodeRay’s deep class hierarchy.
The examples in the demo directory demonstrate common cases using this interface.
Read this to get a general view what CodeRay provides.
Scanning means analysing an input string, splitting it up into Tokens. Each Token knows about what type it is: string, comment, class name, etc. Each +lang+ (language) has its own Scanner; for example, <tt>:ruby</tt> code is handled by CodeRay::Scanners::Ruby.
CodeRay.scan | Scan a string in a given language into Tokens. This is the most common method to use. |
CodeRay.scan_file | Scan a file and guess the language using FileType. |
The Tokens object you get from these methods can encode itself; see Tokens.
Encoding means compiling Tokens into an output. This can be colored HTML or LaTeX, a textual statistic or just the number of non-whitespace tokens.
Each Encoder provides output in a specific format, so you select Encoders via formats like :html or :statistic.
CodeRay.encode | Scan and encode a string in a given language. |
CodeRay.encode_tokens | Encode the given tokens. |
CodeRay.encode_file | Scan a file, guess the language using FileType and encode it. |
Streaming saves RAM by running Scanner and Encoder in some sort of pipe mode; see TokenStream.
CodeRay.scan_stream | Scan in stream mode. |
CodeRay.encode | Highlight a string with a given input and output format. |
You can use an Encoder instance to highlight multiple inputs. This way, the setup for this Encoder must only be done once.
CodeRay.encoder | Create an Encoder instance with format and options. |
CodeRay.scanner | Create an Scanner instance for lang, with ’’ as default code. |
To make use of CodeRay.scanner, use CodeRay::Scanner::code=.
The scanning methods provide more flexibility; we recommend to use these.
If you want to re-use scanners and encoders (because that is faster), see CodeRay::Duo for the most convenient (and recommended) interface.
Version: Major.Minor.Teeny[.Revision] Major: 0 for pre-stable, 1 for stable Minor: feature milestone Teeny: development state, 0 for pre-release Revision: Subversion Revision number (generated on rake gem:make)
Encode a string.
This scans code with the the Scanner for lang and then encodes it with the Encoder for format. options will be passed to the Encoder.
See CodeRay::Encoder.encode
# File lib/coderay.rb, line 209 209: def encode code, lang, format, options = {} 210: encoder(format, options).encode code, lang, options 211: end
Encodes filename (a path to a code file) with the Scanner for lang.
See CodeRay.scan_file. Notice that the second argument is the output format, not the input language.
Example:
require 'coderay' page = CodeRay.encode_file 'some_c_code.c', :html
# File lib/coderay.rb, line 244 244: def encode_file filename, format, options = {} 245: tokens = scan_file filename, :auto, get_scanner_options(options) 246: encode_tokens tokens, format, options 247: end
Encode a string in Streaming mode.
This starts scanning code with the the Scanner for lang while encodes the output with the Encoder for format. options will be passed to the Encoder.
See CodeRay::Encoder.encode_stream
# File lib/coderay.rb, line 198 198: def encode_stream code, lang, format, options = {} 199: encoder(format, options).encode_stream code, lang, options 200: end
Encode pre-scanned Tokens. Use this together with CodeRay.scan:
require 'coderay' # Highlight a short Ruby code example in a HTML span tokens = CodeRay.scan '1 + 2', :ruby puts CodeRay.encode_tokens(tokens, :span)
# File lib/coderay.rb, line 232 232: def encode_tokens tokens, format, options = {} 233: encoder(format, options).encode_tokens tokens, options 234: end
Finds the Encoder class for format and creates an instance, passing options to it.
Example:
require 'coderay' stats = CodeRay.encoder(:statistic) stats.encode("puts 17 + 4\n", :ruby) puts '%d out of %d tokens have the kind :integer.' % [ stats.type_stats[:integer].count, stats.real_token_count ] #-> 2 out of 4 tokens have the kind :integer.
# File lib/coderay.rb, line 273 273: def encoder format, options = {} 274: Encoders[format].new options 275: end
Extract the options for the scanner from the options hash.
Returns an empty Hash if :scanner_options is not set.
This is used if a method like CodeRay.encode has to provide options for Encoder and scanner.
# File lib/coderay.rb, line 291 291: def get_scanner_options options 292: options.fetch :scanner_options, {} 293: end
Highlight a string into a HTML
CSS styles use classes, so you have to include a stylesheet in your output.
See encode.
# File lib/coderay.rb, line 219 219: def highlight code, lang, options = { :css => :class }, format = :div 220: encode code, lang, format, options 221: end
Highlight a file into a HTML
CSS styles use classes, so you have to include a stylesheet in your output.
See encode.
# File lib/coderay.rb, line 255 255: def highlight_file filename, options = { :css => :class }, format = :div 256: encode_file filename, format, options 257: end
# File lib/coderay/scanners/ruby/patterns.rb, line 220 220: def initialize kind, interpreted, delim, heredoc = false 221: if heredoc 222: pattern = HEREDOC_PATTERN[ [delim, interpreted, heredoc == :indented] ] 223: delim = nil 224: else 225: pattern = STRING_PATTERN[ [delim, interpreted] ] 226: if paren = CLOSING_PAREN[delim] 227: delim, paren = paren, delim 228: paren_depth = 1 229: end 230: end 231: super kind, interpreted, delim, heredoc, paren, paren_depth, pattern, :initial 232: end
Convenience method for plugin loading. The syntax used is:
CodeRay.require_plugin '<Host ID>/<Plugin ID>'
Returns the loaded plugin.
# File lib/coderay/helpers/plugin.rb, line 341 341: def self.require_plugin path 342: host_id, plugin_id = path.split '/', 2 343: host = PluginHost.host_by_id(host_id) 344: raise PluginHost::HostNotFound, 345: "No host for #{host_id.inspect} found." unless host 346: host.load plugin_id 347: end
Scans the given code (a String) with the Scanner for lang.
This is a simple way to use CodeRay. Example:
require 'coderay' page = CodeRay.scan("puts 'Hello, world!'", :ruby).html
See also demo/demo_simple.
# File lib/coderay.rb, line 156 156: def scan code, lang, options = {}, &block 157: scanner = Scanners[lang].new code, options, &block 158: scanner.tokenize 159: end
Scans filename (a path to a code file) with the Scanner for lang.
If lang is :auto or omitted, the CodeRay::FileType module is used to determine it. If it cannot find out what type it is, it uses CodeRay::Scanners::Plaintext.
Calls CodeRay.scan.
Example:
require 'coderay' page = CodeRay.scan_file('some_c_code.c').html
# File lib/coderay.rb, line 172 172: def scan_file filename, lang = :auto, options = {}, &block 173: file = IO.read filename 174: if lang == :auto 175: require 'coderay/helpers/file_type' 176: lang = FileType.fetch filename, :plaintext, true 177: end 178: scan file, lang, options = {}, &block 179: end
Scan the code (a string) with the scanner for lang.
Calls scan.
See CodeRay.scan.
# File lib/coderay.rb, line 186 186: def scan_stream code, lang, options = {}, &block 187: options[:stream] = true 188: scan code, lang, options, &block 189: end
# File lib/coderay/scanners/python.rb, line 97 97: def scan_tokens tokens, options 98: 99: state = :initial 100: string_delimiter = nil 101: string_raw = false 102: import_clause = class_name_follows = last_token_dot = false 103: unicode = string.respond_to?(:encoding) && string.encoding.name == 'UTF-8' 104: from_import_state = [] 105: 106: until eos? 107: 108: kind = nil 109: match = nil 110: 111: if state == :string 112: if scan(STRING_DELIMITER_REGEXP[string_delimiter]) 113: tokens << [matched, :delimiter] 114: tokens << [:close, :string] 115: state = :initial 116: next 117: elsif string_delimiter.size == 3 && scan(/\n/) 118: kind = :content 119: elsif scan(STRING_CONTENT_REGEXP[string_delimiter]) 120: kind = :content 121: elsif !string_raw && scan(/ \\ #{ESCAPE} /x) 122: kind = :char 123: elsif scan(/ \\ #{UNICODE_ESCAPE} /x) 124: kind = :char 125: elsif scan(/ \\ . /) 126: kind = :content 127: elsif scan(/ \\ | $ /) 128: tokens << [:close, :string] 129: kind = :error 130: state = :initial 131: else 132: raise_inspect "else case \" reached; %p not handled." % peek(1), tokens, state 133: end 134: 135: elsif match = scan(/ [ \t]+ | \\\n /) 136: tokens << [match, :space] 137: next 138: 139: elsif match = scan(/\n/) 140: tokens << [match, :space] 141: state = :initial if state == :include_expected 142: next 143: 144: elsif match = scan(/ \# [^\n]* /x) 145: tokens << [match, :comment] 146: next 147: 148: elsif state == :initial 149: 150: if scan(/#{OPERATOR}/) 151: kind = :operator 152: 153: elsif match = scan(/(u?r?|b)?("""|"|'''|')/) 154: tokens << [:open, :string] 155: string_delimiter = self[2] 156: string_raw = false 157: modifiers = self[1] 158: unless modifiers.empty? 159: string_raw = !!modifiers.index(rr) 160: tokens << [modifiers, :modifier] 161: match = string_delimiter 162: end 163: state = :string 164: kind = :delimiter 165: 166: # TODO: backticks 167: 168: elsif match = scan(unicode ? /#{NAME}/o : /#{NAME}/) 169: kind = IDENT_KIND[match] 170: # TODO: keyword arguments 171: kind = :ident if last_token_dot 172: if kind == :old_keyword 173: kind = check(/\(/) ? :ident : :keyword 174: elsif kind == :predefined && check(/ *=/) 175: kind = :ident 176: elsif kind == :keyword 177: state = DEF_NEW_STATE[match] 178: from_import_state << match.to_sym if state == :include_expected 179: end 180: 181: elsif scan(/@[a-zA-Z0-9_.]+[lL]?/) 182: kind = :decorator 183: 184: elsif scan(/0[xX][0-9A-Fa-f]+[lL]?/) 185: kind = :hex 186: 187: elsif scan(/0[bB][01]+[lL]?/) 188: kind = :bin 189: 190: elsif match = scan(/(?:\d*\.\d+|\d+\.\d*)(?:[eE][+-]?\d+)?|\d+[eE][+-]?\d+/) 191: kind = :float 192: if scan(/[jJ]/) 193: match << matched 194: kind = :imaginary 195: end 196: 197: elsif scan(/0[oO][0-7]+|0[0-7]+(?![89.eE])[lL]?/) 198: kind = :oct 199: 200: elsif match = scan(/\d+([lL])?/) 201: kind = :integer 202: if self[1] == nil && scan(/[jJ]/) 203: match << matched 204: kind = :imaginary 205: end 206: 207: else 208: getch 209: kind = :error 210: 211: end 212: 213: elsif state == :def_expected 214: state = :initial 215: if match = scan(unicode ? /#{NAME}/o : /#{NAME}/) 216: kind = :method 217: else 218: next 219: end 220: 221: elsif state == :class_expected 222: state = :initial 223: if match = scan(unicode ? /#{NAME}/o : /#{NAME}/) 224: kind = :class 225: else 226: next 227: end 228: 229: elsif state == :include_expected 230: if match = scan(unicode ? /#{DESCRIPTOR}/o : /#{DESCRIPTOR}/) 231: kind = :include 232: if match == 'as' 233: kind = :keyword 234: from_import_state << :as 235: elsif from_import_state.first == :from && match == 'import' 236: kind = :keyword 237: from_import_state << :import 238: elsif from_import_state.last == :as 239: # kind = match[0,1][unicode ? /[[:upper:]]/u : /[[:upper:]]/] ? :class : :method 240: kind = :ident 241: from_import_state.pop 242: elsif IDENT_KIND[match] == :keyword 243: unscan 244: match = nil 245: state = :initial 246: next 247: end 248: elsif match = scan(/,/) 249: from_import_state.pop if from_import_state.last == :as 250: kind = :operator 251: else 252: from_import_state = [] 253: state = :initial 254: next 255: end 256: 257: else 258: raise_inspect 'Unknown state', tokens, state 259: 260: end 261: 262: match ||= matched 263: if $CODERAY_DEBUG and not kind 264: raise_inspect 'Error token %p in line %d' % 265: [[match, kind], line], tokens, state 266: end 267: raise_inspect 'Empty token', tokens, state unless match 268: 269: last_token_dot = match == '.' 270: 271: tokens << [match, kind] 272: 273: end 274: 275: if state == :string 276: tokens << [:close, :string] 277: end 278: 279: tokens 280: end
Disabled; run with --debug to generate this.
Generated with the Darkfish Rdoc Generator 1.1.6.