lexer.coffee

跳转到… +

browser.coffee cake.coffee coffee-script.coffee command.coffee grammar.coffee helpers.coffee index.coffee lexer.coffee nodes.coffee optparse.coffee register.coffee repl.coffee rewriter.coffee scope.litcoffee sourcemap.litcoffee

lexer.coffee
¶

CoffeeScript 词法分析器。使用一系列 token 匹配正则表达式来尝试匹配源代码的开头。当找到匹配项时，会生成一个 token，我们消耗匹配项，然后重新开始。token 的形式为
```
[tag, value, locationData]
```
其中 locationData 为 {first_line, first_column, last_line, last_column}，这是一种可以直接馈送到 Jison 的格式。这些由 jison 在 coffee-script.coffee 中定义的 parser.lexer 函数中读取。
```
{Rewriter, INVERSES} = require './rewriter'
```

导入我们需要的帮助程序。

{count, starts, compact, repeat, invertLiterate,
locationDataToString,  throwSyntaxError} = require './helpers'

¶

词法分析器类
¶
¶

词法分析器类读取 CoffeeScript 流并将其分成标记的 token。通过将一些额外的智能推入词法分析器，避免了语法中的一些潜在歧义。
```
exports.Lexer = class Lexer
```

tokenize 是词法分析器的主要方法。通过尝试一次匹配一个 token 来扫描，使用一个固定在剩余代码开头的正则表达式，或一个自定义的递归 token 匹配方法（用于插值）。当记录下一个 token 时，我们将在代码中向前移动到 token 之后，然后重新开始。

每个 token 化方法负责返回它已消耗的字符数。

在返回 token 流之前，将其通过重写器。

  tokenize: (code, opts = {}) ->
    @literate   = opts.literate  # Are we lexing literate CoffeeScript?
    @indent     = 0              # The current indentation level.
    @baseIndent = 0              # The overall minimum indentation level
    @indebt     = 0              # The over-indentation at the current level.
    @outdebt    = 0              # The under-outdentation at the current level.
    @indents    = []             # The stack of all current indentation levels.
    @ends       = []             # The stack for pairing up tokens.
    @tokens     = []             # Stream of parsed tokens in the form `['TYPE', value, location data]`.
    @seenFor    = no             # Used to recognize FORIN, FOROF and FORFROM tokens.
    @seenImport = no             # Used to recognize IMPORT FROM? AS? tokens.
    @seenExport = no             # Used to recognize EXPORT FROM? AS? tokens.
    @importSpecifierList = no    # Used to identify when in an IMPORT {...} FROM? ...
    @exportSpecifierList = no    # Used to identify when in an EXPORT {...} FROM? ...

    @chunkLine =
      opts.line or 0             # The start line for the current @chunk.
    @chunkColumn =
      opts.column or 0           # The start column of the current @chunk.
    code = @clean code           # The stripped, cleaned original source code.

在每个位置，运行此尝试匹配列表，如果任何匹配成功，则短路。它们的顺序决定了优先级：@literalToken 是后备万能符。

    i = 0
    while @chunk = code[i..]
      consumed = \
           @identifierToken() or
           @commentToken()    or
           @whitespaceToken() or
           @lineToken()       or
           @stringToken()     or
           @numberToken()     or
           @regexToken()      or
           @jsToken()         or
           @literalToken()

更新位置

      [@chunkLine, @chunkColumn] = @getLineAndColumnFromChunk consumed

      i += consumed

      return {@tokens, index: i} if opts.untilBalanced and @ends.length is 0

    @closeIndentation()
    @error "missing #{end.tag}", end.origin[2] if end = @ends.pop()
    return @tokens if opts.rewrite is off
    (new Rewriter).rewrite @tokens

预处理代码以删除前导和尾随空格、回车等。如果我们正在对文字 CoffeeScript 进行词法分析，则通过删除所有未缩进至少四个空格或一个制表符的行来剥离外部 Markdown。

  clean: (code) ->
    code = code.slice(1) if code.charCodeAt(0) is BOM
    code = code.replace(/\r/g, '').replace TRAILING_SPACES, ''
    if WHITESPACE.test code
      code = "\n#{code}"
      @chunkLine--
    code = invertLiterate code if @literate
    code

¶

词法分析器
¶
¶

匹配识别字面量：变量、关键字、方法名称等。检查以确保 JavaScript 保留字未被用作标识符。由于 CoffeeScript 保留了一些在 JavaScript 中允许使用的关键字，因此我们要注意在将它们用作属性名称时不要将它们标记为关键字，因此您仍然可以执行 jQuery.is()，即使 is 意味着 ===。
```
  identifierToken: ->
    return 0 unless match = IDENTIFIER.exec @chunk
    [input, id, colon] = match
```

保留 id 的长度以用于位置数据

    idLength = id.length
    poppedToken = undefined

    if id is 'own' and @tag() is 'FOR'
      @token 'OWN', id
      return id.length
    if id is 'from' and @tag() is 'YIELD'
      @token 'FROM', id
      return id.length
    if id is 'as' and @seenImport
      if @value() is '*'
        @tokens[@tokens.length - 1][0] = 'IMPORT_ALL'
      else if @value() in COFFEE_KEYWORDS
        @tokens[@tokens.length - 1][0] = 'IDENTIFIER'
      if @tag() in ['DEFAULT', 'IMPORT_ALL', 'IDENTIFIER']
        @token 'AS', id
        return id.length
    if id is 'as' and @seenExport and @tag() in ['IDENTIFIER', 'DEFAULT']
      @token 'AS', id
      return id.length
    if id is 'default' and @seenExport and @tag() in ['EXPORT', 'AS']
      @token 'DEFAULT', id
      return id.length

    [..., prev] = @tokens

    tag =
      if colon or prev? and
         (prev[0] in ['.', '?.', '::', '?::'] or
         not prev.spaced and prev[0] is '@')
        'PROPERTY'
      else
        'IDENTIFIER'

    if tag is 'IDENTIFIER' and (id in JS_KEYWORDS or id in COFFEE_KEYWORDS) and
       not (@exportSpecifierList and id in COFFEE_KEYWORDS)
      tag = id.toUpperCase()
      if tag is 'WHEN' and @tag() in LINE_BREAK
        tag = 'LEADING_WHEN'
      else if tag is 'FOR'
        @seenFor = yes
      else if tag is 'UNLESS'
        tag = 'IF'
      else if tag is 'IMPORT'
        @seenImport = yes
      else if tag is 'EXPORT'
        @seenExport = yes
      else if tag in UNARY
        tag = 'UNARY'
      else if tag in RELATION
        if tag isnt 'INSTANCEOF' and @seenFor
          tag = 'FOR' + tag
          @seenFor = no
        else
          tag = 'RELATION'
          if @value() is '!'
            poppedToken = @tokens.pop()
            id = '!' + id
    else if tag is 'IDENTIFIER' and @seenFor and id is 'from' and
       isForFrom(prev)
      tag = 'FORFROM'
      @seenFor = no

    if tag is 'IDENTIFIER' and id in RESERVED
      @error "reserved word '#{id}'", length: id.length

    unless tag is 'PROPERTY'
      if id in COFFEE_ALIASES
        alias = id
        id = COFFEE_ALIAS_MAP[id]
      tag = switch id
        when '!'                 then 'UNARY'
        when '==', '!='          then 'COMPARE'
        when 'true', 'false'     then 'BOOL'
        when 'break', 'continue', \
             'debugger'          then 'STATEMENT'
        when '&&', '||'          then id
        else  tag

    tagToken = @token tag, id, 0, idLength
    tagToken.origin = [tag, alias, tagToken[2]] if alias
    if poppedToken
      [tagToken[2].first_line, tagToken[2].first_column] =
        [poppedToken[2].first_line, poppedToken[2].first_column]
    if colon
      colonOffset = input.lastIndexOf ':'
      @token ':', ':', colonOffset, colon.length

    input.length

匹配数字，包括小数、十六进制和指数表示法。注意不要干扰正在进行的范围。

  numberToken: ->
    return 0 unless match = NUMBER.exec @chunk

    number = match[0]
    lexedLength = number.length

    switch
      when /^0[BOX]/.test number
        @error "radix prefix in '#{number}' must be lowercase", offset: 1
      when /^(?!0x).*E/.test number
        @error "exponential notation in '#{number}' must be indicated with a lowercase 'e'",
          offset: number.indexOf('E')
      when /^0\d*[89]/.test number
        @error "decimal literal '#{number}' must not be prefixed with '0'", length: lexedLength
      when /^0\d+/.test number
        @error "octal literal '#{number}' must be prefixed with '0o'", length: lexedLength

    base = switch number.charAt 1
      when 'b' then 2
      when 'o' then 8
      when 'x' then 16
      else null
    numberValue = if base? then parseInt(number[2..], base) else parseFloat(number)
    if number.charAt(1) in ['b', 'o']
      number = "0x#{numberValue.toString 16}"

    tag = if numberValue is Infinity then 'INFINITY' else 'NUMBER'
    @token tag, number, 0, lexedLength
    lexedLength

匹配字符串，包括多行字符串，以及 heredocs，无论是否包含插值。

  stringToken: ->
    [quote] = STRING_START.exec(@chunk) || []
    return 0 unless quote

如果前面的 token 是 from 并且这是一个导入或导出语句，则正确标记 from。

    if @tokens.length and @value() is 'from' and (@seenImport or @seenExport)
      @tokens[@tokens.length - 1][0] = 'FROM'

    regex = switch quote
      when "'"   then STRING_SINGLE
      when '"'   then STRING_DOUBLE
      when "'''" then HEREDOC_SINGLE
      when '"""' then HEREDOC_DOUBLE
    heredoc = quote.length is 3

    {tokens, index: end} = @matchWithInterpolations regex, quote
    $ = tokens.length - 1

    delimiter = quote.charAt(0)
    if heredoc

找到最小的缩进。它将在以后从所有行中删除。

      indent = null
      doc = (token[1] for token, i in tokens when token[0] is 'NEOSTRING').join '#{}'
      while match = HEREDOC_INDENT.exec doc
        attempt = match[1]
        indent = attempt if indent is null or 0 < attempt.length < indent.length
      indentRegex = /// \n#{indent} ///g if indent
      @mergeInterpolationTokens tokens, {delimiter}, (value, i) =>
        value = @formatString value, delimiter: quote
        value = value.replace indentRegex, '\n' if indentRegex
        value = value.replace LEADING_BLANK_LINE,  '' if i is 0
        value = value.replace TRAILING_BLANK_LINE, '' if i is $
        value
    else
      @mergeInterpolationTokens tokens, {delimiter}, (value, i) =>
        value = @formatString value, delimiter: quote
        value = value.replace SIMPLE_STRING_OMIT, (match, offset) ->
          if (i is 0 and offset is 0) or
             (i is $ and offset + match.length is value.length)
            ''
          else
            ' '
        value

    end

匹配并消耗注释。

  commentToken: ->
    return 0 unless match = @chunk.match COMMENT
    [comment, here] = match
    if here
      if match = HERECOMMENT_ILLEGAL.exec comment
        @error "block comments cannot contain #{match[0]}",
          offset: match.index, length: match[0].length
      if here.indexOf('\n') >= 0
        here = here.replace /// \n #{repeat ' ', @indent} ///g, '\n'
      @token 'HERECOMMENT', here, 0, comment.length
    comment.length

匹配直接通过反引号插入到源代码中的 JavaScript。

  jsToken: ->
    return 0 unless @chunk.charAt(0) is '`' and
      (match = HERE_JSTOKEN.exec(@chunk) or JSTOKEN.exec(@chunk))

¶

将转义的反引号转换为反引号，并将转义的反斜杠（在转义的反引号之前）转换为反斜杠
```
    script = match[1].replace /\\+(`|$)/g, (string) ->
```
¶

string 始终是一个值，例如 ‘`‘、‘\`‘、‘\\`‘ 等。通过将其缩减为后半部分，我们将 ‘`‘ 转换为 ‘'、'\\\‘ 转换为 ‘`‘ 等。
```
      string[-Math.ceil(string.length / 2)..]
    @token 'JS', script, 0, match[0].length
    match[0].length
```

匹配正则表达式字面量，以及多行扩展的正则表达式。词法分析正则表达式很难与除法区分，因此我们借鉴了 JavaScript 和 Ruby 中的一些基本启发式方法。

  regexToken: ->
    switch
      when match = REGEX_ILLEGAL.exec @chunk
        @error "regular expressions cannot begin with #{match[2]}",
          offset: match.index + match[1].length
      when match = @matchWithInterpolations HEREGEX, '///'
        {tokens, index} = match
      when match = REGEX.exec @chunk
        [regex, body, closed] = match
        @validateEscapes body, isRegex: yes, offsetInChunk: 1
        body = @formatRegex body, delimiter: '/'
        index = regex.length
        [..., prev] = @tokens
        if prev
          if prev.spaced and prev[0] in CALLABLE
            return 0 if not closed or POSSIBLY_DIVISION.test regex
          else if prev[0] in NOT_REGEX
            return 0
        @error 'missing / (unclosed regex)' unless closed
      else
        return 0

    [flags] = REGEX_FLAGS.exec @chunk[index..]
    end = index + flags.length
    origin = @makeToken 'REGEX', null, 0, end
    switch
      when not VALID_FLAGS.test flags
        @error "invalid regular expression flags #{flags}", offset: index, length: flags.length
      when regex or tokens.length is 1
        body ?= @formatHeregex tokens[0][1]
        @token 'REGEX', "#{@makeDelimitedLiteral body, delimiter: '/'}#{flags}", 0, end, origin
      else
        @token 'REGEX_START', '(', 0, 0, origin
        @token 'IDENTIFIER', 'RegExp', 0, 0
        @token 'CALL_START', '(', 0, 0
        @mergeInterpolationTokens tokens, {delimiter: '"', double: yes}, @formatHeregex
        if flags
          @token ',', ',', index - 1, 0
          @token 'STRING', '"' + flags + '"', index - 1, flags.length
        @token ')', ')', end - 1, 0
        @token 'REGEX_END', ')', end - 1, 0

    end

匹配换行符、缩进和缩出，并确定哪个是哪个。如果我们可以检测到当前行继续到下一行，则换行符将被抑制

elements
  .each( ... )
  .map( ... )

跟踪缩进级别，因为单个缩出 token 可以关闭多个缩进，因此我们需要知道我们碰巧在多远的位置。

  lineToken: ->
    return 0 unless match = MULTI_DENT.exec @chunk
    indent = match[0]

    @seenFor = no
    @seenImport = no unless @importSpecifierList
    @seenExport = no unless @exportSpecifierList

    size = indent.length - 1 - indent.lastIndexOf '\n'
    noNewlines = @unfinished()

    if size - @indebt is @indent
      if noNewlines then @suppressNewlines() else @newlineToken 0
      return indent.length

    if size > @indent
      if noNewlines
        @indebt = size - @indent
        @suppressNewlines()
        return indent.length
      unless @tokens.length
        @baseIndent = @indent = size
        return indent.length
      diff = size - @indent + @outdebt
      @token 'INDENT', diff, indent.length - size, size
      @indents.push diff
      @ends.push {tag: 'OUTDENT'}
      @outdebt = @indebt = 0
      @indent = size
    else if size < @baseIndent
      @error 'missing indentation', offset: indent.length
    else
      @indebt = 0
      @outdentToken @indent - size, noNewlines, indent.length
    indent.length

记录缩出 token 或多个 token，如果我们碰巧向内移动到多个记录的缩进之外。设置新的 @indent 值。

  outdentToken: (moveOut, noNewlines, outdentLength) ->
    decreasedIndent = @indent - moveOut
    while moveOut > 0
      lastIndent = @indents[@indents.length - 1]
      if not lastIndent
        moveOut = 0
      else if lastIndent is @outdebt
        moveOut -= @outdebt
        @outdebt = 0
      else if lastIndent < @outdebt
        @outdebt -= lastIndent
        moveOut  -= lastIndent
      else
        dent = @indents.pop() + @outdebt
        if outdentLength and @chunk[outdentLength] in INDENTABLE_CLOSERS
          decreasedIndent -= dent - moveOut
          moveOut = dent
        @outdebt = 0

pair 可能会调用 outdentToken，因此保留 decreasedIndent

        @pair 'OUTDENT'
        @token 'OUTDENT', moveOut, 0, outdentLength
        moveOut -= dent
    @outdebt -= moveOut if dent
    @tokens.pop() while @value() is ';'

    @token 'TERMINATOR', '\n', outdentLength, 0 unless @tag() is 'TERMINATOR' or noNewlines
    @indent = decreasedIndent
    this

匹配并消耗无意义的空格。将前一个 token 标记为“有空格”，因为在某些情况下这会产生影响。

  whitespaceToken: ->
    return 0 unless (match = WHITESPACE.exec @chunk) or
                    (nline = @chunk.charAt(0) is '\n')
    [..., prev] = @tokens
    prev[if match then 'spaced' else 'newLine'] = true if prev
    if match then match[0].length else 0

生成一个换行符 token。连续的换行符将合并在一起。

  newlineToken: (offset) ->
    @tokens.pop() while @value() is ';'
    @token 'TERMINATOR', '\n', offset, 0 unless @tag() is 'TERMINATOR'
    this

¶

使用行尾处的 \ 来抑制换行符。一旦完成工作，斜杠将在此处删除。
```
  suppressNewlines: ->
    @tokens.pop() if @value() is '\\'
    this
```

我们将所有其他单个字符视为一个 token。例如：( ) , . ! 多字符运算符也是字面量 token，以便 Jison 可以分配正确的运算顺序。这里有一些我们在这里特别标记的符号。; 和换行符都被视为 TERMINATOR，我们将指示方法调用的括号与普通括号区分开来，等等。

  literalToken: ->
    if match = OPERATOR.exec @chunk
      [value] = match
      @tagParameters() if CODE.test value
    else
      value = @chunk.charAt 0
    tag  = value
    [..., prev] = @tokens

    if prev and value in ['=', COMPOUND_ASSIGN...]
      skipToken = false
      if value is '=' and prev[1] in ['||', '&&'] and not prev.spaced
        prev[0] = 'COMPOUND_ASSIGN'
        prev[1] += '='
        prev = @tokens[@tokens.length - 2]
        skipToken = true
      if prev and prev[0] isnt 'PROPERTY'
        origin = prev.origin ? prev
        message = isUnassignable prev[1], origin[1]
        @error message, origin[2] if message
      return value.length if skipToken

    if value is '{' and @seenImport
      @importSpecifierList = yes
    else if @importSpecifierList and value is '}'
      @importSpecifierList = no
    else if value is '{' and prev?[0] is 'EXPORT'
      @exportSpecifierList = yes
    else if @exportSpecifierList and value is '}'
      @exportSpecifierList = no

    if value is ';'
      @seenFor = @seenImport = @seenExport = no
      tag = 'TERMINATOR'
    else if value is '*' and prev[0] is 'EXPORT'
      tag = 'EXPORT_ALL'
    else if value in MATH            then tag = 'MATH'
    else if value in COMPARE         then tag = 'COMPARE'
    else if value in COMPOUND_ASSIGN then tag = 'COMPOUND_ASSIGN'
    else if value in UNARY           then tag = 'UNARY'
    else if value in UNARY_MATH      then tag = 'UNARY_MATH'
    else if value in SHIFT           then tag = 'SHIFT'
    else if value is '?' and prev?.spaced then tag = 'BIN?'
    else if prev and not prev.spaced
      if value is '(' and prev[0] in CALLABLE
        prev[0] = 'FUNC_EXIST' if prev[0] is '?'
        tag = 'CALL_START'
      else if value is '[' and prev[0] in INDEXABLE
        tag = 'INDEX_START'
        switch prev[0]
          when '?'  then prev[0] = 'INDEX_SOAK'
    token = @makeToken tag, value
    switch value
      when '(', '{', '[' then @ends.push {tag: INVERSES[value], origin: token}
      when ')', '}', ']' then @pair value
    @tokens.push token
    value.length

¶

Token 操纵器
¶

我们语法中的一个歧义来源曾经是函数定义中的参数列表与函数调用中的参数列表。向后走，专门标记参数，以便为解析器简化操作。

  tagParameters: ->
    return this if @tag() isnt ')'
    stack = []
    {tokens} = this
    i = tokens.length
    tokens[--i][0] = 'PARAM_END'
    while tok = tokens[--i]
      switch tok[0]
        when ')'
          stack.push tok
        when '(', 'CALL_START'
          if stack.length then stack.pop()
          else if tok[0] is '('
            tok[0] = 'PARAM_START'
            return this
          else return this
    this

关闭文件末尾的所有剩余打开的块。

  closeIndentation: ->
    @outdentToken @indent

¶

匹配分隔 token 的内容，并使用 Ruby 式的表示法扩展其中的变量和表达式，以替换任意表达式。
```
"Hello #{name.capitalize()}."
```
如果遇到插值，此方法将递归地创建一个新的词法分析器并进行 token 化，直到 #{ 的 { 与 } 平衡。
- regex 匹配 token 的内容（但不匹配 delimiter，如果需要插值，则不匹配 #{）。
- delimiter 是 token 的分隔符。例如 '、"、'''、""" 和 ///。
此方法允许我们在插值中的字符串中拥有字符串，无限循环。
```
  matchWithInterpolations: (regex, delimiter) ->
    tokens = []
    offsetInChunk = delimiter.length
    return null unless @chunk[...offsetInChunk] is delimiter
    str = @chunk[offsetInChunk..]
    loop
      [strPart] = regex.exec str

      @validateEscapes strPart, {isRegex: delimiter.charAt(0) is '/', offsetInChunk}
```

推送一个假的 ‘NEOSTRING’ token，它将在以后转换为真正的字符串。

      tokens.push @makeToken 'NEOSTRING', strPart, offsetInChunk

      str = str[strPart.length..]
      offsetInChunk += strPart.length

      break unless str[...2] is '#{'

1 用于删除 #{ 中的 #。

      [line, column] = @getLineAndColumnFromChunk offsetInChunk + 1
      {tokens: nested, index} =
        new Lexer().tokenize str[1..], line: line, column: column, untilBalanced: on

¶

跳过尾随的 }。
```
      index += 1
```

将前导和尾随的 { 和 } 转换为括号。不必要的括号将在以后删除。

      [open, ..., close] = nested
      open[0]  = open[1]  = '('
      close[0] = close[1] = ')'
      close.origin = ['', 'end of interpolation', close[2]]

删除前导的 ‘TERMINATOR’（如果有）。

      nested.splice 1, 1 if nested[1]?[0] is 'TERMINATOR'

推送一个假的 ‘TOKENS’ token，它将在以后转换为真正的 token。

      tokens.push ['TOKENS', nested]

      str = str[index..]
      offsetInChunk += index

    unless str[...delimiter.length] is delimiter
      @error "missing #{delimiter}", length: delimiter.length

    [firstToken, ..., lastToken] = tokens
    firstToken[2].first_column -= delimiter.length
    if lastToken[1].substr(-1) is '\n'
      lastToken[2].last_line += 1
      lastToken[2].last_column = delimiter.length - 1
    else
      lastToken[2].last_column += delimiter.length
    lastToken[2].last_column -= 1 if lastToken[1].length is 0

    {tokens, index: offsetInChunk + delimiter.length}

将假 token 类型 ‘TOKENS’ 和 ‘NEOSTRING’ 的数组 tokens（如 matchWithInterpolations 返回的那样）合并到 token 流中。‘NEOSTRING’ 的值首先使用 fn 转换，然后使用 options 转换为字符串。

  mergeInterpolationTokens: (tokens, options, fn) ->
    if tokens.length > 1
      lparen = @token 'STRING_START', '(', 0, 0

    firstIndex = @tokens.length
    for token, i in tokens
      [tag, value] = token
      switch tag
        when 'TOKENS'

¶

优化掉空插值（一对空的括号）。
```
          continue if value.length is 2
```

推送假 ‘TOKENS’ token 中的所有 token。这些已经具有合理的 location 数据。

          locationToken = value[0]
          tokensToPush = value
        when 'NEOSTRING'

将 ‘NEOSTRING’ 转换为 ‘STRING’。

          converted = fn.call this, token[1], i

优化掉空字符串。我们确保 token 流始终以一个字符串 token 开头，以确保结果确实是一个字符串。

          if converted.length is 0
            if i is 0
              firstEmptyStringIndex = @tokens.length
            else
              continue

但是，有一种情况我们可以优化掉一个起始空字符串。

          if i is 2 and firstEmptyStringIndex?
            @tokens.splice firstEmptyStringIndex, 2 # Remove empty string and the plus.
          token[0] = 'STRING'
          token[1] = @makeDelimitedLiteral converted, options
          locationToken = token
          tokensToPush = [token]
      if @tokens.length > firstIndex

创建一个长度为 0 的“+” token。

        plusToken = @token '+', '+'
        plusToken[2] =
          first_line:   locationToken[2].first_line
          first_column: locationToken[2].first_column
          last_line:    locationToken[2].first_line
          last_column:  locationToken[2].first_column
      @tokens.push tokensToPush...

    if lparen
      [..., lastToken] = tokens
      lparen.origin = ['STRING', null,
        first_line:   lparen[2].first_line
        first_column: lparen[2].first_column
        last_line:    lastToken[2].last_line
        last_column:  lastToken[2].last_column
      ]
      rparen = @token 'STRING_END', ')'
      rparen[2] =
        first_line:   lastToken[2].last_line
        first_column: lastToken[2].last_column
        last_line:    lastToken[2].last_line
        last_column:  lastToken[2].last_column

将一个结束 token 配对，确保所有列出的 token 对在整个 token 流过程中都正确平衡。

  pair: (tag) ->
    [..., prev] = @ends
    unless tag is wanted = prev?.tag
      @error "unmatched #{tag}" unless 'OUTDENT' is wanted

自动关闭 INDENT 以支持以下语法

el.click((event) ->
  el.hide())

      [..., lastIndent] = @indents
      @outdentToken lastIndent, true
      return @pair tag
    @ends.pop()

¶

帮助程序
¶

从当前块中的偏移量返回行号和列号。

offset 是 @chunk 中的字符数。

  getLineAndColumnFromChunk: (offset) ->
    if offset is 0
      return [@chunkLine, @chunkColumn]

    if offset >= @chunk.length
      string = @chunk
    else
      string = @chunk[..offset-1]

    lineCount = count string, '\n'

    column = @chunkColumn
    if lineCount > 0
      [..., lastLine] = string.split '\n'
      column = lastLine.length
    else
      column += string.length

    [@chunkLine + lineCount, column]

与“token”相同，只是它只返回 token，而不将其添加到结果中。

  makeToken: (tag, value, offsetInChunk = 0, length = value.length) ->
    locationData = {}
    [locationData.first_line, locationData.first_column] =
      @getLineAndColumnFromChunk offsetInChunk

使用 length - 1 作为最终偏移量 - 我们提供 last_line 和 last_column，因此如果 last_column == first_column，那么我们正在查看一个长度为 1 的字符。

    lastCharacter = if length > 0 then (length - 1) else 0
    [locationData.last_line, locationData.last_column] =
      @getLineAndColumnFromChunk offsetInChunk + lastCharacter

    token = [tag, value, locationData]

    token

¶

将一个 token 添加到结果中。offset 是当前 @chunk 中 token 开始的偏移量。length 是 @chunk 中 token 的长度，在偏移量之后。如果未指定，则将使用 value 的长度。

返回新的 token。
```
  token: (tag, value, offsetInChunk, length, origin) ->
    token = @makeToken tag, value, offsetInChunk, length
    token.origin = origin if origin
    @tokens.push token
    token
```

查看 token 流中的最后一个标签。

  tag: ->
    [..., token] = @tokens
    token?[0]

查看 token 流中的最后一个值。

  value: ->
    [..., token] = @tokens
    token?[1]

我们是否处于未完成的表达式中？

  unfinished: ->
    LINE_CONTINUER.test(@chunk) or
    @tag() in UNFINISHED

  formatString: (str, options) ->
    @replaceUnicodeCodePointEscapes str.replace(STRING_OMIT, '$1'), options

  formatHeregex: (str) ->
    @formatRegex str.replace(HEREGEX_OMIT, '$1$2'), delimiter: '///'

  formatRegex: (str, options) ->
    @replaceUnicodeCodePointEscapes str, options

  unicodeCodePointToUnicodeEscapes: (codePoint) ->
    toUnicodeEscape = (val) ->
      str = val.toString 16
      "\\u#{repeat '0', 4 - str.length}#{str}"
    return toUnicodeEscape(codePoint) if codePoint < 0x10000

代理对

    high = Math.floor((codePoint - 0x10000) / 0x400) + 0xD800
    low = (codePoint - 0x10000) % 0x400 + 0xDC00
    "#{toUnicodeEscape(high)}#{toUnicodeEscape(low)}"

在字符串和正则表达式中用 \uxxxx[\uxxxx] 替换 \u{…}

  replaceUnicodeCodePointEscapes: (str, options) ->
    str.replace UNICODE_CODE_POINT_ESCAPE, (match, escapedBackslash, codePointHex, offset) =>
      return escapedBackslash if escapedBackslash

      codePointDecimal = parseInt codePointHex, 16
      if codePointDecimal > 0x10ffff
        @error "unicode code point escapes greater than \\u{10ffff} are not allowed",
          offset: offset + options.delimiter.length
          length: codePointHex.length + 4

      @unicodeCodePointToUnicodeEscapes codePointDecimal

验证字符串和正则表达式中的转义。

  validateEscapes: (str, options = {}) ->
    invalidEscapeRegex =
      if options.isRegex
        REGEX_INVALID_ESCAPE
      else
        STRING_INVALID_ESCAPE
    match = invalidEscapeRegex.exec str
    return unless match
    [[], before, octal, hex, unicodeCodePoint, unicode] = match
    message =
      if octal
        "octal escape sequences are not allowed"
      else
        "invalid escape sequence"
    invalidEscape = "\\#{octal or hex or unicodeCodePoint or unicode}"
    @error "#{message} #{invalidEscape}",
      offset: (options.offsetInChunk ? 0) + match.index + before.length
      length: invalidEscape.length

通过转义某些字符来构造字符串或正则表达式。

  makeDelimitedLiteral: (body, options = {}) ->
    body = '(?:)' if body is '' and options.delimiter is '/'
    regex = ///
        (\\\\)                               # escaped backslash
      | (\\0(?=[1-7]))                       # nul character mistaken as octal escape
      | \\?(#{options.delimiter})            # (possibly escaped) delimiter
      | \\?(?: (\n)|(\r)|(\u2028)|(\u2029) ) # (possibly escaped) newlines
      | (\\.)                                # other escapes
    ///g
    body = body.replace regex, (match, backslash, nul, delimiter, lf, cr, ls, ps, other) -> switch

忽略转义的反斜杠。

      when backslash then (if options.double then backslash + backslash else backslash)
      when nul       then '\\x00'
      when delimiter then "\\#{delimiter}"
      when lf        then '\\n'
      when cr        then '\\r'
      when ls        then '\\u2028'
      when ps        then '\\u2029'
      when other     then (if options.double then "\\#{other}" else other)
    "#{options.delimiter}#{body}#{options.delimiter}"

在当前块中的给定偏移量处或在 token 的位置（token[2]）处抛出错误。

  error: (message, options = {}) ->
    location =
      if 'first_line' of options
        options
      else
        [first_line, first_column] = @getLineAndColumnFromChunk options.offset ? 0
        {first_line, first_column, last_column: first_column + (options.length ? 1) - 1}
    throwSyntaxError message, location

¶

辅助函数


isUnassignable = (name, displayName = name) -> switch
  when name in [JS_KEYWORDS..., COFFEE_KEYWORDS...]
    "keyword '#{displayName}' can't be assigned"
  when name in STRICT_PROSCRIBED
    "'#{displayName}' can't be assigned"
  when name in RESERVED
    "reserved word '#{displayName}' can't be assigned"
  else
    false

exports.isUnassignable = isUnassignable

¶

from 不是 CoffeeScript 关键字，但在 import 和 export 语句（如上所述）以及 for 循环的声明行中，它的行为类似于关键字。尝试检测 from 是一个变量标识符还是这个“有时”关键字。
```
isForFrom = (prev) ->
  if prev[0] is 'IDENTIFIER'
```

for i from from、for from from iterable

    if prev[1] is 'from'
      prev[1][0] = 'IDENTIFIER'
      yes

¶

for i from iterable
```
    yes
```
¶

for from…
```
  else if prev[0] is 'FOR'
    no
```

for {from}…、for [from]…、for {a, from}…、for {a: from}…

  else if prev[1] in ['{', '[', ',', ':']
    no
  else
    yes

¶

常量
¶

CoffeeScript 与 JavaScript 共有的关键字。

JS_KEYWORDS = [
  'true', 'false', 'null', 'this'
  'new', 'delete', 'typeof', 'in', 'instanceof'
  'return', 'throw', 'break', 'continue', 'debugger', 'yield'
  'if', 'else', 'switch', 'for', 'while', 'do', 'try', 'catch', 'finally'
  'class', 'extends', 'super'
  'import', 'export', 'default'
]

仅 CoffeeScript 的关键字。

COFFEE_KEYWORDS = [
  'undefined', 'Infinity', 'NaN'
  'then', 'unless', 'until', 'loop', 'of', 'by', 'when'
]

COFFEE_ALIAS_MAP =
  and  : '&&'
  or   : '||'
  is   : '=='
  isnt : '!='
  not  : '!'
  yes  : 'true'
  no   : 'false'
  on   : 'true'
  off  : 'false'

COFFEE_ALIASES  = (key for key of COFFEE_ALIAS_MAP)
COFFEE_KEYWORDS = COFFEE_KEYWORDS.concat COFFEE_ALIASES

JavaScript 保留但未使用或由 CoffeeScript 在内部使用的关键字列表。当遇到这些关键字时，我们会抛出错误，以避免在运行时出现 JavaScript 错误。

RESERVED = [
  'case', 'function', 'var', 'void', 'with', 'const', 'let', 'enum'
  'native', 'implements', 'interface', 'package', 'private'
  'protected', 'public', 'static'
]

STRICT_PROSCRIBED = ['arguments', 'eval']

¶

JavaScript 关键字和保留字的超集，这些关键字都不能用作标识符或属性。
```
exports.JS_FORBIDDEN = JS_KEYWORDS.concat(RESERVED).concat(STRICT_PROSCRIBED)
```
¶

令人讨厌的 Microsoft 疯狂（也称为 BOM）的字符代码。
```
BOM = 65279
```

Token 匹配正则表达式。

IDENTIFIER = /// ^
  (?!\d)
  ( (?: (?!\s)[$\w\x7f-\uffff] )+ )
  ( [^\n\S]* : (?!:) )?  # Is this a property name?
///

NUMBER     = ///
  ^ 0b[01]+    |              # binary
  ^ 0o[0-7]+   |              # octal
  ^ 0x[\da-f]+ |              # hex
  ^ \d*\.?\d+ (?:e[+-]?\d+)?  # decimal
///i

OPERATOR   = /// ^ (
  ?: [-=]>             # function
   | [-+*/%<>&|^!?=]=  # compound assign / compare
   | >>>=?             # zero-fill right shift
   | ([-+:])\1         # doubles
   | ([&|<>*/%])\2=?   # logic / shift / power / floor division / modulo
   | \?(\.|::)         # soak access
   | \.{2,3}           # range or splat
) ///

WHITESPACE = /^[^\n\S]+/

COMMENT    = /^###([^#][\s\S]*?)(?:###[^\n\S]*|###$)|^(?:\s*#(?!##[^#]).*)+/

CODE       = /^[-=]>/

MULTI_DENT = /^(?:\n[^\n\S]*)+/

JSTOKEN      = ///^ `(?!``) ((?: [^`\\] | \\[\s\S]           )*) `   ///
HERE_JSTOKEN = ///^ ```     ((?: [^`\\] | \\[\s\S] | `(?!``) )*) ``` ///

字符串匹配正则表达式。

STRING_START   = /^(?:'''|"""|'|")/

STRING_SINGLE  = /// ^(?: [^\\']  | \\[\s\S]                      )* ///
STRING_DOUBLE  = /// ^(?: [^\\"#] | \\[\s\S] |           \#(?!\{) )* ///
HEREDOC_SINGLE = /// ^(?: [^\\']  | \\[\s\S] | '(?!'')            )* ///
HEREDOC_DOUBLE = /// ^(?: [^\\"#] | \\[\s\S] | "(?!"") | \#(?!\{) )* ///

STRING_OMIT    = ///
    ((?:\\\\)+)      # consume (and preserve) an even number of backslashes
  | \\[^\S\n]*\n\s*  # remove escaped newlines
///g
SIMPLE_STRING_OMIT = /\s*\n\s*/g
HEREDOC_INDENT     = /\n+([^\n\S]*)(?=\S)/g

正则表达式匹配正则表达式。

REGEX = /// ^
  / (?!/) ((
  ?: [^ [ / \n \\ ]  # every other thing
   | \\[^\n]         # anything but newlines escaped
   | \[              # character class
       (?: \\[^\n] | [^ \] \n \\ ] )*
     \]
  )*) (/)?
///

REGEX_FLAGS  = /^\w*/
VALID_FLAGS  = /^(?!.*(.).*\1)[imguy]*$/

HEREGEX      = /// ^(?: [^\\/#] | \\[\s\S] | /(?!//) | \#(?!\{) )* ///

HEREGEX_OMIT = ///
    ((?:\\\\)+)     # consume (and preserve) an even number of backslashes
  | \\(\s)          # preserve escaped whitespace
  | \s+(?:#.*)?     # remove whitespace and comments
///g

REGEX_ILLEGAL = /// ^ ( / | /{3}\s*) (\*) ///

POSSIBLY_DIVISION   = /// ^ /=?\s ///

其他正则表达式。

HERECOMMENT_ILLEGAL = /\*\//

LINE_CONTINUER      = /// ^ \s* (?: , | \??\.(?![.\d]) | :: ) ///

STRING_INVALID_ESCAPE = ///
  ( (?:^|[^\\]) (?:\\\\)* )        # make sure the escape isn’t escaped
  \\ (
     ?: (0[0-7]|[1-7])             # octal escape
      | (x(?![\da-fA-F]{2}).{0,2}) # hex escape
      | (u\{(?![\da-fA-F]{1,}\})[^}]*\}?) # unicode code point escape
      | (u(?!\{|[\da-fA-F]{4}).{0,4}) # unicode escape
  )
///
REGEX_INVALID_ESCAPE = ///
  ( (?:^|[^\\]) (?:\\\\)* )        # make sure the escape isn’t escaped
  \\ (
     ?: (0[0-7])                   # octal escape
      | (x(?![\da-fA-F]{2}).{0,2}) # hex escape
      | (u\{(?![\da-fA-F]{1,}\})[^}]*\}?) # unicode code point escape
      | (u(?!\{|[\da-fA-F]{4}).{0,4}) # unicode escape
  )
///

UNICODE_CODE_POINT_ESCAPE = ///
  ( \\\\ )        # make sure the escape isn’t escaped
  |
  \\u\{ ( [\da-fA-F]+ ) \}
///g

LEADING_BLANK_LINE  = /^[^\n\S]*\n/
TRAILING_BLANK_LINE = /\n[^\n\S]*$/

TRAILING_SPACES     = /\s+$/

复合赋值 token。

COMPOUND_ASSIGN = [
  '-=', '+=', '/=', '*=', '%=', '||=', '&&=', '?=', '<<=', '>>=', '>>>='
  '&=', '^=', '|=', '**=', '//=', '%%='
]

一元 token。

UNARY = ['NEW', 'TYPEOF', 'DELETE', 'DO']

UNARY_MATH = ['!', '~']

¶

位移 token。
```
SHIFT = ['<<', '>>', '>>>']
```

比较 token。

COMPARE = ['==', '!=', '<', '>', '<=', '>=']

¶

数学 token。
```
MATH = ['*', '/', '%', '//', '%%']
```
¶

可以使用 not 前缀取反的关系 token。
```
RELATION = ['IN', 'OF', 'INSTANCEOF']
```
¶

布尔 token。
```
BOOL = ['TRUE', 'FALSE']
```

可以合法地调用或索引的 token。在这些 token 之后，一个开括号或方括号将被记录为函数调用或索引操作的开始。

CALLABLE  = ['IDENTIFIER', 'PROPERTY', ')', ']', '?', '@', 'THIS', 'SUPER']
INDEXABLE = CALLABLE.concat [
  'NUMBER', 'INFINITY', 'NAN', 'STRING', 'STRING_END', 'REGEX', 'REGEX_END'
  'BOOL', 'NULL', 'UNDEFINED', '}', '::'
]

¶

正则表达式永远不会紧随其后的 token（除了某些情况下有空格的 CALLABLE），但除法运算符可以紧随其后。

参见：http://www-archive.mozilla.org/js/language/js20-2002-04/rationale/syntax.html#regular-expressions
```
NOT_REGEX = INDEXABLE.concat ['++', '--']
```
¶

当紧接在 WHEN 之前时，表示 WHEN 出现在行首的 token。我们将这些与尾随的 whens 区分开来，以避免语法中的歧义。
```
LINE_BREAK = ['INDENT', 'OUTDENT', 'TERMINATOR']
```
¶

这些前面的额外缩进将被忽略。
```
INDENTABLE_CLOSERS = [')', '}', ']']
```

当出现在行尾时，这些标记会抑制随后的 TERMINATOR/INDENT 标记。

UNFINISHED = ['\\', '.', '?.', '?::', 'UNARY', 'MATH', 'UNARY_MATH', '+', '-',
           '**', 'SHIFT', 'RELATION', 'COMPARE', '&', '^', '|', '&&', '||',
           'BIN?', 'THROW', 'EXTENDS', 'DEFAULT']

词法分析器类

词法分析器

Token 操纵器

帮助程序

辅助函数

常量