= HIGHLIGHT PLUGIN MANUAL
André Simon
v3.57, May 2020
:lang: en
:toc: left
:toc-title: Contents
:toclevels: 4
:sectnums:
:sectnumlevels: 2
:sectanchors:
// Cross References:
:xrefstyle: short
:section-refsig: Sect.
// Misc Settings:
:experimental: true
:icons: font
:linkattrs: true
// GitHub Settings to enable Admonitions Icons in preview:
ifdef::env-github[]
:caution-caption: :fire:
:important-caption: :heavy_exclamation_mark:
:note-caption: :information_source:
:tip-caption: :bulb:
:warning-caption: :warning:
endif::[]

// =====================================
// Custom Attributes for Reference Links
// =====================================
// Folders:
:plugins: pass:q[link:plugins/[`plugins/`^]]
// Plugins files:
:bash_functions_lua: pass:q[link:plugins/bash_functions.lua[`bash_functions.lua`^]]
:cpp_ref_cplusplus_com_lua: pass:q[link:plugins/cpp_ref_cplusplus_com.lua[`cpp_ref_cplusplus_com.lua`^]]
:ctags_html_tooltips_lua: pass:q[link:plugins/ctags_html_tooltips.lua[`ctags_html_tooltips.lua`^]]
:outhtml_codefold_lua: pass:q[link:plugins/outhtml_codefold.lua[`outhtml_codefold.lua`^]]
:outhtml_curly_brackets_matcher_lua: pass:q[link:plugins/outhtml_curly_brackets_matcher.lua[`outhtml_curly_brackets_matcher.lua`^]]
:outhtml_keyword_matcher_lua: pass:q[link:plugins/outhtml_keyword_matcher.lua[`outhtml_keyword_matcher.lua`^]]
:outhtml_keyword_matcher_lua: pass:q[link:plugins/outhtml_keyword_matcher.lua[`outhtml_keyword_matcher.lua`^]]
:theme_invert_lua: pass:q[link:plugins/theme_invert.lua[`theme_invert.lua`^]]

== ABOUT

The plug-in interface allows modifications of syntax parsing, colouring and
the document's header and footer.

A common task would be to define a new set of syntax keywords, or to add items
to existing keyword groups. More advanced plug-ins add tooltips based on ctags
input or source code folding to the output (see <<EXAMPLES>>).


== SCRIPT STRUCTURE

The following script contains the basic plug-in structure which has no effect
on the output:

[source,lua]
--------------------------------------------------------------------------------
Description="Plugin Boilerplate"

-- optional parameter desc: syntax description
function syntaxUpdate(desc)
end

function themeUpdate(desc)
end

function formatUpdate(desc)
end

Plugins={
  { Type="theme", Chunk=themeUpdate },
  { Type="lang", Chunk=syntaxUpdate },
  { Type="format", Chunk=formatUpdate },
}
--------------------------------------------------------------------------------

The first line contains a description which gives a short summary of the
plug-in effects.

The next lines contain function definitions:

* The `syntaxUpdate` function is applied on syntax definition scripts (`+*.lang+`),
  whereas the `themeUpdate` function is applied on colour themes (`+*.theme+`).
* The `formatUpdate` is not executed in a syntax or theme Lua state.
  It just returns strings to override or enhance the document header and footer.
* The `desc` parameter contains the description string of the loaded syntax
  definition or colour theme. This information can be used to restrict
  modifications to certain kinds of input (i.e. only C code should be
  enhanced with syslog keywords).

Finally the `Plugins` array connects the functions to the Lua states which
are created when highlight loads a lang or theme script.
In this example, `themeUpdate` is connected to the theme state, and `syntaxUpdate`
to the lang state.
It is not required to always define both connections if your plugin only affects
one of the config script types.


== SYNTAX CHUNK ELEMENTS

The following list includes all variables and constants you may use to adjust
the parser's syntax highlighting.

Syntax definition items:

[horizontal]
`Comments`                :: table
`Description`             :: string
`Digits`                  :: string
`EnableIndentation`       :: boolean
`Identifiers`             :: string
`IgnoreCase`              :: boolean
`Keywords`                :: table
`NestedSections`          :: table
`Operators`               :: string
`PreProcessor`            :: table
`Strings`                 :: table


Document modification items:

[horizontal]
`HeaderInjection`         :: string
`FooterInjection`         :: string

Read only (internal highlighting states):

[horizontal]
`HL_STANDARD`             :: number
`HL_BLOCK_COMMENT`        :: number
`HL_BLOCK_COMMENT_END`    :: number
`HL_EMBEDDED_CODE_BEGIN`  :: number
`HL_EMBEDDED_CODE_END`    :: number
`HL_ESC_SEQ`              :: number
`HL_ESC_SEQ_END`          :: number
`HL_IDENTIFIER_BEGIN`     :: number
`HL_IDENTIFIER_END`       :: number
`HL_KEYWORD`              :: number
`HL_KEYWORD_END`          :: number
`HL_LINENUMBER`           :: number
`HL_LINE_COMMENT`         :: number
`HL_LINE_COMMENT_END`     :: number
`HL_NUMBER`               :: number
`HL_OPERATOR`             :: number
`HL_OPERATOR_END`         :: number
`HL_PREPROC`              :: number
`HL_PREPROC_END`          :: number
`HL_PREPROC_STRING`       :: number
`HL_STRING`               :: number
`HL_STRING_END`           :: number
`HL_UNKNOWN`              :: number
`HL_REJECT`               :: number

Read only (output document format):

[horizontal]
`HL_OUTPUT`               :: number (selected format)
`HL_FORMAT_HTML`          :: number
`HL_FORMAT_XHTML`         :: number
`HL_FORMAT_TEX`           :: number
`HL_FORMAT_LATEX`         :: number
`HL_FORMAT_RTF`           :: number
`HL_FORMAT_ANSI`          :: number
`HL_FORMAT_XTERM256`      :: number
`HL_FORMAT_TRUECOLOR`     :: number
`HL_FORMAT_SVG`           :: number
`HL_FORMAT_BBCODE`        :: number
`HL_FORMAT_PANGO`         :: number
`HL_FORMAT_ODT`           :: number

Read only (other):

[horizontal]
`HL_PLUGIN_PARAM`         :: string (set with `--plug-in-param`)
`HL_LANG_DIR`             :: string (path of language definition directory)

Functions:

[horizontal]
`AddKeyword`              :: function
`RemoveKeyword`           :: function
`OnStateChange`           :: function
`Decorate`                :: function
`DecorateLineBegin`       :: function
`DecorateLineEnd`         :: function

[IMPORTANT]
================================================================================
Functions will only be executed if they are defined as local functions within the
`lang` chunk referenced in the `Plugins` array.
They will be ignored when defined elsewhere in the script.
================================================================================

The functions `AddKeyword`, `RemoveKeyword` and `OnStateChange` are also useful
in language definitions without a plug-in use case.


=== FUNCTIONS ADDKEYWORD/REMOVEKEYWORD

[[AddKeyword]]
The `AddKeyword` function will add a keyword to one of the the internal keyword
lists. It has no effect if the keyword was added before.
Keywords added with `AddKeyword` will remain active for all files of the same
syntax if highlight is in batch mode.

................................................................................
AddKeyword(keyword, kwGroupID)

  Parameters: keyword:   string which should be added to a keyword list
              kwGroupID: keyword group ID of the keyword
  Returns:    true if successful
................................................................................

[[RemoveKeyword]]
The `RemoveKeyword` function erases the given keyword from the internal list.

................................................................................
RemoveKeyword(keyword)

  Parameters: keyword:   string which should be removed from the keyword list
  Returns:    true if successful
................................................................................


=== FUNCTION ADDPERSISTENTSTATE

[[AddPersistentState]]
This function enables storage of keywords and keyword ranges in a plug-in file. 
If the syntax contains elements which depend on a context, you can highlight 
them although this context is lost in other input files or code sections.
The invocation of AddPersistentState will cause highlight to save a plugin as 
temporary file and parse input files using this plug-in again if necessary.

................................................................................
AddPersistentState(keyword, kwGroupID)

  Parameters: keyword:   string which should be added to a keyword list
              kwGroupID: keyword group ID of the keyword
  Returns:    true if successful
................................................................................

................................................................................
AddPersistentState(lineno, kwGroupID, column, length)

  Parameters: lineno:    line number
              kwGroupID: the keyword group ID
              column:    column
              length:    length of the keyword
  Returns:    true if successful
................................................................................


=== FUNCTION ONSTATECHANGE

[[OnStateChange]]
This function is a hook which is called if an internal state changes (e.g. from
`HL_STANDARD` to `HL_KEYWORD` if a keyword is found). It can be used to alter
the new state or to manipulate syntax elements like keyword lists.

................................................................................
OnStateChange(oldState, newState, token, kwGroupID, lineno, column)

  Hook Event: Highlighting parser state change
  Parameters: oldState:  old state
              newState:  intended new state
              token:     the current token which triggered the new state
              kwGroupID: if newState is HL_KEYWORD, the parameter
                         contains the keyword group ID
              lineno:    line number (since 3.50)
              column:    line column (since 3.50)
  Returns:    Correct state to continue OR HL_REJECT
................................................................................

Returns `HL_REJECT` if the recognized token and state should be discarded;
the first character of token will be outputted and highlighted as `oldState`.

Example for its usage in a plugin context:

[source,lua]
--------------------------------------------------------------------------------
function OnStateChange(oldState, newState, token, kwgroup)
   if newState==HL_KEYWORD and kwgroup==5 then
      AddKeyword(token, 5)
   end
   return newState
end
--------------------------------------------------------------------------------

This function adds the current keyword to the internal keyword list if the
keyword belongs to keyword group 5. If keyword group 5 is defined by a regex,
this token will be recognized later as a keyword even if the regular expression
does no longer match.


=== FUNCTION OVERRIDEPARAM

[[OverrideParam]]
This function enables modification of internal default values concerning the
parser or output formats. This function is experimental and subject to change.
Currently it supports these parameters:

[horizontal]
`state.string.raw`    :: `true` or `false`
`format.maskws`       :: `true` or `false`
`format.spacer`       :: whitespace format string

See `langDefs/toml.lang` and `plugins/outhtml_ie7_webctrl.lua` for examples.

................................................................................
OverrideParam(keyword, kwGroupID)

  Parameters: paramName:   name of configuration item
              paramVal:    value
  Returns:    true if successful
................................................................................


=== DECORATE FUNCTIONS

[[Decorate]]
The `Decorate` function is a hook which is called if a syntax token has been
identified. It can be used to alter the token or to add additional text in the
target output format (e.g. hyperlinks).

................................................................................
Decorate(token, state, kwGroupID, stateTrace)

  Hook Event: Token identification
  Parameters: token:      current token
              state:      current state
              kwGroupID:  if state is HL_KEYWORD, the parameter
                          contains the keyword group ID
              stateTrace: string containing past states of the current line;
                          separated by ';'; limited to 100 entries
  Returns:    Altered token string or nothing if original token should be
              outputted
................................................................................

Examples:

[source,lua]
--------------------------------------------------------------------------------
function Decorate(token, state)
  if (state == HL_KEYWORD) then
    return string.upper(token)
  end
end
--------------------------------------------------------------------------------

This function converts all keywords to upper case.

The functions `DecorateLineBegin` and `DecorateLineEnd` are called if a new line
starts or ends. They can be used to add special formatting to lines of code.

[[DecorateLineBegin]]
................................................................................
DecorateLineBegin(lineNumber)

  Hook Event: output of a new line
  Parameters: lineNumber: the current line number
  Returns:    A string to be prepended to a new line (or nothing)
................................................................................

[[DecorateLineEnd]]
................................................................................
DecorateLineEnd(lineNumber)

  Hook Event: output of a line ending
  Parameters: lineNumber: the current line number
  Returns:    A string to be appended to a line (or nothing)
................................................................................


[IMPORTANT]
================================================================================
The return value of `Decorate` functions will be embedded in the formatting tags
of the output format.
The return values are not modified or validated by highlight.
================================================================================


== THEME CHUNK ELEMENTS

The following list includes all those items which you can overwrite or extend to
adjust the formatting (colour and font attributes) of the output:

Output formatting items:

[horizontal]
`Default`                 :: table
`Canvas`                  :: table
`Number`                  :: table
`Escape`                  :: table
`String`                  :: table
`StringPreProc`           :: table
`BlockComment`            :: table
`PreProcessor`            :: table
`LineNum`                 :: table
`Operator`                :: table
`LineComment`             :: table
`Keywords`                :: table

Read only (output document format):

[horizontal]
`HL_OUTPUT`               :: number
`HL_FORMAT_HTML`          :: number
`HL_FORMAT_XHTML`         :: number
`HL_FORMAT_TEX`           :: number
`HL_FORMAT_LATEX`         :: number
`HL_FORMAT_RTF`           :: number
`HL_FORMAT_ANSI`          :: number
`HL_FORMAT_XTERM256`      :: number
`HL_FORMAT_TRUECOLOR`     :: number
`HL_FORMAT_SVG`           :: number
`HL_FORMAT_BBCODE`        :: number
`HL_FORMAT_PANGO`         :: number
`HL_FORMAT_ODT`           :: number

Add additional styling information:

[horizontal]
`Injections`              :: table


== FORMAT CHUNK ELEMENTS

Read only (output document format):

[horizontal]
`HL_OUTPUT`               :: number
`HL_FORMAT_HTML`          :: number
`HL_FORMAT_XHTML`         :: number
`HL_FORMAT_TEX`           :: number
`HL_FORMAT_LATEX`         :: number
`HL_FORMAT_RTF`           :: number
`HL_FORMAT_ANSI`          :: number
`HL_FORMAT_XTERM256`      :: number
`HL_FORMAT_TRUECOLOR`     :: number
`HL_FORMAT_SVG`           :: number
`HL_FORMAT_BBCODE`        :: number
`HL_FORMAT_PANGO`         :: number
`HL_FORMAT_ODT`           :: number

Functions:

[horizontal]
`DocumentHeader`          :: function
`DocumentFooter`          :: function


=== DOCUMENTHEADER FUNCTION

[[DocumentHeader]]
................................................................................
DocumentHeader(numFiles, currFile, options)

  Hook Event: output of a new file's header
  Parameters: numFiles: number of files to be generated
              currFile: current file counter
              options: Map of the following options
              options.title: document title
              options.encoding: document encoding
              options.fragment: true if header/footer should not be outputted
              options.font: font name
              options.fontsize: font size

  Returns:    [string, boolean?] (or nothing)
              The string contains the new document header
              The boolean value indicates if the string should replace the default
              header (false=default) or if it should be appended to it (true).
................................................................................


=== DOCUMENTFOOTER FUNCTION

[[DocumentFooter]]
................................................................................
DocumentFooter(numFiles, currFile, options)

  Hook Event: output of a new file's footer
  Parameters: see DocumentHeader

  Returns:    [string, boolean?] (or nothing)
              The string contains the new document footer
              The boolean value indicates if the string should replace the default
              footer (false=default) or if it should precede it (true).
................................................................................


== STEP BY STEP

This example will add reference hyperlinks to Qt keywords:

[source,lua]
--------------------------------------------------------------------------------
-- first add a description of what the plug-in does
Description="Add qtproject.org reference links to HTML, LaTeX or RTF output"

-- define the plugin categories (ie. supported output formats; languages)
Categories = {"c++", "qt" }

-- the syntaxUpdate function contains code related to syntax recognition
function syntaxUpdate(desc)

  -- if the current file is no C++ file we exit
  if desc~="C and C++" then
     return
  end

  -- this function returns a qt-project reference link of the given token
  function getURL(token)
     -- generate the URL
     url='http://qt-project.org/doc/qt-4.8/'..string.lower(token).. '.html'

     -- embed the URL in a hyperlink according to the output format
     -- first HTML, then LaTeX and RTF
     if (HL_OUTPUT== HL_FORMAT_HTML or HL_OUTPUT == HL_FORMAT_XHTML) then
         return '<a class="hl" target="new" href="'
                .. url .. '">'.. token .. '</a>'
     elseif (HL_OUTPUT == HL_FORMAT_LATEX) then
         return '\\href{'..url..'}{'..token..'}'
     elseif (HL_OUTPUT == HL_FORMAT_RTF) then
         return '{{\\field{\\*\\fldinst HYPERLINK "'
                ..url..'" }{\\fldrslt\\ul\\ulc0 '..token..'}}}'
     end
   end

  -- the Decorate function will be invoked for every recognized token
  function Decorate(token, state)

    -- we are only interested in keywords, preprocessor or default items
    if (state ~= HL_STANDARD and state ~= HL_KEYWORD and
        state ~=HL_PREPROC) then
      return
    end

    -- Qt keywords start with Q, followed by an upper and a lower case letter
    -- if this pattern applies to the token, we return the URL
    -- if we return nothing, the token is outputted as is
    if string.find(token, "Q%u%l")==1 then
      return getURL(token)
    end

  end
end

-- the themeUpdate function contains code related to the theme
function themeUpdate(desc)
  -- the Injections table can be used to add style information to the theme

  -- HTML: we add additional CSS style information to beautify hyperlinks,
  -- they should have the same color as their surrounding tags
  if (HL_OUTPUT == HL_FORMAT_HTML or HL_OUTPUT == HL_FORMAT_XHTML) then
    Injections[#Injections+1]=
      "a.hl, a.hl:visited {color:inherit;font-weight:inherit;}"

  -- LaTeX: hyperlinks require the hyperref package, so we add this here
  -- the colorlinks and pdfborderstyle options remove ugly boxes in the output
  elseif (HL_OUTPUT==HL_FORMAT_LATEX) then
    Injections[#Injections+1]=
      "\\usepackage[colorlinks=false, pdfborderstyle={/S/U/W 1}]{hyperref}"
  end
end

-- let highlight load the chunks
Plugins={
  { Type="lang", Chunk=syntaxUpdate },
  { Type="theme", Chunk=themeUpdate },
}
--------------------------------------------------------------------------------


== EXAMPLES


The {plugins} directory contains example scripts, including:

{bash_functions_lua}::
+
[horizontal]
Description ::: Add function names to keyword list.
Features    ::: Adds new keyword group based on a regex, defines `OnStateChange`,
                uses `AddKeyword`.

{theme_invert_lua}::
+
[horizontal]
Description ::: Invert colours of the original theme.
Features    ::: Modifies all color attributes of the theme script, uses Lua
                pattern matching.

{cpp_ref_cplusplus_com_lua}::
+
[horizontal]
Description ::: Add qtproject.org reference links to HTML, LaTeX or RTF output of
                C++ code.
Features    ::: Uses `Decorate` to add hyperlinks for a defined set of C++ keywords. +
                Adds CSS styles with `Injections`.

{ctags_html_tooltips_lua}::
+
[horizontal]
Description ::: Add tooltips based on a ctags file (default input file: tags).
Features    ::: Uses file input (defined by cli option `--plug-in-param`) and
                parses tags data before `Decorate` is called.

{outhtml_curly_brackets_matcher_lua}::
+
[horizontal]
Description ::: Shows matching curly brackets in HTML output.
Features    ::: Uses `Decorate` to add span tags with unique ids to opening and
                closing brackets. +
                Adds JavaScript with `HeaderInjection` variable. +
                Inserts additional CSS styles with `Injections` variable.

{outhtml_keyword_matcher_lua}::
+
[horizontal]
Description ::: Shows matching keywords in HTML output.
Features    ::: Uses `Decorate` to add span tags with unique ids to keywords. +
                Uses `OnStateChange` to assign an internal ID to each keyword. +
                Adds JavaScript with `HeaderInjection` variable. +
                Inserts additional CSS styles with `Injections` variable. +

{outhtml_codefold_lua}::
+
[horizontal]
Description ::: Adds code folding for C style languages, Pascal, Lua and Ruby to
                HTML output.
Features    ::: Uses `DecorateLineBegin` and `DecorateLineEnd` to add ID-spans to each
                line. +
                Applies `Decorate` to each code block delimiter to add `onClick` event
                handlers. +
                Adds JavaScript with `HeaderInjection` and `FooterInjection` variables. +
                Inserts additional CSS styles with `Injections` variable.


== PLUG-IN USAGE

[discrete]
=== Command line interface

Run highlight `--list-scripts=plugins` to show all installed plug-ins.

Use `--plug-in` to load a plug-in script file.
This option can be applied more than once to apply several plug-ins.
Omit the `.lua` suffix.
You can store your plug-in scripts for testing in `~/.highlight/plugins`.

.Example

................................................................................
highlight my.cpp -Ilz --plug-in=html_curly_brackets_matcher > ~/test_out/my.html
................................................................................


[discrete]
=== GUI

Add the plug-in scripts in the plug-in selection tab and enable them using the
checkboxes.


// EOF //
