diff options
Diffstat (limited to 'ctags/docs/running-multi-parsers.html')
| -rw-r--r-- | ctags/docs/running-multi-parsers.html | 548 |
1 files changed, 548 insertions, 0 deletions
diff --git a/ctags/docs/running-multi-parsers.html b/ctags/docs/running-multi-parsers.html new file mode 100644 index 0000000..7e1d620 --- /dev/null +++ b/ctags/docs/running-multi-parsers.html @@ -0,0 +1,548 @@ + +<!DOCTYPE html> + +<html> + <head> + <meta charset="utf-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" /> + + <title>Running multiple parsers on an input file — Universal Ctags 0.3.0 documentation</title> + <link rel="stylesheet" type="text/css" href="_static/pygments.css" /> + <link rel="stylesheet" type="text/css" href="_static/classic.css" /> + + <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script> + <script src="_static/jquery.js"></script> + <script src="_static/underscore.js"></script> + <script src="_static/doctools.js"></script> + + <link rel="index" title="Index" href="genindex.html" /> + <link rel="search" title="Search" href="search.html" /> + <link rel="next" title="Interactive mode" href="interactive-mode.html" /> + <link rel="prev" title="JSON output" href="output-json.html" /> + </head><body> + <div class="related" role="navigation" aria-label="related navigation"> + <h3>Navigation</h3> + <ul> + <li class="right" style="margin-right: 10px"> + <a href="genindex.html" title="General Index" + accesskey="I">index</a></li> + <li class="right" > + <a href="interactive-mode.html" title="Interactive mode" + accesskey="N">next</a> |</li> + <li class="right" > + <a href="output-json.html" title="JSON output" + accesskey="P">previous</a> |</li> + <li class="nav-item nav-item-0"><a href="index.html">Universal Ctags 0.3.0 documentation</a> »</li> + <li class="nav-item nav-item-this"><a href="">Running multiple parsers on an input file</a></li> + </ul> + </div> + + <div class="document"> + <div class="documentwrapper"> + <div class="bodywrapper"> + <div class="body" role="main"> + + <section id="running-multiple-parsers-on-an-input-file"> +<span id="multiple-parsers"></span><h1>Running multiple parsers on an input file<a class="headerlink" href="#running-multiple-parsers-on-an-input-file" title="Permalink to this headline">¶</a></h1> +<p>Universal Ctags provides parser developers two ways, <em>guest parser</em> (guest/host) +and <em>subparser</em> (sub/base), to run multiple parsers for an input file.</p> +<p>This section shows concepts behind the running multiple parsers and real +examples.</p> +<section id="guest-parser-applying-a-parser-to-specified-areas-of-input-file"> +<span id="host-guest-parsers"></span><h2>Guest parser: Applying a parser to specified areas of input file<a class="headerlink" href="#guest-parser-applying-a-parser-to-specified-areas-of-input-file" title="Permalink to this headline">¶</a></h2> +<p><em>Guest parser</em> (guest/host) considers the case that an input file has areas +written in languages different from the language for the input file.</p> +<p>A <em>host parser</em> parses the input file and detects the areas. +The host parser schedules <em>guest parsers</em> parsing the areas. +The guest parsers parse the areas.</p> +<figure class="align-default"> +<a class="reference internal image-reference" href="_images/area-and-parsers.svg"><img alt="_images/area-and-parsers.svg" src="_images/area-and-parsers.svg" /></a> +</figure> +<section id="command-line-interface"> +<h3>Command line interface<a class="headerlink" href="#command-line-interface" title="Permalink to this headline">¶</a></h3> +<p>Running guest parser can be controlled with guest (<code class="docutils literal notranslate"><span class="pre">g</span></code>) extras flag. +By default it is disabled. To turning on the feature running +guest parser, specify <code class="docutils literal notranslate"><span class="pre">--extras=+g</span></code>.</p> +<p>If <code class="docutils literal notranslate"><span class="pre">--fields=+E</span></code> is given, all tags generated by a guest parser is marked +<code class="docutils literal notranslate"><span class="pre">guest</span></code> in their <code class="docutils literal notranslate"><span class="pre">extras:</span></code> fields.</p> +</section> +<section id="examples-of-guest-parser"> +<h3>Examples of guest parser<a class="headerlink" href="#examples-of-guest-parser" title="Permalink to this headline">¶</a></h3> +<section id="css-javascript-html-parser-combination"> +<h4>{CSS,JavaScript}/HTML parser combination<a class="headerlink" href="#css-javascript-html-parser-combination" title="Permalink to this headline">¶</a></h4> +<p>For an HTML file, you may want to run HTML parser, of course. The +HTML file may have CSS areas and JavaScript areas. In other hand +Universal Ctags has both CSS and JavaScript parsers. Don’t you +think it is useful if you can apply these parsers to the areas?</p> +<p>In this case, HTML has responsible to detect the CSS and +JavaScript areas and record the positions of the areas. +The HTML parser schedules delayed invocations of CSS and +JavaScript parsers on the area with promise API.</p> +<p>Here HTML parser is a <em>host parser</em>. CSS and JavaScript parsers +are <em>guest parsers</em>.</p> +<p>See “<a class="reference internal" href="parser-html.html#html"><span class="std std-ref">The new HTML parser</span></a>” and “<a class="reference external" href="https://github.com/universal-ctags/ctags/blob/master/parsers/html.c">parsers/html.c</a>”.</p> +</section> +<section id="c-yacc-parser-combination"> +<h4>C/Yacc parser combination<a class="headerlink" href="#c-yacc-parser-combination" title="Permalink to this headline">¶</a></h4> +<p>A yacc file has some areas written in C. Universal Ctags has both YACC +and C parsers. You may want to run C parser for the areas from YACC +parser.</p> +<p>Here YACC parser is a host parser. C parser is a guest parser. +See “<a class="reference external" href="https://github.com/universal-ctags/ctags/blob/master/parsers/yacc.c">parsers/yacc.c</a>”.</p> +</section> +<section id="pod-perl-parser-combination"> +<h4>Pod/Perl parser combination<a class="headerlink" href="#pod-perl-parser-combination" title="Permalink to this headline">¶</a></h4> +<p>Pod (Plain Old Documentation) is a language for documentation. The language +can be used not only in a stand alone file but also it can be +used inside a Perl script.</p> +<p>Universal Ctags has both parsers for Perl and Pod. +The Perl parser recognizes the area where Pod document is +embedded in a Perl script and schedules applying pod parser +as a guest parser on the area.</p> +</section> +</section> +</section> +<section id="subparser-tagging-definitions-of-higher-upper-level-language"> +<span id="base-sub-parsers"></span><h2>Subparser: Tagging definitions of higher (upper) level language<a class="headerlink" href="#subparser-tagging-definitions-of-higher-upper-level-language" title="Permalink to this headline">¶</a></h2> +<section id="background"> +<h3>Background<a class="headerlink" href="#background" title="Permalink to this headline">¶</a></h3> +<p>Consider an application written in language X. The application has +its domain own concepts. Developers of the application may try to +express the concepts in the syntax of language X.</p> +<p>In language X level, the developer can define functions, variables, types, and +so on. Further more, if the syntax of X allows, the developers want to +define higher level (= application level) things for implementing the +domain own concepts.</p> +<p>Let me show the part of source code of SPY-WARS, an imaginary game application. +It is written in scheme language, a dialect of lisp. +(Here <a class="reference external" href="https://practical-scheme.net/gauche/index.html">gauche</a> is considered +as the implementation of scheme interpreter).</p> +<div class="highlight-scheme notranslate"><div class="highlight"><pre><span></span><span class="p">(</span><span class="k">define </span><span class="nv">agent-tables</span> <span class="p">(</span><span class="nf">make-hash-table</span><span class="p">))</span> +<span class="p">(</span><span class="nf">define-class</span> <span class="nv"><agent></span> <span class="p">()</span> + <span class="p">((</span><span class="nf">rights</span> <span class="nv">:init-keyword</span> <span class="nv">:rights</span><span class="p">)</span> + <span class="p">(</span><span class="nf">responsibilities</span> <span class="nv">:init-keyword</span> <span class="nv">:responsibilities</span><span class="p">)))</span> + +<span class="p">(</span><span class="nf">define-macro</span> <span class="p">(</span><span class="nf">define-agent</span> <span class="nv">name</span> <span class="nv">rights</span> <span class="nv">responsibilities</span><span class="p">)</span> + <span class="o">`</span><span class="p">(</span><span class="nf">hash-table-put!</span> <span class="nv">agent-tables</span> <span class="ss">',name</span> + <span class="p">(</span><span class="nf">make</span> <span class="nv"><agent></span> + <span class="nv">:rights</span> <span class="ss">',rights</span> + <span class="nv">:responsibilities</span> <span class="ss">',responsibilities</span><span class="p">)))</span> + +<span class="p">(</span><span class="nf">define-agent</span> <span class="nv">Bond</span> <span class="p">(</span><span class="nf">kill</span> <span class="o">...</span><span class="p">)</span> <span class="o">...</span><span class="p">)</span> +<span class="p">(</span><span class="nf">define-agent</span> <span class="nv">Bourne</span> <span class="o">...</span><span class="p">)</span> + +<span class="o">...</span> +</pre></div> +</div> +<p><code class="docutils literal notranslate"><span class="pre">define</span></code>, <code class="docutils literal notranslate"><span class="pre">define-class</span></code>, and <code class="docutils literal notranslate"><span class="pre">define-macro</span></code> are keywords of scheme +for defining a variable, class and macro. Therefore scheme parser of +ctags should make tags for <code class="docutils literal notranslate"><span class="pre">agent-tables</span></code> with variable kind, +<code class="docutils literal notranslate"><span class="pre"><agent></span></code> with class kind, and <code class="docutils literal notranslate"><span class="pre">define-agent</span></code> with macro kind. +There is no discussion here.</p> +<blockquote> +<div><p>NOTE: To be exactly <code class="docutils literal notranslate"><span class="pre">define-class</span></code> and <code class="docutils literal notranslate"><span class="pre">define-macro</span></code> are not the part +of scheme language. They are part of gauche. That means three parsers +are stacked: scheme, gosh, and SPY-WARS.</p> +</div></blockquote> +<p>The interesting things here are <code class="docutils literal notranslate"><span class="pre">Bond</span></code> and <code class="docutils literal notranslate"><span class="pre">Bourne</span></code>.</p> +<div class="highlight-scheme notranslate"><div class="highlight"><pre><span></span><span class="p">(</span><span class="nf">define-agent</span> <span class="nv">Bond</span> <span class="p">(</span><span class="nf">kill</span> <span class="o">...</span><span class="p">)</span> <span class="o">...</span><span class="p">)</span> +<span class="p">(</span><span class="nf">define-agent</span> <span class="nv">Bourne</span> <span class="o">...</span><span class="p">)</span> +</pre></div> +</div> +<p>In scheme parser level, the two expressions define nothing; the two +expressions are just macro (<code class="docutils literal notranslate"><span class="pre">define-agent</span></code>) expansions.</p> +<p>However, in the application level, they define agents as the +macro name shown. In this level Universal Ctags should capture +<code class="docutils literal notranslate"><span class="pre">Bond</span></code> and <code class="docutils literal notranslate"><span class="pre">Bourne</span></code>. The question is which parser should +capture them? scheme parser should not; define-agent is not part of +scheme language. Newly defined SPY-WARS parser is the answer.</p> +<p>Though <code class="docutils literal notranslate"><span class="pre">define-agent</span></code> is just a macro in scheme parser level, +it is keyword in SPY-WARS parser. SPY-WARS parser makes a +tag for a token next to <code class="docutils literal notranslate"><span class="pre">define-agent</span></code>.</p> +<p>The above example illustrates levels of language in an input +file. scheme is used as the base language. With the base language we +can assume an imaginary higher level language named SPY-WARS is used +to write the application. To parse the source code of the application +written in two stacked language, ctags uses the two stacked parsers.</p> +<p>Making higher level language is very popular technique in the +languages of lisp family (see “<a class="reference external" href="http://www.paulgraham.com/onlisp.html">On Lisp</a>” for more details). +However, it is not special to lisp.</p> +<p>Following code is taken from linux kernel written in C:</p> +<div class="highlight-C notranslate"><div class="highlight"><pre><span></span><span class="n">DEFINE_EVENT</span><span class="p">(</span><span class="n">mac80211_msg_event</span><span class="p">,</span> <span class="n">mac80211_info</span><span class="p">,</span> + <span class="n">TP_PROTO</span><span class="p">(</span><span class="k">struct</span> <span class="nc">va_format</span> <span class="o">*</span><span class="n">vaf</span><span class="p">),</span> + <span class="n">TP_ARGS</span><span class="p">(</span><span class="n">vaf</span><span class="p">)</span> +<span class="p">);</span> +</pre></div> +</div> +<p>There is no concept EVENT in C language, however it make sense in the +source tree of linux kernel. So we can consider linux parser, based on +C parser, which tags <code class="docutils literal notranslate"><span class="pre">mac80211_msg_event</span></code> as <code class="docutils literal notranslate"><span class="pre">event</span></code> kind.</p> +</section> +<section id="terms"> +<h3>Terms<a class="headerlink" href="#terms" title="Permalink to this headline">¶</a></h3> +<section id="base-parser-and-subparser"> +<h4>Base parser and subparser<a class="headerlink" href="#base-parser-and-subparser" title="Permalink to this headline">¶</a></h4> +<p>In the context of the SPY-WARS example, scheme parser is called a <em>base +parser</em>. The SPY-WARS is called a <em>subparser</em>. A base parser tags +definitions found in lower level view. A subparser on the base parser tags +definitions found in higher level view. This relationship can be nested. +A subparser can be a base parser for another subparser.</p> +<figure class="align-default"> +<a class="reference internal image-reference" href="_images/stack-and-parsers.svg"><img alt="_images/stack-and-parsers.svg" src="_images/stack-and-parsers.svg" /></a> +</figure> +<p>At a glance the relationship between two parsers are similar to the +relationship guest parser and host parser description in +“<a class="reference internal" href="#host-guest-parsers"><span class="std std-ref">Guest parser: Applying a parser to specified areas of input file</span></a>”. +However, they are different. Though a guest +parser can run stand-alone, a subparser cannot; a subparser needs help +from base parser to work.</p> +</section> +<section id="top-down-parser-choice-and-bottom-up-parser-choice"> +<h4>Top down parser choice and bottom up parser choice<a class="headerlink" href="#top-down-parser-choice-and-bottom-up-parser-choice" title="Permalink to this headline">¶</a></h4> +<p>There are two ways to run a subparser: <em>top down</em> or <em>bottom up</em> parser +choices.</p> +<p>Universal Ctags can chose a subparser <a class="reference internal" href="man/ctags.1.html#guessing"><span class="std std-ref">automatically</span></a>. +Matching file name patterns and extensions are the typical ways for +choosing. A user can choose a subparser with <code class="docutils literal notranslate"><span class="pre">--language-force=</span></code> option. +Choosing a parser in these deterministic way is called <em>top down</em>. +When a parser is chosen as a subparser in the top down way, the +subparser must call its base parser. The base parser may call methods +defined in the subparser.</p> +<p>Universal Ctags uses <em>bottom up</em> choice when the top down way +doesn’t work; a given file name doesn’t match any patterns and +extensions of subparsers and the user doesn’t specify +<code class="docutils literal notranslate"><span class="pre">--language-force=</span></code> explicitly. In choosing a subparser bottom up way +it is assumed that a base parser for the subparser can be chosen +by top down way. During a base parser running, the base parser tries +to detect use of higher level languages in the input file. As shown +later in this section, the base parser utilizes methods defined in its +subparsers for the detection. If the base parser detects the use of a +higher level language, a subparser for the higher level language is +chosen. Choosing a parser in this non-deterministic way (dynamic way) +is called <em>bottom up</em>.</p> +<table class="docutils align-default"> +<colgroup> +<col style="width: 64%" /> +<col style="width: 36%" /> +</colgroup> +<thead> +<tr class="row-odd"><th class="head"><p>input file</p></th> +<th class="head"><p>subparser choices</p></th> +</tr> +</thead> +<tbody> +<tr class="row-even"><td><p><em><SUB_LANG></em> (<code class="docutils literal notranslate"><span class="pre">input.sub</span></code>)</p></td> +<td><p>top down</p></td> +</tr> +<tr class="row-odd"><td><p><em><BASE_LANG></em> (<code class="docutils literal notranslate"><span class="pre">input.base</span></code>)</p></td> +<td><p>bottom up</p></td> +</tr> +</tbody> +</table> +<p>Here is an example. Universal Ctags has both m4 parser and Autoconf +parser. The m4 parser is a base parser. The Autoconf parser is a +subparser based on the m4 parser. If <code class="docutils literal notranslate"><span class="pre">configure.ac</span></code> is given as an +input file, Autoconf parser is chosen automatically because the +Autoconf parser has <code class="docutils literal notranslate"><span class="pre">configure.ac</span></code> in its patterns list. Based on the +pattern matching, Universal Ctags chooses the Autoconf parser +automatically (top down choice).</p> +<p>If <code class="docutils literal notranslate"><span class="pre">input.m4</span></code> is given as an input file, the Autoconf parser is +not chosen. Instead the m4 parser is chosen automatically because +the m4 parser has <code class="docutils literal notranslate"><span class="pre">.m4</span></code> in its extension list. The m4 parser passes +every token finding in the input file to the +Autoconf parser. The Autoconf parser gets the chance to probe +whether the Autoconf parser itself can handle the input or not; if +a token name is started with <code class="docutils literal notranslate"><span class="pre">AC_</span></code>, the Autoconf parser +reports “this is Autoconf input though its file extension +is <code class="docutils literal notranslate"><span class="pre">.m4</span></code>” to the m4 parser. As the result the Autoconf parser is +chosen (bottom up choice).</p> +<p>Some subparsers can be chosen both top down and bottom up ways. Some +subparser can be chosen only top down way or bottom up ways.</p> +</section> +<section id="direction-flags"> +<span id="multiple-parsers-directions"></span><h4>Direction flags<a class="headerlink" href="#direction-flags" title="Permalink to this headline">¶</a></h4> +<p><em>Direction flags</em> specify how a base parser and a subparser work together. You +can choose directions by putting a long flag after +<code class="docutils literal notranslate"><span class="pre">--langdef=<SUB_LANG>{base=<BASE_LANG>}</span></code>.</p> +<p>The following three direction flags are available;</p> +<dl> +<dt><code class="docutils literal notranslate"><span class="pre">shared</span></code> (default):</dt><dd><p>For an input file of <em><BASE_LANG></em> (ex. <code class="docutils literal notranslate"><span class="pre">input.base</span></code>), tags captured by +both <em><BASE_LANG></em> and <em><SUB_LANG></em> parser are recorded to tags file.</p> +<p>For an input file of <em><SUB_LANG></em> (ex. <code class="docutils literal notranslate"><span class="pre">input.sub</span></code>), tags captured by only +<em><SUB_LANG></em> parser are recorded to tags file.</p> +<table class="docutils align-default"> +<colgroup> +<col style="width: 26%" /> +<col style="width: 36%" /> +<col style="width: 38%" /> +</colgroup> +<thead> +<tr class="row-odd"><th class="head"><p>input file</p></th> +<th class="head"><p>tags of <em><BASE_LANG></em></p></th> +<th class="head"><p>tags of <em><SUB_LANG></em></p></th> +</tr> +</thead> +<tbody> +<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">input.base</span></code></p></td> +<td><p>recorded</p></td> +<td><p>recorded</p></td> +</tr> +<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">input.sub</span></code></p></td> +<td><p>---</p></td> +<td><p>recorded</p></td> +</tr> +</tbody> +</table> +</dd> +<dt><code class="docutils literal notranslate"><span class="pre">dedicated</span></code>:</dt><dd><p>For an input file of <em><BASE_LANG></em> (ex. <code class="docutils literal notranslate"><span class="pre">input.base</span></code>), tags captured by +only <em><BASE_LANG></em> parser are recorded to tags file.</p> +<p>For an input file of <em><SUB_LANG></em> (ex. <code class="docutils literal notranslate"><span class="pre">input.sub</span></code>), tags captured by both +<em><BASE_LANG></em> and <em><SUB_LANG></em> parser are recorded to tags file.</p> +<table class="docutils align-default"> +<colgroup> +<col style="width: 26%" /> +<col style="width: 36%" /> +<col style="width: 38%" /> +</colgroup> +<thead> +<tr class="row-odd"><th class="head"><p>input file</p></th> +<th class="head"><p>tags of <em><BASE_LANG></em></p></th> +<th class="head"><p>tags of <em><SUB_LANG></em></p></th> +</tr> +</thead> +<tbody> +<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">input.base</span></code></p></td> +<td><p>recorded</p></td> +<td><p>---</p></td> +</tr> +<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">input.sub</span></code></p></td> +<td><p>recorded</p></td> +<td><p>recorded</p></td> +</tr> +</tbody> +</table> +</dd> +<dt><code class="docutils literal notranslate"><span class="pre">bidirectional</span></code>:</dt><dd><p>For an input file of both <em><BASE_LANG></em> (ex. <code class="docutils literal notranslate"><span class="pre">input.base</span></code>) and +<em><SUB_LANG></em> (ex. <code class="docutils literal notranslate"><span class="pre">input.sub</span></code>), tags captured by both <em><BASE_LANG></em> and +<SUB_LANG> parser are recorded to tags file.</p> +<table class="docutils align-default"> +<colgroup> +<col style="width: 26%" /> +<col style="width: 36%" /> +<col style="width: 38%" /> +</colgroup> +<thead> +<tr class="row-odd"><th class="head"><p>input file</p></th> +<th class="head"><p>tags of <em><BASE_LANG></em></p></th> +<th class="head"><p>tags of <em><SUB_LANG></em></p></th> +</tr> +</thead> +<tbody> +<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">input.base</span></code></p></td> +<td><p>recorded</p></td> +<td><p>recorded</p></td> +</tr> +<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">input.sub</span></code></p></td> +<td><p>recorded</p></td> +<td><p>recorded</p></td> +</tr> +</tbody> +</table> +</dd> +</dl> +<p>If no direction flag is specified, it implies <code class="docutils literal notranslate"><span class="pre">{shared}</span></code>.</p> +<p>See “<a class="reference internal" href="optlib.html#optlib-directions"><span class="std std-ref">Direction flags</span></a>” in “<a class="reference internal" href="optlib.html#optlib"><span class="std std-ref">Extending ctags with Regex parser (optlib)</span></a>” for examples of using the +direction flags.</p> +</section> +</section> +<section id="listing-subparsers"> +<h3>Listing subparsers<a class="headerlink" href="#listing-subparsers" title="Permalink to this headline">¶</a></h3> +<p>Subparsers can be listed with <code class="docutils literal notranslate"><span class="pre">--list-subparser</span></code>:</p> +<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>ctags --options<span class="o">=</span>./linux.ctags --list-subparsers<span class="o">=</span>C +<span class="gp">#</span>NAME BASEPARSER DIRECTION +<span class="go">linux C base => sub {shared}</span> +</pre></div> +</div> +</section> +<section id="id1"> +<h3>Command line interface<a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h3> +<p>Running subparser can be controlled with subparser (<code class="docutils literal notranslate"><span class="pre">s</span></code>) extras flag. +By default it is enabled. To turning off the feature running +subparser, specify <code class="docutils literal notranslate"><span class="pre">--extras=-s</span></code>.</p> +</section> +<section id="examples-of-subparser"> +<h3>Examples of subparser<a class="headerlink" href="#examples-of-subparser" title="Permalink to this headline">¶</a></h3> +<section id="automake-make-parser-combination"> +<h4>Automake/Make parser combination<a class="headerlink" href="#automake-make-parser-combination" title="Permalink to this headline">¶</a></h4> +<p>Simply to say the syntax of Automake is the subset of Make. However, +the Automake parser has interests in Make macros having special +suffixes: <code class="docutils literal notranslate"><span class="pre">_PROGRAMS</span></code>, <code class="docutils literal notranslate"><span class="pre">_LTLIBRARIES</span></code>, and <code class="docutils literal notranslate"><span class="pre">_SCRIPTS</span></code> so on.</p> +<p>Here is an example of input for Automake:</p> +<div class="highlight-Make notranslate"><div class="highlight"><pre><span></span><span class="nv">bin_PROGRAMS</span> <span class="o">=</span> ctags +<span class="nv">ctags_CPPFLAGS</span> <span class="o">=</span> <span class="se">\</span> + -I. <span class="se">\</span> + -I<span class="k">$(</span>srcdir<span class="k">)</span> <span class="se">\</span> + -I<span class="k">$(</span>srcdir<span class="k">)</span>/main +</pre></div> +</div> +<p>From the point of the view of the Make parser, <code class="docutils literal notranslate"><span class="pre">bin_PROGRAMS</span></code> is a just +a macro; the Make parser tags <code class="docutils literal notranslate"><span class="pre">bin_PROGRAMS</span></code> as a macro. The Make parser +doesn’t tag <code class="docutils literal notranslate"><span class="pre">ctags</span></code> being right side of ‘<code class="docutils literal notranslate"><span class="pre">=</span></code>’ because it is not a new +name: just a value assigned to bin_PROGRAMS. However, for the Automake +parser <code class="docutils literal notranslate"><span class="pre">ctags</span></code> is a new name; the Automake parser tags <code class="docutils literal notranslate"><span class="pre">ctags</span></code> with +kind <code class="docutils literal notranslate"><span class="pre">Program</span></code>. The Automake parser can tag it with getting help from +the Make parser.</p> +<p>The Automake parser is an exclusive subparser. It is chosen in top +down way; an input file name <code class="docutils literal notranslate"><span class="pre">Makefile.am</span></code> gives enough information for +choosing the Automake parser.</p> +<p>To give chances to the Automake parser to capture Automake own +definitions, The Make parser provides following interface in +<code class="docutils literal notranslate"><span class="pre">parsers/make.h</span></code>:</p> +<div class="highlight-C notranslate"><div class="highlight"><pre><span></span><span class="k">struct</span> <span class="nc">sMakeSubparser</span> <span class="p">{</span> + <span class="n">subparser</span> <span class="n">subparser</span><span class="p">;</span> + + <span class="kt">void</span> <span class="p">(</span><span class="o">*</span> <span class="n">valueNotify</span><span class="p">)</span> <span class="p">(</span><span class="n">makeSubparser</span> <span class="o">*</span><span class="n">s</span><span class="p">,</span> <span class="kt">char</span><span class="o">*</span> <span class="n">name</span><span class="p">);</span> + <span class="kt">void</span> <span class="p">(</span><span class="o">*</span> <span class="n">directiveNotify</span><span class="p">)</span> <span class="p">(</span><span class="n">makeSubparser</span> <span class="o">*</span><span class="n">s</span><span class="p">,</span> <span class="kt">char</span><span class="o">*</span> <span class="n">name</span><span class="p">);</span> + <span class="kt">void</span> <span class="p">(</span><span class="o">*</span> <span class="n">newMacroNotify</span><span class="p">)</span> <span class="p">(</span><span class="n">makeSubparser</span> <span class="o">*</span><span class="n">s</span><span class="p">,</span> + <span class="kt">char</span><span class="o">*</span> <span class="n">name</span><span class="p">,</span> + <span class="kt">bool</span> <span class="n">withDefineDirective</span><span class="p">,</span> + <span class="kt">bool</span> <span class="n">appending</span><span class="p">);</span> +<span class="p">};</span> +</pre></div> +</div> +<p>The Automake parser defines methods for tagging Automake own definitions +in a <code class="docutils literal notranslate"><span class="pre">struct</span> <span class="pre">sMakeSubparser</span></code> type variable, and runs the Make parser by +calling <code class="docutils literal notranslate"><span class="pre">scheduleRunningBaseparser</span></code> function.</p> +<p>The Make parser tags Make own definitions in an input file. In +addition Make parser calls the methods during parsing the input file.</p> +<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>ctags --fields<span class="o">=</span>+lK --extras<span class="o">=</span>+r -o - Makefile.am +<span class="go">bin Makefile.am /^bin_PROGRAMS = ctags$/;" directory language:Automake</span> +<span class="go">bin_PROGRAMS Makefile.am /^bin_PROGRAMS = ctags$/;" macro language:Make</span> +<span class="go">ctags Makefile.am /^bin_PROGRAMS = ctags$/;" program language:Automake directory:bin</span> +<span class="go">ctags_CPPFLAGS Makefile.am /^ctags_CPPFLAGS = \\$/;" macro language:Make</span> +</pre></div> +</div> +<p><code class="docutils literal notranslate"><span class="pre">bin_PROGRAMS</span></code> and <code class="docutils literal notranslate"><span class="pre">ctags_CPPFLAGS</span></code> are tagged as macros of Make. +In addition <code class="docutils literal notranslate"><span class="pre">bin</span></code> is tagged as directory, and <code class="docutils literal notranslate"><span class="pre">ctags</span></code> as program of Automake.</p> +<p><code class="docutils literal notranslate"><span class="pre">bin</span></code> is tagged in a callback function assigned to <code class="docutils literal notranslate"><span class="pre">newMacroFound</span></code> method. +<code class="docutils literal notranslate"><span class="pre">ctags</span></code> is tagged in a callback function assigned to <code class="docutils literal notranslate"><span class="pre">valuesFound</span></code> method.</p> +<p><code class="docutils literal notranslate"><span class="pre">--extras=+r</span></code> is used in the example. Reference (<code class="docutils literal notranslate"><span class="pre">r</span></code>) extra is needed to +tag <code class="docutils literal notranslate"><span class="pre">bin</span></code>. <code class="docutils literal notranslate"><span class="pre">bin</span></code> is not defined in the line, <code class="docutils literal notranslate"><span class="pre">bin_PROGRAMS</span> <span class="pre">=</span></code>. +<code class="docutils literal notranslate"><span class="pre">bin</span></code> is referenced as a name of directory where programs are +stored. Therefore <code class="docutils literal notranslate"><span class="pre">r</span></code> is needed.</p> +<p>For tagging <code class="docutils literal notranslate"><span class="pre">ctags</span></code>, the Automake parser must recognize +<code class="docutils literal notranslate"><span class="pre">bin</span></code> in <code class="docutils literal notranslate"><span class="pre">bin_PROGRAMS</span></code> first. <code class="docutils literal notranslate"><span class="pre">ctags</span></code> is tagged +because it is specified as a value for <code class="docutils literal notranslate"><span class="pre">bin_PROGRAMS</span></code>. +As the result <code class="docutils literal notranslate"><span class="pre">r</span></code> is also needed to tag <code class="docutils literal notranslate"><span class="pre">ctags</span></code>.</p> +<p>Only Automake related tags are emitted if Make parser is +disabled.</p> +<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>ctags --languages<span class="o">=</span>-Make --fields<span class="o">=</span>+lKr --extras<span class="o">=</span>+r -o - Makefile.am +<span class="go">bin Makefile.am /^bin_PROGRAMS = ctags$/;" directory language:Automake roles:program</span> +<span class="go">ctags Makefile.am /^bin_PROGRAMS = ctags$/;" program language:Automake directory:bin</span> +</pre></div> +</div> +</section> +<section id="autoconf-m4-parser-combination"> +<h4>Autoconf/M4 parser combination<a class="headerlink" href="#autoconf-m4-parser-combination" title="Permalink to this headline">¶</a></h4> +<p>Universal Ctags uses m4 parser as a base parser and Autoconf parse as +a subparser for <code class="docutils literal notranslate"><span class="pre">configure.ac</span></code> input file.</p> +<div class="highlight-Autoconf notranslate"><div class="highlight"><pre><span></span>AC_DEFUN([PRETTY_VAR_EXPAND], + [$(eval "$as_echo_n" $(eval "$as_echo_n" "${$1}"))]) +</pre></div> +</div> +<p>The m4 parser finds no definition here. However, Autoconf parser finds +<code class="docutils literal notranslate"><span class="pre">PRETTY_VAR_EXPAND</span></code> as a macro definition. Syntax like <code class="docutils literal notranslate"><span class="pre">(...)</span></code> is part +of M4 language. So Autoconf parser is implemented as a subparser of +m4 parser. The most parts of tokens in input files are handled by +M4. Autoconf parser gives hints for parsing <code class="docutils literal notranslate"><span class="pre">configure.ac</span></code> and +registers callback functions to +Autoconf parser.</p> +</section> +</section> +</section> +</section> + + + <div class="clearer"></div> + </div> + </div> + </div> + <div class="sphinxsidebar" role="navigation" aria-label="main navigation"> + <div class="sphinxsidebarwrapper"> + <h3><a href="index.html">Table of Contents</a></h3> + <ul> +<li><a class="reference internal" href="#">Running multiple parsers on an input file</a><ul> +<li><a class="reference internal" href="#guest-parser-applying-a-parser-to-specified-areas-of-input-file">Guest parser: Applying a parser to specified areas of input file</a><ul> +<li><a class="reference internal" href="#command-line-interface">Command line interface</a></li> +<li><a class="reference internal" href="#examples-of-guest-parser">Examples of guest parser</a><ul> +<li><a class="reference internal" href="#css-javascript-html-parser-combination">{CSS,JavaScript}/HTML parser combination</a></li> +<li><a class="reference internal" href="#c-yacc-parser-combination">C/Yacc parser combination</a></li> +<li><a class="reference internal" href="#pod-perl-parser-combination">Pod/Perl parser combination</a></li> +</ul> +</li> +</ul> +</li> +<li><a class="reference internal" href="#subparser-tagging-definitions-of-higher-upper-level-language">Subparser: Tagging definitions of higher (upper) level language</a><ul> +<li><a class="reference internal" href="#background">Background</a></li> +<li><a class="reference internal" href="#terms">Terms</a><ul> +<li><a class="reference internal" href="#base-parser-and-subparser">Base parser and subparser</a></li> +<li><a class="reference internal" href="#top-down-parser-choice-and-bottom-up-parser-choice">Top down parser choice and bottom up parser choice</a></li> +<li><a class="reference internal" href="#direction-flags">Direction flags</a></li> +</ul> +</li> +<li><a class="reference internal" href="#listing-subparsers">Listing subparsers</a></li> +<li><a class="reference internal" href="#id1">Command line interface</a></li> +<li><a class="reference internal" href="#examples-of-subparser">Examples of subparser</a><ul> +<li><a class="reference internal" href="#automake-make-parser-combination">Automake/Make parser combination</a></li> +<li><a class="reference internal" href="#autoconf-m4-parser-combination">Autoconf/M4 parser combination</a></li> +</ul> +</li> +</ul> +</li> +</ul> +</li> +</ul> + + <h4>Previous topic</h4> + <p class="topless"><a href="output-json.html" + title="previous chapter">JSON output</a></p> + <h4>Next topic</h4> + <p class="topless"><a href="interactive-mode.html" + title="next chapter">Interactive mode</a></p> +<div id="searchbox" style="display: none" role="search"> + <h3 id="searchlabel">Quick search</h3> + <div class="searchformwrapper"> + <form class="search" action="search.html" method="get"> + <input type="text" name="q" aria-labelledby="searchlabel" /> + <input type="submit" value="Go" /> + </form> + </div> +</div> +<script>$('#searchbox').show(0);</script> + </div> + </div> + <div class="clearer"></div> + </div> + <div class="related" role="navigation" aria-label="related navigation"> + <h3>Navigation</h3> + <ul> + <li class="right" style="margin-right: 10px"> + <a href="genindex.html" title="General Index" + >index</a></li> + <li class="right" > + <a href="interactive-mode.html" title="Interactive mode" + >next</a> |</li> + <li class="right" > + <a href="output-json.html" title="JSON output" + >previous</a> |</li> + <li class="nav-item nav-item-0"><a href="index.html">Universal Ctags 0.3.0 documentation</a> »</li> + <li class="nav-item nav-item-this"><a href="">Running multiple parsers on an input file</a></li> + </ul> + </div> + <div class="footer" role="contentinfo"> + © Copyright 2015, Universal Ctags Team. + Last updated on 11 Jun 2021. + Created using <a href="https://www.sphinx-doc.org/">Sphinx</a> 4.0.2. + </div> + </body> +</html>
\ No newline at end of file |