1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
|
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
<title>Running multiple parsers on an input file — Universal Ctags 0.3.0 documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="_static/classic.css" />
<script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
<script src="_static/jquery.js"></script>
<script src="_static/underscore.js"></script>
<script src="_static/doctools.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Interactive mode" href="interactive-mode.html" />
<link rel="prev" title="JSON output" href="output-json.html" />
</head><body>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="interactive-mode.html" title="Interactive mode"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="output-json.html" title="JSON output"
accesskey="P">previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">Universal Ctags 0.3.0 documentation</a> »</li>
<li class="nav-item nav-item-this"><a href="">Running multiple parsers on an input file</a></li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<section id="running-multiple-parsers-on-an-input-file">
<span id="multiple-parsers"></span><h1>Running multiple parsers on an input file<a class="headerlink" href="#running-multiple-parsers-on-an-input-file" title="Permalink to this headline">¶</a></h1>
<p>Universal Ctags provides parser developers two ways, <em>guest parser</em> (guest/host)
and <em>subparser</em> (sub/base), to run multiple parsers for an input file.</p>
<p>This section shows concepts behind the running multiple parsers and real
examples.</p>
<section id="guest-parser-applying-a-parser-to-specified-areas-of-input-file">
<span id="host-guest-parsers"></span><h2>Guest parser: Applying a parser to specified areas of input file<a class="headerlink" href="#guest-parser-applying-a-parser-to-specified-areas-of-input-file" title="Permalink to this headline">¶</a></h2>
<p><em>Guest parser</em> (guest/host) considers the case that an input file has areas
written in languages different from the language for the input file.</p>
<p>A <em>host parser</em> parses the input file and detects the areas.
The host parser schedules <em>guest parsers</em> parsing the areas.
The guest parsers parse the areas.</p>
<figure class="align-default">
<a class="reference internal image-reference" href="_images/area-and-parsers.svg"><img alt="_images/area-and-parsers.svg" src="_images/area-and-parsers.svg" /></a>
</figure>
<section id="command-line-interface">
<h3>Command line interface<a class="headerlink" href="#command-line-interface" title="Permalink to this headline">¶</a></h3>
<p>Running guest parser can be controlled with guest (<code class="docutils literal notranslate"><span class="pre">g</span></code>) extras flag.
By default it is disabled. To turning on the feature running
guest parser, specify <code class="docutils literal notranslate"><span class="pre">--extras=+g</span></code>.</p>
<p>If <code class="docutils literal notranslate"><span class="pre">--fields=+E</span></code> is given, all tags generated by a guest parser is marked
<code class="docutils literal notranslate"><span class="pre">guest</span></code> in their <code class="docutils literal notranslate"><span class="pre">extras:</span></code> fields.</p>
</section>
<section id="examples-of-guest-parser">
<h3>Examples of guest parser<a class="headerlink" href="#examples-of-guest-parser" title="Permalink to this headline">¶</a></h3>
<section id="css-javascript-html-parser-combination">
<h4>{CSS,JavaScript}/HTML parser combination<a class="headerlink" href="#css-javascript-html-parser-combination" title="Permalink to this headline">¶</a></h4>
<p>For an HTML file, you may want to run HTML parser, of course. The
HTML file may have CSS areas and JavaScript areas. In other hand
Universal Ctags has both CSS and JavaScript parsers. Don’t you
think it is useful if you can apply these parsers to the areas?</p>
<p>In this case, HTML has responsible to detect the CSS and
JavaScript areas and record the positions of the areas.
The HTML parser schedules delayed invocations of CSS and
JavaScript parsers on the area with promise API.</p>
<p>Here HTML parser is a <em>host parser</em>. CSS and JavaScript parsers
are <em>guest parsers</em>.</p>
<p>See “<a class="reference internal" href="parser-html.html#html"><span class="std std-ref">The new HTML parser</span></a>” and “<a class="reference external" href="https://github.com/universal-ctags/ctags/blob/master/parsers/html.c">parsers/html.c</a>”.</p>
</section>
<section id="c-yacc-parser-combination">
<h4>C/Yacc parser combination<a class="headerlink" href="#c-yacc-parser-combination" title="Permalink to this headline">¶</a></h4>
<p>A yacc file has some areas written in C. Universal Ctags has both YACC
and C parsers. You may want to run C parser for the areas from YACC
parser.</p>
<p>Here YACC parser is a host parser. C parser is a guest parser.
See “<a class="reference external" href="https://github.com/universal-ctags/ctags/blob/master/parsers/yacc.c">parsers/yacc.c</a>”.</p>
</section>
<section id="pod-perl-parser-combination">
<h4>Pod/Perl parser combination<a class="headerlink" href="#pod-perl-parser-combination" title="Permalink to this headline">¶</a></h4>
<p>Pod (Plain Old Documentation) is a language for documentation. The language
can be used not only in a stand alone file but also it can be
used inside a Perl script.</p>
<p>Universal Ctags has both parsers for Perl and Pod.
The Perl parser recognizes the area where Pod document is
embedded in a Perl script and schedules applying pod parser
as a guest parser on the area.</p>
</section>
</section>
</section>
<section id="subparser-tagging-definitions-of-higher-upper-level-language">
<span id="base-sub-parsers"></span><h2>Subparser: Tagging definitions of higher (upper) level language<a class="headerlink" href="#subparser-tagging-definitions-of-higher-upper-level-language" title="Permalink to this headline">¶</a></h2>
<section id="background">
<h3>Background<a class="headerlink" href="#background" title="Permalink to this headline">¶</a></h3>
<p>Consider an application written in language X. The application has
its domain own concepts. Developers of the application may try to
express the concepts in the syntax of language X.</p>
<p>In language X level, the developer can define functions, variables, types, and
so on. Further more, if the syntax of X allows, the developers want to
define higher level (= application level) things for implementing the
domain own concepts.</p>
<p>Let me show the part of source code of SPY-WARS, an imaginary game application.
It is written in scheme language, a dialect of lisp.
(Here <a class="reference external" href="https://practical-scheme.net/gauche/index.html">gauche</a> is considered
as the implementation of scheme interpreter).</p>
<div class="highlight-scheme notranslate"><div class="highlight"><pre><span></span><span class="p">(</span><span class="k">define </span><span class="nv">agent-tables</span> <span class="p">(</span><span class="nf">make-hash-table</span><span class="p">))</span>
<span class="p">(</span><span class="nf">define-class</span> <span class="nv"><agent></span> <span class="p">()</span>
<span class="p">((</span><span class="nf">rights</span> <span class="nv">:init-keyword</span> <span class="nv">:rights</span><span class="p">)</span>
<span class="p">(</span><span class="nf">responsibilities</span> <span class="nv">:init-keyword</span> <span class="nv">:responsibilities</span><span class="p">)))</span>
<span class="p">(</span><span class="nf">define-macro</span> <span class="p">(</span><span class="nf">define-agent</span> <span class="nv">name</span> <span class="nv">rights</span> <span class="nv">responsibilities</span><span class="p">)</span>
<span class="o">`</span><span class="p">(</span><span class="nf">hash-table-put!</span> <span class="nv">agent-tables</span> <span class="ss">',name</span>
<span class="p">(</span><span class="nf">make</span> <span class="nv"><agent></span>
<span class="nv">:rights</span> <span class="ss">',rights</span>
<span class="nv">:responsibilities</span> <span class="ss">',responsibilities</span><span class="p">)))</span>
<span class="p">(</span><span class="nf">define-agent</span> <span class="nv">Bond</span> <span class="p">(</span><span class="nf">kill</span> <span class="o">...</span><span class="p">)</span> <span class="o">...</span><span class="p">)</span>
<span class="p">(</span><span class="nf">define-agent</span> <span class="nv">Bourne</span> <span class="o">...</span><span class="p">)</span>
<span class="o">...</span>
</pre></div>
</div>
<p><code class="docutils literal notranslate"><span class="pre">define</span></code>, <code class="docutils literal notranslate"><span class="pre">define-class</span></code>, and <code class="docutils literal notranslate"><span class="pre">define-macro</span></code> are keywords of scheme
for defining a variable, class and macro. Therefore scheme parser of
ctags should make tags for <code class="docutils literal notranslate"><span class="pre">agent-tables</span></code> with variable kind,
<code class="docutils literal notranslate"><span class="pre"><agent></span></code> with class kind, and <code class="docutils literal notranslate"><span class="pre">define-agent</span></code> with macro kind.
There is no discussion here.</p>
<blockquote>
<div><p>NOTE: To be exactly <code class="docutils literal notranslate"><span class="pre">define-class</span></code> and <code class="docutils literal notranslate"><span class="pre">define-macro</span></code> are not the part
of scheme language. They are part of gauche. That means three parsers
are stacked: scheme, gosh, and SPY-WARS.</p>
</div></blockquote>
<p>The interesting things here are <code class="docutils literal notranslate"><span class="pre">Bond</span></code> and <code class="docutils literal notranslate"><span class="pre">Bourne</span></code>.</p>
<div class="highlight-scheme notranslate"><div class="highlight"><pre><span></span><span class="p">(</span><span class="nf">define-agent</span> <span class="nv">Bond</span> <span class="p">(</span><span class="nf">kill</span> <span class="o">...</span><span class="p">)</span> <span class="o">...</span><span class="p">)</span>
<span class="p">(</span><span class="nf">define-agent</span> <span class="nv">Bourne</span> <span class="o">...</span><span class="p">)</span>
</pre></div>
</div>
<p>In scheme parser level, the two expressions define nothing; the two
expressions are just macro (<code class="docutils literal notranslate"><span class="pre">define-agent</span></code>) expansions.</p>
<p>However, in the application level, they define agents as the
macro name shown. In this level Universal Ctags should capture
<code class="docutils literal notranslate"><span class="pre">Bond</span></code> and <code class="docutils literal notranslate"><span class="pre">Bourne</span></code>. The question is which parser should
capture them? scheme parser should not; define-agent is not part of
scheme language. Newly defined SPY-WARS parser is the answer.</p>
<p>Though <code class="docutils literal notranslate"><span class="pre">define-agent</span></code> is just a macro in scheme parser level,
it is keyword in SPY-WARS parser. SPY-WARS parser makes a
tag for a token next to <code class="docutils literal notranslate"><span class="pre">define-agent</span></code>.</p>
<p>The above example illustrates levels of language in an input
file. scheme is used as the base language. With the base language we
can assume an imaginary higher level language named SPY-WARS is used
to write the application. To parse the source code of the application
written in two stacked language, ctags uses the two stacked parsers.</p>
<p>Making higher level language is very popular technique in the
languages of lisp family (see “<a class="reference external" href="http://www.paulgraham.com/onlisp.html">On Lisp</a>” for more details).
However, it is not special to lisp.</p>
<p>Following code is taken from linux kernel written in C:</p>
<div class="highlight-C notranslate"><div class="highlight"><pre><span></span><span class="n">DEFINE_EVENT</span><span class="p">(</span><span class="n">mac80211_msg_event</span><span class="p">,</span> <span class="n">mac80211_info</span><span class="p">,</span>
<span class="n">TP_PROTO</span><span class="p">(</span><span class="k">struct</span> <span class="nc">va_format</span> <span class="o">*</span><span class="n">vaf</span><span class="p">),</span>
<span class="n">TP_ARGS</span><span class="p">(</span><span class="n">vaf</span><span class="p">)</span>
<span class="p">);</span>
</pre></div>
</div>
<p>There is no concept EVENT in C language, however it make sense in the
source tree of linux kernel. So we can consider linux parser, based on
C parser, which tags <code class="docutils literal notranslate"><span class="pre">mac80211_msg_event</span></code> as <code class="docutils literal notranslate"><span class="pre">event</span></code> kind.</p>
</section>
<section id="terms">
<h3>Terms<a class="headerlink" href="#terms" title="Permalink to this headline">¶</a></h3>
<section id="base-parser-and-subparser">
<h4>Base parser and subparser<a class="headerlink" href="#base-parser-and-subparser" title="Permalink to this headline">¶</a></h4>
<p>In the context of the SPY-WARS example, scheme parser is called a <em>base
parser</em>. The SPY-WARS is called a <em>subparser</em>. A base parser tags
definitions found in lower level view. A subparser on the base parser tags
definitions found in higher level view. This relationship can be nested.
A subparser can be a base parser for another subparser.</p>
<figure class="align-default">
<a class="reference internal image-reference" href="_images/stack-and-parsers.svg"><img alt="_images/stack-and-parsers.svg" src="_images/stack-and-parsers.svg" /></a>
</figure>
<p>At a glance the relationship between two parsers are similar to the
relationship guest parser and host parser description in
“<a class="reference internal" href="#host-guest-parsers"><span class="std std-ref">Guest parser: Applying a parser to specified areas of input file</span></a>”.
However, they are different. Though a guest
parser can run stand-alone, a subparser cannot; a subparser needs help
from base parser to work.</p>
</section>
<section id="top-down-parser-choice-and-bottom-up-parser-choice">
<h4>Top down parser choice and bottom up parser choice<a class="headerlink" href="#top-down-parser-choice-and-bottom-up-parser-choice" title="Permalink to this headline">¶</a></h4>
<p>There are two ways to run a subparser: <em>top down</em> or <em>bottom up</em> parser
choices.</p>
<p>Universal Ctags can chose a subparser <a class="reference internal" href="man/ctags.1.html#guessing"><span class="std std-ref">automatically</span></a>.
Matching file name patterns and extensions are the typical ways for
choosing. A user can choose a subparser with <code class="docutils literal notranslate"><span class="pre">--language-force=</span></code> option.
Choosing a parser in these deterministic way is called <em>top down</em>.
When a parser is chosen as a subparser in the top down way, the
subparser must call its base parser. The base parser may call methods
defined in the subparser.</p>
<p>Universal Ctags uses <em>bottom up</em> choice when the top down way
doesn’t work; a given file name doesn’t match any patterns and
extensions of subparsers and the user doesn’t specify
<code class="docutils literal notranslate"><span class="pre">--language-force=</span></code> explicitly. In choosing a subparser bottom up way
it is assumed that a base parser for the subparser can be chosen
by top down way. During a base parser running, the base parser tries
to detect use of higher level languages in the input file. As shown
later in this section, the base parser utilizes methods defined in its
subparsers for the detection. If the base parser detects the use of a
higher level language, a subparser for the higher level language is
chosen. Choosing a parser in this non-deterministic way (dynamic way)
is called <em>bottom up</em>.</p>
<table class="docutils align-default">
<colgroup>
<col style="width: 64%" />
<col style="width: 36%" />
</colgroup>
<thead>
<tr class="row-odd"><th class="head"><p>input file</p></th>
<th class="head"><p>subparser choices</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p><em><SUB_LANG></em> (<code class="docutils literal notranslate"><span class="pre">input.sub</span></code>)</p></td>
<td><p>top down</p></td>
</tr>
<tr class="row-odd"><td><p><em><BASE_LANG></em> (<code class="docutils literal notranslate"><span class="pre">input.base</span></code>)</p></td>
<td><p>bottom up</p></td>
</tr>
</tbody>
</table>
<p>Here is an example. Universal Ctags has both m4 parser and Autoconf
parser. The m4 parser is a base parser. The Autoconf parser is a
subparser based on the m4 parser. If <code class="docutils literal notranslate"><span class="pre">configure.ac</span></code> is given as an
input file, Autoconf parser is chosen automatically because the
Autoconf parser has <code class="docutils literal notranslate"><span class="pre">configure.ac</span></code> in its patterns list. Based on the
pattern matching, Universal Ctags chooses the Autoconf parser
automatically (top down choice).</p>
<p>If <code class="docutils literal notranslate"><span class="pre">input.m4</span></code> is given as an input file, the Autoconf parser is
not chosen. Instead the m4 parser is chosen automatically because
the m4 parser has <code class="docutils literal notranslate"><span class="pre">.m4</span></code> in its extension list. The m4 parser passes
every token finding in the input file to the
Autoconf parser. The Autoconf parser gets the chance to probe
whether the Autoconf parser itself can handle the input or not; if
a token name is started with <code class="docutils literal notranslate"><span class="pre">AC_</span></code>, the Autoconf parser
reports “this is Autoconf input though its file extension
is <code class="docutils literal notranslate"><span class="pre">.m4</span></code>” to the m4 parser. As the result the Autoconf parser is
chosen (bottom up choice).</p>
<p>Some subparsers can be chosen both top down and bottom up ways. Some
subparser can be chosen only top down way or bottom up ways.</p>
</section>
<section id="direction-flags">
<span id="multiple-parsers-directions"></span><h4>Direction flags<a class="headerlink" href="#direction-flags" title="Permalink to this headline">¶</a></h4>
<p><em>Direction flags</em> specify how a base parser and a subparser work together. You
can choose directions by putting a long flag after
<code class="docutils literal notranslate"><span class="pre">--langdef=<SUB_LANG>{base=<BASE_LANG>}</span></code>.</p>
<p>The following three direction flags are available;</p>
<dl>
<dt><code class="docutils literal notranslate"><span class="pre">shared</span></code> (default):</dt><dd><p>For an input file of <em><BASE_LANG></em> (ex. <code class="docutils literal notranslate"><span class="pre">input.base</span></code>), tags captured by
both <em><BASE_LANG></em> and <em><SUB_LANG></em> parser are recorded to tags file.</p>
<p>For an input file of <em><SUB_LANG></em> (ex. <code class="docutils literal notranslate"><span class="pre">input.sub</span></code>), tags captured by only
<em><SUB_LANG></em> parser are recorded to tags file.</p>
<table class="docutils align-default">
<colgroup>
<col style="width: 26%" />
<col style="width: 36%" />
<col style="width: 38%" />
</colgroup>
<thead>
<tr class="row-odd"><th class="head"><p>input file</p></th>
<th class="head"><p>tags of <em><BASE_LANG></em></p></th>
<th class="head"><p>tags of <em><SUB_LANG></em></p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">input.base</span></code></p></td>
<td><p>recorded</p></td>
<td><p>recorded</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">input.sub</span></code></p></td>
<td><p>---</p></td>
<td><p>recorded</p></td>
</tr>
</tbody>
</table>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">dedicated</span></code>:</dt><dd><p>For an input file of <em><BASE_LANG></em> (ex. <code class="docutils literal notranslate"><span class="pre">input.base</span></code>), tags captured by
only <em><BASE_LANG></em> parser are recorded to tags file.</p>
<p>For an input file of <em><SUB_LANG></em> (ex. <code class="docutils literal notranslate"><span class="pre">input.sub</span></code>), tags captured by both
<em><BASE_LANG></em> and <em><SUB_LANG></em> parser are recorded to tags file.</p>
<table class="docutils align-default">
<colgroup>
<col style="width: 26%" />
<col style="width: 36%" />
<col style="width: 38%" />
</colgroup>
<thead>
<tr class="row-odd"><th class="head"><p>input file</p></th>
<th class="head"><p>tags of <em><BASE_LANG></em></p></th>
<th class="head"><p>tags of <em><SUB_LANG></em></p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">input.base</span></code></p></td>
<td><p>recorded</p></td>
<td><p>---</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">input.sub</span></code></p></td>
<td><p>recorded</p></td>
<td><p>recorded</p></td>
</tr>
</tbody>
</table>
</dd>
<dt><code class="docutils literal notranslate"><span class="pre">bidirectional</span></code>:</dt><dd><p>For an input file of both <em><BASE_LANG></em> (ex. <code class="docutils literal notranslate"><span class="pre">input.base</span></code>) and
<em><SUB_LANG></em> (ex. <code class="docutils literal notranslate"><span class="pre">input.sub</span></code>), tags captured by both <em><BASE_LANG></em> and
<SUB_LANG> parser are recorded to tags file.</p>
<table class="docutils align-default">
<colgroup>
<col style="width: 26%" />
<col style="width: 36%" />
<col style="width: 38%" />
</colgroup>
<thead>
<tr class="row-odd"><th class="head"><p>input file</p></th>
<th class="head"><p>tags of <em><BASE_LANG></em></p></th>
<th class="head"><p>tags of <em><SUB_LANG></em></p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">input.base</span></code></p></td>
<td><p>recorded</p></td>
<td><p>recorded</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">input.sub</span></code></p></td>
<td><p>recorded</p></td>
<td><p>recorded</p></td>
</tr>
</tbody>
</table>
</dd>
</dl>
<p>If no direction flag is specified, it implies <code class="docutils literal notranslate"><span class="pre">{shared}</span></code>.</p>
<p>See “<a class="reference internal" href="optlib.html#optlib-directions"><span class="std std-ref">Direction flags</span></a>” in “<a class="reference internal" href="optlib.html#optlib"><span class="std std-ref">Extending ctags with Regex parser (optlib)</span></a>” for examples of using the
direction flags.</p>
</section>
</section>
<section id="listing-subparsers">
<h3>Listing subparsers<a class="headerlink" href="#listing-subparsers" title="Permalink to this headline">¶</a></h3>
<p>Subparsers can be listed with <code class="docutils literal notranslate"><span class="pre">--list-subparser</span></code>:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>ctags --options<span class="o">=</span>./linux.ctags --list-subparsers<span class="o">=</span>C
<span class="gp">#</span>NAME BASEPARSER DIRECTION
<span class="go">linux C base => sub {shared}</span>
</pre></div>
</div>
</section>
<section id="id1">
<h3>Command line interface<a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h3>
<p>Running subparser can be controlled with subparser (<code class="docutils literal notranslate"><span class="pre">s</span></code>) extras flag.
By default it is enabled. To turning off the feature running
subparser, specify <code class="docutils literal notranslate"><span class="pre">--extras=-s</span></code>.</p>
</section>
<section id="examples-of-subparser">
<h3>Examples of subparser<a class="headerlink" href="#examples-of-subparser" title="Permalink to this headline">¶</a></h3>
<section id="automake-make-parser-combination">
<h4>Automake/Make parser combination<a class="headerlink" href="#automake-make-parser-combination" title="Permalink to this headline">¶</a></h4>
<p>Simply to say the syntax of Automake is the subset of Make. However,
the Automake parser has interests in Make macros having special
suffixes: <code class="docutils literal notranslate"><span class="pre">_PROGRAMS</span></code>, <code class="docutils literal notranslate"><span class="pre">_LTLIBRARIES</span></code>, and <code class="docutils literal notranslate"><span class="pre">_SCRIPTS</span></code> so on.</p>
<p>Here is an example of input for Automake:</p>
<div class="highlight-Make notranslate"><div class="highlight"><pre><span></span><span class="nv">bin_PROGRAMS</span> <span class="o">=</span> ctags
<span class="nv">ctags_CPPFLAGS</span> <span class="o">=</span> <span class="se">\</span>
-I. <span class="se">\</span>
-I<span class="k">$(</span>srcdir<span class="k">)</span> <span class="se">\</span>
-I<span class="k">$(</span>srcdir<span class="k">)</span>/main
</pre></div>
</div>
<p>From the point of the view of the Make parser, <code class="docutils literal notranslate"><span class="pre">bin_PROGRAMS</span></code> is a just
a macro; the Make parser tags <code class="docutils literal notranslate"><span class="pre">bin_PROGRAMS</span></code> as a macro. The Make parser
doesn’t tag <code class="docutils literal notranslate"><span class="pre">ctags</span></code> being right side of ‘<code class="docutils literal notranslate"><span class="pre">=</span></code>’ because it is not a new
name: just a value assigned to bin_PROGRAMS. However, for the Automake
parser <code class="docutils literal notranslate"><span class="pre">ctags</span></code> is a new name; the Automake parser tags <code class="docutils literal notranslate"><span class="pre">ctags</span></code> with
kind <code class="docutils literal notranslate"><span class="pre">Program</span></code>. The Automake parser can tag it with getting help from
the Make parser.</p>
<p>The Automake parser is an exclusive subparser. It is chosen in top
down way; an input file name <code class="docutils literal notranslate"><span class="pre">Makefile.am</span></code> gives enough information for
choosing the Automake parser.</p>
<p>To give chances to the Automake parser to capture Automake own
definitions, The Make parser provides following interface in
<code class="docutils literal notranslate"><span class="pre">parsers/make.h</span></code>:</p>
<div class="highlight-C notranslate"><div class="highlight"><pre><span></span><span class="k">struct</span> <span class="nc">sMakeSubparser</span> <span class="p">{</span>
<span class="n">subparser</span> <span class="n">subparser</span><span class="p">;</span>
<span class="kt">void</span> <span class="p">(</span><span class="o">*</span> <span class="n">valueNotify</span><span class="p">)</span> <span class="p">(</span><span class="n">makeSubparser</span> <span class="o">*</span><span class="n">s</span><span class="p">,</span> <span class="kt">char</span><span class="o">*</span> <span class="n">name</span><span class="p">);</span>
<span class="kt">void</span> <span class="p">(</span><span class="o">*</span> <span class="n">directiveNotify</span><span class="p">)</span> <span class="p">(</span><span class="n">makeSubparser</span> <span class="o">*</span><span class="n">s</span><span class="p">,</span> <span class="kt">char</span><span class="o">*</span> <span class="n">name</span><span class="p">);</span>
<span class="kt">void</span> <span class="p">(</span><span class="o">*</span> <span class="n">newMacroNotify</span><span class="p">)</span> <span class="p">(</span><span class="n">makeSubparser</span> <span class="o">*</span><span class="n">s</span><span class="p">,</span>
<span class="kt">char</span><span class="o">*</span> <span class="n">name</span><span class="p">,</span>
<span class="kt">bool</span> <span class="n">withDefineDirective</span><span class="p">,</span>
<span class="kt">bool</span> <span class="n">appending</span><span class="p">);</span>
<span class="p">};</span>
</pre></div>
</div>
<p>The Automake parser defines methods for tagging Automake own definitions
in a <code class="docutils literal notranslate"><span class="pre">struct</span> <span class="pre">sMakeSubparser</span></code> type variable, and runs the Make parser by
calling <code class="docutils literal notranslate"><span class="pre">scheduleRunningBaseparser</span></code> function.</p>
<p>The Make parser tags Make own definitions in an input file. In
addition Make parser calls the methods during parsing the input file.</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>ctags --fields<span class="o">=</span>+lK --extras<span class="o">=</span>+r -o - Makefile.am
<span class="go">bin Makefile.am /^bin_PROGRAMS = ctags$/;" directory language:Automake</span>
<span class="go">bin_PROGRAMS Makefile.am /^bin_PROGRAMS = ctags$/;" macro language:Make</span>
<span class="go">ctags Makefile.am /^bin_PROGRAMS = ctags$/;" program language:Automake directory:bin</span>
<span class="go">ctags_CPPFLAGS Makefile.am /^ctags_CPPFLAGS = \\$/;" macro language:Make</span>
</pre></div>
</div>
<p><code class="docutils literal notranslate"><span class="pre">bin_PROGRAMS</span></code> and <code class="docutils literal notranslate"><span class="pre">ctags_CPPFLAGS</span></code> are tagged as macros of Make.
In addition <code class="docutils literal notranslate"><span class="pre">bin</span></code> is tagged as directory, and <code class="docutils literal notranslate"><span class="pre">ctags</span></code> as program of Automake.</p>
<p><code class="docutils literal notranslate"><span class="pre">bin</span></code> is tagged in a callback function assigned to <code class="docutils literal notranslate"><span class="pre">newMacroFound</span></code> method.
<code class="docutils literal notranslate"><span class="pre">ctags</span></code> is tagged in a callback function assigned to <code class="docutils literal notranslate"><span class="pre">valuesFound</span></code> method.</p>
<p><code class="docutils literal notranslate"><span class="pre">--extras=+r</span></code> is used in the example. Reference (<code class="docutils literal notranslate"><span class="pre">r</span></code>) extra is needed to
tag <code class="docutils literal notranslate"><span class="pre">bin</span></code>. <code class="docutils literal notranslate"><span class="pre">bin</span></code> is not defined in the line, <code class="docutils literal notranslate"><span class="pre">bin_PROGRAMS</span> <span class="pre">=</span></code>.
<code class="docutils literal notranslate"><span class="pre">bin</span></code> is referenced as a name of directory where programs are
stored. Therefore <code class="docutils literal notranslate"><span class="pre">r</span></code> is needed.</p>
<p>For tagging <code class="docutils literal notranslate"><span class="pre">ctags</span></code>, the Automake parser must recognize
<code class="docutils literal notranslate"><span class="pre">bin</span></code> in <code class="docutils literal notranslate"><span class="pre">bin_PROGRAMS</span></code> first. <code class="docutils literal notranslate"><span class="pre">ctags</span></code> is tagged
because it is specified as a value for <code class="docutils literal notranslate"><span class="pre">bin_PROGRAMS</span></code>.
As the result <code class="docutils literal notranslate"><span class="pre">r</span></code> is also needed to tag <code class="docutils literal notranslate"><span class="pre">ctags</span></code>.</p>
<p>Only Automake related tags are emitted if Make parser is
disabled.</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>ctags --languages<span class="o">=</span>-Make --fields<span class="o">=</span>+lKr --extras<span class="o">=</span>+r -o - Makefile.am
<span class="go">bin Makefile.am /^bin_PROGRAMS = ctags$/;" directory language:Automake roles:program</span>
<span class="go">ctags Makefile.am /^bin_PROGRAMS = ctags$/;" program language:Automake directory:bin</span>
</pre></div>
</div>
</section>
<section id="autoconf-m4-parser-combination">
<h4>Autoconf/M4 parser combination<a class="headerlink" href="#autoconf-m4-parser-combination" title="Permalink to this headline">¶</a></h4>
<p>Universal Ctags uses m4 parser as a base parser and Autoconf parse as
a subparser for <code class="docutils literal notranslate"><span class="pre">configure.ac</span></code> input file.</p>
<div class="highlight-Autoconf notranslate"><div class="highlight"><pre><span></span>AC_DEFUN([PRETTY_VAR_EXPAND],
[$(eval "$as_echo_n" $(eval "$as_echo_n" "${$1}"))])
</pre></div>
</div>
<p>The m4 parser finds no definition here. However, Autoconf parser finds
<code class="docutils literal notranslate"><span class="pre">PRETTY_VAR_EXPAND</span></code> as a macro definition. Syntax like <code class="docutils literal notranslate"><span class="pre">(...)</span></code> is part
of M4 language. So Autoconf parser is implemented as a subparser of
m4 parser. The most parts of tokens in input files are handled by
M4. Autoconf parser gives hints for parsing <code class="docutils literal notranslate"><span class="pre">configure.ac</span></code> and
registers callback functions to
Autoconf parser.</p>
</section>
</section>
</section>
</section>
<div class="clearer"></div>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<h3><a href="index.html">Table of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">Running multiple parsers on an input file</a><ul>
<li><a class="reference internal" href="#guest-parser-applying-a-parser-to-specified-areas-of-input-file">Guest parser: Applying a parser to specified areas of input file</a><ul>
<li><a class="reference internal" href="#command-line-interface">Command line interface</a></li>
<li><a class="reference internal" href="#examples-of-guest-parser">Examples of guest parser</a><ul>
<li><a class="reference internal" href="#css-javascript-html-parser-combination">{CSS,JavaScript}/HTML parser combination</a></li>
<li><a class="reference internal" href="#c-yacc-parser-combination">C/Yacc parser combination</a></li>
<li><a class="reference internal" href="#pod-perl-parser-combination">Pod/Perl parser combination</a></li>
</ul>
</li>
</ul>
</li>
<li><a class="reference internal" href="#subparser-tagging-definitions-of-higher-upper-level-language">Subparser: Tagging definitions of higher (upper) level language</a><ul>
<li><a class="reference internal" href="#background">Background</a></li>
<li><a class="reference internal" href="#terms">Terms</a><ul>
<li><a class="reference internal" href="#base-parser-and-subparser">Base parser and subparser</a></li>
<li><a class="reference internal" href="#top-down-parser-choice-and-bottom-up-parser-choice">Top down parser choice and bottom up parser choice</a></li>
<li><a class="reference internal" href="#direction-flags">Direction flags</a></li>
</ul>
</li>
<li><a class="reference internal" href="#listing-subparsers">Listing subparsers</a></li>
<li><a class="reference internal" href="#id1">Command line interface</a></li>
<li><a class="reference internal" href="#examples-of-subparser">Examples of subparser</a><ul>
<li><a class="reference internal" href="#automake-make-parser-combination">Automake/Make parser combination</a></li>
<li><a class="reference internal" href="#autoconf-m4-parser-combination">Autoconf/M4 parser combination</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="output-json.html"
title="previous chapter">JSON output</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="interactive-mode.html"
title="next chapter">Interactive mode</a></p>
<div id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" />
<input type="submit" value="Go" />
</form>
</div>
</div>
<script>$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="interactive-mode.html" title="Interactive mode"
>next</a> |</li>
<li class="right" >
<a href="output-json.html" title="JSON output"
>previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">Universal Ctags 0.3.0 documentation</a> »</li>
<li class="nav-item nav-item-this"><a href="">Running multiple parsers on an input file</a></li>
</ul>
</div>
<div class="footer" role="contentinfo">
© Copyright 2015, Universal Ctags Team.
Last updated on 11 Jun 2021.
Created using <a href="https://www.sphinx-doc.org/">Sphinx</a> 4.0.2.
</div>
</body>
</html>
|
