"
+ TopDoc = "\
+ Please choose one of the following actions:
\
+ "
+ TopFooter = ""
+ }
+
+ `SetUpServer' is similar to the previous example, except for calling
+another function, `SetUpEliza'. This approach can be used to implement
+other kinds of servers. The only changes needed to do so are hidden in
+the functions `SetUpServer' and `HandleGET'. Perhaps it might be
+necessary to implement other HTTP methods. The `igawk' program that
+comes with `gawk' may be useful for this process.
+
+ When extending this example to a complete application, the first
+thing to do is to implement the function `SetUpServer' to initialize
+the HTML pages and some variables. These initializations determine the
+way your HTML pages look (colors, titles, menu items, etc.).
+
+ The function `HandleGET' is a nested case selection that decides
+which page the user wants to see next. Each nesting level refers to a
+menu level of the GUI. Each case implements a certain action of the
+menu. On the deepest level of case selection, the handler essentially
+knows what the user wants and stores the answer into the variable that
+holds the HTML page contents:
+
+ function HandleGET() {
+ # A real HTTP server would treat some parts of the URI as a file name.
+ # We take parts of the URI as menu choices and go on accordingly.
+ if(MENU[2] == "AboutServer") {
+ Document = "This is not a CGI script.\
+ This is an httpd, an HTML file, and a CGI script all \
+ in one GAWK script. It needs no separate www-server, \
+ no installation, and no root privileges.\
+ To run it, do this:
\
+ - start this script with \"gawk -f httpserver.awk\",
\
+ - and on the same host let your www browser open location\
+ \"http://localhost:8080\"
\
+
\\ Details of HTTP come from:
\
+ - Hethmon: Illustrated Guide to HTTP\
+
- RFC 2068
JK 14.9.1997
"
+ } else if (MENU[2] == "AboutELIZA") {
+ Document = "This is an implementation of the famous ELIZA\
+ program by Joseph Weizenbaum. It is written in GAWK and\
+ /bin/sh: expad: command not found
+ } else if (MENU[2] == "StartELIZA") {
+ gsub(/\+/, " ", GETARG["YouSay"])
+ # Here we also have to substitute coded special characters
+ Document = ""
+ }
+ }
+
+ Now we are down to the heart of ELIZA, so you can see how it works.
+Initially the user does not say anything; then ELIZA resets its money
+counter and asks the user to tell what comes to mind open heartedly.
+The subsequent answers are converted to uppercase characters and stored
+for later comparison. ELIZA presents the bill when being confronted with
+a sentence that contains the phrase "shut up." Otherwise, it looks for
+keywords in the sentence, conjugates the rest of the sentence, remembers
+the keyword for later use, and finally selects an answer from the set of
+possible answers:
+
+ function ElizaSays(YouSay) {
+ if (YouSay == "") {
+ cost = 0
+ answer = "HI, IM ELIZA, TELL ME YOUR PROBLEM"
+ } else {
+ q = toupper(YouSay)
+ gsub("'", "", q)
+ if(q == qold) {
+ answer = "PLEASE DONT REPEAT YOURSELF !"
+ } else {
+ if (index(q, "SHUT UP") > 0) {
+ answer = "WELL, PLEASE PAY YOUR BILL. ITS EXACTLY ... $"\
+ int(100*rand()+30+cost/100)
+ } else {
+ qold = q
+ w = "-" # no keyword recognized yet
+ for (i in k) { # search for keywords
+ if (index(q, i) > 0) {
+ w = i
+ break
+ }
+ }
+ if (w == "-") { # no keyword, take old subject
+ w = wold
+ subj = subjold
+ } else { # find subject
+ subj = substr(q, index(q, w) + length(w)+1)
+ wold = w
+ subjold = subj # remember keyword and subject
+ }
+ for (i in conj)
+ gsub(i, conj[i], q) # conjugation
+ # from all answers to this keyword, select one randomly
+ answer = r[indices[int(split(k[w], indices) * rand()) + 1]]
+ # insert subject into answer
+ gsub("_", subj, answer)
+ }
+ }
+ }
+ cost += length(answer) # for later payment : 1 cent per character
+ return answer
+ }
+
+ In the long but simple function `SetUpEliza', you can see tables for
+conjugation, keywords, and answers.(1) The associative array `k'
+contains indices into the array of answers `r'. To choose an answer,
+ELIZA just picks an index randomly:
+
+ function SetUpEliza() {
+ srand()
+ wold = "-"
+ subjold = " "
+
+ # table for conjugation
+ conj[" ARE " ] = " AM "
+ conj["WERE " ] = "WAS "
+ conj[" YOU " ] = " I "
+ conj["YOUR " ] = "MY "
+ conj[" IVE " ] =\
+ conj[" I HAVE " ] = " YOU HAVE "
+ conj[" YOUVE " ] =\
+ conj[" YOU HAVE "] = " I HAVE "
+ conj[" IM " ] =\
+ conj[" I AM " ] = " YOU ARE "
+ conj[" YOURE " ] =\
+ conj[" YOU ARE " ] = " I AM "
+
+ # table of all answers
+ r[1] = "DONT YOU BELIEVE THAT I CAN _"
+ r[2] = "PERHAPS YOU WOULD LIKE TO BE ABLE TO _ ?"
+ ...
+
+ # table for looking up answers that
+ # fit to a certain keyword
+ k["CAN YOU"] = "1 2 3"
+ k["CAN I"] = "4 5"
+ k["YOU ARE"] =\
+ k["YOURE"] = "6 7 8 9"
+ ...
+
+ }
+
+ Some interesting remarks and details (including the original source
+code of ELIZA) are found on Mark Humphrys' home page. Yahoo! also has
+a page with a collection of ELIZA-like programs. Many of them are
+written in Java, some of them disclosing the Java source code, and a
+few even explain how to modify the Java source code.
+
+ ---------- Footnotes ----------
+
+ (1) The version shown here is abbreviated. The full version comes
+with the `gawk' distribution.
+
+
+File: gawkinet.info, Node: Caveats, Next: Challenges, Prev: Simple Server, Up: Using Networking
+
+2.11 Network Programming Caveats
+================================
+
+By now it should be clear that debugging a networked application is more
+complicated than debugging a single-process single-hosted application.
+The behavior of a networked application sometimes looks noncausal
+because it is not reproducible in a strong sense. Whether a network
+application works or not sometimes depends on the following:
+
+ * How crowded the underlying network is
+
+ * If the party at the other end is running or not
+
+ * The state of the party at the other end
+
+ The most difficult problems for a beginner arise from the hidden
+states of the underlying network. After closing a TCP connection, it's
+often necessary to wait a short while before reopening the connection.
+Even more difficult is the establishment of a connection that
+previously ended with a "broken pipe." Those connections have to "time
+out" for a minute or so before they can reopen. Check this with the
+command `netstat -a', which provides a list of still "active"
+connections.
+
+
+File: gawkinet.info, Node: Challenges, Prev: Caveats, Up: Using Networking
+
+2.12 Where To Go From Here
+==========================
+
+Now, you have learned enough to build your own application. You could,
+for example, take part in the Loebner Contest to win $100,000. The
+Loebner Prize is the first formal instantiation of a Turing Test. Hugh
+Loebner agreed with The Cambridge Center for Behavioral Studies to
+underwrite a contest designed to implement the Turing Test. Dr.
+Loebner pledged a Grand Prize of $100,000 for the first computer whose
+responses were indistinguishable from a human's. Each year an annual
+prize of $2000 and a bronze medal is awarded to the _most_ human
+computer. The winner of the annual contest is the best entry relative
+to other entries that year, irrespective of how good it is in an
+absolute sense. Here is an example of a conversation with the winning
+program of 1997:
+
+ (c)1997 Cambridge Center For Behavioral Studies all rights reserved [CONVERSE]
+ [Intelligent Research Ltd.] Start at: [1997/04/29 08:59:40]
+ PROGRAM[10:39:20]Hi, my name is Catherine, what's yours?
+ JUDGE02[10:39:36]My name is Nathan.
+ PROGRAM[10:39:37]Well, it's nice to meet you, Nathan.
+ PROGRAM[10:39:45]Did you see that story on CNN last night about the lesbian
+ PROGRAM[10:40:03]couple who came out at a White House party on Sunday?
+ JUDGE02[10:40:45]No. I'm sorry I missed that. How did the Prez take it?
+ PROGRAM[10:40:47]Ellen Degeneres was one of them - she was kissing her lover
+ PROGRAM[10:41:08]in front of Bill Clinton. Do you think he was very
+ PROGRAM[10:41:21]embarrassed?
+ JUDGE02[10:41:37]You tell me. I didn't see it.
+ PROGRAM[10:41:38]Are you still a Friend of Bill Clinton?
+ JUDGE02[10:42:22]Yes. I would like to talk about something else. Do you know anything about sculpture.
+ PROGRAM[10:42:24]Well, I suppose he's doing his best like most of us, but he
+ PROGRAM[10:42:35]does seem to attract an UNUSUAL amount of bad publicity,
+ PROGRAM[10:42:51]wouldnt you say--can it all be bad luck?
+ JUDGE02[10:43:20]I asked you if you knew anything about sculpture.
+ PROGRAM[10:43:21]OK, but what happens now--does it all drift on, with
+ PROGRAM[10:43:36]everyone waiting to see how much his friends have said about
+ PROGRAM[10:43:47]him under threat of lots and lots of jail-or will something
+ PROGRAM[10:44:01]dramatic actually happen?
+ JUDGE02[10:44:19]Do you know what sculpture is?
+ PROGRAM[10:44:20]Well, youre more a fan of his than I am I guess---though I
+ PROGRAM[10:44:34]cant SWEAR what Id have done in that locked hotel room, I
+ PROGRAM[10:44:51]must say.
+ JUDGE02[10:45:26]Have you ever been to the Salmagundi Club?
+ PROGRAM[10:45:29]Never. to go back to the original discussion Time to get
+ PROGRAM[10:45:43]off this one I think and onto something more down to earth!
+ JUDGE02[10:46:20]What is more down to earth than art?
+ PROGRAM[10:46:21]I don't know what -- to be honest. -
+ PROGRAM[10:46:30]I suppose it depends on where you're coming from, but as the
+ PROGRAM[10:46:46]song goes in My Fair Lady, "Why can't a woman be more like a
+ PROGRAM[10:47:03]man? "
+ JUDGE02[10:47:52]I liked that song but I prefer "Why can't children be more like we are"
+ JUDGE02[10:48:10]from another play the name of which I cannot remember.
+ PROGRAM[10:48:12]What, if anything, do you enjoy about being a man?
+ JUDGE02[10:49:01]I would enjoy all aspects of life no matter what my gender.
+ PROGRAM[10:49:02]Do you think of yourself as being attractive?
+
+ This program insists on always speaking about the same story around
+Bill Clinton. You see, even a program with a rather narrow mind can
+behave so much like a human being that it can win this prize. It is
+quite common to let these programs talk to each other via network
+connections. But during the competition itself, the program and its
+computer have to be present at the place the competition is held. We
+all would love to see a `gawk' program win in such an event. Maybe it
+is up to you to accomplish this?
+
+ Some other ideas for useful networked applications:
+ * Read the file `doc/awkforai.txt' in the `gawk' distribution. It
+ was written by Ronald P. Loui (Associate Professor of Computer
+ Science, at Washington University in St. Louis,
+ ) and summarizes why he teaches `gawk' to
+ students of Artificial Intelligence. Here are some passages from
+ the text:
+
+ The GAWK manual can be consumed in a single lab session and
+ the language can be mastered by the next morning by the
+ average student. GAWK's automatic initialization, implicit
+ coercion, I/O support and lack of pointers forgive many of
+ the mistakes that young programmers are likely to make.
+ Those who have seen C but not mastered it are happy to see
+ that GAWK retains some of the same sensibilities while adding
+ what must be regarded as spoonsful of syntactic sugar.
+ ...
+ There are further simple answers. Probably the best is the
+ fact that increasingly, undergraduate AI programming is
+ involving the Web. Oren Etzioni (University of Washington,
+ Seattle) has for a while been arguing that the "softbot" is
+ replacing the mechanical engineers' robot as the most
+ glamorous AI testbed. If the artifact whose behavior needs
+ to be controlled in an intelligent way is the software agent,
+ then a language that is well-suited to controlling the
+ software environment is the appropriate language. That would
+ imply a scripting language. If the robot is KAREL, then the
+ right language is "turn left; turn right." If the robot is
+ Netscape, then the right language is something that can
+ generate `netscape -remote
+ 'openURL(http://cs.wustl.edu/~loui)'' with elan.
+ ...
+ AI programming requires high-level thinking. There have
+ always been a few gifted programmers who can write high-level
+ programs in assembly language. Most however need the ambient
+ abstraction to have a higher floor.
+ ...
+ Second, inference is merely the expansion of notation. No
+ matter whether the logic that underlies an AI program is
+ fuzzy, probabilistic, deontic, defeasible, or deductive, the
+ logic merely defines how strings can be transformed into
+ other strings. A language that provides the best support for
+ string processing in the end provides the best support for
+ logic, for the exploration of various logics, and for most
+ forms of symbolic processing that AI might choose to call
+ "reasoning" instead of "logic." The implication is that
+ PROLOG, which saves the AI programmer from having to write a
+ unifier, saves perhaps two dozen lines of GAWK code at the
+ expense of strongly biasing the logic and representational
+ expressiveness of any approach.
+
+ Now that `gawk' itself can connect to the Internet, it should be
+ obvious that it is suitable for writing intelligent web agents.
+
+ * `awk' is strong at pattern recognition and string processing. So,
+ it is well suited to the classic problem of language translation.
+ A first try could be a program that knows the 100 most frequent
+ English words and their counterparts in German or French. The
+ service could be implemented by regularly reading email with the
+ program above, replacing each word by its translation and sending
+ the translation back via SMTP. Users would send English email to
+ their translation service and get back a translated email message
+ in return. As soon as this works, more effort can be spent on a
+ real translation program.
+
+ * Another dialogue-oriented application (on the verge of ridicule)
+ is the email "support service." Troubled customers write an email
+ to an automatic `gawk' service that reads the email. It looks for
+ keywords in the mail and assembles a reply email accordingly. By
+ carefully investigating the email header, and repeating these
+ keywords through the reply email, it is rather simple to give the
+ customer a feeling that someone cares. Ideally, such a service
+ would search a database of previous cases for solutions. If none
+ exists, the database could, for example, consist of all the
+ newsgroups, mailing lists and FAQs on the Internet.
+
+
+File: gawkinet.info, Node: Some Applications and Techniques, Next: Links, Prev: Using Networking, Up: Top
+
+3 Some Applications and Techniques
+**********************************
+
+In this major node, we look at a number of self-contained scripts, with
+an emphasis on concise networking. Along the way, we work towards
+creating building blocks that encapsulate often needed functions of the
+networking world, show new techniques that broaden the scope of
+problems that can be solved with `gawk', and explore leading edge
+technology that may shape the future of networking.
+
+ We often refer to the site-independent core of the server that we
+built in *note A Simple Web Server: Simple Server. When building new
+and nontrivial servers, we always copy this building block and append
+new instances of the two functions `SetUpServer' and `HandleGET'.
+
+ This makes a lot of sense, since this scheme of event-driven
+execution provides `gawk' with an interface to the most widely accepted
+standard for GUIs: the web browser. Now, `gawk' can rival even Tcl/Tk.
+
+ Tcl and `gawk' have much in common. Both are simple scripting
+languages that allow us to quickly solve problems with short programs.
+But Tcl has Tk on top of it, and `gawk' had nothing comparable up to
+now. While Tcl needs a large and ever-changing library (Tk, which was
+bound to the X Window System until recently), `gawk' needs just the
+networking interface and some kind of browser on the client's side.
+Besides better portability, the most important advantage of this
+approach (embracing well-established standards such HTTP and HTML) is
+that _we do not need to change the language_. We let others do the work
+of fighting over protocols and standards. We can use HTML, JavaScript,
+VRML, or whatever else comes along to do our work.
+
+* Menu:
+
+* PANIC:: An Emergency Web Server.
+* GETURL:: Retrieving Web Pages.
+* REMCONF:: Remote Configuration Of Embedded Systems.
+* URLCHK:: Look For Changed Web Pages.
+* WEBGRAB:: Extract Links From A Page.
+* STATIST:: Graphing A Statistical Distribution.
+* MAZE:: Walking Through A Maze In Virtual Reality.
+* MOBAGWHO:: A Simple Mobile Agent.
+* STOXPRED:: Stock Market Prediction As A Service.
+* PROTBASE:: Searching Through A Protein Database.
+
+
+File: gawkinet.info, Node: PANIC, Next: GETURL, Prev: Some Applications and Techniques, Up: Some Applications and Techniques
+
+3.1 PANIC: An Emergency Web Server
+==================================
+
+At first glance, the `"Hello, world"' example in *note A Primitive Web
+Service: Primitive Service, seems useless. By adding just a few lines,
+we can turn it into something useful.
+
+ The PANIC program tells everyone who connects that the local site is
+not working. When a web server breaks down, it makes a difference if
+customers get a strange "network unreachable" message, or a short
+message telling them that the server has a problem. In such an
+emergency, the hard disk and everything on it (including the regular
+web service) may be unavailable. Rebooting the web server off a
+diskette makes sense in this setting.
+
+ To use the PANIC program as an emergency web server, all you need
+are the `gawk' executable and the program below on a diskette. By
+default, it connects to port 8080. A different value may be supplied on
+the command line:
+
+ BEGIN {
+ RS = ORS = "\r\n"
+ if (MyPort == 0) MyPort = 8080
+ HttpService = "/inet/tcp/" MyPort "/0/0"
+ Hello = "Out Of Service" \
+ "" \
+ "This site is temporarily out of service." \
+ "
"
+ Len = length(Hello) + length(ORS)
+ while ("awk" != "complex") {
+ print "HTTP/1.0 200 OK" |& HttpService
+ print "Content-Length: " Len ORS |& HttpService
+ print Hello |& HttpService
+ while ((HttpService |& getline) > 0)
+ continue;
+ close(HttpService)
+ }
+ }
+
+
+File: gawkinet.info, Node: GETURL, Next: REMCONF, Prev: PANIC, Up: Some Applications and Techniques
+
+3.2 GETURL: Retrieving Web Pages
+================================
+
+GETURL is a versatile building block for shell scripts that need to
+retrieve files from the Internet. It takes a web address as a
+command-line parameter and tries to retrieve the contents of this
+address. The contents are printed to standard output, while the header
+is printed to `/dev/stderr'. A surrounding shell script could analyze
+the contents and extract the text or the links. An ASCII browser could
+be written around GETURL. But more interestingly, web robots are
+straightforward to write on top of GETURL. On the Internet, you can find
+several programs of the same name that do the same job. They are usually
+much more complex internally and at least 10 times longer.
+
+ At first, GETURL checks if it was called with exactly one web
+address. Then, it checks if the user chose to use a special proxy
+server whose name is handed over in a variable. By default, it is
+assumed that the local machine serves as proxy. GETURL uses the `GET'
+method by default to access the web page. By handing over the name of a
+different method (such as `HEAD'), it is possible to choose a different
+behavior. With the `HEAD' method, the user does not receive the body of
+the page content, but does receive the header:
+
+ BEGIN {
+ if (ARGC != 2) {
+ print "GETURL - retrieve Web page via HTTP 1.0"
+ print "IN:\n the URL as a command-line parameter"
+ print "PARAM(S):\n -v Proxy=MyProxy"
+ print "OUT:\n the page content on stdout"
+ print " the page header on stderr"
+ print "JK 16.05.1997"
+ print "ADR 13.08.2000"
+ exit
+ }
+ URL = ARGV[1]; ARGV[1] = ""
+ if (Proxy == "") Proxy = "127.0.0.1"
+ if (ProxyPort == 0) ProxyPort = 80
+ if (Method == "") Method = "GET"
+ HttpService = "/inet/tcp/0/" Proxy "/" ProxyPort
+ ORS = RS = "\r\n\r\n"
+ print Method " " URL " HTTP/1.0" |& HttpService
+ HttpService |& getline Header
+ print Header > "/dev/stderr"
+ while ((HttpService |& getline) > 0)
+ printf "%s", $0
+ close(HttpService)
+ }
+
+ This program can be changed as needed, but be careful with the last
+lines. Make sure transmission of binary data is not corrupted by
+additional line breaks. Even as it is now, the byte sequence
+`"\r\n\r\n"' would disappear if it were contained in binary data. Don't
+get caught in a trap when trying a quick fix on this one.
+
+
+File: gawkinet.info, Node: REMCONF, Next: URLCHK, Prev: GETURL, Up: Some Applications and Techniques
+
+3.3 REMCONF: Remote Configuration of Embedded Systems
+=====================================================
+
+Today, you often find powerful processors in embedded systems.
+Dedicated network routers and controllers for all kinds of machinery
+are examples of embedded systems. Processors like the Intel 80x86 or
+the AMD Elan are able to run multitasking operating systems, such as
+XINU or GNU/Linux in embedded PCs. These systems are small and usually
+do not have a keyboard or a display. Therefore it is difficult to set
+up their configuration. There are several widespread ways to set them
+up:
+
+ * DIP switches
+
+ * Read Only Memories such as EPROMs
+
+ * Serial lines or some kind of keyboard
+
+ * Network connections via `telnet' or SNMP
+
+ * HTTP connections with HTML GUIs
+
+ In this node, we look at a solution that uses HTTP connections to
+control variables of an embedded system that are stored in a file.
+Since embedded systems have tight limits on resources like memory, it
+is difficult to employ advanced techniques such as SNMP and HTTP
+servers. `gawk' fits in quite nicely with its single executable which
+needs just a short script to start working. The following program
+stores the variables in a file, and a concurrent process in the
+embedded system may read the file. The program uses the
+site-independent part of the simple web server that we developed in
+*note A Web Service with Interaction: Interacting Service. As
+mentioned there, all we have to do is to write two new procedures
+`SetUpServer' and `HandleGET':
+
+ function SetUpServer() {
+ TopHeader = "Remote Configuration"
+ TopDoc = "\
+ Please choose one of the following actions:
\
+ "
+ TopFooter = ""
+ if (ConfigFile == "") ConfigFile = "config.asc"
+ }
+
+ The function `SetUpServer' initializes the top level HTML texts as
+usual. It also initializes the name of the file that contains the
+configuration parameters and their values. In case the user supplies a
+name from the command line, that name is used. The file is expected to
+contain one parameter per line, with the name of the parameter in
+column one and the value in column two.
+
+ The function `HandleGET' reflects the structure of the menu tree as
+usual. The first menu choice tells the user what this is all about. The
+second choice reads the configuration file line by line and stores the
+parameters and their values. Notice that the record separator for this
+file is `"\n"', in contrast to the record separator for HTTP. The third
+menu choice builds an HTML table to show the contents of the
+configuration file just read. The fourth choice does the real work of
+changing parameters, and the last one just saves the configuration into
+a file:
+
+ function HandleGET() {
+ if(MENU[2] == "AboutServer") {
+ Document = "This is a GUI for remote configuration of an\
+ embedded system. It is is implemented as one GAWK script."
+ } else if (MENU[2] == "ReadConfig") {
+ RS = "\n"
+ while ((getline < ConfigFile) > 0)
+ config[$1] = $2;
+ close(ConfigFile)
+ RS = "\r\n"
+ Document = "Configuration has been read."
+ } else if (MENU[2] == "CheckConfig") {
+ Document = ""
+ for (i in config)
+ Document = Document "| " i " | " \
+ "" config[i] " |
"
+ Document = Document "
"
+ } else if (MENU[2] == "ChangeConfig") {
+ if ("Param" in GETARG) { # any parameter to set?
+ if (GETARG["Param"] in config) { # is parameter valid?
+ config[GETARG["Param"]] = GETARG["Value"]
+ Document = (GETARG["Param"] " = " GETARG["Value"] ".")
+ } else {
+ Document = "Parameter " GETARG["Param"] " is invalid."
+ }
+ } else {
+ Document = ""
+ }
+ } else if (MENU[2] == "SaveConfig") {
+ for (i in config)
+ printf("%s %s\n", i, config[i]) > ConfigFile
+ close(ConfigFile)
+ Document = "Configuration has been saved."
+ }
+ }
+
+ We could also view the configuration file as a database. From this
+point of view, the previous program acts like a primitive database
+server. Real SQL database systems also make a service available by
+providing a TCP port that clients can connect to. But the application
+level protocols they use are usually proprietary and also change from
+time to time. This is also true for the protocol that MiniSQL uses.
+
+
+File: gawkinet.info, Node: URLCHK, Next: WEBGRAB, Prev: REMCONF, Up: Some Applications and Techniques
+
+3.4 URLCHK: Look for Changed Web Pages
+======================================
+
+Most people who make heavy use of Internet resources have a large
+bookmark file with pointers to interesting web sites. It is impossible
+to regularly check by hand if any of these sites have changed. A program
+is needed to automatically look at the headers of web pages and tell
+which ones have changed. URLCHK does the comparison after using GETURL
+with the `HEAD' method to retrieve the header.
+
+ Like GETURL, this program first checks that it is called with exactly
+one command-line parameter. URLCHK also takes the same command-line
+variables `Proxy' and `ProxyPort' as GETURL, because these variables
+are handed over to GETURL for each URL that gets checked. The one and
+only parameter is the name of a file that contains one line for each
+URL. In the first column, we find the URL, and the second and third
+columns hold the length of the URL's body when checked for the two last
+times. Now, we follow this plan:
+
+ 1. Read the URLs from the file and remember their most recent lengths
+
+ 2. Delete the contents of the file
+
+ 3. For each URL, check its new length and write it into the file
+
+ 4. If the most recent and the new length differ, tell the user
+
+ It may seem a bit peculiar to read the URLs from a file together
+with their two most recent lengths, but this approach has several
+advantages. You can call the program again and again with the same
+file. After running the program, you can regenerate the changed URLs by
+extracting those lines that differ in their second and third columns:
+
+ BEGIN {
+ if (ARGC != 2) {
+ print "URLCHK - check if URLs have changed"
+ print "IN:\n the file with URLs as a command-line parameter"
+ print " file contains URL, old length, new length"
+ print "PARAMS:\n -v Proxy=MyProxy -v ProxyPort=8080"
+ print "OUT:\n same as file with URLs"
+ print "JK 02.03.1998"
+ exit
+ }
+ URLfile = ARGV[1]; ARGV[1] = ""
+ if (Proxy != "") Proxy = " -v Proxy=" Proxy
+ if (ProxyPort != "") ProxyPort = " -v ProxyPort=" ProxyPort
+ while ((getline < URLfile) > 0)
+ Length[$1] = $3 + 0
+ close(URLfile) # now, URLfile is read in and can be updated
+ GetHeader = "gawk " Proxy ProxyPort " -v Method=\"HEAD\" -f geturl.awk "
+ for (i in Length) {
+ GetThisHeader = GetHeader i " 2>&1"
+ while ((GetThisHeader | getline) > 0)
+ if (toupper($0) ~ /CONTENT-LENGTH/) NewLength = $2 + 0
+ close(GetThisHeader)
+ print i, Length[i], NewLength > URLfile
+ if (Length[i] != NewLength) # report only changed URLs
+ print i, Length[i], NewLength
+ }
+ close(URLfile)
+ }
+
+ Another thing that may look strange is the way GETURL is called.
+Before calling GETURL, we have to check if the proxy variables need to
+be passed on. If so, we prepare strings that will become part of the
+command line later. In `GetHeader', we store these strings together
+with the longest part of the command line. Later, in the loop over the
+URLs, `GetHeader' is appended with the URL and a redirection operator
+to form the command that reads the URL's header over the Internet.
+GETURL always produces the headers over `/dev/stderr'. That is the
+reason why we need the redirection operator to have the header piped in.
+
+ This program is not perfect because it assumes that changing URLs
+results in changed lengths, which is not necessarily true. A more
+advanced approach is to look at some other header line that holds time
+information. But, as always when things get a bit more complicated,
+this is left as an exercise to the reader.
+
+
+File: gawkinet.info, Node: WEBGRAB, Next: STATIST, Prev: URLCHK, Up: Some Applications and Techniques
+
+3.5 WEBGRAB: Extract Links from a Page
+======================================
+
+Sometimes it is necessary to extract links from web pages. Browsers do
+it, web robots do it, and sometimes even humans do it. Since we have a
+tool like GETURL at hand, we can solve this problem with some help from
+the Bourne shell:
+
+ BEGIN { RS = "http://[#%&\\+\\-\\./0-9\\:;\\?A-Z_a-z\\~]*" }
+ RT != "" {
+ command = ("gawk -v Proxy=MyProxy -f geturl.awk " RT \
+ " > doc" NR ".html")
+ print command
+ }
+
+ Notice that the regular expression for URLs is rather crude. A
+precise regular expression is much more complex. But this one works
+rather well. One problem is that it is unable to find internal links of
+an HTML document. Another problem is that `ftp', `telnet', `news',
+`mailto', and other kinds of links are missing in the regular
+expression. However, it is straightforward to add them, if doing so is
+necessary for other tasks.
+
+ This program reads an HTML file and prints all the HTTP links that
+it finds. It relies on `gawk''s ability to use regular expressions as
+record separators. With `RS' set to a regular expression that matches
+links, the second action is executed each time a non-empty link is
+found. We can find the matching link itself in `RT'.
+
+ The action could use the `system' function to let another GETURL
+retrieve the page, but here we use a different approach. This simple
+program prints shell commands that can be piped into `sh' for
+execution. This way it is possible to first extract the links, wrap
+shell commands around them, and pipe all the shell commands into a
+file. After editing the file, execution of the file retrieves exactly
+those files that we really need. In case we do not want to edit, we can
+retrieve all the pages like this:
+
+ gawk -f geturl.awk http://www.suse.de | gawk -f webgrab.awk | sh
+
+ After this, you will find the contents of all referenced documents in
+files named `doc*.html' even if they do not contain HTML code. The
+most annoying thing is that we always have to pass the proxy to GETURL.
+If you do not like to see the headers of the web pages appear on the
+screen, you can redirect them to `/dev/null'. Watching the headers
+appear can be quite interesting, because it reveals interesting details
+such as which web server the companies use. Now, it is clear how the
+clever marketing people use web robots to determine the market shares
+of Microsoft and Netscape in the web server market.
+
+ Port 80 of any web server is like a small hole in a repellent
+firewall. After attaching a browser to port 80, we usually catch a
+glimpse of the bright side of the server (its home page). With a tool
+like GETURL at hand, we are able to discover some of the more concealed
+or even "indecent" services (i.e., lacking conformity to standards of
+quality). It can be exciting to see the fancy CGI scripts that lie
+there, revealing the inner workings of the server, ready to be called:
+
+ * With a command such as:
+
+ gawk -f geturl.awk http://any.host.on.the.net/cgi-bin/
+
+ some servers give you a directory listing of the CGI files.
+ Knowing the names, you can try to call some of them and watch for
+ useful results. Sometimes there are executables in such directories
+ (such as Perl interpreters) that you may call remotely. If there
+ are subdirectories with configuration data of the web server, this
+ can also be quite interesting to read.
+
+ * The well-known Apache web server usually has its CGI files in the
+ directory `/cgi-bin'. There you can often find the scripts
+ `test-cgi' and `printenv'. Both tell you some things about the
+ current connection and the installation of the web server. Just
+ call:
+
+ gawk -f geturl.awk http://any.host.on.the.net/cgi-bin/test-cgi
+ gawk -f geturl.awk http://any.host.on.the.net/cgi-bin/printenv
+
+ * Sometimes it is even possible to retrieve system files like the web
+ server's log file--possibly containing customer data--or even the
+ file `/etc/passwd'. (We don't recommend this!)
+
+ *Caution:* Although this may sound funny or simply irrelevant, we
+are talking about severe security holes. Try to explore your own system
+this way and make sure that none of the above reveals too much
+information about your system.
+
+
+File: gawkinet.info, Node: STATIST, Next: MAZE, Prev: WEBGRAB, Up: Some Applications and Techniques
+
+3.6 STATIST: Graphing a Statistical Distribution
+================================================
+
+In the HTTP server examples we've shown thus far, we never present an
+image to the browser and its user. Presenting images is one task.
+Generating images that reflect some user input and presenting these
+dynamically generated images is another. In this node, we use GNUPlot
+for generating `.png', `.ps', or `.gif' files.(1)
+
+ The program we develop takes the statistical parameters of two
+samples and computes the t-test statistics. As a result, we get the
+probabilities that the means and the variances of both samples are the
+same. In order to let the user check plausibility, the program presents
+an image of the distributions. The statistical computation follows
+`Numerical Recipes in C: The Art of Scientific Computing' by William H.
+Press, Saul A. Teukolsky, William T. Vetterling, and Brian P. Flannery.
+Since `gawk' does not have a built-in function for the computation of
+the beta function, we use the `ibeta' function of GNUPlot. As a side
+effect, we learn how to use GNUPlot as a sophisticated calculator. The
+comparison of means is done as in `tutest', paragraph 14.2, page 613,
+and the comparison of variances is done as in `ftest', page 611 in
+`Numerical Recipes'.
+
+ As usual, we take the site-independent code for servers and append
+our own functions `SetUpServer' and `HandleGET':
+
+ function SetUpServer() {
+ TopHeader = "Statistics with GAWK"
+ TopDoc = "\
+ Please choose one of the following actions:
\
+ "
+ TopFooter = ""
+ GnuPlot = "gnuplot 2>&1"
+ m1=m2=0; v1=v2=1; n1=n2=10
+ }
+
+ Here, you see the menu structure that the user sees. Later, we will
+see how the program structure of the `HandleGET' function reflects the
+menu structure. What is missing here is the link for the image we
+generate. In an event-driven environment, request, generation, and
+delivery of images are separated.
+
+ Notice the way we initialize the `GnuPlot' command string for the
+pipe. By default, GNUPlot outputs the generated image via standard
+output, as well as the results of `print'(ed) calculations via standard
+error. The redirection causes standard error to be mixed into standard
+output, enabling us to read results of calculations with `getline'. By
+initializing the statistical parameters with some meaningful defaults,
+we make sure the user gets an image the first time he uses the program.
+
+ Following is the rather long function `HandleGET', which implements
+the contents of this service by reacting to the different kinds of
+requests from the browser. Before you start playing with this script,
+make sure that your browser supports JavaScript and that it also has
+this option switched on. The script uses a short snippet of JavaScript
+code for delayed opening of a window with an image. A more detailed
+explanation follows:
+
+ function HandleGET() {
+ if(MENU[2] == "AboutServer") {
+ Document = "This is a GUI for a statistical computation.\
+ It compares means and variances of two distributions.\
+ It is implemented as one GAWK script and uses GNUPLOT."
+ } else if (MENU[2] == "EnterParameters") {
+ Document = ""
+ if ("m1" in GETARG) { # are there parameters to compare?
+ Document = Document ""
+ m1 = GETARG["m1"]; v1 = GETARG["v1"]; n1 = GETARG["n1"]
+ m2 = GETARG["m2"]; v2 = GETARG["v2"]; n2 = GETARG["n2"]
+ t = (m1-m2)/sqrt(v1/n1+v2/n2)
+ df = (v1/n1+v2/n2)*(v1/n1+v2/n2)/((v1/n1)*(v1/n1)/(n1-1) \
+ + (v2/n2)*(v2/n2) /(n2-1))
+ if (v1>v2) {
+ f = v1/v2
+ df1 = n1 - 1
+ df2 = n2 - 1
+ } else {
+ f = v2/v1
+ df1 = n2 - 1
+ df2 = n1 - 1
+ }
+ print "pt=ibeta(" df/2 ",0.5," df/(df+t*t) ")" |& GnuPlot
+ print "pF=2.0*ibeta(" df2/2 "," df1/2 "," \
+ df2/(df2+df1*f) ")" |& GnuPlot
+ print "print pt, pF" |& GnuPlot
+ RS="\n"; GnuPlot |& getline; RS="\r\n" # $1 is pt, $2 is pF
+ print "invsqrt2pi=1.0/sqrt(2.0*pi)" |& GnuPlot
+ print "nd(x)=invsqrt2pi/sd*exp(-0.5*((x-mu)/sd)**2)" |& GnuPlot
+ print "set term png small color" |& GnuPlot
+ #print "set term postscript color" |& GnuPlot
+ #print "set term gif medium size 320,240" |& GnuPlot
+ print "set yrange[-0.3:]" |& GnuPlot
+ print "set label 'p(m1=m2) =" $1 "' at 0,-0.1 left" |& GnuPlot
+ print "set label 'p(v1=v2) =" $2 "' at 0,-0.2 left" |& GnuPlot
+ print "plot mu=" m1 ",sd=" sqrt(v1) ", nd(x) title 'sample 1',\
+ mu=" m2 ",sd=" sqrt(v2) ", nd(x) title 'sample 2'" |& GnuPlot
+ print "quit" |& GnuPlot
+ GnuPlot |& getline Image
+ while ((GnuPlot |& getline) > 0)
+ Image = Image RS $0
+ close(GnuPlot)
+ }
+ Document = Document "\
+ Do these samples have the same Gaussian distribution?
\
+
"
+ } else if (MENU[2] ~ "Image") {
+ Reason = "OK" ORS "Content-type: image/png"
+ #Reason = "OK" ORS "Content-type: application/x-postscript"
+ #Reason = "OK" ORS "Content-type: image/gif"
+ Header = Footer = ""
+ Document = Image
+ }
+ }
+
+ As usual, we give a short description of the service in the first
+menu choice. The third menu choice shows us that generation and
+presentation of an image are two separate actions. While the latter
+takes place quite instantly in the third menu choice, the former takes
+place in the much longer second choice. Image data passes from the
+generating action to the presenting action via the variable `Image'
+that contains a complete `.png' image, which is otherwise stored in a
+file. If you prefer `.ps' or `.gif' images over the default `.png'
+images, you may select these options by uncommenting the appropriate
+lines. But remember to do so in two places: when telling GNUPlot which
+kind of images to generate, and when transmitting the image at the end
+of the program.
+
+ Looking at the end of the program, the way we pass the
+`Content-type' to the browser is a bit unusual. It is appended to the
+`OK' of the first header line to make sure the type information becomes
+part of the header. The other variables that get transmitted across
+the network are made empty, because in this case we do not have an HTML
+document to transmit, but rather raw image data to contain in the body.
+
+ Most of the work is done in the second menu choice. It starts with a
+strange JavaScript code snippet. When first implementing this server,
+we used a short `"
"' here. But then
+browsers got smarter and tried to improve on speed by requesting the
+image and the HTML code at the same time. When doing this, the browser
+tries to build up a connection for the image request while the request
+for the HTML text is not yet completed. The browser tries to connect to
+the `gawk' server on port 8080 while port 8080 is still in use for
+transmission of the HTML text. The connection for the image cannot be
+built up, so the image appears as "broken" in the browser window. We
+solved this problem by telling the browser to open a separate window
+for the image, but only after a delay of 1000 milliseconds. By this
+time, the server should be ready for serving the next request.
+
+ But there is one more subtlety in the JavaScript code. Each time
+the JavaScript code opens a window for the image, the name of the image
+is appended with a timestamp (`systime'). Why this constant change of
+name for the image? Initially, we always named the image `Image', but
+then the Netscape browser noticed the name had _not_ changed since the
+previous request and displayed the previous image (caching behavior).
+The server core is implemented so that browsers are told _not_ to cache
+anything. Obviously HTTP requests do not always work as expected. One
+way to circumvent the cache of such overly smart browsers is to change
+the name of the image with each request. These three lines of JavaScript
+caused us a lot of trouble.
+
+ The rest can be broken down into two phases. At first, we check if
+there are statistical parameters. When the program is first started,
+there usually are no parameters because it enters the page coming from
+the top menu. Then, we only have to present the user a form that he
+can use to change statistical parameters and submit them. Subsequently,
+the submission of the form causes the execution of the first phase
+because _now_ there _are_ parameters to handle.
+
+ Now that we have parameters, we know there will be an image
+available. Therefore we insert the JavaScript code here to initiate
+the opening of the image in a separate window. Then, we prepare some
+variables that will be passed to GNUPlot for calculation of the
+probabilities. Prior to reading the results, we must temporarily change
+`RS' because GNUPlot separates lines with newlines. After instructing
+GNUPlot to generate a `.png' (or `.ps' or `.gif') image, we initiate
+the insertion of some text, explaining the resulting probabilities. The
+final `plot' command actually generates the image data. This raw binary
+has to be read in carefully without adding, changing, or deleting a
+single byte. Hence the unusual initialization of `Image' and completion
+with a `while' loop.
+
+ When using this server, it soon becomes clear that it is far from
+being perfect. It mixes source code of six scripting languages or
+protocols:
+
+ * GNU `awk' implements a server for the protocol:
+
+ * HTTP which transmits:
+
+ * HTML text which contains a short piece of:
+
+ * JavaScript code opening a separate window.
+
+ * A Bourne shell script is used for piping commands into:
+
+ * GNUPlot to generate the image to be opened.
+
+ After all this work, the GNUPlot image opens in the JavaScript window
+where it can be viewed by the user.
+
+ It is probably better not to mix up so many different languages.
+The result is not very readable. Furthermore, the statistical part of
+the server does not take care of invalid input. Among others, using
+negative variances will cause invalid results.
+
+ ---------- Footnotes ----------
+
+ (1) Due to licensing problems, the default installation of GNUPlot
+disables the generation of `.gif' files. If your installed version
+does not accept `set term gif', just download and install the most
+recent version of GNUPlot and the GD library
+(http://www.boutell.com/gd/) by Thomas Boutell. Otherwise you still
+have the chance to generate some ASCII-art style images with GNUPlot by
+using `set term dumb'. (We tried it and it worked.)
+
+
+File: gawkinet.info, Node: MAZE, Next: MOBAGWHO, Prev: STATIST, Up: Some Applications and Techniques
+
+3.7 MAZE: Walking Through a Maze In Virtual Reality
+===================================================
+
+ In the long run, every program becomes rococo, and then rubble.
+ Alan Perlis
+
+ By now, we know how to present arbitrary `Content-type's to a
+browser. In this node, our server will present a 3D world to our
+browser. The 3D world is described in a scene description language
+(VRML, Virtual Reality Modeling Language) that allows us to travel
+through a perspective view of a 2D maze with our browser. Browsers with
+a VRML plugin enable exploration of this technology. We could do one of
+those boring `Hello world' examples here, that are usually presented
+when introducing novices to VRML. If you have never written any VRML
+code, have a look at the VRML FAQ. Presenting a static VRML scene is a
+bit trivial; in order to expose `gawk''s new capabilities, we will
+present a dynamically generated VRML scene. The function `SetUpServer'
+is very simple because it only sets the default HTML page and
+initializes the random number generator. As usual, the surrounding
+server lets you browse the maze.
+
+ function SetUpServer() {
+ TopHeader = "Walk through a maze"
+ TopDoc = "\
+ Please choose one of the following actions:
\
+ "
+ TopFooter = ""
+ srand()
+ }
+
+ The function `HandleGET' is a bit longer because it first computes
+the maze and afterwards generates the VRML code that is sent across the
+network. As shown in the STATIST example (*note STATIST::), we set the
+type of the content to VRML and then store the VRML representation of
+the maze as the page content. We assume that the maze is stored in a 2D
+array. Initially, the maze consists of walls only. Then, we add an
+entry and an exit to the maze and let the rest of the work be done by
+the function `MakeMaze'. Now, only the wall fields are left in the
+maze. By iterating over the these fields, we generate one line of VRML
+code for each wall field.
+
+ function HandleGET() {
+ if (MENU[2] == "AboutServer") {
+ Document = "If your browser has a VRML 2 plugin,\
+ this server shows you a simple VRML scene."
+ } else if (MENU[2] == "VRMLtest") {
+ XSIZE = YSIZE = 11 # initially, everything is wall
+ for (y = 0; y < YSIZE; y++)
+ for (x = 0; x < XSIZE; x++)
+ Maze[x, y] = "#"
+ delete Maze[0, 1] # entry is not wall
+ delete Maze[XSIZE-1, YSIZE-2] # exit is not wall
+ MakeMaze(1, 1)
+ Document = "\
+ #VRML V2.0 utf8\n\
+ Group {\n\
+ children [\n\
+ PointLight {\n\
+ ambientIntensity 0.2\n\
+ color 0.7 0.7 0.7\n\
+ location 0.0 8.0 10.0\n\
+ }\n\
+ DEF B1 Background {\n\
+ skyColor [0 0 0, 1.0 1.0 1.0 ]\n\
+ skyAngle 1.6\n\
+ groundColor [1 1 1, 0.8 0.8 0.8, 0.2 0.2 0.2 ]\n\
+ groundAngle [ 1.2 1.57 ]\n\
+ }\n\
+ DEF Wall Shape {\n\
+ geometry Box {size 1 1 1}\n\
+ appearance Appearance { material Material { diffuseColor 0 0 1 } }\n\
+ }\n\
+ DEF Entry Viewpoint {\n\
+ position 0.5 1.0 5.0\n\
+ orientation 0.0 0.0 -1.0 0.52\n\
+ }\n"
+ for (i in Maze) {
+ split(i, t, SUBSEP)
+ Document = Document " Transform { translation "
+ Document = Document t[1] " 0 -" t[2] " children USE Wall }\n"
+ }
+ Document = Document " ] # end of group for world\n}"
+ Reason = "OK" ORS "Content-type: model/vrml"
+ Header = Footer = ""
+ }
+ }
+
+ Finally, we have a look at `MakeMaze', the function that generates
+the `Maze' array. When entered, this function assumes that the array
+has been initialized so that each element represents a wall element and
+the maze is initially full of wall elements. Only the entrance and the
+exit of the maze should have been left free. The parameters of the
+function tell us which element must be marked as not being a wall.
+After this, we take a look at the four neighbouring elements and
+remember which we have already treated. Of all the neighbouring
+elements, we take one at random and walk in that direction. Therefore,
+the wall element in that direction has to be removed and then, we call
+the function recursively for that element. The maze is only completed
+if we iterate the above procedure for _all_ neighbouring elements (in
+random order) and for our present element by recursively calling the
+function for the present element. This last iteration could have been
+done in a loop, but it is done much simpler recursively.
+
+ Notice that elements with coordinates that are both odd are assumed
+to be on our way through the maze and the generating process cannot
+terminate as long as there is such an element not being `delete'd. All
+other elements are potentially part of the wall.
+
+ function MakeMaze(x, y) {
+ delete Maze[x, y] # here we are, we have no wall here
+ p = 0 # count unvisited fields in all directions
+ if (x-2 SUBSEP y in Maze) d[p++] = "-x"
+ if (x SUBSEP y-2 in Maze) d[p++] = "-y"
+ if (x+2 SUBSEP y in Maze) d[p++] = "+x"
+ if (x SUBSEP y+2 in Maze) d[p++] = "+y"
+ if (p>0) { # if there are univisited fields, go there
+ p = int(p*rand()) # choose one unvisited field at random
+ if (d[p] == "-x") { delete Maze[x - 1, y]; MakeMaze(x - 2, y)
+ } else if (d[p] == "-y") { delete Maze[x, y - 1]; MakeMaze(x, y - 2)
+ } else if (d[p] == "+x") { delete Maze[x + 1, y]; MakeMaze(x + 2, y)
+ } else if (d[p] == "+y") { delete Maze[x, y + 1]; MakeMaze(x, y + 2)
+ } # we are back from recursion
+ MakeMaze(x, y); # try again while there are unvisited fields
+ }
+ }
+
+
+File: gawkinet.info, Node: MOBAGWHO, Next: STOXPRED, Prev: MAZE, Up: Some Applications and Techniques
+
+3.8 MOBAGWHO: a Simple Mobile Agent
+===================================
+
+ There are two ways of constructing a software design: One way is to
+ make it so simple that there are obviously no deficiencies, and the
+ other way is to make it so complicated that there are no obvious
+ deficiencies.
+ C. A. R. Hoare
+
+ A "mobile agent" is a program that can be dispatched from a computer
+and transported to a remote server for execution. This is called
+"migration", which means that a process on another system is started
+that is independent from its originator. Ideally, it wanders through a
+network while working for its creator or owner. In places like the UMBC
+Agent Web, people are quite confident that (mobile) agents are a
+software engineering paradigm that enables us to significantly increase
+the efficiency of our work. Mobile agents could become the mediators
+between users and the networking world. For an unbiased view at this
+technology, see the remarkable paper `Mobile Agents: Are they a good
+idea?'.(1)
+
+ When trying to migrate a process from one system to another, a
+server process is needed on the receiving side. Depending on the kind
+of server process, several ways of implementation come to mind. How
+the process is implemented depends upon the kind of server process:
+
+ * HTTP can be used as the protocol for delivery of the migrating
+ process. In this case, we use a common web server as the receiving
+ server process. A universal CGI script mediates between migrating
+ process and web server. Each server willing to accept migrating
+ agents makes this universal service available. HTTP supplies the
+ `POST' method to transfer some data to a file on the web server.
+ When a CGI script is called remotely with the `POST' method
+ instead of the usual `GET' method, data is transmitted from the
+ client process to the standard input of the server's CGI script.
+ So, to implement a mobile agent, we must not only write the agent
+ program to start on the client side, but also the CGI script to
+ receive the agent on the server side.
+
+ * The `PUT' method can also be used for migration. HTTP does not
+ require a CGI script for migration via `PUT'. However, with common
+ web servers there is no advantage to this solution, because web
+ servers such as Apache require explicit activation of a special
+ `PUT' script.
+
+ * `Agent Tcl' pursues a different course; it relies on a dedicated
+ server process with a dedicated protocol specialized for receiving
+ mobile agents.
+
+ Our agent example abuses a common web server as a migration tool.
+So, it needs a universal CGI script on the receiving side (the web
+server). The receiving script is activated with a `POST' request when
+placed into a location like `/httpd/cgi-bin/PostAgent.sh'. Make sure
+that the server system uses a version of `gawk' that supports network
+access (Version 3.1 or later; verify with `gawk --version').
+
+ #!/bin/sh
+ MobAg=/tmp/MobileAgent.$$
+ # direct script to mobile agent file
+ cat > $MobAg
+ # execute agent concurrently
+ gawk -f $MobAg $MobAg > /dev/null &
+ # HTTP header, terminator and body
+ gawk 'BEGIN { print "\r\nAgent started" }'
+ rm $MobAg # delete script file of agent
+
+ By making its process id (`$$') part of the unique file name, the
+script avoids conflicts between concurrent instances of the script.
+First, all lines from standard input (the mobile agent's source code)
+are copied into this unique file. Then, the agent is started as a
+concurrent process and a short message reporting this fact is sent to
+the submitting client. Finally, the script file of the mobile agent is
+removed because it is no longer needed. Although it is a short script,
+there are several noteworthy points:
+
+Security
+ _There is none_. In fact, the CGI script should never be made
+ available on a server that is part of the Internet because everyone
+ would be allowed to execute arbitrary commands with it. This
+ behavior is acceptable only when performing rapid prototyping.
+
+Self-Reference
+ Each migrating instance of an agent is started in a way that
+ enables it to read its own source code from standard input and use
+ the code for subsequent migrations. This is necessary because it
+ needs to treat the agent's code as data to transmit. `gawk' is not
+ the ideal language for such a job. Lisp and Tcl are more suitable
+ because they do not make a distinction between program code and
+ data.
+
+Independence
+ After migration, the agent is not linked to its former home in any
+ way. By reporting `Agent started', it waves "Goodbye" to its
+ origin. The originator may choose to terminate or not.
+
+ The originating agent itself is started just like any other
+command-line script, and reports the results on standard output. By
+letting the name of the original host migrate with the agent, the agent
+that migrates to a host far away from its origin can report the result
+back home. Having arrived at the end of the journey, the agent
+establishes a connection and reports the results. This is the reason
+for determining the name of the host with `uname -n' and storing it in
+`MyOrigin' for later use. We may also set variables with the `-v'
+option from the command line. This interactivity is only of importance
+in the context of starting a mobile agent; therefore this `BEGIN'
+pattern and its action do not take part in migration:
+
+ BEGIN {
+ if (ARGC != 2) {
+ print "MOBAG - a simple mobile agent"
+ print "CALL:\n gawk -f mobag.awk mobag.awk"
+ print "IN:\n the name of this script as a command-line parameter"
+ print "PARAM:\n -v MyOrigin=myhost.com"
+ print "OUT:\n the result on stdout"
+ print "JK 29.03.1998 01.04.1998"
+ exit
+ }
+ if (MyOrigin == "") {
+ "uname -n" | getline MyOrigin
+ close("uname -n")
+ }
+ }
+
+ Since `gawk' cannot manipulate and transmit parts of the program
+directly, the source code is read and stored in strings. Therefore,
+the program scans itself for the beginning and the ending of functions.
+Each line in between is appended to the code string until the end of
+the function has been reached. A special case is this part of the
+program itself. It is not a function. Placing a similar framework
+around it causes it to be treated like a function. Notice that this
+mechanism works for all the functions of the source code, but it cannot
+guarantee that the order of the functions is preserved during migration:
+
+ #ReadMySelf
+ /^function / { FUNC = $2 }
+ /^END/ || /^#ReadMySelf/ { FUNC = $1 }
+ FUNC != "" { MOBFUN[FUNC] = MOBFUN[FUNC] RS $0 }
+ (FUNC != "") && (/^}/ || /^#EndOfMySelf/) \
+ { FUNC = "" }
+ #EndOfMySelf
+
+ The web server code in *note A Web Service with Interaction:
+Interacting Service, was first developed as a site-independent core.
+Likewise, the `gawk'-based mobile agent starts with an
+agent-independent core, to which can be appended application-dependent
+functions. What follows is the only application-independent function
+needed for the mobile agent:
+
+ function migrate(Destination, MobCode, Label) {
+ MOBVAR["Label"] = Label
+ MOBVAR["Destination"] = Destination
+ RS = ORS = "\r\n"
+ HttpService = "/inet/tcp/0/" Destination
+ for (i in MOBFUN)
+ MobCode = (MobCode "\n" MOBFUN[i])
+ MobCode = MobCode "\n\nBEGIN {"
+ for (i in MOBVAR)
+ MobCode = (MobCode "\n MOBVAR[\"" i "\"] = \"" MOBVAR[i] "\"")
+ MobCode = MobCode "\n}\n"
+ print "POST /cgi-bin/PostAgent.sh HTTP/1.0" |& HttpService
+ print "Content-length:", length(MobCode) ORS |& HttpService
+ printf "%s", MobCode |& HttpService
+ while ((HttpService |& getline) > 0)
+ print $0
+ close(HttpService)
+ }
+
+ The `migrate' function prepares the aforementioned strings
+containing the program code and transmits them to a server. A
+consequence of this modular approach is that the `migrate' function
+takes some parameters that aren't needed in this application, but that
+will be in future ones. Its mandatory parameter `Destination' holds the
+name (or IP address) of the server that the agent wants as a host for
+its code. The optional parameter `MobCode' may contain some `gawk' code
+that is inserted during migration in front of all other code. The
+optional parameter `Label' may contain a string that tells the agent
+what to do in program execution after arrival at its new home site. One
+of the serious obstacles in implementing a framework for mobile agents
+is that it does not suffice to migrate the code. It is also necessary
+to migrate the state of execution of the agent. In contrast to `Agent
+Tcl', this program does not try to migrate the complete set of
+variables. The following conventions are used:
+
+ * Each variable in an agent program is local to the current host and
+ does _not_ migrate.
+
+ * The array `MOBFUN' shown above is an exception. It is handled by
+ the function `migrate' and does migrate with the application.
+
+ * The other exception is the array `MOBVAR'. Each variable that
+ takes part in migration has to be an element of this array.
+ `migrate' also takes care of this.
+
+ Now it's clear what happens to the `Label' parameter of the function
+`migrate'. It is copied into `MOBVAR["Label"]' and travels alongside
+the other data. Since travelling takes place via HTTP, records must be
+separated with `"\r\n"' in `RS' and `ORS' as usual. The code assembly
+for migration takes place in three steps:
+
+ * Iterate over `MOBFUN' to collect all functions verbatim.
+
+ * Prepare a `BEGIN' pattern and put assignments to mobile variables
+ into the action part.
+
+ * Transmission itself resembles GETURL: the header with the request
+ and the `Content-length' is followed by the body. In case there is
+ any reply over the network, it is read completely and echoed to
+ standard output to avoid irritating the server.
+
+ The application-independent framework is now almost complete. What
+follows is the `END' pattern that is executed when the mobile agent has
+finished reading its own code. First, it checks whether it is already
+running on a remote host or not. In case initialization has not yet
+taken place, it starts `MyInit'. Otherwise (later, on a remote host), it
+starts `MyJob':
+
+ END {
+ if (ARGC != 2) exit # stop when called with wrong parameters
+ if (MyOrigin != "") # is this the originating host?
+ MyInit() # if so, initialize the application
+ else # we are on a host with migrated data
+ MyJob() # so we do our job
+ }
+
+ All that's left to extend the framework into a complete application
+is to write two application-specific functions: `MyInit' and `MyJob'.
+Keep in mind that the former is executed once on the originating host,
+while the latter is executed after each migration:
+
+ function MyInit() {
+ MOBVAR["MyOrigin"] = MyOrigin
+ MOBVAR["Machines"] = "localhost/80 max/80 moritz/80 castor/80"
+ split(MOBVAR["Machines"], Machines) # which host is the first?
+ migrate(Machines[1], "", "") # go to the first host
+ while (("/inet/tcp/8080/0/0" |& getline) > 0) # wait for result
+ print $0 # print result
+ close("/inet/tcp/8080/0/0")
+ }
+
+ As mentioned earlier, this agent takes the name of its origin
+(`MyOrigin') with it. Then, it takes the name of its first destination
+and goes there for further work. Notice that this name has the port
+number of the web server appended to the name of the server, because
+the function `migrate' needs it this way to create the `HttpService'
+variable. Finally, it waits for the result to arrive. The `MyJob'
+function runs on the remote host:
+
+ function MyJob() {
+ # forget this host
+ sub(MOBVAR["Destination"], "", MOBVAR["Machines"])
+ MOBVAR["Result"]=MOBVAR["Result"] SUBSEP SUBSEP MOBVAR["Destination"] ":"
+ while (("who" | getline) > 0) # who is logged in?
+ MOBVAR["Result"] = MOBVAR["Result"] SUBSEP $0
+ close("who")
+ if (index(MOBVAR["Machines"], "/") > 0) { # any more machines to visit?
+ split(MOBVAR["Machines"], Machines) # which host is next?
+ migrate(Machines[1], "", "") # go there
+ } else { # no more machines
+ gsub(SUBSEP, "\n", MOBVAR["Result"]) # send result to origin
+ print MOBVAR["Result"] |& "/inet/tcp/0/" MOBVAR["MyOrigin"] "/8080"
+ close("/inet/tcp/0/" MOBVAR["MyOrigin"] "/8080")
+ }
+ }
+
+ After migrating, the first thing to do in `MyJob' is to delete the
+name of the current host from the list of hosts to visit. Now, it is
+time to start the real work by appending the host's name to the result
+string, and reading line by line who is logged in on this host. A very
+annoying circumstance is the fact that the elements of `MOBVAR' cannot
+hold the newline character (`"\n"'). If they did, migration of this
+string did not work because the string didn't obey the syntax rule for
+a string in `gawk'. `SUBSEP' is used as a temporary replacement. If
+the list of hosts to visit holds at least one more entry, the agent
+migrates to that place to go on working there. Otherwise, we replace
+the `SUBSEP's with a newline character in the resulting string, and
+report it to the originating host, whose name is stored in
+`MOBVAR["MyOrigin"]'.
+
+ ---------- Footnotes ----------
+
+ (1) `http://www.research.ibm.com/massive/mobag.ps'
+
+
+File: gawkinet.info, Node: STOXPRED, Next: PROTBASE, Prev: MOBAGWHO, Up: Some Applications and Techniques
+
+3.9 STOXPRED: Stock Market Prediction As A Service
+==================================================
+
+ Far out in the uncharted backwaters of the unfashionable end of
+ the Western Spiral arm of the Galaxy lies a small unregarded
+ yellow sun.
+
+ Orbiting this at a distance of roughly ninety-two million miles is
+ an utterly insignificant little blue-green planet whose
+ ape-descendent life forms are so amazingly primitive that they
+ still think digital watches are a pretty neat idea.
+
+ This planet has -- or rather had -- a problem, which was this:
+ most of the people living on it were unhappy for pretty much of
+ the time. Many solutions were suggested for this problem, but
+ most of these were largely concerned with the movements of small
+ green pieces of paper, which is odd because it wasn't the small
+ green pieces of paper that were unhappy.
+ Douglas Adams, `The Hitch Hiker's Guide to the Galaxy'
+
+ Valuable services on the Internet are usually _not_ implemented as
+mobile agents. There are much simpler ways of implementing services.
+All Unix systems provide, for example, the `cron' service. Unix system
+users can write a list of tasks to be done each day, each week, twice a
+day, or just once. The list is entered into a file named `crontab'.
+For example, to distribute a newsletter on a daily basis this way, use
+`cron' for calling a script each day early in the morning.
+
+ # run at 8 am on weekdays, distribute the newsletter
+ 0 8 * * 1-5 $HOME/bin/daily.job >> $HOME/log/newsletter 2>&1
+
+ The script first looks for interesting information on the Internet,
+assembles it in a nice form and sends the results via email to the
+customers.
+
+ The following is an example of a primitive newsletter on stock
+market prediction. It is a report which first tries to predict the
+change of each share in the Dow Jones Industrial Index for the
+particular day. Then it mentions some especially promising shares as
+well as some shares which look remarkably bad on that day. The report
+ends with the usual disclaimer which tells every child _not_ to try
+this at home and hurt anybody.
+
+ Good morning Uncle Scrooge,
+
+ This is your daily stock market report for Monday, October 16, 2000.
+ Here are the predictions for today:
+
+ AA neutral
+ GE up
+ JNJ down
+ MSFT neutral
+ ...
+ UTX up
+ DD down
+ IBM up
+ MO down
+ WMT up
+ DIS up
+ INTC up
+ MRK down
+ XOM down
+ EK down
+ IP down
+
+ The most promising shares for today are these:
+
+ INTC http://biz.yahoo.com/n/i/intc.html
+
+ The stock shares to avoid today are these:
+
+ EK http://biz.yahoo.com/n/e/ek.html
+ IP http://biz.yahoo.com/n/i/ip.html
+ DD http://biz.yahoo.com/n/d/dd.html
+ ...
+
+ The script as a whole is rather long. In order to ease the pain of
+studying other people's source code, we have broken the script up into
+meaningful parts which are invoked one after the other. The basic
+structure of the script is as follows:
+
+ BEGIN {
+ Init()
+ ReadQuotes()
+ CleanUp()
+ Prediction()
+ Report()
+ SendMail()
+ }
+
+ The earlier parts store data into variables and arrays which are
+subsequently used by later parts of the script. The `Init' function
+first checks if the script is invoked correctly (without any
+parameters). If not, it informs the user of the correct usage. What
+follows are preparations for the retrieval of the historical quote
+data. The names of the 30 stock shares are stored in an array `name'
+along with the current date in `day', `month', and `year'.
+
+ All users who are separated from the Internet by a firewall and have
+to direct their Internet accesses to a proxy must supply the name of
+the proxy to this script with the `-v Proxy=NAME' option. For most
+users, the default proxy and port number should suffice.
+
+ function Init() {
+ if (ARGC != 1) {
+ print "STOXPRED - daily stock share prediction"
+ print "IN:\n no parameters, nothing on stdin"
+ print "PARAM:\n -v Proxy=MyProxy -v ProxyPort=80"
+ print "OUT:\n commented predictions as email"
+ print "JK 09.10.2000"
+ exit
+ }
+ # Remember ticker symbols from Dow Jones Industrial Index
+ StockCount = split("AA GE JNJ MSFT AXP GM JPM PG BA HD KO \
+ SBC C HON MCD T CAT HWP MMM UTX DD IBM MO WMT DIS INTC \
+ MRK XOM EK IP", name);
+ # Remember the current date as the end of the time series
+ day = strftime("%d")
+ month = strftime("%m")
+ year = strftime("%Y")
+ if (Proxy == "") Proxy = "chart.yahoo.com"
+ if (ProxyPort == 0) ProxyPort = 80
+ YahooData = "/inet/tcp/0/" Proxy "/" ProxyPort
+ }
+
+ There are two really interesting parts in the script. One is the
+function which reads the historical stock quotes from an Internet
+server. The other is the one that does the actual prediction. In the
+following function we see how the quotes are read from the Yahoo
+server. The data which comes from the server is in CSV format
+(comma-separated values):
+
+ Date,Open,High,Low,Close,Volume
+ 9-Oct-00,22.75,22.75,21.375,22.375,7888500
+ 6-Oct-00,23.8125,24.9375,21.5625,22,10701100
+ 5-Oct-00,24.4375,24.625,23.125,23.50,5810300
+
+ Lines contain values of the same time instant, whereas columns are
+separated by commas and contain the kind of data that is described in
+the header (first) line. At first, `gawk' is instructed to separate
+columns by commas (`FS = ","'). In the loop that follows, a connection
+to the Yahoo server is first opened, then a download takes place, and
+finally the connection is closed. All this happens once for each ticker
+symbol. In the body of this loop, an Internet address is built up as a
+string according to the rules of the Yahoo server. The starting and
+ending date are chosen to be exactly the same, but one year apart in
+the past. All the action is initiated within the `printf' command which
+transmits the request for data to the Yahoo server.
+
+ In the inner loop, the server's data is first read and then scanned
+line by line. Only lines which have six columns and the name of a month
+in the first column contain relevant data. This data is stored in the
+two-dimensional array `quote'; one dimension being time, the other
+being the ticker symbol. During retrieval of the first stock's data,
+the calendar names of the time instances are stored in the array `day'
+because we need them later.
+
+ function ReadQuotes() {
+ # Retrieve historical data for each ticker symbol
+ FS = ","
+ for (stock = 1; stock <= StockCount; stock++) {
+ URL = "http://chart.yahoo.com/table.csv?s=" name[stock] \
+ "&a=" month "&b=" day "&c=" year-1 \
+ "&d=" month "&e=" day "&f=" year \
+ "g=d&q=q&y=0&z=" name[stock] "&x=.csv"
+ printf("GET " URL " HTTP/1.0\r\n\r\n") |& YahooData
+ while ((YahooData |& getline) > 0) {
+ if (NF == 6 && $1 ~ /Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec/) {
+ if (stock == 1)
+ days[++daycount] = $1;
+ quote[$1, stock] = $5
+ }
+ }
+ close(YahooData)
+ }
+ FS = " "
+ }
+
+ Now that we _have_ the data, it can be checked once again to make
+sure that no individual stock is missing or invalid, and that all the
+stock quotes are aligned correctly. Furthermore, we renumber the time
+instances. The most recent day gets day number 1 and all other days get
+consecutive numbers. All quotes are rounded toward the nearest whole
+number in US Dollars.
+
+ function CleanUp() {
+ # clean up time series; eliminate incomplete data sets
+ for (d = 1; d <= daycount; d++) {
+ for (stock = 1; stock <= StockCount; stock++)
+ if (! ((days[d], stock) in quote))
+ stock = StockCount + 10
+ if (stock > StockCount + 1)
+ continue
+ datacount++
+ for (stock = 1; stock <= StockCount; stock++)
+ data[datacount, stock] = int(0.5 + quote[days[d], stock])
+ }
+ delete quote
+ delete days
+ }
+
+ Now we have arrived at the second really interesting part of the
+whole affair. What we present here is a very primitive prediction
+algorithm: _If a stock fell yesterday, assume it will also fall today;
+if it rose yesterday, assume it will rise today_. (Feel free to
+replace this algorithm with a smarter one.) If a stock changed in the
+same direction on two consecutive days, this is an indication which
+should be highlighted. Two-day advances are stored in `hot' and
+two-day declines in `avoid'.
+
+ The rest of the function is a sanity check. It counts the number of
+correct predictions in relation to the total number of predictions one
+could have made in the year before.
+
+ function Prediction() {
+ # Predict each ticker symbol by prolonging yesterday's trend
+ for (stock = 1; stock <= StockCount; stock++) {
+ if (data[1, stock] > data[2, stock]) {
+ predict[stock] = "up"
+ } else if (data[1, stock] < data[2, stock]) {
+ predict[stock] = "down"
+ } else {
+ predict[stock] = "neutral"
+ }
+ if ((data[1, stock] > data[2, stock]) && (data[2, stock] > data[3, stock]))
+ hot[stock] = 1
+ if ((data[1, stock] < data[2, stock]) && (data[2, stock] < data[3, stock]))
+ avoid[stock] = 1
+ }
+ # Do a plausibility check: how many predictions proved correct?
+ for (s = 1; s <= StockCount; s++) {
+ for (d = 1; d <= datacount-2; d++) {
+ if (data[d+1, s] > data[d+2, s]) {
+ UpCount++
+ } else if (data[d+1, s] < data[d+2, s]) {
+ DownCount++
+ } else {
+ NeutralCount++
+ }
+ if (((data[d, s] > data[d+1, s]) && (data[d+1, s] > data[d+2, s])) ||
+ ((data[d, s] < data[d+1, s]) && (data[d+1, s] < data[d+2, s])) ||
+ ((data[d, s] == data[d+1, s]) && (data[d+1, s] == data[d+2, s])))
+ CorrectCount++
+ }
+ }
+ }
+
+ At this point the hard work has been done: the array `predict'
+contains the predictions for all the ticker symbols. It is up to the
+function `Report' to find some nice words to introduce the desired
+information.
+
+ function Report() {
+ # Generate report
+ report = "\nThis is your daily "
+ report = report "stock market report for "strftime("%A, %B %d, %Y")".\n"
+ report = report "Here are the predictions for today:\n\n"
+ for (stock = 1; stock <= StockCount; stock++)
+ report = report "\t" name[stock] "\t" predict[stock] "\n"
+ for (stock in hot) {
+ if (HotCount++ == 0)
+ report = report "\nThe most promising shares for today are these:\n\n"
+ report = report "\t" name[stock] "\t\thttp://biz.yahoo.com/n/" \
+ tolower(substr(name[stock], 1, 1)) "/" tolower(name[stock]) ".html\n"
+ }
+ for (stock in avoid) {
+ if (AvoidCount++ == 0)
+ report = report "\nThe stock shares to avoid today are these:\n\n"
+ report = report "\t" name[stock] "\t\thttp://biz.yahoo.com/n/" \
+ tolower(substr(name[stock], 1, 1)) "/" tolower(name[stock]) ".html\n"
+ }
+ report = report "\nThis sums up to " HotCount+0 " winners and " AvoidCount+0
+ report = report " losers. When using this kind\nof prediction scheme for"
+ report = report " the 12 months which lie behind us,\nwe get " UpCount
+ report = report " 'ups' and " DownCount " 'downs' and " NeutralCount
+ report = report " 'neutrals'. Of all\nthese " UpCount+DownCount+NeutralCount
+ report = report " predictions " CorrectCount " proved correct next day.\n"
+ report = report "A success rate of "\
+ int(100*CorrectCount/(UpCount+DownCount+NeutralCount)) "%.\n"
+ report = report "Random choice would have produced a 33% success rate.\n"
+ report = report "Disclaimer: Like every other prediction of the stock\n"
+ report = report "market, this report is, of course, complete nonsense.\n"
+ report = report "If you are stupid enough to believe these predictions\n"
+ report = report "you should visit a doctor who can treat your ailment."
+ }
+
+ The function `SendMail' goes through the list of customers and opens
+a pipe to the `mail' command for each of them. Each one receives an
+email message with a proper subject heading and is addressed with his
+full name.
+
+ function SendMail() {
+ # send report to customers
+ customer["uncle.scrooge@ducktown.gov"] = "Uncle Scrooge"
+ customer["more@utopia.org" ] = "Sir Thomas More"
+ customer["spinoza@denhaag.nl" ] = "Baruch de Spinoza"
+ customer["marx@highgate.uk" ] = "Karl Marx"
+ customer["keynes@the.long.run" ] = "John Maynard Keynes"
+ customer["bierce@devil.hell.org" ] = "Ambrose Bierce"
+ customer["laplace@paris.fr" ] = "Pierre Simon de Laplace"
+ for (c in customer) {
+ MailPipe = "mail -s 'Daily Stock Prediction Newsletter'" c
+ print "Good morning " customer[c] "," | MailPipe
+ print report "\n.\n" | MailPipe
+ close(MailPipe)
+ }
+ }
+
+ Be patient when running the script by hand. Retrieving the data for
+all the ticker symbols and sending the emails may take several minutes
+to complete, depending upon network traffic and the speed of the
+available Internet link. The quality of the prediction algorithm is
+likely to be disappointing. Try to find a better one. Should you find
+one with a success rate of more than 50%, please tell us about it! It
+is only for the sake of curiosity, of course. `:-)'
+
+
+File: gawkinet.info, Node: PROTBASE, Prev: STOXPRED, Up: Some Applications and Techniques
+
+3.10 PROTBASE: Searching Through A Protein Database
+===================================================
+
+ Hoare's Law of Large Problems: Inside every large problem is a
+ small problem struggling to get out.
+
+ Yahoo's database of stock market data is just one among the many
+large databases on the Internet. Another one is located at NCBI
+(National Center for Biotechnology Information). Established in 1988 as
+a national resource for molecular biology information, NCBI creates
+public databases, conducts research in computational biology, develops
+software tools for analyzing genome data, and disseminates biomedical
+information. In this section, we look at one of NCBI's public services,
+which is called BLAST (Basic Local Alignment Search Tool).
+
+ You probably know that the information necessary for reproducing
+living cells is encoded in the genetic material of the cells. The
+genetic material is a very long chain of four base nucleotides. It is
+the order of appearance (the sequence) of nucleotides which contains
+the information about the substance to be produced. Scientists in
+biotechnology often find a specific fragment, determine the nucleotide
+sequence, and need to know where the sequence at hand comes from. This
+is where the large databases enter the game. At NCBI, databases store
+the knowledge about which sequences have ever been found and where they
+have been found. When the scientist sends his sequence to the BLAST
+service, the server looks for regions of genetic material in its
+database which look the most similar to the delivered nucleotide
+sequence. After a search time of some seconds or minutes the server
+sends an answer to the scientist. In order to make access simple, NCBI
+chose to offer their database service through popular Internet
+protocols. There are four basic ways to use the so-called BLAST
+services:
+
+ * The easiest way to use BLAST is through the web. Users may simply
+ point their browsers at the NCBI home page and link to the BLAST
+ pages. NCBI provides a stable URL that may be used to perform
+ BLAST searches without interactive use of a web browser. This is
+ what we will do later in this section. A demonstration client and
+ a `README' file demonstrate how to access this URL.
+
+ * Currently, `blastcl3' is the standard network BLAST client. You
+ can download `blastcl3' from the anonymous FTP location.
+
+ * BLAST 2.0 can be run locally as a full executable and can be used
+ to run BLAST searches against private local databases, or
+ downloaded copies of the NCBI databases. BLAST 2.0 executables may
+ be found on the NCBI anonymous FTP server.
+
+ * The NCBI BLAST Email server is the best option for people without
+ convenient access to the web. A similarity search can be performed
+ by sending a properly formatted mail message containing the
+ nucleotide or protein query sequence to .
+ The query sequence is compared against the specified database
+ using the BLAST algorithm and the results are returned in an email
+ message. For more information on formulating email BLAST searches,
+ you can send a message consisting of the word "HELP" to the same
+ address, .
+
+ Our starting point is the demonstration client mentioned in the
+first option. The `README' file that comes along with the client
+explains the whole process in a nutshell. In the rest of this section,
+we first show what such requests look like. Then we show how to use
+`gawk' to implement a client in about 10 lines of code. Finally, we
+show how to interpret the result returned from the service.
+
+ Sequences are expected to be represented in the standard IUB/IUPAC
+amino acid and nucleic acid codes, with these exceptions: lower-case
+letters are accepted and are mapped into upper-case; a single hyphen or
+dash can be used to represent a gap of indeterminate length; and in
+amino acid sequences, `U' and `*' are acceptable letters (see below).
+Before submitting a request, any numerical digits in the query sequence
+should either be removed or replaced by appropriate letter codes (e.g.,
+`N' for unknown nucleic acid residue or `X' for unknown amino acid
+residue). The nucleic acid codes supported are:
+
+ A --> adenosine M --> A C (amino)
+ C --> cytidine S --> G C (strong)
+ G --> guanine W --> A T (weak)
+ T --> thymidine B --> G T C
+ U --> uridine D --> G A T
+ R --> G A (purine) H --> A C T
+ Y --> T C (pyrimidine) V --> G C A
+ K --> G T (keto) N --> A G C T (any)
+ - gap of indeterminate length
+
+ Now you know the alphabet of nucleotide sequences. The last two lines
+of the following example query show you such a sequence, which is
+obviously made up only of elements of the alphabet just described.
+Store this example query into a file named `protbase.request'. You are
+now ready to send it to the server with the demonstration client.
+
+ PROGRAM blastn
+ DATALIB month
+ EXPECT 0.75
+ BEGIN
+ >GAWK310 the gawking gene GNU AWK
+ tgcttggctgaggagccataggacgagagcttcctggtgaagtgtgtttcttgaaatcat
+ caccaccatggacagcaaa
+
+ The actual search request begins with the mandatory parameter
+`PROGRAM' in the first column followed by the value `blastn' (the name
+of the program) for searching nucleic acids. The next line contains
+the mandatory search parameter `DATALIB' with the value `month' for the
+newest nucleic acid sequences. The third line contains an optional
+`EXPECT' parameter and the value desired for it. The fourth line
+contains the mandatory `BEGIN' directive, followed by the query
+sequence in FASTA/Pearson format. Each line of information must be
+less than 80 characters in length.
+
+ The "month" database contains all new or revised sequences released
+in the last 30 days and is useful for searching against new sequences.
+There are five different blast programs, `blastn' being the one that
+compares a nucleotide query sequence against a nucleotide sequence
+database.
+
+ The last server directive that must appear in every request is the
+`BEGIN' directive. The query sequence should immediately follow the
+`BEGIN' directive and must appear in FASTA/Pearson format. A sequence
+in FASTA/Pearson format begins with a single-line description. The
+description line, which is required, is distinguished from the lines of
+sequence data that follow it by having a greater-than (`>') symbol in
+the first column. For the purposes of the BLAST server, the text of
+the description is arbitrary.
+
+ If you prefer to use a client written in `gawk', just store the
+following 10 lines of code into a file named `protbase.awk' and use
+this client instead. Invoke it with `gawk -f protbase.awk
+protbase.request'. Then wait a minute and watch the result coming in.
+In order to replicate the demonstration client's behaviour as closely
+as possible, this client does not use a proxy server. We could also
+have extended the client program in *note Retrieving Web Pages: GETURL,
+to implement the client request from `protbase.awk' as a special case.
+
+ { request = request "\n" $0 }
+
+ END {
+ BLASTService = "/inet/tcp/0/www.ncbi.nlm.nih.gov/80"
+ printf "POST /cgi-bin/BLAST/nph-blast_report HTTP/1.0\n" |& BLASTService
+ printf "Content-Length: " length(request) "\n\n" |& BLASTService
+ printf request |& BLASTService
+ while ((BLASTService |& getline) > 0)
+ print $0
+ close(BLASTService)
+ }
+
+ The demonstration client from NCBI is 214 lines long (written in C)
+and it is not immediately obvious what it does. Our client is so short
+that it _is_ obvious what it does. First it loops over all lines of the
+query and stores the whole query into a variable. Then the script
+establishes an Internet connection to the NCBI server and transmits the
+query by framing it with a proper HTTP request. Finally it receives and
+prints the complete result coming from the server.
+
+ Now, let us look at the result. It begins with an HTTP header, which
+you can ignore. Then there are some comments about the query having been
+filtered to avoid spuriously high scores. After this, there is a
+reference to the paper that describes the software being used for
+searching the data base. After a repetition of the original query's
+description we find the list of significant alignments:
+
+ Sequences producing significant alignments: (bits) Value
+
+ gb|AC021182.14|AC021182 Homo sapiens chromosome 7 clone RP11-733... 38 0.20
+ gb|AC021056.12|AC021056 Homo sapiens chromosome 3 clone RP11-115... 38 0.20
+ emb|AL160278.10|AL160278 Homo sapiens chromosome 9 clone RP11-57... 38 0.20
+ emb|AL391139.11|AL391139 Homo sapiens chromosome X clone RP11-35... 38 0.20
+ emb|AL365192.6|AL365192 Homo sapiens chromosome 6 clone RP3-421H... 38 0.20
+ emb|AL138812.9|AL138812 Homo sapiens chromosome 11 clone RP1-276... 38 0.20
+ gb|AC073881.3|AC073881 Homo sapiens chromosome 15 clone CTD-2169... 38 0.20
+
+ This means that the query sequence was found in seven human
+chromosomes. But the value 0.20 (20%) means that the probability of an
+accidental match is rather high (20%) in all cases and should be taken
+into account. You may wonder what the first column means. It is a key
+to the specific database in which this occurrence was found. The
+unique sequence identifiers reported in the search results can be used
+as sequence retrieval keys via the NCBI server. The syntax of sequence
+header lines used by the NCBI BLAST server depends on the database from
+which each sequence was obtained. The table below lists the
+identifiers for the databases from which the sequences were derived.
+
+ Database Name Identifier Syntax
+ ============================ ========================
+ GenBank gb|accession|locus
+ EMBL Data Library emb|accession|locus
+ DDBJ, DNA Database of Japan dbj|accession|locus
+ NBRF PIR pir||entry
+ Protein Research Foundation prf||name
+ SWISS-PROT sp|accession|entry name
+ Brookhaven Protein Data Bank pdb|entry|chain
+ Kabat's Sequences of Immuno... gnl|kabat|identifier
+ Patents pat|country|number
+ GenInfo Backbone Id bbs|number
+
+ For example, an identifier might be `gb|AC021182.14|AC021182', where
+the `gb' tag indicates that the identifier refers to a GenBank sequence,
+`AC021182.14' is its GenBank ACCESSION, and `AC021182' is the GenBank
+LOCUS. The identifier contains no spaces, so that a space indicates
+the end of the identifier.
+
+ Let us continue in the result listing. Each of the seven alignments
+mentioned above is subsequently described in detail. We will have a
+closer look at the first of them.
+
+ >gb|AC021182.14|AC021182 Homo sapiens chromosome 7 clone RP11-733N23, WORKING DRAFT SEQUENCE, 4
+ unordered pieces
+ Length = 176383
+
+ Score = 38.2 bits (19), Expect = 0.20
+ Identities = 19/19 (100%)
+ Strand = Plus / Plus
+
+ Query: 35 tggtgaagtgtgtttcttg 53
+ |||||||||||||||||||
+ Sbjct: 69786 tggtgaagtgtgtttcttg 69804
+
+ This alignment was located on the human chromosome 7. The fragment
+on which part of the query was found had a total length of 176383. Only
+19 of the nucleotides matched and the matching sequence ran from
+character 35 to 53 in the query sequence and from 69786 to 69804 in the
+fragment on chromosome 7. If you are still reading at this point, you
+are probably interested in finding out more about Computational Biology
+and you might appreciate the following hints.
+
+ 1. There is a book called `Introduction to Computational Biology' by
+ Michael S. Waterman, which is worth reading if you are seriously
+ interested. You can find a good book review on the Internet.
+
+ 2. While Waterman's book can explain to you the algorithms employed
+ internally in the database search engines, most practitioners
+ prefer to approach the subject differently. The applied side of
+ Computational Biology is called Bioinformatics, and emphasizes the
+ tools available for day-to-day work as well as how to actually
+ _use_ them. One of the very few affordable books on Bioinformatics
+ is `Developing Bioinformatics Computer Skills'.
+
+ 3. The sequences _gawk_ and _gnuawk_ are in widespread use in the
+ genetic material of virtually every earthly living being. Let us
+ take this as a clear indication that the divine creator has
+ intended `gawk' to prevail over other scripting languages such as
+ `perl', `tcl', or `python' which are not even proper sequences.
+ (:-)
+
+
+File: gawkinet.info, Node: Links, Next: GNU Free Documentation License, Prev: Some Applications and Techniques, Up: Top
+
+4 Related Links
+***************
+
+This section lists the URLs for various items discussed in this major
+node. They are presented in the order in which they appear.
+
+`Internet Programming with Python'
+ `http://www.fsbassociates.com/books/python.htm'
+
+`Advanced Perl Programming'
+ `http://www.oreilly.com/catalog/advperl'
+
+`Web Client Programming with Perl'
+ `http://www.oreilly.com/catalog/webclient'
+
+Richard Stevens's home page and book
+ `http://www.kohala.com/~rstevens'
+
+The SPAK home page
+ `http://www.userfriendly.net/linux/RPM/contrib/libc6/i386/spak-0.6b-1.i386.html'
+
+Volume III of `Internetworking with TCP/IP', by Comer and Stevens
+ `http://www.cs.purdue.edu/homes/dec/tcpip3s.cont.html'
+
+XBM Graphics File Format
+ `http://www.wotsit.org/download.asp?f=xbm'
+
+GNUPlot
+ `http://www.cs.dartmouth.edu/gnuplot_info.html'
+
+Mark Humphrys' Eliza page
+ `http://www.compapp.dcu.ie/~humphrys/eliza.html'
+
+Yahoo! Eliza Information
+ `http://dir.yahoo.com/Recreation/Games/Computer_Games/Internet_Games/Web_Games/Artificial_Intelligence'
+
+Java versions of Eliza
+ `http://www.tjhsst.edu/Psych/ch1/eliza.html'
+
+Java versions of Eliza with source code
+ `http://home.adelphia.net/~lifeisgood/eliza/eliza.htm'
+
+Eliza Programs with Explanations
+ `http://chayden.net/chayden/eliza/Eliza.shtml'
+
+Loebner Contest
+ `http://acm.org/~loebner/loebner-prize.htmlx'
+
+Tck/Tk Information
+ `http://www.scriptics.com/'
+
+Intel 80x86 Processors
+ `http://developer.intel.com/design/platform/embedpc/what_is.htm'
+
+AMD Elan Processors
+ `http://www.amd.com/products/epd/processors/4.32bitcont/32bitcont/index.html'
+
+XINU
+ `http://willow.canberra.edu.au/~chrisc/xinu.html'
+
+GNU/Linux
+ `http://uclinux.lineo.com/'
+
+Embedded PCs
+ `http://dir.yahoo.com/Business_and_Economy/Business_to_Business/Computers/Hardware/Embedded_Control/'
+
+MiniSQL
+ `http://www.hughes.com.au/library/'
+
+Market Share Surveys
+ `http://www.netcraft.com/survey'
+
+`Numerical Recipes in C: The Art of Scientific Computing'
+ `http://www.nr.com'
+
+VRML
+ `http://www.vrml.org'
+
+The VRML FAQ
+ `http://www.vrml.org/technicalinfo/specifications/specifications.htm#FAQ'
+
+The UMBC Agent Web
+ `http://www.cs.umbc.edu/agents'
+
+Apache Web Server
+ `http://www.apache.org'
+
+National Center for Biotechnology Information (NCBI)
+ `http://www.ncbi.nlm.nih.gov'
+
+Basic Local Alignment Search Tool (BLAST)
+ `http://www.ncbi.nlm.nih.gov/BLAST/blast_overview.html'
+
+NCBI Home Page
+ `http://www.ncbi.nlm.nih.gov'
+
+BLAST Pages
+ `http://www.ncbi.nlm.nih.gov/BLAST'
+
+BLAST Demonstration Client
+ `ftp://ncbi.nlm.nih.gov/blast/blasturl/'
+
+BLAST anonymous FTP location
+ `ftp://ncbi.nlm.nih.gov/blast/network/netblast/'
+
+BLAST 2.0 Executables
+ `ftp://ncbi.nlm.nih.gov/blast/executables/'
+
+IUB/IUPAC Amino Acid and Nucleic Acid Codes
+ `http://www.uthscsa.edu/geninfo/blastmail.html#item6'
+
+FASTA/Pearson Format
+ `http://www.ncbi.nlm.nih.gov/BLAST/fasta.html'
+
+Fasta/Pearson Sequence in Java
+ `http://www.kazusa.or.jp/java/codon_table_java/'
+
+Book Review of `Introduction to Computational Biology'
+ `http://www.acm.org/crossroads/xrds5-1/introcb.html'
+
+`Developing Bioinformatics Computer Skills'
+ `http://www.oreilly.com/catalog/bioskills/'
+
+
+
+File: gawkinet.info, Node: GNU Free Documentation License, Next: Index, Prev: Links, Up: Top
+
+GNU Free Documentation License
+******************************
+
+ Version 1.2, November 2002
+
+ Copyright (C) 2000,2001,2002 Free Software Foundation, Inc.
+ 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA
+
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+ 0. PREAMBLE
+
+ The purpose of this License is to make a manual, textbook, or other
+ functional and useful document "free" in the sense of freedom: to
+ assure everyone the effective freedom to copy and redistribute it,
+ with or without modifying it, either commercially or
+ noncommercially. Secondarily, this License preserves for the
+ author and publisher a way to get credit for their work, while not
+ being considered responsible for modifications made by others.
+
+ This License is a kind of "copyleft", which means that derivative
+ works of the document must themselves be free in the same sense.
+ It complements the GNU General Public License, which is a copyleft
+ license designed for free software.
+
+ We have designed this License in order to use it for manuals for
+ free software, because free software needs free documentation: a
+ free program should come with manuals providing the same freedoms
+ that the software does. But this License is not limited to
+ software manuals; it can be used for any textual work, regardless
+ of subject matter or whether it is published as a printed book.
+ We recommend this License principally for works whose purpose is
+ instruction or reference.
+
+ 1. APPLICABILITY AND DEFINITIONS
+
+ This License applies to any manual or other work, in any medium,
+ that contains a notice placed by the copyright holder saying it
+ can be distributed under the terms of this License. Such a notice
+ grants a world-wide, royalty-free license, unlimited in duration,
+ to use that work under the conditions stated herein. The
+ "Document", below, refers to any such manual or work. Any member
+ of the public is a licensee, and is addressed as "you". You
+ accept the license if you copy, modify or distribute the work in a
+ way requiring permission under copyright law.
+
+ A "Modified Version" of the Document means any work containing the
+ Document or a portion of it, either copied verbatim, or with
+ modifications and/or translated into another language.
+
+ A "Secondary Section" is a named appendix or a front-matter section
+ of the Document that deals exclusively with the relationship of the
+ publishers or authors of the Document to the Document's overall
+ subject (or to related matters) and contains nothing that could
+ fall directly within that overall subject. (Thus, if the Document
+ is in part a textbook of mathematics, a Secondary Section may not
+ explain any mathematics.) The relationship could be a matter of
+ historical connection with the subject or with related matters, or
+ of legal, commercial, philosophical, ethical or political position
+ regarding them.
+
+ The "Invariant Sections" are certain Secondary Sections whose
+ titles are designated, as being those of Invariant Sections, in
+ the notice that says that the Document is released under this
+ License. If a section does not fit the above definition of
+ Secondary then it is not allowed to be designated as Invariant.
+ The Document may contain zero Invariant Sections. If the Document
+ does not identify any Invariant Sections then there are none.
+
+ The "Cover Texts" are certain short passages of text that are
+ listed, as Front-Cover Texts or Back-Cover Texts, in the notice
+ that says that the Document is released under this License. A
+ Front-Cover Text may be at most 5 words, and a Back-Cover Text may
+ be at most 25 words.
+
+ A "Transparent" copy of the Document means a machine-readable copy,
+ represented in a format whose specification is available to the
+ general public, that is suitable for revising the document
+ straightforwardly with generic text editors or (for images
+ composed of pixels) generic paint programs or (for drawings) some
+ widely available drawing editor, and that is suitable for input to
+ text formatters or for automatic translation to a variety of
+ formats suitable for input to text formatters. A copy made in an
+ otherwise Transparent file format whose markup, or absence of
+ markup, has been arranged to thwart or discourage subsequent
+ modification by readers is not Transparent. An image format is
+ not Transparent if used for any substantial amount of text. A
+ copy that is not "Transparent" is called "Opaque".
+
+ Examples of suitable formats for Transparent copies include plain
+ ASCII without markup, Texinfo input format, LaTeX input format,
+ SGML or XML using a publicly available DTD, and
+ standard-conforming simple HTML, PostScript or PDF designed for
+ human modification. Examples of transparent image formats include
+ PNG, XCF and JPG. Opaque formats include proprietary formats that
+ can be read and edited only by proprietary word processors, SGML or
+ XML for which the DTD and/or processing tools are not generally
+ available, and the machine-generated HTML, PostScript or PDF
+ produced by some word processors for output purposes only.
+
+ The "Title Page" means, for a printed book, the title page itself,
+ plus such following pages as are needed to hold, legibly, the
+ material this License requires to appear in the title page. For
+ works in formats which do not have any title page as such, "Title
+ Page" means the text near the most prominent appearance of the
+ work's title, preceding the beginning of the body of the text.
+
+ A section "Entitled XYZ" means a named subunit of the Document
+ whose title either is precisely XYZ or contains XYZ in parentheses
+ following text that translates XYZ in another language. (Here XYZ
+ stands for a specific section name mentioned below, such as
+ "Acknowledgements", "Dedications", "Endorsements", or "History".)
+ To "Preserve the Title" of such a section when you modify the
+ Document means that it remains a section "Entitled XYZ" according
+ to this definition.
+
+ The Document may include Warranty Disclaimers next to the notice
+ which states that this License applies to the Document. These
+ Warranty Disclaimers are considered to be included by reference in
+ this License, but only as regards disclaiming warranties: any other
+ implication that these Warranty Disclaimers may have is void and
+ has no effect on the meaning of this License.
+
+ 2. VERBATIM COPYING
+
+ You may copy and distribute the Document in any medium, either
+ commercially or noncommercially, provided that this License, the
+ copyright notices, and the license notice saying this License
+ applies to the Document are reproduced in all copies, and that you
+ add no other conditions whatsoever to those of this License. You
+ may not use technical measures to obstruct or control the reading
+ or further copying of the copies you make or distribute. However,
+ you may accept compensation in exchange for copies. If you
+ distribute a large enough number of copies you must also follow
+ the conditions in section 3.
+
+ You may also lend copies, under the same conditions stated above,
+ and you may publicly display copies.
+
+ 3. COPYING IN QUANTITY
+
+ If you publish printed copies (or copies in media that commonly
+ have printed covers) of the Document, numbering more than 100, and
+ the Document's license notice requires Cover Texts, you must
+ enclose the copies in covers that carry, clearly and legibly, all
+ these Cover Texts: Front-Cover Texts on the front cover, and
+ Back-Cover Texts on the back cover. Both covers must also clearly
+ and legibly identify you as the publisher of these copies. The
+ front cover must present the full title with all words of the
+ title equally prominent and visible. You may add other material
+ on the covers in addition. Copying with changes limited to the
+ covers, as long as they preserve the title of the Document and
+ satisfy these conditions, can be treated as verbatim copying in
+ other respects.
+
+ If the required texts for either cover are too voluminous to fit
+ legibly, you should put the first ones listed (as many as fit
+ reasonably) on the actual cover, and continue the rest onto
+ adjacent pages.
+
+ If you publish or distribute Opaque copies of the Document
+ numbering more than 100, you must either include a
+ machine-readable Transparent copy along with each Opaque copy, or
+ state in or with each Opaque copy a computer-network location from
+ which the general network-using public has access to download
+ using public-standard network protocols a complete Transparent
+ copy of the Document, free of added material. If you use the
+ latter option, you must take reasonably prudent steps, when you
+ begin distribution of Opaque copies in quantity, to ensure that
+ this Transparent copy will remain thus accessible at the stated
+ location until at least one year after the last time you
+ distribute an Opaque copy (directly or through your agents or
+ retailers) of that edition to the public.
+
+ It is requested, but not required, that you contact the authors of
+ the Document well before redistributing any large number of
+ copies, to give them a chance to provide you with an updated
+ version of the Document.
+
+ 4. MODIFICATIONS
+
+ You may copy and distribute a Modified Version of the Document
+ under the conditions of sections 2 and 3 above, provided that you
+ release the Modified Version under precisely this License, with
+ the Modified Version filling the role of the Document, thus
+ licensing distribution and modification of the Modified Version to
+ whoever possesses a copy of it. In addition, you must do these
+ things in the Modified Version:
+
+ A. Use in the Title Page (and on the covers, if any) a title
+ distinct from that of the Document, and from those of
+ previous versions (which should, if there were any, be listed
+ in the History section of the Document). You may use the
+ same title as a previous version if the original publisher of
+ that version gives permission.
+
+ B. List on the Title Page, as authors, one or more persons or
+ entities responsible for authorship of the modifications in
+ the Modified Version, together with at least five of the
+ principal authors of the Document (all of its principal
+ authors, if it has fewer than five), unless they release you
+ from this requirement.
+
+ C. State on the Title page the name of the publisher of the
+ Modified Version, as the publisher.
+
+ D. Preserve all the copyright notices of the Document.
+
+ E. Add an appropriate copyright notice for your modifications
+ adjacent to the other copyright notices.
+
+ F. Include, immediately after the copyright notices, a license
+ notice giving the public permission to use the Modified
+ Version under the terms of this License, in the form shown in
+ the Addendum below.
+
+ G. Preserve in that license notice the full lists of Invariant
+ Sections and required Cover Texts given in the Document's
+ license notice.
+
+ H. Include an unaltered copy of this License.
+
+ I. Preserve the section Entitled "History", Preserve its Title,
+ and add to it an item stating at least the title, year, new
+ authors, and publisher of the Modified Version as given on
+ the Title Page. If there is no section Entitled "History" in
+ the Document, create one stating the title, year, authors,
+ and publisher of the Document as given on its Title Page,
+ then add an item describing the Modified Version as stated in
+ the previous sentence.
+
+ J. Preserve the network location, if any, given in the Document
+ for public access to a Transparent copy of the Document, and
+ likewise the network locations given in the Document for
+ previous versions it was based on. These may be placed in
+ the "History" section. You may omit a network location for a
+ work that was published at least four years before the
+ Document itself, or if the original publisher of the version
+ it refers to gives permission.
+
+ K. For any section Entitled "Acknowledgements" or "Dedications",
+ Preserve the Title of the section, and preserve in the
+ section all the substance and tone of each of the contributor
+ acknowledgements and/or dedications given therein.
+
+ L. Preserve all the Invariant Sections of the Document,
+ unaltered in their text and in their titles. Section numbers
+ or the equivalent are not considered part of the section
+ titles.
+
+ M. Delete any section Entitled "Endorsements". Such a section
+ may not be included in the Modified Version.
+
+ N. Do not retitle any existing section to be Entitled
+ "Endorsements" or to conflict in title with any Invariant
+ Section.
+
+ O. Preserve any Warranty Disclaimers.
+
+ If the Modified Version includes new front-matter sections or
+ appendices that qualify as Secondary Sections and contain no
+ material copied from the Document, you may at your option
+ designate some or all of these sections as invariant. To do this,
+ add their titles to the list of Invariant Sections in the Modified
+ Version's license notice. These titles must be distinct from any
+ other section titles.
+
+ You may add a section Entitled "Endorsements", provided it contains
+ nothing but endorsements of your Modified Version by various
+ parties--for example, statements of peer review or that the text
+ has been approved by an organization as the authoritative
+ definition of a standard.
+
+ You may add a passage of up to five words as a Front-Cover Text,
+ and a passage of up to 25 words as a Back-Cover Text, to the end
+ of the list of Cover Texts in the Modified Version. Only one
+ passage of Front-Cover Text and one of Back-Cover Text may be
+ added by (or through arrangements made by) any one entity. If the
+ Document already includes a cover text for the same cover,
+ previously added by you or by arrangement made by the same entity
+ you are acting on behalf of, you may not add another; but you may
+ replace the old one, on explicit permission from the previous
+ publisher that added the old one.
+
+ The author(s) and publisher(s) of the Document do not by this
+ License give permission to use their names for publicity for or to
+ assert or imply endorsement of any Modified Version.
+
+ 5. COMBINING DOCUMENTS
+
+ You may combine the Document with other documents released under
+ this License, under the terms defined in section 4 above for
+ modified versions, provided that you include in the combination
+ all of the Invariant Sections of all of the original documents,
+ unmodified, and list them all as Invariant Sections of your
+ combined work in its license notice, and that you preserve all
+ their Warranty Disclaimers.
+
+ The combined work need only contain one copy of this License, and
+ multiple identical Invariant Sections may be replaced with a single
+ copy. If there are multiple Invariant Sections with the same name
+ but different contents, make the title of each such section unique
+ by adding at the end of it, in parentheses, the name of the
+ original author or publisher of that section if known, or else a
+ unique number. Make the same adjustment to the section titles in
+ the list of Invariant Sections in the license notice of the
+ combined work.
+
+ In the combination, you must combine any sections Entitled
+ "History" in the various original documents, forming one section
+ Entitled "History"; likewise combine any sections Entitled
+ "Acknowledgements", and any sections Entitled "Dedications". You
+ must delete all sections Entitled "Endorsements."
+
+ 6. COLLECTIONS OF DOCUMENTS
+
+ You may make a collection consisting of the Document and other
+ documents released under this License, and replace the individual
+ copies of this License in the various documents with a single copy
+ that is included in the collection, provided that you follow the
+ rules of this License for verbatim copying of each of the
+ documents in all other respects.
+
+ You may extract a single document from such a collection, and
+ distribute it individually under this License, provided you insert
+ a copy of this License into the extracted document, and follow
+ this License in all other respects regarding verbatim copying of
+ that document.
+
+ 7. AGGREGATION WITH INDEPENDENT WORKS
+
+ A compilation of the Document or its derivatives with other
+ separate and independent documents or works, in or on a volume of
+ a storage or distribution medium, is called an "aggregate" if the
+ copyright resulting from the compilation is not used to limit the
+ legal rights of the compilation's users beyond what the individual
+ works permit. When the Document is included an aggregate, this
+ License does not apply to the other works in the aggregate which
+ are not themselves derivative works of the Document.
+
+ If the Cover Text requirement of section 3 is applicable to these
+ copies of the Document, then if the Document is less than one half
+ of the entire aggregate, the Document's Cover Texts may be placed
+ on covers that bracket the Document within the aggregate, or the
+ electronic equivalent of covers if the Document is in electronic
+ form. Otherwise they must appear on printed covers that bracket
+ the whole aggregate.
+
+ 8. TRANSLATION
+
+ Translation is considered a kind of modification, so you may
+ distribute translations of the Document under the terms of section
+ 4. Replacing Invariant Sections with translations requires special
+ permission from their copyright holders, but you may include
+ translations of some or all Invariant Sections in addition to the
+ original versions of these Invariant Sections. You may include a
+ translation of this License, and all the license notices in the
+ Document, and any Warranty Disclaimers, provided that you also
+ include the original English version of this License and the
+ original versions of those notices and disclaimers. In case of a
+ disagreement between the translation and the original version of
+ this License or a notice or disclaimer, the original version will
+ prevail.
+
+ If a section in the Document is Entitled "Acknowledgements",
+ "Dedications", or "History", the requirement (section 4) to
+ Preserve its Title (section 1) will typically require changing the
+ actual title.
+
+ 9. TERMINATION
+
+ You may not copy, modify, sublicense, or distribute the Document
+ except as expressly provided for under this License. Any other
+ attempt to copy, modify, sublicense or distribute the Document is
+ void, and will automatically terminate your rights under this
+ License. However, parties who have received copies, or rights,
+ from you under this License will not have their licenses
+ terminated so long as such parties remain in full compliance.
+
+ 10. FUTURE REVISIONS OF THIS LICENSE
+
+ The Free Software Foundation may publish new, revised versions of
+ the GNU Free Documentation License from time to time. Such new
+ versions will be similar in spirit to the present version, but may
+ differ in detail to address new problems or concerns. See
+ `http://www.gnu.org/copyleft/'.
+
+ Each version of the License is given a distinguishing version
+ number. If the Document specifies that a particular numbered
+ version of this License "or any later version" applies to it, you
+ have the option of following the terms and conditions either of
+ that specified version or of any later version that has been
+ published (not as a draft) by the Free Software Foundation. If
+ the Document does not specify a version number of this License,
+ you may choose any version ever published (not as a draft) by the
+ Free Software Foundation.
+
+ADDENDUM: How to use this License for your documents
+====================================================
+
+To use this License in a document you have written, include a copy of
+the License in the document and put the following copyright and license
+notices just after the title page:
+
+ Copyright (C) YEAR YOUR NAME.
+ Permission is granted to copy, distribute and/or modify this document
+ under the terms of the GNU Free Documentation License, Version 1.2
+ or any later version published by the Free Software Foundation;
+ with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
+ A copy of the license is included in the section entitled ``GNU
+ Free Documentation License''.
+
+ If you have Invariant Sections, Front-Cover Texts and Back-Cover
+Texts, replace the "with...Texts." line with this:
+
+ with the Invariant Sections being LIST THEIR TITLES, with
+ the Front-Cover Texts being LIST, and with the Back-Cover Texts
+ being LIST.
+
+ If you have Invariant Sections without Cover Texts, or some other
+combination of the three, merge those two alternatives to suit the
+situation.
+
+ If your document contains nontrivial examples of program code, we
+recommend releasing these examples in parallel under your choice of
+free software license, such as the GNU General Public License, to
+permit their use in free software.
+
+
+File: gawkinet.info, Node: Index, Prev: GNU Free Documentation License, Up: Top
+
+Index
+*****
+
+ [index ]
+* Menu:
+
+* /inet/ files (gawk): Gawk Special Files. (line 34)
+* /inet/raw special files (gawk): File /inet/raw. (line 6)
+* /inet/tcp special files (gawk): File /inet/tcp. (line 6)
+* /inet/udp special files (gawk): File /inet/udp. (line 6)
+* advanced features, network connections: Troubleshooting. (line 6)
+* agent <1>: MOBAGWHO. (line 6)
+* agent: Challenges. (line 76)
+* AI: Challenges. (line 76)
+* apache <1>: MOBAGWHO. (line 42)
+* apache: WEBGRAB. (line 72)
+* Bioinformatics: PROTBASE. (line 227)
+* BLAST, Basic Local Alignment Search Tool: PROTBASE. (line 6)
+* blocking: Making Connections. (line 35)
+* Boutell, Thomas: STATIST. (line 6)
+* CGI (Common Gateway Interface): MOBAGWHO. (line 42)
+* CGI (Common Gateway Interface), dynamic web pages and: Web page.
+ (line 46)
+* CGI (Common Gateway Interface), library: CGI Lib. (line 11)
+* clients: Making Connections. (line 21)
+* Clinton, Bill: Challenges. (line 59)
+* Common Gateway Interface, See CGI: Web page. (line 46)
+* Computational Biology: PROTBASE. (line 227)
+* contest: Challenges. (line 6)
+* cron utility: STOXPRED. (line 23)
+* CSV format: STOXPRED. (line 128)
+* dark corner, RAW protocol: File /inet/raw. (line 13)
+* Dow Jones Industrial Index: STOXPRED. (line 44)
+* ELIZA program: Simple Server. (line 11)
+* email: Email. (line 11)
+* FASTA/Pearson format: PROTBASE. (line 102)
+* FDL (Free Documentation License): GNU Free Documentation License.
+ (line 6)
+* filenames, for network access: Gawk Special Files. (line 29)
+* files, /inet/ (gawk): Gawk Special Files. (line 34)
+* files, /inet/raw (gawk): File /inet/raw. (line 6)
+* files, /inet/tcp (gawk): File /inet/tcp. (line 6)
+* files, /inet/udp (gawk): File /inet/udp. (line 6)
+* finger utility: Setting Up. (line 22)
+* Free Documentation License (FDL): GNU Free Documentation License.
+ (line 6)
+* FTP (File Transfer Protocol): Basic Protocols. (line 29)
+* gawk, networking: Using Networking. (line 6)
+* gawk, networking, connections <1>: TCP Connecting. (line 6)
+* gawk, networking, connections: Special File Fields. (line 49)
+* gawk, networking, filenames: Gawk Special Files. (line 29)
+* gawk, networking, See Also email: Email. (line 6)
+* gawk, networking, service, establishing: Setting Up. (line 6)
+* gawk, networking, troubleshooting: Caveats. (line 6)
+* gawk, web and, See web service: Interacting Service. (line 6)
+* getline command: TCP Connecting. (line 11)
+* GETURL program: GETURL. (line 6)
+* GIF image format <1>: STATIST. (line 6)
+* GIF image format: Web page. (line 46)
+* GNU Free Documentation License: GNU Free Documentation License.
+ (line 6)
+* GNU/Linux <1>: REMCONF. (line 6)
+* GNU/Linux <2>: Interacting. (line 27)
+* GNU/Linux: Troubleshooting. (line 54)
+* GNUPlot utility <1>: STATIST. (line 6)
+* GNUPlot utility: Interacting Service. (line 189)
+* Hoare, C.A.R. <1>: PROTBASE. (line 6)
+* Hoare, C.A.R.: MOBAGWHO. (line 6)
+* hostname field: Special File Fields. (line 29)
+* HTML (Hypertext Markup Language): Web page. (line 30)
+* HTTP (Hypertext Transfer Protocol) <1>: Web page. (line 6)
+* HTTP (Hypertext Transfer Protocol): Basic Protocols. (line 29)
+* HTTP (Hypertext Transfer Protocol), record separators and: Web page.
+ (line 30)
+* HTTP server, core logic: Interacting Service. (line 6)
+* Humphrys, Mark: Simple Server. (line 179)
+* Hypertext Markup Language (HTML): Web page. (line 30)
+* Hypertext Transfer Protocol, See HTTP: Web page. (line 6)
+* image format: STATIST. (line 6)
+* images, in web pages: Interacting Service. (line 189)
+* images, retrieving over networks: Web page. (line 46)
+* input/output, two-way, See Also gawk, networking: Gawk Special Files.
+ (line 19)
+* Internet, See networks: Interacting. (line 48)
+* JavaScript: STATIST. (line 56)
+* Linux <1>: REMCONF. (line 6)
+* Linux <2>: Interacting. (line 27)
+* Linux: Troubleshooting. (line 54)
+* Lisp: MOBAGWHO. (line 98)
+* localport field: Gawk Special Files. (line 34)
+* Loebner, Hugh: Challenges. (line 6)
+* Loui, Ronald: Challenges. (line 76)
+* MAZE: MAZE. (line 6)
+* Microsoft Windows: WEBGRAB. (line 43)
+* Microsoft Windows, networking: Troubleshooting. (line 54)
+* Microsoft Windows, networking, ports: Setting Up. (line 37)
+* MiniSQL: REMCONF. (line 111)
+* MOBAGWHO program: MOBAGWHO. (line 6)
+* NCBI, National Center for Biotechnology Information: PROTBASE.
+ (line 6)
+* networks, gawk and: Using Networking. (line 6)
+* networks, gawk and, connections <1>: TCP Connecting. (line 6)
+* networks, gawk and, connections: Special File Fields. (line 49)
+* networks, gawk and, filenames: Gawk Special Files. (line 29)
+* networks, gawk and, See Also email: Email. (line 6)
+* networks, gawk and, service, establishing: Setting Up. (line 6)
+* networks, gawk and, troubleshooting: Caveats. (line 6)
+* networks, ports, reserved: Setting Up. (line 37)
+* networks, ports, specifying: Special File Fields. (line 18)
+* networks, See Also web pages: PANIC. (line 6)
+* Numerical Recipes: STATIST. (line 24)
+* ORS variable, HTTP and: Web page. (line 30)
+* ORS variable, POP and: Email. (line 36)
+* PANIC program: PANIC. (line 6)
+* Perl: Using Networking. (line 14)
+* Perl, gawk networking and: Using Networking. (line 24)
+* Perlis, Alan: MAZE. (line 6)
+* pipes, networking and: TCP Connecting. (line 30)
+* PNG image format <1>: STATIST. (line 6)
+* PNG image format: Web page. (line 46)
+* POP (Post Office Protocol): Email. (line 6)
+* Post Office Protocol (POP): Email. (line 6)
+* PostScript: STATIST. (line 138)
+* PROLOG: Challenges. (line 76)
+* PROTBASE: PROTBASE. (line 6)
+* protocol field: Special File Fields. (line 11)
+* PS image format: STATIST. (line 6)
+* Python: Using Networking. (line 14)
+* Python, gawk networking and: Using Networking. (line 24)
+* RAW protocol: File /inet/raw. (line 6)
+* record separators, HTTP and: Web page. (line 30)
+* record separators, POP and: Email. (line 36)
+* REMCONF program: REMCONF. (line 6)
+* remoteport field: Gawk Special Files. (line 34)
+* robot <1>: WEBGRAB. (line 6)
+* robot: Challenges. (line 85)
+* RS variable, HTTP and: Web page. (line 30)
+* RS variable, POP and: Email. (line 36)
+* servers <1>: Setting Up. (line 22)
+* servers: Making Connections. (line 14)
+* servers, as hosts: Special File Fields. (line 29)
+* servers, HTTP: Interacting Service. (line 6)
+* servers, web: Simple Server. (line 6)
+* Simple Mail Transfer Protocol (SMTP): Email. (line 6)
+* SMTP (Simple Mail Transfer Protocol) <1>: Email. (line 6)
+* SMTP (Simple Mail Transfer Protocol): Basic Protocols. (line 29)
+* SPAK utility: File /inet/raw. (line 21)
+* STATIST program: STATIST. (line 6)
+* STOXPRED program: STOXPRED. (line 6)
+* synchronous communications: Making Connections. (line 35)
+* Tcl/Tk: Using Networking. (line 14)
+* Tcl/Tk, gawk and <1>: Some Applications and Techniques.
+ (line 22)
+* Tcl/Tk, gawk and: Using Networking. (line 24)
+* TCP (Transmission Control Protocol) <1>: File /inet/tcp. (line 6)
+* TCP (Transmission Control Protocol): Using Networking. (line 29)
+* TCP (Transmission Control Protocol), connection, establishing: TCP Connecting.
+ (line 6)
+* TCP (Transmission Control Protocol), UDP and: Interacting. (line 48)
+* TCP/IP, protocols, selecting: Special File Fields. (line 11)
+* TCP/IP, sockets and: Gawk Special Files. (line 19)
+* Transmission Control Protocol, See TCP: Using Networking. (line 29)
+* troubleshooting, gawk, networks: Caveats. (line 6)
+* troubleshooting, networks, connections: Troubleshooting. (line 6)
+* troubleshooting, networks, timeouts: Caveats. (line 18)
+* UDP (User Datagram Protocol): File /inet/udp. (line 6)
+* UDP (User Datagram Protocol), TCP and: Interacting. (line 48)
+* Unix, network ports and: Setting Up. (line 37)
+* URLCHK program: URLCHK. (line 6)
+* User Datagram Protocol, See UDP: File /inet/udp. (line 6)
+* vertical bar (|), |& operator (I/O): TCP Connecting. (line 25)
+* VRML: MAZE. (line 6)
+* web browsers, See web service: Interacting Service. (line 6)
+* web pages: Web page. (line 6)
+* web pages, images in: Interacting Service. (line 189)
+* web pages, retrieving: GETURL. (line 6)
+* web servers: Simple Server. (line 6)
+* web service <1>: PANIC. (line 6)
+* web service: Primitive Service. (line 6)
+* WEBGRAB program: WEBGRAB. (line 6)
+* Weizenbaum, Joseph: Simple Server. (line 11)
+* XBM image format: Interacting Service. (line 189)
+* Yahoo! <1>: STOXPRED. (line 6)
+* Yahoo!: REMCONF. (line 6)
+* | (vertical bar), |& operator (I/O): TCP Connecting. (line 25)
+
+
+
+Tag Table:
+Node: Top2007
+Node: Preface5697
+Node: Introduction7072
+Node: Stream Communications8098
+Node: Datagram Communications9271
+Node: The TCP/IP Protocols10902
+Ref: The TCP/IP Protocols-Footnote-111586
+Node: Basic Protocols11743
+Node: Ports13065
+Node: Making Connections14470
+Ref: Making Connections-Footnote-117051
+Ref: Making Connections-Footnote-217098
+Node: Using Networking17279
+Node: Gawk Special Files19633
+Node: Special File Fields21637
+Ref: table-inet-components25387
+Node: Comparing Protocols27299
+Node: File /inet/tcp27888
+Node: File /inet/udp28914
+Node: File /inet/raw30035
+Ref: File /inet/raw-Footnote-133068
+Node: TCP Connecting33148
+Node: Troubleshooting35486
+Ref: Troubleshooting-Footnote-138537
+Node: Interacting39081
+Node: Setting Up41811
+Node: Email45305
+Node: Web page47631
+Ref: Web page-Footnote-150436
+Node: Primitive Service50633
+Node: Interacting Service53367
+Ref: Interacting Service-Footnote-162496
+Node: CGI Lib62528
+Node: Simple Server69489
+Ref: Simple Server-Footnote-177219
+Node: Caveats77320
+Node: Challenges78463
+Node: Some Applications and Techniques87130
+Node: PANIC89587
+Node: GETURL91305
+Node: REMCONF93928
+Node: URLCHK99404
+Node: WEBGRAB103239
+Node: STATIST107689
+Ref: STATIST-Footnote-1119397
+Node: MAZE119842
+Node: MOBAGWHO126030
+Ref: MOBAGWHO-Footnote-1139974
+Node: STOXPRED140029
+Node: PROTBASE154284
+Node: Links167366
+Node: GNU Free Documentation License170800
+Node: Index193204
+
+End Tag Table
diff --git a/coreutils-5.3.0-bin/contrib/gawk/3.1.6/gawk-3.1.6-src/missing_d/README b/coreutils-5.3.0-bin/contrib/gawk/3.1.6/gawk-3.1.6-src/missing_d/README
new file mode 100644
index 0000000..735889d
--- /dev/null
+++ b/coreutils-5.3.0-bin/contrib/gawk/3.1.6/gawk-3.1.6-src/missing_d/README
@@ -0,0 +1,14 @@
+Fri Aug 25 13:23:06 IDT 2006
+============================
+
+The files memmove.c, mktime.c, snprintf.c, strerror.c, strftime.c,
+strncasecmp.c, and system.c are copyright by the Free Software
+Foundation. They are licensed under the GPL or the LGPL. See the
+COPYING.LIB file in this directory and the COPYING file in the parent
+directory for licensing information.
+
+All other files are public domain.
+
+Arnold Robbins
+arnold@skeeve.com
+
diff --git a/coreutils-5.3.0-bin/contrib/gawk/3.1.6/gawk-3.1.6-src/test/README b/coreutils-5.3.0-bin/contrib/gawk/3.1.6/gawk-3.1.6-src/test/README
new file mode 100644
index 0000000..2343be2
--- /dev/null
+++ b/coreutils-5.3.0-bin/contrib/gawk/3.1.6/gawk-3.1.6-src/test/README
@@ -0,0 +1,18 @@
+Mon Jan 22 13:08:58 EST 1996
+
+This directory contains the tests for gawk. The tests use the
+following conventions.
+
+Given some aspect of gawk named `foo', there will be one or more
+of the following files:
+
+foo.awk --- actual code for the test if not inline in the Makefile
+foo.in --- the data for the test, if it needs data
+foo.ok --- the expected results
+_foo --- the actual results; generated at run time
+
+The _foo file will be left around if a test fails, allowing you to
+compare actual and expected results, in case they differ.
+
+If they do differ (other than strftime.ok and _strftime!), send in a
+bug report. See the manual for the bug report procedure.
diff --git a/coreutils-5.3.0-bin/contrib/gawk/3.1.6/gawk-3.1.6-src/unsupported/atari/README.1st b/coreutils-5.3.0-bin/contrib/gawk/3.1.6/gawk-3.1.6-src/unsupported/atari/README.1st
new file mode 100644
index 0000000..a158cda
--- /dev/null
+++ b/coreutils-5.3.0-bin/contrib/gawk/3.1.6/gawk-3.1.6-src/unsupported/atari/README.1st
@@ -0,0 +1,5 @@
+Tue Nov 7 14:19:41 2000
+
+The atari port is no longer supported. If you have an atari,
+you are welcome to try and use the port here, but we no longer have
+the hardware to test gawk on.
--
cgit v1.2.3