


SWISH-RUN(1)          SWISH-E Documentation          SWISH-RUN(1)


NAME
       SWISH-RUN - Running Swish-e and Command Line Switches

OVERVIEW
       The Swish-e program is controlled by command line arguments (called
       switches).  Often, it is run manually from a shell (command prompt), or
       from a program such as a CGI script that passes the command line argu-
       ments to swish.

       Note: A number of the command line switches may be specified in the
       Swish-e configuration file specified with the "-c" command line argu-
       ment.  Please see SWISH-CONFIG for a complete description of available
       configuration file directives.

       There are two basic operating modes of Swish-e: indexing and searching.
       There are command line arguments that are unique to each mode, and oth-
       ers that apply to both (yet may have different meaning depending on the
       operating mode).  These command line arguments are listed below,
       grouped by:

       INDEXING -- describes the command line arguments used while indexing.

       SEARCHING -- lists the command line arguments used while searching.

       OTHER SWITCHES -- lists switches that don't apply to searching or
       indexing.

       Beginning with Swish-e version 2.1, you may embed its search engine
       into your applications.  Please see SWISH-LIBRARY.

INDEXING
       Swish-e indexing is initiated by passing command line arguments to
       swish.  The command line arguments used for searching are described in
       SEARCHING.  Also, see SWISH-SEARCH for examples of searching with
       Swish-e.

       Swish-e usage:

           swish-e [-i dir file ... ] [-c file] [-f file] [-l] \
                   [-v (num)] [-S method(fs|http|prog)] [-N path]

       The "-h" switch (help) will list the available Swish-e command line
       arguments:

           swish-e -h

       Typically, most if not all indexing settings are placed in a configura-
       tion file (specified with the "-c" switch).  Once the configuration
       file is setup indexing is initiated as:

           swish-e -c /path/to/config/file

       See SWISH-CONFIG for information on the configuration file.




2.4.7                       2009-04-04                          1





SWISH-RUN(1)          SWISH-E Documentation          SWISH-RUN(1)


       Security Note: If the swish binary is named swish-search then swish
       will not allow any operation that would cause swish to write to the
       index file.

       When indexing it may be advisable to index to a temporary file, and
       then after indexing has successfully completed rename the file to the
       final location.  This is especially important when replacing an index
       that is currently in use.

           swish-e -c swish.config -f index.tmp
           [check return code from swish or look for err: output]
           mv index.tmp index.swish-e

       Indexing Command Line Arguments


       -i *directories and/or files* (input file)
           This specifies the directories and/or files to index. Directories
           will be indexed recursively.  This is typically specified in the
           configuration file with the IndexDir directive instead of on the
           command line.  Use of this switch overrides the configuration file
           settings.

       -S [fs|http|prog] (document source/access mode)
           This specifies the method to use for accessing documents to index.
           Can be either "fs" for local indexing via the file system (the
           default), "http" for spidering, or "prog" for reading documents
           from an external program.

           Located in the "conf" directory are example configuration files
           that demonstrate indexing with the different document source meth-
           ods.

           See the SWISH-FAQ for a discussion on the different indexing meth-
           ods, and the difference between spidering with the http method vs.
           using the file system method.

           fs - file system
               The "fs" method simply reads files from a local (or networked)
               drive.  This is the default method if the "-S" switch is not
               specified.  See SWISH-CONFIG for configuration directives spe-
               cific to the "fs" method.

           http - spider a web server
               The "http" method is used to spider web servers.  It uses an
               included helper program called swishspider.  See SWISH-CONFIG
               for configuration directives specific to the "http" method.

               Security Note: Under Windows swish passes the URLs fetched from
               remote documents through the shell (swish uses the system()
               command for running swishspider under Windows), and this may be
               considered an additional security risk.

               The "http" method is deprecated (or at least not very well



2.4.7                       2009-04-04                          2





SWISH-RUN(1)          SWISH-E Documentation          SWISH-RUN(1)


               appreciated).  Consider using the "prog" method described below
               for spidering.  There's a spider program available in the prog-
               bin directory for use with the "prog" method.  Here's a number
               of limitation with this method that are solved with the "prog"
               method:

               *   swishspider only spiders standard <a href="..."> links.
                   Frames and other links are not followed.

               *   By default, this method of spidering only indexes files
                   that have a content type of "text/*" (e.g. text/plain,
                   text/html, text/xml).  You should use "DefaultContents" and
                   "IndexContents" to map file extensions to parsers used by
                   swish (e.g.  "IndexContents HTML* .html .htm"), but this
                   will fail where a document does not have a file extension.

               *   Swish-e's "FileFilter" directive can be used with the
                   "http" access method, although it requires a separate pro-
                   cess (in addition to the swsihspider process) for each doc-
                   ument filtered.

               *   The SWISH::Filter modules can be used with the swishspider
                   program.  SWISH::Filter provides a general purpose filter-
                   ing system (see SWISH::Filter documentation).  To use
                   SWISH::Filter set PERL5LIB to point to the location of the
                   SWISH module name space (typically /usr/local/lib/swish-e
                   under Unix).  For example:

                      export PERL5LIB=/usr/local/lib/swish-e  # bash, bourne shells
                      setenv PERL5LIB /usr/local/lib/swish-e  # csh, tcsh

                   or under Windows

                      set PERL5LIB=c:\program files\swish-e2.4\lib\swish-e

                   SWISH::Filter is not enabled by default due to the overhead
                   of loading the modules for every document fetched.

                   The Swish-e distribution includes perl modules in the
                   SWISH::Filters::* namespace to make converting non-text
                   documents into a format that Swish-e can parse easy.  As
                   mentioned above, the helper script swishspider will use
                   these modules if can be found via PERL5LIB.  These modules
                   only provide an interface to programs that do the conver-
                   sion.  For example, you will need to download and install
                   the "catdoc" program to convert MSWord documents into text
                   for indexing. Please see filters/README to see how to use
                   this filter system.

           prog - general purpose access method
               The "prog" method is new to Swish-e version 2.2.  It's designed
               as a general purpose method to feed documents to swish from an
               external program.




2.4.7                       2009-04-04                          3





SWISH-RUN(1)          SWISH-E Documentation          SWISH-RUN(1)


               For example, the external program can read a database (e.g.
               MySQL), spider a web server, or convert documents from one for-
               mat to another (e.g. pdf to html).  Or, you can simply use it
               to read the files of the file system (like "-S fs"), yet pro-
               vide you with full control of what files are indexed.

               The external program name to run is passed to swish either by
               the IndexDir directive, or via the "-i" option.

               The program specified should be an absolute path as swish-e
               will attempt to stat() the program to make sure it exists.
               Swish does this to help in error reporting.

               If the program specified with -i or IndexDir is not an absolute
               path (i.e.  does not include "/" ) then swish-e will append the
               "libexecdir" directory defined during configuration.  Typi-
               cally, libexecdir is set to "$prefix/lib/swish-e"
               (/usr/local/lib/swish-e), but is platform and installation
               dependent.  Running swish-e -h will report the directory.

               For example, the -S prog program "spider.pl" is a Perl helper
               program for use with -S prog and is installed in libexecdir.

                   IndexDir spider.pl
                   SwishProgParameters default http://localhost/index.html

               and swish-e will find spider.pl in libexecdir.

               Additional parameters may be passed to the external program via
               the SwishProgParameters directive.  In the example above swish-
               e will pass two parameters to spider.pl, "default" and
               "http://localhost/index.html".

               A special name "stdin" may be used with "-i" or IndexDir which
               tells swish to read from standard input instead of from an
               external program.  See example below.

               The external program prints to standard output (which swish
               captures) a set of headers followed by the content of the file
               to index.  The output looks similar to an email message or a
               HTTP document returned by a web server in that it includes
               name/value pairs of headers, a blank line, and the content.

               The content length is determined by a content-length header
               supplied to swish by the program; there is no "end of record"
               character or flag sent between documents. Therefore, it is
               critical that the content-length header is correct.  This is a
               common source of errors.

               One advantage of this method (over using filters, for example)
               is that the external program is run only once for the entire
               indexing job, instead of once for every document.  This avoids
               forking and creating a new process for every document, and
               makes a huge difference when your external program is something



2.4.7                       2009-04-04                          4





SWISH-RUN(1)          SWISH-E Documentation          SWISH-RUN(1)


               like perl that has a large startup cost.

               Here's a simple example written in Perl:

                   #!/usr/local/bin/perl -w
                   use strict;

                   # Build a document
                   my $doc = <<EOF;
                   <html>
                   <head>
                       <title>Document Title</title>
                   </head>
                       <body>
                           This is the text.
                       </body>
                   </html>
                   EOF

                   # Prepare the headers for swish
                   my $path = 'Example.file';
                   my $size = length $doc;
                   my $mtime = time;

                   # Output the document (to swish)
                   print <<EOF;
                   Path-Name: $path
                   Content-Length: $size
                   Last-Mtime: $mtime
                   Document-Type: HTML*

                   EOF

                       print $doc;

               The external program passes to swish a header.  The header is
               separated from the body of the document with a blank line.  The
               available headers are:

               Path-Name:
                   This is the name of the file you are indexing. This can be
                   any string, so for example it could be an ID of a record in
                   a database, a URL or a simple file name.

                   This header is required.

               Content-Length:
                   This header specifies the length in bytes of the document
                   that follows the header.  This length must be exactly the
                   length of the document -- do not make the mistake of adding
                   an extra line feed at the end of the document.

                   This header is required.




2.4.7                       2009-04-04                          5





SWISH-RUN(1)          SWISH-E Documentation          SWISH-RUN(1)


               Last-Mtime:
                   Thi parameter is the last modification time of the file,
                   and must be a time stamp (seconds since the Epoch on your
                   platform).

                   This header is not required.

               Document-Type:
                   You may override swish's determination of document type
                   ("Indexcontents") by using the "Document-Type:" header.
                   The document type is used to select which parser Swish-e
                   uses to parse the document's contents.

                   For example, a spider program might map the content-type
                   returned from a web server to one of the types Swish-e
                   understands.  For example,

                       my $doc_type = 'HTML*' if $response->content_type =~ m!text/html!'

                   This header is not required.

               Update-Mode:
                   When updating an incremental index this header can be used
                   to select the mode for updating the index.  There are three
                   possible values:

                       Update
                       Remove
                       Index

                   "Update" will update the index with the given file if the
                   date of the given file is newer than the date of the file
                   already in the index.  Setting to "Update" is the same as
                   using -u on the command line.

                   "Remove" mode will remove the file specified by the Path-
                   Name header.  Setting "Remove" is the same as using -r on
                   the command line.

                   "Index" will add the file to the index. NOTE: swish-e will
                   not check to see if the file already exists.

                   If this header is not specified, the default is the mode
                   specified on the command line (-u, -r, or none).

                   This option is still experimental and is subject to change
                   in the future.  Ask on the Swish-e list before using.

               The above example program only returns one document and exits,
               which is not very useful.  Normally, your program would read
               data from some source, such as files or a database, format as
               XML, HTML, or text, and pass them to swish, one after another.
               The "Content-Length:" header tells swish where each document
               ends -- there is not any special "end of record" character or



2.4.7                       2009-04-04                          6





SWISH-RUN(1)          SWISH-E Documentation          SWISH-RUN(1)


               marker.

               To index with the above example you need to make sure that the
               program is executable (and that the path to perl is correct),
               and then call swish telling to run in "prog" mode, and the name
               of the program to use for input.

                   % chmod 755 example.pl
                   % ./swish-e -S prog -i ./example.pl

               Programs can and should be tested prior to running swish. For
               example:

                   % ./example.pl > test.out

               A few more useful example programs are provided in the swish-e
               distribution located in the prog-bin directory.  Some include
               documentation:

                   % cd prog-bin
                   % perldoc spider.pl

               Others are small examples that include comments:

                   % cd prog-bin
                   % less DirTree.pl

               The spider.pl program can be used as a replacement for the -S
               http method.  It is far more feature-rich and offers much more
               control over indexing.

               If you use the special program name "stdin" with "-i" or
               IndexDir then swish-e will read from standard input instead of
               from a program.  For example:

                   % ./example.pl --count=1000 /path/to/data | ./swish-e -S prog -i stdin

               This is basically the same as using a swish-e configuration
               file of:

                   SwishProgParameters --count=1000 /path/to/data
                   IndexDir ./example.pl

               in a config file and running

                   % ./swish-e -S prog -c swish.conf

               This gives an easy way to run swish without a configuration
               file with a "-S prog" program that requires parameters.  It
               also means you can capture data to a file and then index more
               once with the same data:






2.4.7                       2009-04-04                          7





SWISH-RUN(1)          SWISH-E Documentation          SWISH-RUN(1)


                   % ./example.pl /path/to/data --count=1000 > docs.txt
                   % cat docs.txt | ./swish-e -S prog -i stdin -c normal_index
                   % cat docs.txt | ./swish-e -S prog -i stdin -c fuzzy_index

               Using "stdin" might also be useful for programs that call swish
               (instead of swish calling the program).

               (The reason "stdin" is used instead of the more common "-" dash
               is due to the rotten way swish parses the command line.  This
               should be fixed in the future.)

               The "prog" method bypasses some of the configuration parameters
               available to the file system method -- settings such as "Index-
               Only", "FileRules", "FileMatch" and "FollowSymLinks" are
               ignored when using the "prog" method.  It's expected that these
               operations are better accomplished in the external program
               before passing the document onto swish.  In other words, when
               using the "prog" method, only send the documents to swish that
               you want indexed.

               You may use swish's filter feature with the "prog" method, but
               performance will be better if you run filtering programs from
               within your external program.  See also filters/README for an
               example how to easily add document converstion and filtering
               into your Perl-based programs.

               Notes when using -S prog on MS Windows

               Windows does not use the shebang (#!) line of a program to
               determine the program to run.  So, when running, for example, a
               perl program you may need to specify the perl.exe binary as the
               program, and use the "SwishProgParameters" to name the file.

                   IndexDir e:/perl/bin/perl.exe
                   SwishProgParameters read_database.pl

               Swish will replace the forward slashes with backslashes before
               running the command specified with "IndexDir".  Swish uses the
               popen(3) command which passes the command through the shell.

       -f *indexfile* (index file)
           If you are indexing, this specifies the file to save the generated
           index in, and you can only specify one file.  See also IndexFile in
           the configuration file.

           If you are searching, this specifies the index files (one or more)
           to search from. The default index file is index.swish-e in the cur-
           rent directory.

       -c *file ...* (configuration files)
           Specify the configuration file(s) to use for indexing.  This file
           contains many directives that control how Swish-e proceeds.  See
           SWISH-CONFIG for a complete listing of configuration file direc-
           tives.



2.4.7                       2009-04-04                          8





SWISH-RUN(1)          SWISH-E Documentation          SWISH-RUN(1)


           Example:

               swish-e -c docs.conf

           If you specify a directory to index, an index file, or the verbose
           option on the command-line, these values will override any speci-
           fied in the configuration file.

           You can specify multiple configuration files.  For example, you may
           have one configuration file that has common site-wide settings, and
           another for a specific index.

           Examples:

               1) swish-e -c swish-e.conf
               2) swish-e -i /usr/local/www -f index.swish-e -v -c swish-e.conf
               3) swish-e -c swish-e.conf stopwords.conf

           1  The settings in the configuration file will be used to index a
              site.

           2  These command-line options will override anything in the config-
              uration file.

           3  The variables in swish-e.conf will be read, then the variable in
              stopwords.conf will be read.  Note that if the same variables
              occur in both files, older values may be written over.

       -e (economy mode)
           For large sites indexing may require more RAM than is available.
           The "-e" switch tells swish to use disk space to store data struc-
           tures while indexing, saving memory.  This option is recommended if
           swish uses so much RAM that the computer begins to swap exces-
           sively, and you cannot increase available memory.  The trade-off is
           slightly longer indexing times, and a busy disk drive.

       -l (symbolic links)
           Specifying this option tells swish to follow symbolic links when
           indexing.  The configuration file value FollowSymLinks will over-
           ride the command-line value.

           The default is not to follow symlinks.  A small improvement in
           indexing time my result from enabling FollowSymLinks since swish
           does not need to stat every directory and file processed to deter-
           mine if it is a symbolic link.

       -N path (index only newer files)
           The "-N" option takes a path to a file, and only files newer than
           the specified file will be indexed.  This is helpful for creating
           incremental indexes -- that is, indexes that contain just files
           added since the last full index was created of all files.

           Example (bad example)




2.4.7                       2009-04-04                          9





SWISH-RUN(1)          SWISH-E Documentation          SWISH-RUN(1)


               swish-e -c config.file -N index.swish-e -f index.new

           This will index as normal, but only files with a modified date
           newer than index.swish-e will be indexed.

           This is a bad example because it uses index.swish-e which one might
           assume was the date of last indexing.  The problem is that files
           might have been added between the time indexing read the directory
           and when the index.swish-e file was created -- which can be quite a
           bit of time for very large indexing jobs.

           The only solution is to prevent any new file additions while full
           indexing is running.  If this is impossible then it will be
           slightly better to do this:

           Full indexing:

               touch indexing_time.file
               swish-e -c config.file -f index.tmp
               mv index.tmp index.full

           Incremental indexing:

               swish-e -c config.file -N indexing_time.file -f index.tmp
               mv index.tmp index.incremental

           Then search with

               swish-e -w foo -f index.full index.incremental

           or merge the indexes

               swish-e -M index.full index.incremental index.tmp
               mv index.tmp index.swish-e
               swish-e -w foo

       -r  **incremental index format only** The "-r" option puts swish-e into
           "removal" mode. Any input files (given with "-i" or the "IndexDir"
           parameter) are removed from an existing index.

           Example:

             swish-e -r -i file.html

           would remove file.html from the existing index.

       -u  **incremental index format only** The "-u" option puts swish-e into
           "update" mode. The timestamp of each input file is compared against
           the corresponding file in the existing index.  If swish-e encoun-
           ters an input file that either does not exist yet in the index or
           exists with a timestamp older than the input file, the input file
           is updated in the index. Any words in the input file that have been
           added or removed are reflected as such in the index.




2.4.7                       2009-04-04                         10





SWISH-RUN(1)          SWISH-E Documentation          SWISH-RUN(1)


           Example:

             swish-e -i file.html -u

           would update the index.swish-e index with the contents of
           file.html. If file.html was new, it would be added. If file.html
           already existed in the index, its contents would be updated in the
           index.

       -v [0|1|2|3] (verbosity level)
           The "-v" option can take a numerical value from 0 to 3.  Specify 0
           for completely silent operation and 3 for detailed reports.

           If no value is given then 1 is assumed.  See also IndexReport in
           the configuration file.

           Warnings and errors are reported regardless of the verbosity level.
           In addition, all error and warnings are written to standard out.
           This is for historical reasons (many scripts exist that parse stan-
           dard out for error messages).

       -W (0|1|2|3) (parser warning level)
           If using the libxml2 parser, the default parser warning level is
           set at 2. Use the "-W" option to override that default. Most often,
           you might want to turn it off altogether:

             swish-e -W0 -i path/to/files

           would fail silently if the parser encountered any errors.

SEARCHING
       The following command line arguments are available when searching with
       Swish-e.  These switches are used to select the index to search, what
       fields to search, and how and what to print as results.

       This section just lists the available command line arguments and their
       usage.  Please see SWISH-SEARCH for detailed searching instructions.

       Warning: If using Swish-e via a CGI interface, please see CGI Danger!

       Security Note: If the swish binary is named swish-search then swish
       will not allow any operation that would cause swish to write to the
       index file.

       Searching Command Line Arguments


       -w *word1 word2 ...*  (query words)
           This performs a case-insensitive search using a number of keywords.
           If no index file to search is specified (via the "-f" switch),
           swish-e will try to search a file called index.swish-e in the cur-
           rent directory.

               swish-e -w word



2.4.7                       2009-04-04                         11





SWISH-RUN(1)          SWISH-E Documentation          SWISH-RUN(1)


           Phrase searching is accomplished by placing the quote delimiter (a
           double-quote by default) around the search phrase.

               swish-e -w 'word or "this phrase"'

           Search would should be protected from the shell by quotes.  Typi-
           cally, this is single quotes when running under Unix.

           Under Windows command.com you may not need to use quotes, but you
           will need to backslash the quotes used to delimit phrases:

               swish-e -w \"a phrase\"

           The phrase delimiter can be set with the "-P" switch.

           The search may be limited to a MetaName.  For example:

               swish-e -w meta1=(foo or baz)

           will only search within the meta1 tag.

           Please see SWISH-SEARCH for a description of MetaNames

       -f *file1 file2 ...* (index files)
           Specifies the index file(s) used while searching.  More than one
           file may be listed, and each file will be searched.  If no "-f"
           switch is specified then the file index.swish-e in the current
           directory will be used as the index file.

       -m *number* (max results)
           While searching, this specifies the maximum number of results to
           return.  The default is to return all results.

           This switch is often used in conjunction with the "-b" switch to
           return results one page at a time (strongly recommended for large
           indexes).

       -b *number* (beginning result)
           Sets the begining search result to return (records are numbered
           from 1).  This switch can be used with the "-m" switch to return
           results in groups or pages.

           Example:

               swish-e -w 'word' -b 1 -m 20    # first 'page'
               swish-e -w 'word' -b 21 -m 20   # second 'page'

       -t HBthec (context searching)
           The "-t" option allows you to search for words that exist only in
           specific HTML tags. Each character in the string you specify in the
           argument to this option represents a different tag in which to
           search for the word. H means all HEAD tags, B stands for BODY tags,
           t is all TITLE tags, h is H1 to H6 (header) tags, e is emphasized
           tags (this may be B, I, EM, or STRONG), and c is HTML comment tags



2.4.7                       2009-04-04                         12





SWISH-RUN(1)          SWISH-E Documentation          SWISH-RUN(1)


           search only in header (<H*>) tags

               swish-e -w word -t h

       -d *string* (delimiter)
           Set the delimiter used when printing results.  By default, Swish-e
           separates the output fields by a space, and places double-quotes
           around the document title.  This output may be hard to parse, so it
           is recommended to use "-d" to specify a character or string used as
           a separator between fields.

           The string "dq" means "double-quotes".

               swish-e -w word -d ,    # single char
               swish-e -w word -d ::   # string
               swish-e -w word -d '"'  # double quotes under Unix
               swish-e -w word -d \"   # double quotes under Windows
               swish-e -w word -d dq   # double quotes

           The following control characters may also be specified: "\t \r \n
           \f".

           Warning: This string is passed directly to sprintf() and therefore
           exposes a securty hole.  Do not allow user data to set -d format
           strings directly.

       -P *character*
           Sets the delimiter used for phrase searches.  The default is double
           quotes """.

           Some examples under bash: (be careful about you shell metacharac-
           ters)

               swish-e -P ^ -w 'title=^words in a phrase^'
               swish-e -P \' -w "title='words in a pharse"'

       -p *property1 property2 ...*  (display properties)
           This causes swish to print the listed property in the search
           results.  The properties are returned in the order they are listed
           in the "-p" argument.

           Properties are defined by the ProperNames directive in the configu-
           ration file (see SWISH-CONFIG) and properties must also be defined
           in MetaNames.  Swish stores the text of the meta name as a prop-
           erty, and then will return this text while searching if this option
           is used.

           Properties are very useful for returning data included in a source
           documnet without having to re-read the source document while
           searching.  For example, this could be used to return a short docu-
           ment description.  See also see Document Summeries and Property-
           Names in SWISH-CONFIG.

           To return the subject and category properties while indexing.



2.4.7                       2009-04-04                         13





SWISH-RUN(1)          SWISH-E Documentation          SWISH-RUN(1)


               swish-e -w word -p subject category

           Properties are returned in double quotes.   If a property contains
           a double quote it is HTML escaped (&quot;).  See the "-x" switch
           for a more advanced method of returning a list of properties.

           NOTE: it is necessary to have indexed with the proper PropertyNames
           directive in the user config file in order to use this option.

       -s *property [asc|desc] ...*  (sort)
           Normally, search results are printed out in order of relevancy,
           with the most relevant listed first.  The "-s" sort switch allows
           you to sort results in order of a specified property, where a prop-
           erty was defined using the MetaNames and PropertyNames directives
           during indexing (see SWISH-CONFIG).

           The string passed can include the strings "asc" and "desc" to spec-
           ify the sort order, and more than one property may be specified to
           sort on more than one key.

           Examples:

           sort by title property ascending order

               -s title

           sort descending by title, ascending by name

               -s title desc name asc

           Note: Swish limits sort keys to 100 characters.  This limit can be
           changed by changing MAX_SORT_STRING_LEN in src/config.h and
           rebuilding swish-e.

       -L limit to a range of property values (Limit)
           This is an experimental feature!

           The "-L" switch can be used to limit search results to a range of
           property values

           Example:

               swish-e -w foo -L swishtitle a m

           finds all documents that contain the word "foo", and where the doc-
           ument's title is in the range of "a" to "m", inclusive.  By
           default, the case of the property is ignored, but this can be
           changed by using PropertyNamesCompareCase configuation directive.

           Limiting may be done with user-defined properties, as well.

           For example, if you indexed documents that contain a created times-
           tamp in a meta tag:




2.4.7                       2009-04-04                         14





SWISH-RUN(1)          SWISH-E Documentation          SWISH-RUN(1)


               <meta name="created_on" content="982648324">

           Then you tell Swish that you have a property called "created_on",
           and that it's a timestamp.

               PropertyNamesDate created_on

           After indexing you will be able to limit documents to a range of
           timestamps:

               -w foo -L created_on  946684800 949363199

           will find documents containing the word foo and that have a cre-
           ated_on date from the start of Jan 1, 2000 to the end of Jan 31,
           2000.

           Note: swish currently does not parse dates; Unix timestamps must be
           used.

           Two special formats can be used:

               -L swishtitle <= m
               -L swishtitle >= m

           Finds titles less than or equal, or grater than or equal to the
           letter "m".

           This feature will not work with "swishrank" or "swishdbfile" prop-
           erties.

           This feature takes advantages of the pre-sorted tables built by
           swish during indexing to make this feature fast while searching.
           You should see in the indexing output a line such as:

              6 properties sorted.

           That indicates that six pre-sorted tables were built during index-
           ing.  By default, all properties are presorted while indexing.
           What properties are pre-sorted can be controlled by the configura-
           tion parameter "PreSortedIndex".

           Using the "-L" switch on a property that was not pre-sorted will
           still work, but may be much slower during searching.

           Note that the PropertyNamesSortKeyLength setting is used for sort-
           ing properties.  Using too small a PropertyNamesSortKeyLength could
           result in -L selecting the wrong properties due to incomplete sort-
           ing.

           This is an experimental feature, and its use and interface are sub-
           ject to change.

       -x formatstring (extended output format)
           The "-x" switch defines the output format string.  The format



2.4.7                       2009-04-04                         15





SWISH-RUN(1)          SWISH-E Documentation          SWISH-RUN(1)


           string can contain plain text and property names (including swish-
           defined internal property names) and is used to generate the output
           for every result.  In addition, the output format of the property
           name can be controlled with C-like printf format strings.  This
           feature overrides the cmdline switches "-d" and "-p", and a warning
           will be generated if "-d" or "-p" are used with "-x".

           Warning: The format string (fmt) is passed directly to sprintf()
           and therefore exposes a securty hole.  Do not allow user data to
           set -x format strings directly.

           For example, to return just the title, one per line, in the search
           results:

               swish-e  -w ...   -x '<swishtitle>\n' ...

           Note: the "\n" may need to be protected from your shell.

           See also ResultExtFormatName for a way to define named format
           strings in the swish configuration file.

           Format of "formatstring":

               "text<propertyname>text<propertyname fmt=propfmtstr>text..."

           Where propertyname is:

           *   the name of a user property as specified with the config file
               directive "PropertyNames"

           *   the name of a swish Auto property (see below).  These proper-
               ties are defined automatically by swish -- you do not need to
               specify them with PropertyNames directive.  (This may change in
               the future.)

           propertynames must be placed within "<" and ">".

           User properties:

           Swish-e allows you to specify certain META tags within your docu-
           ments that can be used as document properties.  The contents of any
           META tag that has been identified as a document property can be
           returned as part of the search results.  Doucment properties must
           be defined while indexing using the PropertyNames configuration
           directive (see SWISH-CONFIG).

           Examples of user-defined PropertyNames:

               <keywords>
               <author>
               <deliveredby>
               <reference>
               <id>




2.4.7                       2009-04-04                         16





SWISH-RUN(1)          SWISH-E Documentation          SWISH-RUN(1)


           Auto properties:

           Swish defines a number of "Auto" properties for each document
           indexed.  These are available for output when using the "-x" for-
           mat.

               Name               Type     Contents
               --------------     -------  ----------------------------------------------
               swishreccount      Integer  Result record counter
               swishtitle         String   Document title
               swishrank          Integer  Result rank for this hit
               swishdocpath       String   URL or filepath to document
               swishdocsize       Integer  Document size in bytes
               swishlastmodified  Date     Last modified date of document
               swishdescription   String   Description of document (see:StoreDescription)
               swishdbfile        String   Path of swish database indexfile

           The Auto properties can also be specified using shortcuts:

               Shortcut    Property Name
               --------    --------------
                 %c        swishreccount
                 %d        swishdescription
                 %D        swishlastmodified
                 %I        swishdbfile
                 %p        swishdocpath
                 %r        swishrank
                 %l        swishdocsize
                 %t        swishtitle

           For example, these are equivalent:

              -x '<swishrank>:<swishdocpath>:<swishtitle>\n'
              -x '%r:%p:%t\n'

           Use a double percent sign "%%" to enter a literal percent sign in
           the output.

           Formatstrings of properties:

           Properties listed in an "-x" format string can include format con-
           trol strings.  These "propertyformats" are used to control how the
           contents of the associated property are printed.  Property formats
           are used like C-language printf formats.  The property format is
           specified by including the attribute "fmt" within the property tag.

           Format strings cannot be used with the "%" shortcuts described
           above.

           General syntax:

               -x '<propertyname fmt="propfmtstr">'

           where "subfmt" controls the output format of "propertyname".



2.4.7                       2009-04-04                         17





SWISH-RUN(1)          SWISH-E Documentation          SWISH-RUN(1)


           Examples of property format strings:

                   date type:    <swishlastmodified fmt="%d.%m.%Y">
                   string type:  <swishtitle fmt="%-40.35s">
                   integer type: <swishreccount fmt=/%8.8d/>

           Please see the manual pages for strftime(3) and sprintf(3) for an
           explanation of format strings.  Note: some versions of strftime do
           not offer the %s format string (number of seconds since the Epoch),
           so swish provides a special format string "%ld" to display the num-
           ber of seconds since the Epoch.

           The first character of a property format string defines the delim-
           iter for the format string.  For example,

               -x  "<author  fmt=[%20s]> ...\n"
               -x  "<author  fmt='%20s'> ...\n"
               -x  "<author  fmt=/%20s/> ...\n"

           Standard predefined formats:

           If you ommit the sub-format, the following formats are used:

               String type:       "%s"  (like printf char *)
               Integer type:      "%d"  (like printf int)
               Float type:        "%f"  (like printf double)
               Date type:         "%Y-%m-%d %H:%M:%S" (like strftime)

           Text in "formatstring" or "propfmtstr":

           Text will be output as-is in format strings (and property format
           strings).  Special characters can be escaped with a backslash.  To
           get a new line for each result hit, you have to include the New-
           line-Character "\n" at the end of "fmtstr".

               -x "<swishreccount>|<swishrank>|<swishdocpath>\n"
               -x "Count=<swishreccount>, Rank=<swishrank>\n"
               -x "Title=\<b\><swishtitle>\</b\>"
               -x 'Date: <swishlastmodified fmt="%m/%d/%Y">\n'
               -x 'Date in seconds: <swishlastmodified fmt=/%ld/>\n'

           Control/Escape charcters:

           you can use C-like control escapes in the format string:

              known controls:      \a, \b, \f, \n, \r, \t, \v,
              digit escapes:       \xhexdigits   \0octaldigits
              character escapes:   \anychar

           Example,

               swish -x "%c\t%r\t%p\t\"<swishtitle fmt=/%40s/>\"\n"

           Examples of -x format strings:



2.4.7                       2009-04-04                         18





SWISH-RUN(1)          SWISH-E Documentation          SWISH-RUN(1)


               -x "%c|%r|%p|%t|%D|%d\n"
               -x "%c|%r|%p|%t|<swishdate fmt=/%A, %d. %B %Y/>|%d\n"
               -x "<swishrank>\t<swishdocpath>\t<swishtitle>\t<keywords>\n
               -x "xml_out: \<title\><swishtitle>\>\</title\>\n"
               -x "xml_out: <swishtitle fmt='<title>%s</title>'>\n"

       -H [0|1|2|3|<n>]  (header output verbosity)
           The "-H n" switch generates extened header output.  This is most
           useful when searching more than one index file at a time by speci-
           fying more than one index file with the "-f" switch.  "-H 2" will
           generate a set of headers specific to each index file.  This gives
           access to the settings used to generate each index file.

           Even when searching a single index file, "-H n" will provided addi-
           tional information about the index file, how it was indexed, and
           how swish is interperting the query.

               -H 0 : print no header information, output only search result entries.
               -H 1 : print standard result header (default).
               -H 2 : print additional header information for each searched index file.
               -H 3 : enhanced header output (e.g. print stopwords).
               -H 9 : print diagnostic information in the header of the results (changed from: C<-v 4>)

       -R [0|1] (Ranking Scheme)
           This is an experimental feature!

           The default ranking scheme in SWISH-E evaluates each word in a
           query in terms of its frequency and position in each document. The
           default scheme is 0.

           New in version 2.4.3 you may optionally select an experimental
           ranking scheme that, in addition to document frequency and posi-
           tion, uses Inverse Document Frequency (IDF), or the relative fre-
           quency of each word across all the indexes being searched, and Rel-
           ative Density, or the normalization of the frequency of a word in
           relationship to the number of words in the document.

           NOTE: IgnoreTotalWordCountWhenRanking must be set to no or 0 in
           your index(es) for -R 1 to work.

           Specify -R 1 to turn on IDF ranking. See the API documentation for
           how to set the ranking scheme in your Perl or C program.

OTHER SWITCHES
       -V (version)
           Print the current version.

       -k *letter* (print out keywords)
           The "-k" switch is used for testing and will cause swish to print
           out all keywords in the index beginning with that letter.  You may
           enter "-k '*'" to generate a list of all words indexed by swish.

       -D *index file*  (debug index)
           The -D option is no longer supported in version 2.2.



2.4.7                       2009-04-04                         19





SWISH-RUN(1)          SWISH-E Documentation          SWISH-RUN(1)


       -T *options* (trace/debug swish)
           The -T option is used to print out information that may be helpful
           when debugging swish-e's operation.  This option replaced the "-D"
           option of previous versions.

           Running "-T help" will print out a list of available *options*

Merging Index Files
       In previous versions of Swish-e indexing would require a very large
       amount of memory and the indexing process could be very slow.  Merging
       provided a way to index in chunks and then combine the indexes together
       into a single index.

       Indexing is much faster now and uses much less memory, and with the
       "-e" switch very little memory is needed to index a large site.

       Still, at times it can be useful to merge different index files into
       one file for searching.  This could be because you want to keep sepa-
       rate site indexes and a common one for a global search, or you have
       separate collections of documents that you wish to search all at one
       time, but manage separately.

       -M *index1 index2 ... indexN out_index
           Merges the indexes specified on the command line -- the last file
           name entered is the output file.  The output index must not exist
           (otherwise merge will not proceed).

           Only indexes that were indexed with common settings may be merged.
           (e.g. don't mix stemming and non-stemming indexes, or indexes with
           different WordCharacter settings, etc.).

           Use the "-e" switch while merging to reduce memory usage.

           Merge generates progress messages regardless of the setting of
           "-v".

       -c *configuration file*
           Specify a configuration file while indexing to add administrative
           information to the output index file.

Document Info
       $Id: SWISH-RUN.pod 1741 2005-05-17 02:22:40Z karman $

       .













2.4.7                       2009-04-04                         20


