


SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


NAME
       SWISH-CONFIG - Configuration File Directives

OVERVIEW
       This document lists the available configuration directives available in
       Swish-e.

CONFIGURATION FILE
       What files Swish-e indexes and how they are indexed, and where the
       index is written can be controlled by a configuration file.

       The configuration file is a text file composed of comments, blank
       lines, and configuration directives.  The order of the directives is
       not important.  Some directives may be used more than once in the con-
       figuration file, while others can only be used once (e.g. additional
       directives will overwrite preceding directives).  Case of the directive
       is not important -- you may use upper, lower, or mixed case.

       Comments are any line that begin with a "#".

           # This is a comment

       As of 2.4.3 lines may be continued by placing a backslas as the last
       character on the line:

           IgnoreWords \
               am \
               the \
               foo

       Directives may take more than one parameter.  Enclose single parameters
       that include whitespace in quotes (single or double).  Inside of quotes
       the backslash escapes the next character.

           ReplaceRules append "foo bar"   <- define "foo bar" as a single parameter

       If you need to include a quote character in the value either use a
       backslash to escape it, or enclose it in quotes of the other type.

       Backslashes also have special meaning in regular expressions.

           FileFilterMatch pdftotext "'%p' -" /\.pdf$/

       This says that the dot is a real dot (instead of matching any charac-
       ter).  If you place the regular expression in quotes then you must use
       double-backslashes.

           FileFilterMatch pdftotext "'%p' -" "/\\.pdf$/"

       Swish-e will convert the double backslash into a single backslash
       before passing the parameter to the regular expression compiler.

       Commented example configuration files are included in the conf direc-
       tory of the Swish-e distribution.



2.4.7                       2009-04-04                          1





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


       Some command line arguments can override directives specified in the
       configuration file.  Please see also the SWISH-RUN for instructions on
       running Swish-e, and the SWISH-SEARCH page for information and examples
       on how to search your index.

       The configuration file is specified to Swish-e by the "-c" switch.  For
       example,

           swish-e -c myconfig.conf

       You may also split your directives up into different configuration
       files.  This allows you to have a master configuration file used for
       many different indexes, and smaller configuration files for each sepa-
       rate index.  You can specify the different configuration files when
       running from the command line with the "-c" switch (see SWISH-RUN), or
       you may include other Configuration file with the IncludeConfigFile
       directive below.

       Typically, in a configuration file the directives are grouped together
       in some logical order -- that is, directives that control the source of
       the documents would be grouped together first, and directives that con-
       trol how each document is filtered or its words index in another group
       of directives. (The directives listed below are grouped in this order).

       The configuration file directives are listed below in these groups:

       o   "Administrative Headers Directives" -- You may add administrative
           information to the header of the index file.

       o   "Document Source Directives" -- Directives for selecting the source
           documents and the location of the index file.

       o   "Document Contents Directives" -- Directives that control how a
           document content is indexed.

       o   "Directives for the File Access method only" -- These directives
           are only applicable to the File Access indexing method.

       o   "Directives for the HTTP Access Method Only" -- Likewise, these
           only apply to the HTTP Access method.

       o   "Directives for the prog Access Method Only" -- These only apply to
           the prog Access method.

       o   "Document Filter Directives" -- This is a special section that
           describes using document filters with Swish-e.

       Alphabetical Listing of Directives


       o   AbsoluteLinks [yes|NO]

       o   BeginCharacters *string of characters*




2.4.7                       2009-04-04                          2





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


       o   BumpPositionCounterCharacters *string*

       o   Buzzwords [*list of buzzwords*|File: path]

       o   CompressPositions  [yes|NO]

       o   ConvertHTMLEntities [YES|no]

       o   DefaultContents [TXT|HTML|XML|TXT2|HTML2|XML2|TXT*|HTML*|XML*]

       o   Delay *seconds*

       o   DontBumpPositionOnEndTags *list of names*

       o   DontBumpPositionOnStartTags *list of names*

       o   EnableAltSearchSyntax  [yes|NO]

       o   EndCharacters *string of characters*

       o   EquivalentServer *server alias*

       o   ExtractPath *metaname* [replace|remove|prepend|append|regex]

       o   FileFilter *suffix* *program* [options]

       o   FileFilterMatch *program* *options* *regex* [*regex* ...]

       o   FileInfoCompression [yes|NO]

       o   FileMatch [contains|is|regex] *regular expression*

       o   FileRules [contains|is|regex] *regular expression*

       o   FuzzyIndexingMode [NONE|Stemming|Soundex|Metaphone|DoubleMetaphone]

       o   FollowSymLinks [yes|NO]

       o   HTMLLinksMetaName *metaname*

       o   IgnoreFirstChar *string of characters*

       o   IgnoreLastChar *string of characters*

       o   IgnoreLimit *integer integer*

       o   IgnoreMetaTags *list of names*

       o   IgnoreNumberChars *list of characters*

       o   IgnoreTotalWordCountWhenRanking [YES|no]

       o   IgnoreWords [*list of stop words*|File: path]




2.4.7                       2009-04-04                          3





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


       o   ImageLinksMetaName *metaname*

       o   IncludeConfigFile

       o   IndexAdmin *text*

       o   IndexAltTagMetaName *tagname*|as-text

       o   IndexComments [yes|NO]

       o   IndexContents [TXT|HTML|XML|TXT2|HTML2|XML2|TXT*|HTML*|XML*]  *file
           extensions*

       o   IndexDescription *text*

       o   IndexDir [URL|directories or files]

       o   IndexFile *path*

       o   IndexName *text*

       o   IndexOnly *list of file suffixes*

       o   IndexPointer *text*

       o   IndexReport [0|1|2|3]

       o   MaxDepth *integer*

       o   MaxWordLimit *integer*

       o   MetaNameAlias *meta name* *list of aliases*

       o   MetaNames *list of names*

       o   MinWordLimit *integer*

       o   NoContents *list of file suffixes*

       o   obeyRobotsNoIndex [yes|NO]

       o   ParserWarnLevel [0|1|2|3]

       o   PreSortedIndex *list of property names*

       o   PropCompressionLevel [0-9]

       o   PropertyNameAlias *property name* *list of aliases*

       o   PropertyNames *list of meta names*

       o   PropertyNamesCompareCase *list of meta names*

       o   PropertyNamesIgnoreCase *list of meta names*



2.4.7                       2009-04-04                          4





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


       o   PropertyNamesNoStripChars *list of meta names*

       o   PropertyNamesDate *list of meta names*

       o   PropertyNamesNumeric *list of meta names*

       o   PropertyNamesMaxLength integer *list of meta names*

       o   PropertyNamesSortKeyLength integer *list of meta names*

       o   ReplaceRules [replace|remove|prepend|append|regex]

       o   ResultExtFormatName  name -x format string

       o   SpiderDirectory *path*

       o   StoreDescription [XML <tag>|HTML <meta>|TXT size]

       o   "SwishProgParameters *list of parameters*

       o   SwishSearchDefaultRule   [<AND-WORD>|<or-word>]

       o   TmpDir *path*

       o   TranslateCharacters [*string1 string2*|:ascii7:]

       o   TruncateDocSize *number of characters*

       o   UndefinedMetaTags [error|ignore|INDEX|auto]

       o   UndefinedXMLAttributes [DISABLE|error|ignore|index|auto]

       o   UseStemming [yes|NO]

       o   UseSoundex [yes|NO]

       o   UseWords [*list of words*|File: path]

       o   WordCharacters *string of characters*

       o   XMLClassAttributes *list of XML attribute names*

       Directives that Control Swish

       These configuration directives control the general behavior of Swish-e.

       IncludeConfigFile *path to config file*
           This directive can be used to include configuration directives
           located in another file.

               IncludeConfigFile /usr/local/swish/conf/site_config.config

       IndexReport [0|1|2|3]
           This is how detailed you want reporting while indexing. You can



2.4.7                       2009-04-04                          5





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


           specify numbers 0 to 3.  0 is totally silent, 3 is the most ver-
           bose.   The default is 1.

           This may be overridden from the command line via the "-v" switch
           (see SWISH-RUN).

       ParserWarnLevel [0|1|2|3]
           Sets the error level when using the libxml2 parser for XML and
           HTML.  libxml2 will point out structural errors in your documents.

               0 = no report
               1 = fatal errors
               2 = errors
               3 = warnings

           Currently (as of 2.4.4 - early 2005) libxml2 only reports errors at
           level 2.  The default as of 2.4.4 is "2" which should report any
           errors that might indicate a problem parsing a document.

           The exception to this is UTF-8 to Latin-1 conversion errors are
           reported at level 3 (changed from 1 in 2.4.4).  Although these
           errors indicate a problem indexing text, they are only reported at
           level 3 because they can be very common.

           It is recommended that you index at ParserWarnLevel 3 when first
           starting out to see what errors and warnings are reported.  Then
           reduce the level when you understand what documents are causing
           parsing problems and why.

       IndexFile *path*
           Index file specifies the location of the generated index file.  If
           not specified, Swish-e will create the file index.swish-e in the
           current directory.

               IndexFile /usr/local/swish/site.index

       obeyRobotsNoIndex [yes|NO]
           When enabled, Swish-e will not index any HTML file that contains:

               <meta name="robots" content="noindex">

           The default is to ignore these meta tags and index the document.
           This tag is described at http://www.robotstxt.org/wc/exclu-
           sion.html.

           Note: This feature is only available with the libxml2 HTML parser.

           Also, if you are using the libxml2 parser (HTML2 and XML2) then you
           can use the following comments in your documents to prevent index-
           ing:

                  <!-- SwishCommand noindex -->
                  <!-- SwishCommand index -->




2.4.7                       2009-04-04                          6





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


           and/or these may be used also:

                  <!-- noindex -->
                  <!-- index -->

           For example, these are very helpful to prevent indexing of common
           headers, footers, and menus.

       NOTE: This following items are currently not available.  These items
       require Swish-e to parse the configuration file while searching.

       EnableAltSearchSyntax [yes|NO]
           NOTE: This following item is currently not available.

           Enable alternate search syntax.  Allows the usage of a basic
           "Altavista(c)", "Lycos(c)", etc. like search syntax.  This means a
           search query can contain "+" and "-" as syntax parameter.

           Example:

               swish-e -w "+word1 +word2 -word3  word4 word5"
               "+"  = following word has to be in all found documents
               "-"  = following word may not be in any document found
               " "  = following word will be searched in documents

       SwishSearhOperators <and-word> <or-word> <not-word>
           NOTE: This following item is currently not available.

           Using this config directive you can change the boolean search oper-
           ators of Swish-e, e.g. to adapt these to your language.  The
           default is:    AND  OR  NOT

           Example (german):

               SwishSearchOperators   UND  ODER  NICHT

       SwishSearchDefaultRule   [<AND-WORD>|<or-word>]
           NOTE: This following item is currently not available.

           "SwishSearchDefaultRule" defines the default Boolean operator to
           use if none is specified between words or phrases.  The default is
           "AND".

           The word you specify must match one of the available "SwishSearch-
           Operators".

           Example:

               SwishSearchOperators   UND  ODER  NICHT
               # Make it act like a web search engine
               SwishSearchDefaultRule ODER

       ResultExtFormatName name -x format string
           NOTE: This following item is currently not available.



2.4.7                       2009-04-04                          7





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


           The output of Swish-e can be defined by specifying a format string
           with the "-x" command line argument.  Using "ResultExtFormatName"
           you can assign a predefined format string to a name.

           Examples:

               ResultExtFormatName  moreinfo   "%c|%r|%t|%p|<author>|<publishyear>\n"

           Then when searching you can specify the format string's name

               swish-e   ...  -x moreinfo  ...

           See the "-x" switch in SWISH-RUN for more information about output
           formats.

       Administrative Headers Directives

       Swish-e stores configuration information in the header of the index
       file.  This information can be retrieved while searching or by func-
       tions in the Swish-e C library.  There are a number of fields available
       for your own use.  None of these fields are required:

       IndexName *text*
       IndexDescription *text*
       IndexPointer *text*
       IndexAdmin *text*
           These variables specify information that goes into index files to
           help users and administrators.  IndexName should be the name of
           your index, like a book title.  IndexDescription is a short
           description of the index or a URL pointing to a more full descrip-
           tion.  IndexPointer should be a pointer to the original informa-
           tion, most likely a URL.  IndexAdmin should be the name of the
           index maintainer and can include name and email information.  These
           values should not be more than 70 or so characters and should be
           contained in quotes.  Note that the automatically generated date in
           index files is in D/M/Y and 24-hour format.

           Examples:

               IndexName "Linux Documentation"
               IndexDescription "This is an index of /usr/doc on our Linux machine."
               IndexPointer http://localhost/swish/linux/index.html
               IndexAdmin webmaster

       Document Source Directives

       These directives control what documents are indexed and how they are
       accessed.  See also Directives for the File Access method only and
       Directives for the HTTP Access Method Only for directives that are spe-
       cific to those access methods.

       IndexDir [directories or files|URL|external program]
           IndexDir defines the source of the documents for Swish-e.  Swish-e
           currently supports three file access methods: File system, HTTP



2.4.7                       2009-04-04                          8





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


           (also called spidering), and prog for reading files from an exter-
           nal program.

           The "-S" command line argument is used to select the file access
           method.

               swish-e -c swish.config -S fs    - file system
               swish-e -c swish.config -S http  - internal http spider
               swish-e -c swish.config -S prog  - external program of any type

           For the fs method of access IndexDir is a space-separated list of
           files and directories to index.  Use a forward slash as the path
           separator in MS Windows.

           For the http method the IndexDir setting is a list of space-sepa-
           rated URLs.

           For the prog method the IndexDir setting is a list of space-sepa-
           rated programs to run (which generate documents for swish to
           index).

           You may specify more than one IndexDir directive.

           Any sub-directories of any listed directory will also be indexed.

           Note: While processing directories, Swish-e will ignore any files
           or directories that begin with a dot (".").  You may index files or
           directories that begin with a dot by specifying their name with
           "IndexDir" or "-i".

           Examples:

               # Index this directory an any subdirectories
               IndexDir /usr/local/home/http

               # Index the docs directory in current directory
               IndexDir ./docs

               # Index these files in the current directory
               IndexDir ./index.html ./page1.html ./page2.html
               # and index this directory, too
               IndexDir ../public_html

           For the HTTP method of access specify the URL's from which you want
           the spidering to begin.

           Example:

               IndexDir http://www.my-site.com/index.html
               IndexDir http://localhost/index.html

           Obviously, using the HTTP method to index is much slower than
           indexing local files.  Be well aware that some sites do not appre-
           ciate spidering and may block your IP address.  You may wish to



2.4.7                       2009-04-04                          9





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


           contact the remote site before spidering their web site.  More
           information about spidering can be found in Directives for the HTTP
           Access Method Only below.

           For the prog method of access IndexDir specifies the path to the
           program(s) to execute.  The external program must correctly format
           the documents being passed back to Swish-e.  Examples of external
           programs are provided in the prog-bin directory.

               IndexDir ./myprogram.pl

           See prog for details.

           Note: Not all directives work with all methods.

       NoContents *list of file suffixes*
           Files with these suffixes will not have their contents indexed, but
           will have their path name (file name) indexed instead.

           If the file's type is HTML or HTML2 (as set by "IndexContents" or
           "DefaultContents") then the file will be parsed for a HTML title
           and that title will be indexed.  Note that you must set the file's
           type with "IndexContents" or "DefaultContents": ".html" and ".htm"
           are NOT type HTML by default.  For example:

              IndexContents HTML* .htm .html

           If a title is found, it will still be checked for "FileRules
           title", and the file will be skipped if a match is found.  See
           "FileRules".

           If the file's type is not HTML, or it is HTML and no title is
           found, then the file's path will be indexed.

           For example, this will allow searching by image file name.

               NoContents .gif .xbm .au .mov .mpg .pdf .ps

           Note: Using this directive will not cause files with those suffixes
           to be indexed.  That is, if you use "IndexOnly" to limit the types
           of files that are indexed, then you must specify in "IndexOnly" the
           same suffixes listed in "NoContents".

           This does not work:

               # Wrong!
               IndexOnly .htm .html
               NoContents .gif .xbm .au .mov .mpg .pdf .ps

           A "-S prog" program may set the "No-Contents:" header to enable
           this feature for a specific document (although it would be smarter
           for the "-S prog" program to simply only send the pathname or title
           to be indexed.




2.4.7                       2009-04-04                         10





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


       ReplaceRules [replace|remove|prepend|append|regex]
           ReplaceRules allows you to make changes to file pathnames before
           they're indexed.  These changed file names or URLs will be returned
           in search results.

           For example, you may index your files locally (with the File system
           indexing method), yet return a URL in search results.  This direc-
           tive can be used to map the file names to their respective URLs on
           your web server.

           There are five operations you can specify: replace, append, remove,
           prepend, and regex They will parse the pathname in the order you've
           typed these commands.

           This directive uses C library regex.h regular expressions.

              replace "the string you want replaced" "what to change it to"
              remove "a string to remove"
              prepend "a string to add before the result"
              append "a string to add after the result"
              regex  "/search string/replace string/options"

           Remember, quotes are needed if an expression contains white space,
           and backslashes have special meaning.

           Regex is an Extended Regular Expression.  The first character found
           is the delimiter (but it's not smart enough to use matched chars
           such as [], (), and {}).

           The replace string may use substitution variables:

               $0      the entire matched (sub)string
               $1-$9   returns patterns captured in "(" ")" pairs
               $`      the string before the matched pattern
               $'      the string after the matched pattern

           The options change the behavior of expression:

               i       ignore the case when matching
               g       repeat the substitution for the entire pattern

           Examples:

               ReplaceRules replace testdir/ anotherdir/
               ReplaceRules replace [a-z_0-9]*_m.*\.html index.html

               ReplaceRules remove testdir/

               ReplaceRules prepend http://localhost/
               ReplaceRules append .html







2.4.7                       2009-04-04                         11





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


               ReplaceRules regex  !^/web/(.+)/!http://$1.domain.com/!
               replaces a file path:
                   /web/search/foo/index.html
               with
                   http://search.domain.com/foo/index.html

               ReplaceRules regex  #^#http://localhost/www#
               ReplaceRules prepend http://localhost/www  (same thing)

               # Remove all extensions from C source files
               ReplaceRules remove .c     # ERROR! That "." is *any char*
               ReplaceRules remove \.c    # much better...

               ReplaceRules remove "\\.c" # if in quotes you need double-backslash!
               ReplaceRules remove "\.c"  # ERROR! "\." -> "." and is *any char*

       IndexContents [TXT|HTML|XML|TXT2|HTML2|XML2|TXT*|HTML*|XML*]  *file
       exten- sions*
           The "IndexContents" directive assigns one of Swish-e's document
           parsers to a document, based on the its extension.  Swish-e cur-
           rently knows how to parse TXT, HTML, and XML documents.

           The XML2, HTML2, and TXT2 parsers are currently only available when
           Swish-e is configured to use libxml2.

           You may use XML*, HTML*, and TXT* to select the parser automati-
           cally.  If libxml2 is installed then it will be used to parse the
           content.  Otherwise, Swish-e's internal parsers will be used.

           Documents that are not assigned a parser with "IndexContents" will,
           by default, use the HTML2 parser if libxml2 is installed, otherwise
           will use Swish-e's internal HTML parser.  The "DefaultContents"
           directive may be used to assign a parser to documents that do not
           match a file extension defined with the "IndexContents" directive.

           Example:

               IndexContents HTML* .htm .html .shtml
               IndexContents TXT*  .txt .log .text
               IndexContents XML*  .xml

           HTML* is the default type for all files, unless otherwise specified
           (and this default can be changed by the DefaultContents directive.
           Swish-e parses titles from HTML files, if available, and keeps
           track of the context of the text for context searching (see "-t" in
           SWISH-RUN).

           If using filters (with the "FileFilter" directive) to convert docu-
           ments you should include those extensions, too.  For example, if
           using a filter to convert .pdf to .html, you need to tell Swish-e
           that .pdf should be indexed by the internal HTML parser:

               FileFilter  .pdf   pdf2html
               IndexContent  HTML  .pdf



2.4.7                       2009-04-04                         12





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


           See also Document Filter Directives.

           Note: Some of this may be changed in the future to use content-
           types instead of file extensions.  See SWISH-3.0

       DefaultContents [TXT|HTML|XML|TXT2|HTML2|XML2|TXT*|HTML*|XML*]
           This sets the default parser for documents that are not specified
           in IndexContents. If not specified the default is HTML.

           The XML2, HTML2, and TXT2 parsers are currently only available when
           Swish-e is configured to use libxml2.

           You may use XML*, HTML*, and TXT* to select the parser automati-
           cally.  If libxml2 is installed then it will be used to parse the
           content.  Otherwise, Swish-e's internal parsers will be used.

           Example:

               DefaultContents HTML

           The "DefaultContents" directive should be used when spidering, as
           HTML files may be returned without a file extension (such as when
           requesting a directory and the default index.html is returned).

       FileInfoCompression [yes|NO]
           ** This directive is currently not supported **

           Setting FileInfoCompression to "yes" will compress the index file
           to save disk space.  This may result in longer indexing times.  The
           default is "no".

           Also see the "-e" switch in SWISH-RUN for saving RAM during index-
           ing.

       Document Contents Directives

       These directives control what information is extracted from your source
       documents, and how that information is made available during searching.

       ConvertHTMLEntities [YES|no]
           ASCII entities can be converted automatically while indexing docu-
           ments of type HTML (not for HTML2).  For performance reasons you
           may wish to set this to "no" if your documents do not contain HTML
           entities.  The default is "yes".

           If "ConvertHTMLEntities" is set "no" the entities will be indexed
           without conversion.

           NOTE: Entities within XML files and files parsed with libxml2
           (HTML2) are converted regardless of this setting.

       MetaNames *list of names*
           META names are a way to define "fields" in your XML and HTML docu-
           ments.  You can use the META names in your queries to limit the



2.4.7                       2009-04-04                         13





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


           search to just the words contained in that META name of your docu-
           ment.  For example, you might have a META tagged field in your doc-
           uments called "subjects" and then you can search your documents for
           the word "foo" but only return documents where "foo" is within the
           "subjects" META tag.

               swish-e -w subjects=foo

           (See also the "-t" switch in SWISH-RUN for information about con-
           text searching in HTML documents.)

           The MetaNames directive is a space separated list.  For example:

               MetaNames meta1 meta2 keywords subjects

           You may also use "UndefinedMetaTags" to specify automatic extrac-
           tion of meta names from your HTML and XML documents, and also to
           ignore indexing content of meta tags.

           META tags can have two formats in your HTML source documents:

               <META NAME="meta1" CONTENT="some content">

           and (if using the HTML2/libxml2 parser)

               <meta1>
                   some content
               </meta1>

           But this second version is invalid HTML, and will generate a warn-
           ing if ParserWarningLevel is set (libxml2 only).

           And in XML documents, use the format:

               <meta1>
                   Some Content
               </meta1>

           Then you can limit your search to just META meta1 like this:

               swish-e -w 'meta1=(apples or oranges)'

           You may nest the XML and the start/end tag versions:

               <keywords>
                   <tag1>
                       some content
                   </tag1>
                   <tag2>
                       some other content
                   </tag2>
               <keywords>

           Then you can search in both tag2 and tag2 with:



2.4.7                       2009-04-04                         14





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


               swish-e -w 'keywords=(query words)'

           Swish-e indexes all text as some metaname.  The default is
           "swishdefault", so these two queries are the same:

               swish-e -w foo
               swish-e -w swishdefault=foo

           When indexing HTML Swish-e indexes the HTML title as default text,
           so when searching Swish-e will find matches in both the HTML body
           and the HTML title.  Swish also, by default, indexes content of
           meta tags.  So:

               swish-e -w foo

           will find "foo" in the body, the title, or any meta tags.

           Currently, there's no way to prevent Swish-e from indexing the
           title contents along with the body contents, but see "Undefined-
           MetaTags" for how to control the indexing of meta tags.

           If you would like to search just the title text, you may use:

               MetaNames swishtitle

           This will index the title text separately under the built-in swish
           internal meta name "swishtitle".  You may then search like

               swish-e -w foo  -- search for "foo" in title, body (and undefined meta tags)
               swish-e -w swishtitle=foo -- search for "foo" in title only

           In addition to swishtitle, you can limit searches to documents'
           path with:

              MetaNames swishdocpath

           Then to search for "foo" but also limit searches to documents that
           include "manual" or "tutorial" in their path:

              swish-e -w foo swishdocpath=(manual or tutorial)

           See also "ExtractPath".

       MetaNameAlias *meta name* *list of aliases*
           MetaNameAlias assigns aliases for a meta name.  For example, if
           your documents contain meta tags "description", "summary", and
           "overview" that all give a summary of your documents you could do
           this:

               MetaNames summary
               MetaNameAlias summary description overview

           Then all three tags will get indexed as meta tag "summary".  You
           can then search all the fields as:



2.4.7                       2009-04-04                         15





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


               -w summary=foo

           The Alias work at search time, too.  So these will also limit the
           search to the "summary" meta name.

               -w description=foo
               -w overview=foo

       MetaNamesRank integer *list of meta names*
           You can assign a bias to metanames that will affect how ranking is
           calculated.  The range of values is from -10 to +10, with zero
           being no bias.

               MetaNamesRank 4 subject
               MetaNamesRank 3 swishdefault
               MetaNamesRank 2 author publisher
               MetaNamesRank -5 wrongwords

           This feature is still considered experimental. If you use it,
           please send feedback to the discussion list.

       HTMLLinksMetaName *metaname*
           Allows indexing of HTML links.  Normally, HTML links (href tags)
           are not indexed by Swish-e.  This directive defines a metaname, and
           links will be indexed under this meta name.

           Example:

               HTMLLinksMetaName links

           Now, to limit searches to files with a link to "home.html" do this:

               -w links='"home.html"'

           The double quotes force a phrase search.

           To make Swish-e index links as normal text, you may use:

               HTMLLinksMetaName swishdefault

           This feature is only available with the libxml2 HTML parser.

       ImageLinksMetaName *metaname*
           Allows indexing of image links under a metaname.  Normally, image
           URLs are not indexed.

           Example:

               ImagesLinksMetaName images

           Now, if you would like to find pages that include a nice image of a
           beach:

               -w images='beach'



2.4.7                       2009-04-04                         16





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


           To make Swish-e index links as normal text, you may use:

               ImageLinksMetaName swishdefault

           This feature is only available with the libxml2 HTML parser.

       IndexAltTagMetaName *tagname*|as-text
           Allows indexing of images <IMG> ALT tag text.  Specify either a tag
           name which will be used as a metaname, or the special text
           "as-text" which says to index the ALT text as if it were plain text
           at the current location.

           For example, by specifying a tag name:

              IndexAltTagMetaName bar

           would make this markup:

               <foo>
                   <img src="/someimage.png" alt="Alt text here">
               </foo>

           appear like

               <foo>
                   <bar>Alt text here</bar>
               </foo>

           Then the normal rules ("MetaNames" and "PropertyNames") apply to
           how that text is indexed.

           If you use the special tag "as-text" then

               <foo>
                   <img src="/someimage.png" alt="Alt text here">
               </foo>

           simply becomes

               <foo>
                   Alt text here
               </foo>

           This feature is only available when using the libxml2 parser (HTML2
           and XML2).

       AbsoluteLinks [yes|NO]
           If this is set true then Swish-e will attempt to convert relative
           URIs extracted from HTML documents for use with "HTMLLinksMetaName"
           and "ImageLinksMetaName" into absolute URIs.  Swish-e will use any
           <BASE> tag found in the document, otherwise it will use the file's
           pathname.  The pathname used will be the pathname *after*
           "ReplaceRules" has been applied to the document's pathname.




2.4.7                       2009-04-04                         17





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


           For example, say you wish to index image links under the metaname
           "images".

               ImageLinksMetaName images

           If an image is located in http://localhost/vaca-
           tions/france/index.html and "AbsoluteLinks" is set to no, then a
           image within that document:

                <img src="beach.jpeg">

           will only index "beach.jpeg".

           But, if you want more detail when searching, you can enable "Abso-
           luteLinks" and Swish-e will index "http://localhost/vaca-
           tions/france/beach.jpeg".  You can then look for images of beaches,
           but only in France:

               -w images=(beach and france)

           This also means you can search for any images within France:

               -w images=(france)

           This feature is only available with the libxml2 HTML parser.

       UndefinedMetaTags [error|ignore|INDEX|auto]
           This directive defines the behavior of Swish-e during indexing when
           a meta name is found but is not listed in MetaNames.  There are
           four choices:

           error
             If a meta name is found that is not listed in MetaNames then
             indexing will be halted and an error reported.

           ignore
             The contents of the meta tag are ignored and not indexed unless a
             metaname has been defined with the "MetaNames" directive.

           index
             The contents of the meta tag are indexed, but placed in the main
             index unless there's an enclosing metatag already in force. This
             is the default.

           auto
             This method create meta tags automatically for HTML meta names
             and XML elements.  Using this is the same as specifying all the
             meta names explicitly in a MetaNames directive.

       UndefinedXMLAttributes [DISABLE|error|ignore|index|auto]
           This is similar to "UndefinedMetaTags", but only applies to XML
           documents (parsed with libxml2).  This allows indexing of attribute
           content, and provides a way to index the content under a metaname.
           For example, "UndefinedXMLAttributes" can make



2.4.7                       2009-04-04                         18





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


               <person age="23">
                     John Doe
               </person>

           look like the following to swish:

               <person>
                   <person.age>
                       23
                   </person.age>
                   John Doe
               </person>

           What happens to the text "23" will depend on the setting of "Unde-
           finedXMLAttributes":

           disable
             XML attributes are not parsed and not indexed.  This is the
             default.

           error
             If the concatenated meta name (e.g. person.age) is not listed in
             MetaNames then indexing will be halted and an error reported.

           ignore
             The contents of the meta tag are ignored and not indexed unless a
             metaname has been defined with the "MetaNames" directive.

           index
             The contents of the meta tag are indexed, but placed in the main
             index unless there's an enclosing metatag already in force.

           auto
             This method will create meta tags from the combined element and
             attributes (and XML Class name) This options should be used with
             caution as it can generate a lot of metaname entries.

             See also the example below "XMLClassAttribues".

       XMLClassAttributes *list of XML attribute names*
           Combines an XML class name with the element name to make up a
           metaname.  For example:

               XMLClassAttributes class

               <person class="first">
                   John
               </person>
               <person class="last">
                   Doe
               </person>

           Will appear to Swish-e as:




2.4.7                       2009-04-04                         19





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


               <person>
                   <person.first>
                   John
                   </person.first>
               </person>
               <person>
                   <person.last>
                   Doe
                   </person.last>
               </person>

           How the data is indexed depends on "MetaNames" and "Undefined-
           MetaTags".

           Here's an example using the following configuration which combines
           the two directives "XMLClassAttributes" and "UndefinedXMLAt-
           tributes".

               XMLClassAttributes class
               UndefinedMetaTags auto
               UndefinedXMLAttributes auto
               IndexContents XML2 .xml

           The source XML file looks like:

               <xml> <person class="student" phone="555-1212" age="102"> John </person>
               <person greeting="howdy">Bill</person> </xml>

           Swish-e parses as:

               ./swish-e -c 2 -i 1.xml -T parsed_tags  parsed_text  -v 0
               Indexing Data Source: "File-System"

               <xml> (MetaName)

                   <person> (MetaName)
                       <person.student> (MetaName)
                           <person.student.phone> (MetaName)
                               555-1212
                           </person.student.phone>
                           <person.student.age> (MetaName)
                               102
                           </person.student.age>
                           John
                   </person>

                   <person> (MetaName)
                       <person.greeting> (MetaName)
                           howdy
                       </person.greeting>
                       Bill
                   </person>





2.4.7                       2009-04-04                         20





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


               </xml>
               Indexing done!

           One thing to note is that the first <person> block finds a class
           name "student" so all metanames that are created from attributes
           use the combined name "person.student".  The second <person> block
           doesn't contain a "class" so, the attribute name is combined
           directly with the element name (e.g. "person.greeting").

       ExtractPath *metaname* [replace|remove|prepend|append|regex]
           This directive can be used to index extracted parts of a document's
           path.  A common use would be to limit searches to specific areas of
           your file tree.

           The extracted string will be indexed under the specified meta name.

           See "ReplaceRules" for a description of the various pattern
           replacement methods, but you will use the regex method.

           For example, say your file system (or web tree) was organized into
           departments:

               /web/sales/foo...
               /web/parts/foo...
               /web/accounting/foo...

           And you wanted a way to limit searches to just documents under
           "sales".

               ExtractPath department regex !^/web/([^/]+)/.*$!$1!

           Which says, extract out the department name (as substring $1) and
           index it as meta name "department".  Then to limit a search to the
           sales department:

               swish-e -w foo AND department=sales

           Note that the "regex" method uses a substitution pattern, so to
           index only a sub-string match the entire document path in the regu-
           lar expression, as shown above.  Otherwise any part that is not
           matched will end up in the substitution pattern.

           See the "ExtractPathDefault" option for a way to set a value if not
           patterns match.

           Although unlikely, you may use more than one "ExtractPath" direc-
           tive.  More than one directive of the same meta name will operate
           successively (in order listed in the configuration file) on the
           path.  This allows you to use regular expressions on the results of
           the previous pattern substitution (as if piping the output from one
           expression to the patter of the next).

               ExtractPath foo regex !^(...).+$!$1!
               ExtractPath foo regex !^.+(.)$!$1!



2.4.7                       2009-04-04                         21





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


           So, the third letter is indexed as meta name "foo" if both patterns
           match.

               ExtractPath foo regex !^X(...).+$!$1!
               ExtractPath foo regex !^.+(.)$!$1!

           Now (not the "X"), if the first pattern doesn't match, the last
           character of the path name is indexed.  You must be clear on this
           behavior if you are using more than one "ExtractPath" directive
           with the same metaname.

           The document path operated on is the real path swish used to access
           the document.  That is, the "ReplaceRules" directive has no effect
           on the path used with "ExtractPath".

           The full path is used for each meta name if more than one "Extract-
           Path" directive is used.  That is, changes to the path used in
           "ExtractPath foo" do not affect the path used by "ExtractPath bar".

       ExtractPathDefault *metaname* default_value
           This can be used with "ExtractPath" to set a default string to
           index under the given metaname if none of the "ExtractPath" pat-
           terns match.

           For example, say your want to index each document with a metaname
           "department" based on the following path examples:

               /web/sales/foo...
               /web/parts/foo...
               /web/accounting/foo...

           But you are also indexing documents that do not follow that pattern
           and you want to search those separately, too.

               ExtractPath department regex !^/web/([^/]+)/.*$!$1!
               ExtractPathDefault department other

           Now, you may search like this:

               -w foo department=(sales)      - limit searches to the sales documents
               -w foo department=(parts)      - limit searches to the parts documents
               -w foo department=(accounting) - limit searches to the accounting documents
               -w foo department=(other)      - everything but sales, parts, and accounting.

           This basically is a shortcut for:

               -w foo not department=(sales or parts or accounting)

           but you don't need to keep track of what was extracted.

       PropertyNames *list of meta names*
       PropertyNamesCompareCase *list of meta names*
       PropertyNamesIgnoreCase *list of meta names*
           Swish-e allows you to specify certain META tags that can be used as



2.4.7                       2009-04-04                         22





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


           document properties.  The contents of any META tag that has been
           identified as a document property can be returned as part of the
           search results along with the rank, file name, title, and document
           size (see the "-p" and "-x" switches in SWISH-RUN).

           Properties are useful for returning additional data from documents
           in search results -- this saves the effort of reading and parsing
           the source files while reading Swish-e search results, and is espe-
           cially useful when the source documents are no longer available or
           slow to access (e.g. over http).

           Another feature of properties is that Swish-e can use the Property-
           Names for sorting the search results (see the "-s" switch).

               PropertyNames author subjects

           Two variations are available.  "PropertyNamesCompareCase" and
           "PropertyNamesIgnoreCase".  These tell Swish-e to either ignore or
           compare case when sorting results.  The default for "PropertyNames"
           is to ignore the case.

               PropertyNamesIgnoreCase subject
               PropertyNamesCompareCase keyword

           The defaults for "internal" properties are:

               swishtitle          --  ignore the case
               swishdocpath        --  compare case
               swishdescription    --  compare case

           These can be overridden with "PropertyNamesCompareCase" and "Prop-
           ertyNamesIgnoreCase".

               PropertyNamesCompareCase swishtitle

           Use of PropertyNames will increase the size of your index files,
           sometimes significantly.  Properties will be compressed if Swish-e
           is compiled with zlib as described in the INSTALL manual page.

           If Swish-e finds more than one property of the same name in a docu-
           ment the property's contents will be concatinated for strings, and
           a warning issues for numeric (or date) properties.

       PropertyNamesNoStripChars
           PropertyNamesNoStripChars specifies that the listed properties
           should not have strings of low ASCII characters replaced with a
           space character.  Properties will be stored as found in the docu-
           ment.

           When printing properties with the swish-e binary newlines are
           replaced with a space character.  Use the swish-e library (or
           SWISH::API perl module) to fetch properties without newlines
           replaced.




2.4.7                       2009-04-04                         23





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


       PropertyNamesNumeric
           This directive is similar to "PropertyNames", but it flags the
           property as being a string of digits (integer value) that will be
           stored as binary data instead of a string.  This allows sorting
           with "-s" and limiting with "-L" to sort and limit the property
           correctly.

           Swish-e uses strtoul(3) to convert the string into an unsigned long
           integer.  Therefore, only positive integers can be stored.

           Future versions of Swish-e may be able to store different property
           types (such as negative integers and real numbers).  This directive
           may change in future releases of Swish.

       PropertyNamesDate
           This directive is exactly like "PropertyNamesNumeric", but it also
           flags the number as a machine timestamp (seconds since Epoch), and
           will print a formatted date when returning this property.  See "-x"
           in SWISH-RUN.

           Swish-e will not parse dates when indexing; you must use a times-
           tamp.

       PropertyNameAlias  *property name* *list of aliases*
           This allows aliases for a property name.  For example, if you are
           indexing HTML files, plus XML files that are written in English,
           German, and Spanish and thus use the tags "title", "titel", and
           "titulo" you can use:

               PropertyNameAlias swishtitle title titel titulo titulo

           Note that "swishtitle" is the built-in property used to store the
           title of a document, and therefore you do not need to specify it as
           a PropertyName before use.

       PropertyNamesMaxLength  integer *list of meta names*
           This option will set the max length of the text stored in a prop-
           erty.  You must specify a number between 0 and the max integer size
           on your platform, and a list of properties.  The properties speci-
           fied must not be aliases.

           If any of the property names do not exist they will be created
           (e.g. you do not need to define the property with PropertyNames
           first).

           In general, this feature will only be useful when parsing HTML or
           XML with the libxml2 parser.

           For example:

               PropertyNamesMaxLength 1000 swishdescription
               PropertyNameAlias swishdescription body

           Is somewhat like



2.4.7                       2009-04-04                         24





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


               StoreDescription HTML <body> 1000
               StoreDescription XML <body> 1000
               StoreDescription HTML2 <body> 1000
               StoreDescription XML2 <body> 1000

           but StoreDescription allows setting the tag for each parser type.

               PropertyNamesMaxLength 1000 headings
               PropertyNameAlias headings h1 h2 h3 h4

           collects all the heading text into a single property called "head-
           ings", not to exceed 1000 characters.

       PropertyNamesSortKeyLength  integer *list of meta names*
           Sets the length of the string used when sorting.  The default is
           100 characters.  The -T metanames debugging option will list the
           current values for an index.

           This setting is used when sorting during indexing, and perhaps when
           sorting while searching.  It also effects the order when limiting
           to a range of values with the -L option.

       PreSortedIndex *list of property names*
           By default Swish-e generates presorted tables while indexing for
           each property name.  This allows faster sorting when generating
           results.  On large document collections this presorting may add to
           the indexing time, and also adds to the total size of the index.
           This directive can be used to customize exactly which properties
           will be presorted.

           If "PreSortedIndex" it is not present in the config file (default
           action), all the properties will be presorted at indexing time.  If
           it is present without any parameter, no properties will be pre-
           sorted.  Otherwise, only the property names specified will be pre-
           sorted.

           For example, if you only wish to sort results by a property called
           "title":

               PropertyNames title age time
               PreSortedIndex  title

       StoreDescription [XML <tag> size|HTML <meta> size|TXT size]
           StoreDescription allows you to store a document description in the
           index file.  This description can be returned in your search
           results when the "-x" switch is used to include the swishdescrip-
           tion for extended results, or by using "-p swishdescription".

           The document type (XML, HTML and TXT) must match the document type
           currently being indexed as set by "IndexContents" or "DefaultCon-
           tents".  See those directives for possible values.  A common prob-
           lem is using "StoreDescription" yet not setting the document's type
           with "IndexContents" or "DefaultContents".  Another problem is dif-
           ferent types:



2.4.7                       2009-04-04                         25





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


               IndexContents HTML2 .html
               StoreDescription HTML <body>

           Then .html documents are assigned a type of HTML2 (and parsed by
           the libxml2 parser), but the description will not be stored since
           it is type HTML instead of HTML2.

           For text documents you specify the type TXT (or TXT2 or TXT*) and
           the number of characters to capture.

               StoreDescription TXT 20

           The above stores only the first twenty characters from the text
           file in the Swish-e index file.

           For HTML, and XML file types, specify the tag to use for the
           description, and optionally the number of characters to capture.
           If not specified will capture the entire contents of the tag.

               StoreDescription HTML <body> 20000
               StoreDescription XML  <desc> 40

           Again, note that documents must be assigned a document type with
           "IndexContents" or "DefaultContents" to use this feature.

           Swish-e will compress the descriptions (or any other large prop-
           erty) if compiled to use zlib (see INSTALL).  This is recommended
           when using StoreDescription and a large number of documents.  Com-
           pression of 30% to 50% is not uncommon with HTML files.

       PropCompressionLevel [0-9]
           This directive sets the compression level used when storing proper-
           ties to disk.  A setting of zero is no compression, and a setting
           of nine is the most compression.

           The default depends on the default setting compiled with zlib, but
           is typically six.

           This option is useful when using "StoreDescription" to store a
           large amount text in properties (or if using "PropertyNames" with
           large property sizes).

           Properties must be over a value defined in config.h (100 is the
           default) before compression will be attempted.  Swish-e will never
           store the results of the compression if the compressed data is
           larger than the original data.

           This option is only available when Swish-e is compiled with zlib
           support.

       TruncateDocSize *number of characters*
           TruncateDocSize limits the size of a document while indexing docu-
           ments and/or using filters.  This config directive truncates the
           numbers of read bytes of a document to the specified size.  This



2.4.7                       2009-04-04                         26





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


           means: if a document is larger, read only the specified numbers of
           bytes of the document.

           Example:

               TruncateDocSize    10000000

           The default is zero, which means read all data.

           Warning: If you use TruncateDocSize, use it with care!  Truncate-
           DocSize is a safety belt only, to limit e.g.  filteroutput, when
           accessing databases, or to limit "runnaway" filters.  Truncating
           doc input may destroy document structures for Swish-e (e.g.  swish
           may miss closing tags for XML or HTML documents).

           TruncateDocSize does not currently work with the "prog" input
           source method.

       FuzzyIndexingMode NONE|Stemming|Soundex|Metaphone|DoubleMetaphone
           Selects the type of index to create.  Only one type of index may be
           created.

           It's a good idea to create both a normal index and a fuzzy index
           and allow your search interface select which index to use.  Many
           people find the fuzzy searches to be too fuzzy.

           The available fuzzy indexing options can be displayed by running

              swish-e -T LIST_FUZZY_MODES

           Available options include:

           None
               Words are stored in the index without any conversion.  This is
               the default.

           Stemming_*
               This options uses one of the installed Snowball stemmers
               (http://snowball.tartarus.org/).

               The installed stemmers can be viewed by running

                  swish-e -T LIST_FUZZY_MODES

               For example, to use the Spanish stemming module:

                  FuzzyIndexingMode Stemming_es

           Stem or Stemming_en
               **This option is no longer supported.**

               Selects the legacy Swish-e English stemmer.

               This is deprecated in favor of the Snowball English stemmer



2.4.7                       2009-04-04                         27





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


               Stemming_en1.

               Words are converted using the Porter stemming algorithm.

               From: http://www.tartarus.org/~martin/PorterStemmer/

                   The Porter stemming algorithm (or Porter stemmer) is a
                   process for removing the commoner morphological and inflexional
                   endings from words in English. Its main use is as part of a
                   term normalisation process that is usually done when setting up
                   Information Retrieval systems.

               This will help a search for "running" to also find "run" and
               "runs", for example.

               The stemming function does not convert words to their root,
               rather programmatically removes endings on words in an attempt
               to make similar words with different endings stem to the same
               string of characters.  It's not a perfect system, and searches
               on stemmed indexes often return curious results.  For example,
               two entirely different words may stem to the same word.

               Stemming also can be confusing when used with a wildcard (trun-
               cation).  For example, you might expect to find the word "run-
               ning" by searching for "runn*".  But this fails when using a
               stemmed index, as "running" stems to "run", yet searching for
               "runn*" looks for words that start with "runn".

           Soundex
               Soundex was developed in the 1880s so records for people with
               similar sounding names could be found more readily.  Soundex is
               a coded surname based on the way a surname sounds rather than
               spelling.  Surnames that sound similar, like Smith and Smyth,
               are filed together under the same Soundex code.  This is mostly
               useful for US English.

               Soundex should not be used to search for sound-alike words.
               Metaphone would be more appropriate for generic sound matching
               of words.  Soundex should only be used where you need to search
               multiple documents for proper names which sound similar.  This
               is primarily used for indexing genealogical records.  This may
               be useful for indexing other collections of data consisting
               mostly of names.  Many common name variations are matched by
               Soundex.  The only notable exception is the first letter of the
               name.  The first letter is not matched for sound.

           Metaphone and DoubleMetaphone
               Words are transformed into a short series of letters represent-
               ing the sound of the word (in English).  Metaphone algorithms
               are often used for looking up mis-spelled words in dictionary
               programs.

               From: http://aspell.sourceforge.net/metaphone/




2.4.7                       2009-04-04                         28





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


                   Lawrence Philips' Metaphone Algorithm is an algorithm which returns
                   the rough approximation of how an English word sounds.

               The "DoubleMetaphone" mode will sometimes generate two differ-
               ent metaphones for the same word.  This is supposed to be use-
               ful when a word may be pronounced more than one way.

               A metaphone index should give results somewhere in between
               Soundex and Stemming.

       UseStemming [yes|NO]
           Put yes to apply word stemming algorithm during indexing, else no.

               UseStemming no
               UseStemming yes

           When UseStemming is set to "yes" every word is stemmed before plac-
           ing it in to the index.

           This option is deprecated.  It has been superceded by "FuzzyIndex-
           ingMode".

       UseSoundex [yes|NO]
           When UseSoundex is set to "yes" every word is converted to a
           Soundex code before placing it in to the index.

           This option is deprecated.  It has been superceded by "FuzzyIndex-
           ingMode".

       IgnoreTotalWordCountWhenRanking [YES|no]
           Put yes to ignore the total number of words in the file when calcu-
           lating ranking. Often better with merges and small files. Default
           is yes.

               IgnoreTotalWordCountWhenRanking no

           The default was changed from no to yes in version 2.2.

           NOTE: must be set to no if you intend to use the -R 1 option when
           searching.

       MinWordLimit *integer*
           Set the minimum length of an word. Shorter words will not be
           indexed.  The default is 1 (as defined in src/config.h).

               MinWordLimit 5

       MaxWordLimit *integer*
           Set the maximum length of an indexable word. Every longer word will
           not be indexed.  The Default is 40 (as defined in src/config.h).

       WordCharacters *string of characters*
       IgnoreFirstChar *string of characters*




2.4.7                       2009-04-04                         29





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


       IgnoreLastChar *string of characters*
       BeginCharacters *string of characters*
       EndCharacters *string of characters*
           These settings define what a word consists of to the Swish-e index-
           ing engine.  Compiled in defaults are in src/config.h.

           When indexing Swish-e uses WordCharacters to split up the document
           into words.  Words are defined by any string of non-blank charac-
           ters that contain only the characters listed in WordCharacters.  If
           a string of characters includes a character that is not in Word-
           Characters then the word will be spit into two or more separate
           words.

           For example:

               WordCharacters abde

           Would turn "abcde" into two words "ab" and "de".

           Next, of these words, any characters defined in IgnoreFirstChar are
           stripped off the start of the word, and IgnoreLastChar characters
           are stripped off the end of the word.  This allows, for example,
           periods within a word (www.slashdot.com), but not at the end of a
           word.  Characters in IgnoreFirstChar and IgnoreLastChar must be in
           WordCharacters.

           Finally, the resulting words MUST begin with one of the characters
           listed in BeginCharacters and end with one of the characters listed
           in EndCharacters.  BeginCharacters and EndCharacters must be a sub-
           set of the characters in WordCharacters.  Often, WordCharacters,
           BeginCharacters and EndCharacters will all be the same.

           Note that the same process applies to the query while searching.

           Getting these settings correct will take careful consideration and
           practice.  It's helpful to create an index of a single test file,
           and then look at the words that are placed in the index (see the
           "-v 4", "-D" and "-k" searching switches).

           Currently there is only support for eight-bit characters.

           Example:

               WordCharacters  .abcdefghijklmnopqrstuvwxyz
               BeginCharacters abcdefghijklmnopqrstuvwxyz
               EndCharacters   abcdefghijklmnopqrstuvwxyz
               IgnoreFirstChar .
               IgnoreLastChar  .

           So the string

               Please visit http://www.example.com/path/to/file.html.

           will be indexed as the following words:



2.4.7                       2009-04-04                         30





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


               please
               visit
               http
               www.example.com
               path
               to
               file.html

           Which means that you can search for "www.example.com" as a single
           word, but searching for just "example" will not find the document.

           Note: when indexing HTML documents HTML entities are converted to
           their character equivalents before being processed with these
           directives.  This is a change from previous versions of Swish-e
           where you were required to include the characters "0123456789&#;"
           to index entities.  See also "ConvertHTMLEntities"

       Buzzwords [*list of buzzwords*|File: path]
           The Buzzwords option allows you to specify words that will be
           indexed regardless of WordCharacters, BeginCharacters, EndCharac-
           ters, stemming, soundex and many of the other checks done on words
           while indexing.

           Buzzwords are case insensitive.

           Buzzwords should be separated by spaces and may span multiple
           directives.  If the special format "File:filename" is used then the
           Buzzwords will be read from an external file during indexing.

           Examples:

               Buzzwords C++ TCP/IP

               Buzzwords File: ./buzzwords.lst

           If a Buzzword contains search operator characters they must be
           backslashed when searching.  For example:

               Buzzwords C++ TCP/IP web=http

               ./swish-e -w 'web\=http'

           Buzzwords are found by splitting the text on whitespace, removing
           "IgnoreFirstChar" and "IgnoreLastChar" characters from the word,
           and then comparing with the list of "Buzzwords".  Therefore, if
           adding "Buzzwords" to an index you will probably want to define
           "IgnoreFirstChar" and "IgnoreLastChar" settings.

           Note: Buzzwords specific settings for "IgnoreFirstChar" and
           "IgnoreLastChar" may be used in the future.

       CompressPositions  [yes|NO]
           This option enables zlib compression for individual word data in
           the index file.  The default is NO, that is the index word data is



2.4.7                       2009-04-04                         31





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


           not compressed by default.

           Enabling this option can reduced the size of the index file, but at
           the expense of slower wildcard search times.

           The default changed from YES to NO starting with version 2.4.3.

       IgnoreWords [*list of stop words*|File: path]
           The IgnoreWords option allows you to specify words to ignore,
           called stopwords.  The default is to not use any stopwords.

           Words should be separated by spaces and may span multiple direc-
           tives.  If the special format "File:filename" is used then the stop
           words will be read from an external file during indexing.

           In previous versions of Swish-e you could use the directive

               IgnoreWords swishdefault - obsolete!

           to include a default list of compiled in stopwords.  This keyword
           is no longer supported.

           Examples:

               IgnoreWords www http a an the of and or

               IgnoreWords File: ./stopwords.de

       UseWords [*list of words*|File: path]
           UseWords defines the words that Swish-e will index.  Only the words
           listed will be indexed.

           You can specify a list of words following the directive (you may
           specify more than one "UseWords" directive in a config file),
           and/or use the "File:" form to specify a path to a file containing
           the words:

               UseWords perl python pascal fortran basic cobal php
               UseWords File: /path/to/my/wordlist

           Please drop the Swish-e list a note if you actually use this fea-
           ture.  It may be removed from future versions.

       IgnoreLimit *integer integer*
           This automatically omits words that appear too often in the files
           (these words are called stopwords). Specify a whole percentage and
           a number, such as "80 256". This omits words that occur in over 80%
           of the files and appear in over 256 files. Comment out to turn off
           auto-stopwording.

               IgnoreLimit 50 1000

           Swish-e must do extra processing to adjust the entire index when
           this feature is used.  It is recommended that instead of using this



2.4.7                       2009-04-04                         32





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


           feature that you decided what words are stopwords and add them to
           IngoreWords in your configuration file.  To do this, use Ignore-
           Limit one time and note the stop words that are found while index-
           ing.  Add this list to IgnoreWords, and then remove IgnoreLimit
           from the configuration file.

       IgnoreMetaTags *list of names*
           "IgnoreMetaTags" defines a list of metatags to ignore while index-
           ing XML files (and HTML files if using libxml2 for parsing HTML).
           All text within the tags will be ignored -- both for indexing
           ("MetaNames") and properties ("PropertyNames").  To still parse
           properties, yet do not index the text, see "UndefinedMetaTags".

           This option is useful to avoid indexing specific data from a file.
           For example:

               <person>
                   <first_name>
                       William
                   </first_name> <last_name>
                       Shakespeare
                   </last_name> <updated_date>
                       April 25, 1999
                   </updated_date>
               </person>

           In the above example you might not want to index the updated date,
           and therefore prevent finding this record by searching

               -w 'person=(April)'

           This is solved by:

               IgnoreMetaTags updated_date

           See also "UndefinedMetaTags".

       IgnoreNumberChars *list of characters*
           Experimental Feature

           This experimental feature can be used to define a set of characters
           that describe a number.  If a word is found to contain only those
           characters it will not be indexed.  The characters listed must be
           part of "WordCharacters" settings.  In other words, the "word"
           checked is a word that Swish-e would otherwise index.

           For example,

               IgnoreNumberChars 0123456789$.,

           Then Swish-e would not index the following:






2.4.7                       2009-04-04                         33





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


               123
               123,456.78
               $123.45

           You might be tempted to avoid indexing hex numbers with:

               IgnoreNumberChars 0123456789abcdef

           which will not index 0D31, but will also not index the word "bad".

           This is an experimental feature that may change in future versions.
           One possible change is to use regular expressions instead.

       IndexComments [NO|yes]
           This option allows the user decide if to index the contents of HTML
           comments.  Default is no. Set to yes if comment indexing is
           required.

               IndexComments yes

           Note: This is a change in the default behavior prior to version
           2.2.

       TranslateCharacters [*string1 string2*|:ascii7:]
           The TranslateCharacters directive maps the characters in string1 to
           the characters listed in string2.

           For example:

               # This will index a_b as a-b and amo as amo
               TranslateCharacters _a -a

           "TranslateCharacters :ascii7:" is a predefined set of characters
           that will translate eight bit characters to ascii7 characters.
           Using the :ascii7: rule will translate "Aac" to "aac". This means:
           searching "Celik", "celik" or "celik" will all match the same word.

           TranslateCharacters is done early in the indexing process, after
           converting HTML entities but before splitting the input text into
           words based on WordCharacters.  So characters you are translating
           from do not need to be listed in word characters.

           The same character translations take place when searching.

       BumpPositionCounterCharacters *string*
           When indexing Swish-e assigns a word position to each word.  This
           enables phrase searching.  There may be cases where you would like
           to prevent phrase matching.  The BumpPositionCounterCharacters
           directive allows you to specify a set of characters that when found
           in the text will increment the word position -- effectively pre-
           venting phrase matches across that character.

           For example, if you have a tag:




2.4.7                       2009-04-04                         34





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


               <subjects>
                   computer programming | apple computers
               </subjects>

           You might want to prevent matching "programming apple" in that meta
           name.

               BumpPositionCounterCharacters |

           There is no default, and you may list a string of characters.

       DontBumpPositionOnEndTags *list of names*
       DontBumpPositionOnStartTags *list of names*
           Since metatags are typically separate data fields, the word posi-
           tion counter is automatically bumped between metatags (actually,
           bumped when a start tag is found and when an end tag is found).
           This prevents matching a phrase that spans more than one metaname.
           "DontBumpPositionOnEndTags" and "DontBumpPositionOnStartTags" dis-
           ables this feature for the listed metanames.

           For example,

               <person>
                   <first_name>
                       William
                   </first_name>
                   <last_name>
                       Shakespeare
                   </last_name>
                   <updated_date>
                       April 25, 1999
                   </updated_date>
               </person>

           In the configuration file:

               DontBumpPositionOnEndTags first_name
               DontBumpPositionOnStartTags last_name

           This configuration allows this phrase search

               -w 'person=("william shakespeare")'

           but this phrase search will fail

               -w 'person=("shakespeare april")'

       Directives for the File Access method only

       Some directives have different uses depending on the source of the doc-
       uments.  These directives are only valid when using the File system
       method of indexing.





2.4.7                       2009-04-04                         35





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


       IndexOnly *list of file suffixes*
           This directive specifies the allowable file suffixes (extensions)
           while indexing.  The default is to index all files specified in
           IndexDir.

               # Only index .html .htm and .q files
               IndexOnly .html .htm .q

           "IndexOnly" checks that the file end in the characters listed.  It
           does not check "extensions".  "IndexOnly" is tested right before
           "FileRules" is processed.

       FollowSymLinks [yes|NO]
           Put "yes" to follow symbolic links in indexing, else "no".  Default
           is no.

               FollowSymLinks no
               FollowSymLinks yes

           Note that when set to "no" extra stat(2) system calls must be made
           for each file.  For large number of files you may see a small
           reduction in indexing time by setting this to "yes".

           See also the "-l" switch in SWISH-RUN.

       FileRules [type] [contains|is|regex] *regular expression*
       FileMatch [type] [contains|is|regex] *regular expression*
           FileRules and FileMatch are used to, respectively, exclude and
           include files and directories to index.  Since, by default, Swish-e
           indexes all files and recurses all directories (but see also "Fol-
           lowSymLinks") you will typically only use "FileRules" to exclude
           files or directories.  "FileMatch" is useful in a few cases, for
           example, to override the behavior of "IndexOnly".  Some examples
           are included below.

           Except for "FileRules title ...", this feature is only available
           for file access method (-S fs), which is the default indexing mode.
           Also, any pathname modification with "ReplaceRules" happens after
           the check for "FileRules".  (It's unlikely that you would exclude
           files with "FileRules" based on text you added with
           "ReplaceRules"!)

           The regular expression is a C regex.h extended regular expression.
           You may supply more than one regular expression per line, or use
           separate directives.  Preceding the regular expression with the
           word "not" negates the match.

           The regular expression is compared against [type] as described
           below.

           For historical reasons, you can specify "contains" or "is".  "is"
           simply forces the regular expression to match at the start and end
           of the string (by internally prepending "^" and appending "$" to
           the regular expression).



2.4.7                       2009-04-04                         36





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


           The "regex" option requires delimiter characters:

               FileRules title regex /^private/i

           The only advantage of "regex" is if you want to do case insensitive
           matches, or simply like your regular expressions to look like perl
           regular expressions.  You must use matching delimiters; (), {}, and
           [], are not currently supported for no good reason other than lazi-
           ness.

           Use quotes (" or ') around a pattern if it contains any white
           space.  Note that the backslash character becomes the escape char-
           acter within quotes.

           For example, these sets generate the same regular expressions.

               FileRules title is hello
               FileRules title contains ^hello$
               FileRules title regex /^hello$/

           These all need quotes due to the included space character

               FileRules title is "hello there"
               FileRules title contains "^hello there$"
               FileRules title regex "!^hello there$!"

           These show how the backslash must be doubled inside of quotes.
           Swish-e converts a double-backslash into a single backslash, and
           then passes that single onto the regular expression compiler.

               FileRules filename regex /\.pdf/
               FileRules filename regex "/\\.pdf/"

               FileRules filename regex !hello\\there!     # need double for real backslash
               FileRules filename regex "!hello\\\\there!" # need double-double inside of quotes

           Matching Types

           The following types of match strings my be supplied:

               FileRules pathname
               FileRules dirname
               FileRules filename
               FileRules directory
               FileRules title

               FileMatch pathname
               FileMatch filename
               FileMatch dirname
               FileMatch directory

           pathname matches the regular expression against the current path-
           name.  The pathname may or may not be absolute depending on what
           you supplied to "IndexDir".



2.4.7                       2009-04-04                         37





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


           Example:

               # Don't index paths that contain private or hidden
               FileRules pathname contains (private|hidden)

               # Same thing
               FileRules pathname regex /(private|hidden)/

               # Don't index exe files
               FileRules pathname contains \.exe$

           dirname and filename split the path name by the last delimiter
           character into a directory name, and a file name.  Then these are
           compared against the patterns supplied.  Directory names do not
           have a trailing slash.  All path names use the forward slash as a
           delimiter within Swish-e.

           Example:

               # Same as last example - don't index *.exe files.
               FileRules filename contains \.exe$

               # Don't index any file called test.html files
               FileRules filename contains ^test\.html$

               # Same thing
               FileRules filename is test\.html

               # Don't index any directories that contain "old"  (/usr/local/myold/docs)
               FileRules dirname contains old

               # Don't index any directories that contain the path segment "old" (/usr/local/old/foo)
               FileRules dirname contains /old/

               # Index only .htm, .html, plus any all-digit file names
               IndexOnly .htm .html
               FileMatch filename contains ^\d+$

               # Same as previous, but maybe a little slower
               FileRules filename regex not !\.(htm|html)$!
               FileMatch filename contains ^\d+$

           Swish-e checks these settings in the order of "pathname",
           "dirname", and "filename", and "FileMatch" patterns are checked
           before "FileRules", in general.  This allows you to exclude most
           files with "FileRules", yet allow in a few special cases with
           "FileMatch". For example:

               # Exclude all files of .exe, .bin, and .bat
               FileRules filename contains \.(exe|bin|bat)$
               # But, let these two in
               FileMatch filename is baseball\.bat incoming_mail\.bin





2.4.7                       2009-04-04                         38





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


               # Same, but as a single pattern
               FileMatch filename is (baseball\.bat|incoming_mail\.bin)

           The "directory" type is somewhat unique. When Swish-e recurses into
           a directory it will compare all the files in the directory with the
           pattern and then decide if that entire directory should or should
           not be indexed (or recursed).  Note that you are matching against
           file names in a directory -- and some of those names may be direc-
           tory names.

           A "FileRules directory" match will cause Swish-e to ignore all
           files and sub-directories in the current directory.

           Warning: A match with "FileMatch directory" says to index every-
           thing in the *current* directory and ignore any FileRules for this
           directory.

           Example:

               # Don't index any directories (and sub directories) that contain
               # a file (or sub-directory) called "index.skip"
               FileRules directory contains ^index\.skip$

               # Don't index directories that contain a .htaccess file.
               FileRules directory contains ^\.htaccess

           Note: While processing directories, Swish-e will ignore any files
           or directories that begin with a dot (".").  You may index files or
           directories that begin with a dot by specifying their name with
           "IndexDir" or "-i".

           "title" checks for a pattern match in an HTML title.

           Example:

               FileRules title contains construction example pointers

               # This example says to ignore case
               FileRules title regex "/^Internal document/i"

           Note: "FileRules title" works for any input method (fs, prog, or
           http) that is parsed as HTML, and where a title was found in the
           document.

           In case all this seems a bit confusing, processing a directory hap-
           pens in the following order.

           First the directory name is checked:

               FileRules dirname - reject entire directory if matches

           Next the directory is scanned and each file name (which might be
           the name of a sub-directory) is checked:




2.4.7                       2009-04-04                         39





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


               FileRules directory - reject entire dir if *any* files match
               FileMatch directory - accept entire dir if *any* files match

           Then, unless "FileMatch directory" matched, each file is tested
           with FileMatch.  A match says to index the file without further
           testing (i.e.  overrides FileRules and IndexOnly):

               FileMatch pathname  \
               FileMatch dirname   - file is accepted if any match
               FileMatch filename  /

           otherwise

               IndexOnly - file is checked for the correct file extension

               FileRules pathname  \
               FileRules dirname   - file is rejected if any match
               FileRules filename  /

           finally, the file is indexed.

           Files (not directories) listed with "IndexDir" or "-i" are pro-
           cessed in a similar way:

               FileMatch pathname  \
               FileMatch dirname   - file is accepted if any match
               FileMatch filename  /

           otherwise, the file is rejected if it doesn't have the correct
           extension or a FileRules matches.

               IndexOnly - file is checked for the correct file extension

               FileRules pathname  \
               FileRules dirname   - file is rejected if any match
               FileRules filename  /

           Note:  If things are not indexing as you expect, create a directory
           with some test files and use the "-T regex" trace option to see how
           file names are checked.  Start with very simple tests!

       Directives for the HTTP Access Method Only

       The HTTP Access method is enabled by the "-S http" switch when index-
       ing.  It works by running a Perl program called SwishSpider which
       fetches documents from a web server.

       Only text files (content-type of "text/*") are indexed with the HTTP
       Access Method.  Other document types (e.g. PDF or MSWord) may be
       indexed as well.  The SwishSpider will attempt to make use of the
       SWISH::Filter module (included with the Swish-e distribution) to con-
       vert documents into a format that Swish-e can index.

       Note: The -S prog method of spidering (using spider.pl) can be a



2.4.7                       2009-04-04                         40





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


       replacement for the -S http method.  It offers more configuration
       options and better spidering speed.

       These directives below are available when using the HTTP Access Method
       of indexing.

       MaxDepth *integer*
           MaxDepth defines how many links the spider should follow before
           stopping.  A value of 0 configures the spider to traverse all
           links.  The default is MaxDepth 0.

               MaxDepth 5

           Note: The default was changed from 5 to 0 in release 2.4.0

       Delay *seconds*
           The number of seconds to wait between issuing requests to a server.
           This setting allows for more friendly spidering of remote sites.
           The default is 5 seconds.

               Delay 1

           Note: The default was changed from 60 to 5 seconds in release 2.4.0

       TmpDir *path*
           The location of a writable temp directory on your system.  The HTTP
           access method tells the Perl helper to place its files in this
           location, and the "-e" switch causes Swish-e to use this directory
           while indexing.  There is no default.

               TmpDir /tmp/swish

           If this directory does not exist or is not writable Swish-e will
           fail with an error during indexing.

           Note, the environment variables of "TMPDIR", "TMP", and "TEMP" (in
           that order) will override this setting.

       SpiderDirectory *path*
           The location of the Perl helper script called swishspider.  If you
           use a relative directory, it is relative to your directory when you
           run Swish-e, not to the directory that Swish-e is in.  The default
           is the location swishspider was installed.  Normally this does not
           need to be set.

               SpiderDirectory /usr/local/swish

       EquivalentServer *server alias*
           Often times the same site may be referred to by different names.  A
           common example is that often http://www.some-server.com and
           http://some-server.com are the same.  Each line should have a list
           of all the method/names that should be considered equivalent.  Mul-
           tiple EquivalentServer directives may be used.  Each directive
           defines its own set of equivalent servers.



2.4.7                       2009-04-04                         41





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


               EquivalentServer http://library.berkeley.edu http://www.lib.berkeley.edu
               EquivalentServer http://sunsite.berkeley.edu:2000 http://sunsite.berkeley.edu

       Directives for the prog Access Method Only

       This section details the directives that are only available for the
       "prog" document source feature of Swish-e.  The "prog" access method
       runs an external program that "feeds" documents to Swish-e.  This
       allows indexing and filtering of documents from any source.

       See prog - general purpose access method in the SWISH-RUN man page for
       more information.

       A number of example programs for use with the "prog" access method are
       provided in the prog-bin directory.  Please see those example if you
       have questions about implementing a "prog" input program.

       SwishProgParameters *list of parameters*
           This is a list of parameters that will be sent to the external pro-
           gram when running with the "prog" document source method.

               SwishProgParameters /path/to/config hello there
               IndexDir /path/to/program.pl

           Then running:

               swish-e -c config -S prog

           Swish-e will execute "/path/to/program.pl" and pass "/path/to/con-
           fig hello there" as three command line arguments to the program.
           This directive makes it easy to pass settings from the Swish-e con-
           figuration file to the external program.

           For example, the "spider.pl" program (included in the "prog-bin"
           directory) uses the "SwishProgParameters" to specify what file to
           read for configuration information.

               SwishProgParameters spider.config
               IndexDir ./spider.pl

           The "spider.pl" program also has a default action so you can avoid
           using a configuration file:

               SwishProgParameters default http://www.swishe.org/ http://some.other.site/
               IndexDir ./spider.pl

           And the spider program will use default settings for spidering
           those sites.

           Swish-e can read documents from standard input, so another way to
           run an external program with parameters is:

               ./spider.pl spider.conf | ./swish-e -S prog -i stdin




2.4.7                       2009-04-04                         42





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


       Notes when using MS Windows

       You should use unix style path separators to specify your external pro-
       gram.  Swish will convert forward slashes to backslashes before calling
       the external program.  This is only true for the program name specified
       with "IndexDir" or the "-i" command line option.

       In addition, Swish-e will make sure the program specified actually
       exists, which means you need to use the full name of the program.

       For example, to run the perl spider program spider.pl you would need a
       Swish-e configuration file such as:

           IndexDir e:/perl/bin/perl.exe
           SwishProgParameters prog-bin/spider.pl default http://swish-e.org

       and run indexing with the command:

           swish-e -c swish.cfg -S prog -v 9

       The "IndexDir" command tells Swish-e the name of the program to run.
       Under unix you can just specify the name of the script, since unix will
       figure out the program from the first line of the script.

       The "SwishProgParameters" are the parameters passed to the program
       specified by "IndexDir" (perl.exe in this case).  The first parameter
       is the perl script to run (prog-bin/spider.pl).  Perl passes the rest
       of the parameters directly to the perl script.  The second parameter
       default tells the spider.pl program to use default settings for spider-
       ing (or you could specify a spider config file -- see "perldoc spi-
       der.pl" for details), and lastly, the URL is passed into the spider
       program.

       Document Filter Directives

       Internally, Swish-e knows how to parse only text, HTML, and XML docu-
       ments.  With "filters" you can index other types of documents.  For
       example, if all your web pages are in gzip format a filter can uncom-
       press these on the fly for indexing.

       You may wish to read the Swish-e FAQ question on filtering before con-
       tinuing here.  How Do I filter documents?

       There are two suggested methods for filtering.

       Filtering with SWISH::Filter

       The Swish-e distribution includes a Perl module called SWISH::Filter
       and individual filters located in the filters directory.  This system
       uses plug-in filters to extend the types of documents that Swish-e can
       index.  The plug-in filters do not actually do the filtering, but
       rather provide a standard interface for accessing programs that can
       filter or convert documents.  The programs that do the filtering are
       not part of the Swish-e distribution; they must be downloaded and



2.4.7                       2009-04-04                         43





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


       installed separately.

       The advantage of this method is that new filtering methods can be
       installed easily.

       This system is designed to work with the -S http and -prog methods, but
       may also be used with the "FileFilter" feature and -S fs indexing
       method.  See $prefix/share/doc/swish-e/examples/filter-bin/swish_fil-
       ter.pl for an example.

       See the filters/README file for more information.

       Filtering with the FileFilter feature

       A filter is an external program that Swish-e executes while processing
       a document of a given type.  Swish-e will execute the filter program
       for each file that matches the file suffix (extension) set in the File-
       Filter or FileFilterMatch directives.  FileFilterMatch matches using
       regular expressions and is described below.

       Filters may be used with any type of input method (i.e. -S fs, -S http,
       or -S prog).  But because

       Swish-e calls the external program passing as default arguments:

       $0  the name of the filter program

       $1  the physical path name of the file to read.  This may be a tempo-
           rary file location if indexing by the http method.

       $2  When indexing under the file system this will be the same as $1
           (the path to the source file), but when indexing under the http
           method this will be the URL of the source document.

       Swish-e can also pass other parameters to the filter program.  These
       parameters can be defined using the FileFilter or FileFilterMatch
       directives.  See Filter Options below.

       The filter program must open the file, process its contents, and return
       it to Swish-e by printing to STDOUT.

       Note that this can add a significant amount of time to the indexing
       process if your external program is a perl or shell script.  If you
       have many files to filter you should consider writing your filter in C
       instead of a shell or perl script, or using the "prog" Access Method
       along with SWISH::Filter.

       FilterDir  *path-to-directory*
           Deprecated.

           This is the path to a directory where the filter programs are
           stored.  Swish-e looks in this directory to find the filter speci-
           fied in the FileFilter directive.




2.4.7                       2009-04-04                         44





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


           This directive is not needed if the filter program can be found in
           your system's path.  Even if your filter is not in your system's
           path you can specify the full path to the filter in the FileFilter
           or FileFilterMatch directives.

           Example:

               FilterDir /usr/local/swish/filters

       FileFilter   *suffix*   "filter-prog"   ["filter-options"]
           This maps file suffix (extension) to a filter program.  If filter-
           prog starts with a directory delimiter (absolute path), Swish-e
           doesn't use the FilterDir settings, but uses the given filter-prog
           path directly.

           On systems that have a working fork(2) system call the filter pro-
           gram is run by forking swish then executing the filter.  This mean
           the shell is not used for running the filter and no arguments are
           passed through the shell.

           On other systems (e.g. Windows) the arguments are double-quoted and
           popen(3) is used to run the program.  This does pass argument
           though the shell and may be a security concern depending on the
           abilities of the shell.

           Filter options:

           Filter options are a string passed as arguments to the filter-prog.
           Filter options can contain variables, replaced by Swish-e.  If you
           omit filter-options Swish-e will use default parameters for the
           options listed above.

               Default:      %p %P
               Which means:  pass   "workfile path" and "documentfile path" to filter.

           Variables in filter options:

               %%   =  %
               %P   =  Full document pathname (e.g. URL, or path on filesystem)
               %p   =  Full pathname to work file (maybe a tmpfile or the real document path on filesystem)
               %F   =  Filename stripped from full document pathname
               %f   =  Filename stripped from "work" pathname
               %D   =  Directoryname stripped from full document pathname
               %d   =  Directoryname stripped from full "work" pathname

           Examples of strings passed:

               %P =  document pathname:  http://myserver/path1/mydoc.txt
               %p =  work pathname:      /tmp/tmp.1234.mydoc.txt
               %F =     mydoc.txt
               %f =     tmp.1234.mydoc.txt
               %D =     http://myserver/path1
               %d =     /tmp




2.4.7                       2009-04-04                         45





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


           Notes when using MS Windows

           Windows uses double quotes to escape shell metacharacters, so if
           you need to use quotes then use single quotes around the entire
           option string.

               FileFiler .mydoc mydocfilter.exe '--title "text with spaces"'

           You can specify the filter program using forward slashes (unix
           style).  Swish will convert the slashes to backslashes before run-
           ning your program.

               FileFilter .mydoc     c:/some/path/mydocfilter.exe  '-d "%d" -example -url "%P" "%f"'

           Examples of filters:

               FileFilter .doc       /usr/local/bin/catdoc "-s8859-1 -d8859-1 %p"
               FileFilter .pdf       pdftotext   "%p -"
               FileFilter .html.gz   gzip  "-c %p"
               FileFilter .mydoc     "/some/path/mydocfilter"  "-d %d -example -url %P %f"

           The above examples are running a binary filter program.  For more
           complicated filtering needs you may use a scripting language such
           as Perl or a shell script.  Here's some examples of calling a shell
           and perl script:

               FileFilter .pdf       pdf2html.sh
               FileFilter .ps        ghostscript-filter.pl

           Using a scripting language (or any language that has a large
           startup cost) can greatly increase the indexing time.  For small
           indexing jobs, this may not be an issue, but for large collections
           of files that require processing by a scripting language, you may
           be better off using the "-S prog" access method where the script
           will only be compiled once, instead of for each document.

           Filters are probably easier to write than a "-S prog" program.
           Which you decide to use depends on your requirements.  Examples of
           filter scripts can be found in the filter-bin directory, and exam-
           ples of "-S prog" programs can be found in the prog-bin directory.

       FileFilterMatch   *filter-prog*   *filter-options*  *regex* [*regex*
       ...]
           This is similar to "FileMatch" except uses regular expressions to
           match against the file name.  *filter-prog* is the path to the pro-
           gram.  Unlike "FileFilter" this does not use the "FilterDir"
           option.  Also unlike "FileFilter" you must specify the *fil-
           ter-options*.

           Examples:

               FileFilterMatch ./pdftotext "%p -" /\.pdf$/

           Note that will also match a file called ".pdf", so you may want to



2.4.7                       2009-04-04                         46





SWISH-CONFIG(1)       SWISH-E Documentation       SWISH-CONFIG(1)


           use something that requires a filename that has more than just an
           extension.  For example:

               FileFilterMatch ./pdftotext "%p -" /.\.pdf$/

           To specify more than one extension:

               FileFilterMatch ./check_title.pl "%p" /\.html$/  /\.htm$/

           Or a few ways to do the same thing:

               FileFilterMatch ./check_title.pl %p /\.(html|html)$/
               FileFilterMatch ./check_title.pl %p /\.html?$/

           And to ignore case:

               FileFilterMatch ./check_title.pl %p /\.html?$/i

           You may also precede an expression with "not" to negate regular
           expression that follow.  For example, to match files that do not
           have an extension:

               FileFilterMatch ./convert "%p %P" not /\..+$/

Document Info
       $Id: SWISH-CONFIG.pod 1846 2006-10-20 20:18:30Z whmoseley $

       .





























2.4.7                       2009-04-04                         47


