root/libswish3/tags/10Feb2008/doc/libswish3.3.pod.in

=pod

=head1 NAME

libswish3 - Swish3 C library

=head1 SYNOPSIS

<<libswish3.h_HERE>>

=head1 DESCRIPTION

B<libswish3> is the core C library of B<Swish3>.

B<libswish3> uses the GNOME L<Libxml2|http://xmlsoft.org/> library to parse words and metadata 
from XML, HTML and plain text files. B<libswish3> supports full UTF-8 encoding.

B<libswish3> is a parsing tool for use with information retrieval (IR) libraries.
Dynamic language bindings are available in the source distribution in the C<bindings>
directory.

=head1 APIs

The following APIs are defined:

=head1 Parsing API

B<libswish3> provides three basic input functions:

=over

=item

swish_parse_file()

=item

swish_parse_fh()

=item

swish_parse_buffer()

=back

Each of these functions takes a C<swish_Parser> struct pointer
and optional I<user_data>.

In addition:

=over

=item

The swish_parse_file() function takes a file path, which must be a valid file.
Directories and links are not supported. The assumption is that you will use
your calling code to recurse through directories and handle links.

=item

swish_parse_buffer() takes a string representing the document
headers and the full text of the document.

=item

swish_parse_fh() takes a filehandle pointer, which if set to NULL,
defaults to stdin.

=back

See the L<Headers API> section for more
information on using swish_parse_fh() and
swish_parse_buffer().

See the L<I<handler> Function> section for more information on how
to deal with the data extracted by each of the swish_parse_* functions.


=head1 Headers API

The Headers API supports and extends the Swish-e B<-S prog> feature, 
which allows you to feed the indexer with output from another I<prog>ram. 
The API has been extended from Swish-e's to allow for MIME types 
and more congruence with the HTTP 1.1 specification. 

See SWISH-RUN documentation
in the Swish-e distribution for the Swish-e version 2 headers API.

This is the libswish3 implementation. See B<SWISH::Prog::Headers> for a simple
Perl-based way of generating the proper headers.

=over

=item Content-Location

B<Swish-e name:> Path-Name

The name of the document. May be any string: an ID of a record in a database, 
a URL or a simple file name. The string is stored in the swish_DocInfo B<uri> struct member, 
which is often used as the primary identifier of a document in an index.

This header is required.

=item Content-Length

The length in bytes of the document, starting after the blank line separating the headers
from the document itself.
The value must be exactly the length of the document, including any extra
line feeds or carriage returns at the end of the document.

Example:

 Content-Location: foo.html
 Content-Length: 9

 The doc.\n
 ^^^^^^^^ ^
 12345678 9

The value is stored in the swish_DocInfo B<size> struct member.

This header is required.


=item Last-Modified

B<Swish-e name:> Last-Mtime

The last modification time of the document. The value must be an integer:
the seconds since the Epoch on your system.

If not present, will default to the current time.

The value is stored in the swish_DocInfo B<mtime> struct member.

This header is not required.

=item Parser-Type

B<Swish-e name:> Document-Type

Explicitly name the parser used for the document, rather than defaulting to the MIME
type mapping based on B<Content-Type> and/or B<Content-Location>. The three parser types are:

=over

=item

XML

=item

HTML

=item

TXT

=back

The Swish-e values B<XML2>, B<XML*>, B<HTML2>, B<HTML*>, B<TXT2>, B<TXT*> are also
supported for compatibility, but they map to the three libswish3 types.

The value is stored in the swish_DocInfo B<parser> struct member.

If not present, the document parser will be automatically chosen based on the following logic:

=over

=item

If a B<Content-Type> is given, the parser mapped to that MIME type will be used. You may override
the default mappings in your configuration. See B<Configuration API>.

=item

If no B<Content-Type> is given, a MIME type will be guessed at based on the file extension of the
document's B<Content-Location>, and the parser mapped to that MIME type will be used.

=item

Finally, if a MIME type is not identified, the parser defined in B<SWISHP_CONFIG_DEFAULT_PARSER>
in B<libswish3.h> will be used.

=back

See also B<Content-Type> and B<Content-Location>.

This header is not required.

=item Content-Type

The MIME type of the document. The libswish3 MIME type list is based on the Apache 2.0
version. See L<http://www.iana.org/assignments/media-types/> for the official registry.

If not defined with B<Content-Type>, the MIME type will be guessed based on the 
file extension in the B<Content-Location>
header. If the B<Content-Location> string does not contain a file extension (as might be the case
with non-URL value), or the file extension has no MIME mapping, then the MIME type will default
to B<SWISHP_DEFAULT_MIME> as defined in B<libswish3.h>.

You may override the default extension-to-MIME mappings in your configuration. See B<Configuration API>.

The value is stored in the swish_DocInfo B<mime> struct member.

See also B<Content-Location> and B<Parser-Type>.

This header is not required.


=item Update-Mode

B<NOTE:> This header exists only for backwards compatibility with Swish-e's incremental
index feature. B<It may be deprecated in a future version of libswish3.>

=back



=head1 Structures API

Writing an effective I<handler> function requires an understanding of some of the key
B<libswish3> data structures.

For more details on any of these structures, see the SYNOPSIS.

=head2 swish_Config

A configuration object. This object is required for initializing both a C<swish_Analyzer>
object and a C<swish_Parser> object.

=head2 swish_Parser

A parser object. Required for executing any of the three C<swish_parse_*> functions.

=head2 swish_ParseData

A parser data object. This object is passed around internally by the libxml2
SAX2 handlers, and is eventually the object passed to the I<handler> function pointer.
See L<The I<handler> Function>.

=head2 swish_WordList

A list of words or tokens. The object contains a linked list of swish_Word objects.
You can iterate over the contents of the WordList like this:

 swish_debug_msg("%d words in list", list->nwords);
 list->current = list->head;
 while (list->current != NULL)
 {
        swish_debug_msg("   ---------- WORD ---------  ");
        swish_debug_msg("word  : %s", list->current->word);
        swish_debug_msg(" meta : %s", list->current->metaname);
        swish_debug_msg(" context : %s", list->current->context);
        swish_debug_msg("  pos : %d", list->current->position);
        swish_debug_msg("soffset: %d", list->current->start_offset);
        swish_debug_msg("eoffset: %d", list->current->end_offset);
            
        list->current = list->current->next;
 }

=head2 swish_Word

An object representing one word or token in an object. The word's start and end offset,
position relative to other words, tag context and MetaName are all available in the object.

=head2 swish_DocInfo

An object describing metadata about the document itself: URI, MIME type, size, etc.

=head2 swish_Analyzer

The Analyzer object controls how the character content of a document is parsed: whether
or not a WordList is created with a tokenizer, if the words (tokens) are lowercased or 
stemmed, etc.

=head1 The I<handler> Function

The I<handler> function pointer is the final link in the parsing chain. The function
pointer is set in the Parser object constructor, and is called by each of the 
swish_parse_* functions after the entire document has been parsed and (optionally)
tokenized.

The I<handler> receives one argument: a swish_ParseData object containing all the metadata
and words in the document.

If all you wanted to do was print out a report about each document as it was parsed,
your I<handler> function might be as simple as:

 void
 my_handler( swish_ParseData * parse_data )
 {
    swish_debug_docinfo( parse_data->docinfo );
    swish_debug_wordlist( parse_data->wordlist );
    swish_debug_nb( parse_data->properties, "Property" );
    swish_debug_nb( parse_data->metanames, "MetaName" );
 }
 
B<IMPORTANT:> After the I<handler> function is called, all the structures referenced
by the swish_ParseData object are automatically freed, so if you intend to keep any of the
data for storing in an index, you will need to strdup() words, properties, docinfo, etc.
as part of your indexing code.

See the example C<swish_lint.c> file for how to create and pass in a I<handler>
function pointer to the swish_Parser constructor.

=head1 Configuration API

Configuration is different with B<libswish3> than with Swish-e. The biggest change
is that B<libswish3> configuration files are written in XML. This is done for several
reasons:

=over

=item 1

Since B<libswish3> already has a powerful XML parser built-in, it's much easier to 
parse a configuration file written in XML than to port the Swish-e config-style parser
to B<libswish3>.

=item 2

B<libswish3> stores index header information in a XML format nearly identical
to the configuration file format. So the parser needs to understand only one XML
schema.

=item 3

You can store UTF-8 text in your configuration file and it will be parsed correctly.

=item 4

The configuration directive list is extensible. Simple key/value configuration directives
can be added without any modification to the B<libswish3> config parser. They are simply
stored in the C<swish_Config> struct hash for your own use and amusement.

B<CAUTION:> The configuration directive names documented in the L<Directives> section below
are reserved for use by B<libswish3>. Some of them have special handling considerations
(like MetaNames and PropertyNames). So the important idea to grasp with the extensible
configuration feature is "simple key/value pairs."

=back

This section describes how to build a B<libswish3> configuration file.

=head2 Configuration Example

Here's an example B<libswish3> configuration file:

 <swish>
  <FollowSymLinks>yes</FollowSymLinks>
  
  <Meta name="foo" bias="+10" />
  <Meta name="bar" bias="-5" />
  <Meta name="swishtitle" bias="+50" alias="title" />
  <Meta name="other">color size weight</Meta>
  
  <Prop name="foo" type="text" ignorecase="1" />
  <Prop name="bar" type="int" />
  <Prop name="lastmod" type="date" />
  <Prop name="bing" comparecase="1" />
  <Prop name="description" verbatim="1" max="10000" alias="body" length="20" />
  <Prop name="notsorted" sort="0" />
  
  <Tokenize>1</Tokenize>
 </swish>

And here's that same example, dissected:

 <swish>

The top level tag.

 <FollowSymLinks>yes</FollowSymLinks>

Equivalent to the Swish-e style:

 FollowSymLinks yes

which simply informs whatever aggregator you are using that when confronted
with a symlink on the filesystem, it should be followed.

C<FollowSymLinks> is an example of a simple key/value pair (see the B<CAUTION> above).

=head3 MetaNames

Here's the first big difference from Swish-e. MetaNames, MetaNameAlias, and 
MetaNamesRank have been combined into a single XML tag with appropriate
attributes.

 <Meta name="foo" bias="10" />

is the same thing as (in Swish-e style):

 MetaNames foo
 MetaNamesRank 10 foo

while:

 <Meta name="swishtitle" bias="50" alias="title" />

is equivalent to:

 MetaNames swishtitle
 MetaNameAlias swishtitle title
 MetaNamesRank 50 swishtitle

You can see that the XML style allows for a terser, more compact expression.
You can still assign multiple aliases to a single MetaName:

 <Meta name="other">color size weight</meta>

is equivalent to:

 MetaNames other
 MetaNameAlias other color size weight

In addition, there are some special features intended for use with HTML documents.

 <Meta name="links" html="1" alias="href" />      # same as HTMLLinksMetaName
 <Meta name="images" html="1" alias="src" />      # same as ImageLinksMetaName
 <Meta name="alttext" html="1" alias="alt" />     # same as IndexAltTagMetaName
 <Meta name="as-text" html="1" alias="alt" />     # same as IndexAltTagMetaName

=head3 PropertyNames

PropertyNames, PropertyNamesCompareCase, PropertyNamesIgnoreCase, PropertyNamesNoStripChars,
PropertyNamesNumeric, PropertyNamesDate, PropertyNameAlias, PropertyNamesMaxLength,
PropertyNamesSortKeyLength, StoreDescription and PreSortedIndex
have all been combined into a single XML directive.

Here's the example from above with equivalent Swish-e directives annotated:

 <Prop name="foo" ignorecase="1" />
 # PropertyNamesIgnoreCase foo

 <Prop name="bar" type="int" />
 # PropertyNamesNumeric bar
 
 <Prop name="lastmod" type="date" />
 # PropertyNamesDate lastmod
 
 <Prop name="bing" comparecase="1" />
 # PropertyNamesCompareCase bing
 
 <Prop name="description" verbatim="1" max="10000" alias="body" length="20" />
 # PropertyNamesNoStripChars description
 # PropertyNamesMaxLength 10000 description
 # PropertyNameAlias description body
 # PropertyNamesSortKeyLength 20 description

 <Prop name="notsorted" sort="0" />
 # PreSortedIndex foo bar lastmod bind description

Again, the XML format greatly simplifies the syntax. You can assign attributes
as you need, though be aware that some attributes are inherently mismatched
and might generate an error or unexpected behaviour:

 <Prop name="foo" ignorecase="1" type="int" />      # wrong
 <Prop name="foo" comparecase="1" type="date" />    # wrong
 <Prop name="foo" verbatim="1" type="int" />        # wrong
 <Prop name="foo" sort="0" length="20" />           # wrong

=head2 Directives

The following configuration directives are currently supported.

 TODO

=head1 EXAMPLES

See the C<swish_lint.c> file included in the libswish3 distribution.

=head1 FAQ

=head2 What is IR?

Information Retrieval.

=head2 How is libswish3 related to Swish-e?

libswish3 is the core parsing library for Swish-e version 3 (Swish3).

=head2 Is libswish3 a search engine?

No. libswish3 is a document parser. It might work well in or with any number of search engines,
but it is not in itself any kind of search tool.

=head2 So what does libswish3 DO exactly?

libswish3 reads text, HTML and XML files and extracts just the words and document
properties from each document. It then hands off the wordlist and properties
to a I<handler> function. Finally, it frees all the memory associated with the wordlist
and properties.

The I<handler> function can do whatever you wish, though typically a I<handler>
would iterate over the words in the wordlist and add each one to an index using
an IR library API.


=head1 BACKGROUND

libswish3 is part of the Swish-e project. 
It was born out of the need for UTF-8 and incremental
indexing support and a desire to experiment with alternate indexing
libraries like Lucene, KinoSearch, Xapian and Hyperestraier.

libswish3 was developed with the idea that many quality IR libraries already exist,
but few if any provide an easy and fast way of preparing documents for indexing. 
The following assumptions informed the development of libswish3.

=head2 The IR Toolchain

A decent IR toolchain requires 5 parts:

=over 4

=item aggregator

Collects documents from a filesystem, database, website or other sources.

=item filter

Normalizes documents to a standard format (plain text or a delimited/markup
like YAML, HTML or XML) for indexing.

=item parser

Breaks a document into a list of words, including their context and position.

=item indexer

Writes the list of words in a storage system for quick, efficient retrieval.

=item searcher

Parses queries and fetches data from the indexer's storage system.

=back

Of course, the division between these parts is not always clean or apparent. Parsing search
queries, for example, will necessarily involve elements of the parser and searcher
components, while the indexer and searcher are of necessity intrinsically bound.

But any complete IR system will have these five parts in some combination.

=head2 Swish-e aggregators and filters are already good

The existing Swish-e document aggregators (B<DirTree.pl> and B<spider.pl>) and filtering
system (B<SWISH::Filter>) are good. They are all written in Perl and are easily modified,
and they have ample configuration options and documentation.

=head2 Why reinvent the wheel?

Several good IR libraries exist that provide an indexer and searcher. These libraries
do UTF-8, incremental indexing, and have search syntax on par with (or better than)
Swish-e 2.x. Examples include Xapian, KinoSearch and Lucene. 
While they might be a little slower
than Swish-e (at least in terms of indexing speed) they make up that for with:

=over

=item

well-documented APIs

=item

bindings in a variety of programming languages

=item

active development communities

=item

the flexibility that comes with being a library instead of a fixed program

=back


=head2 The missing link

The piece that Swish-e provides that other IR libraries lack is a fast, stable, integrated
document parser. Xapian has Omega, but it does not parse XML, nor does it recognize
ad hoc word context (metanames).

However, the Swish-e 2.x parser does not work independently of the Swish-e indexer
and searcher, nor does it support UTF-8.

One piece is missing: a parser that works with the Swish-e aggregator/filter system, supports
UTF-8, and offers flexible options for connecting with other IR libraries.

Ergo, libswish3: a document parser compatible with the existing Swish-e -S prog API
and capable of generating UTF-8 wordlists for indexing with a variety of IR libraries.

=head2 Where does libswish3 fit?

libswish3 is the core C library in Swish3.

However, libswish3 may be used without the rest of the Swish3.
The assumption is that libswish3 could fit into an IR toolchain like this:

 aggregator -> filter -> libswish3 -> some IR library

You could then use the native search API of the IR library.

For example, you might use the Swish-e B<spider.pl> script to spider a website, filtering
documents with B<SWISH::Filter> and then handing the output to a B<libswish3>-based
program that will parse the documents into words and store the data in a 
Xapian or KinoSearch index (or both!). That model is, in fact, what Swish3 does.

Or you might use the B<SWISH::Prog> Perl module (from the CPAN) to build your own
aggregator/filter system, then hand the output to libswish3.

=head1 AUTHOR

Peter Karman (peter@peknet.com).

=head1 CREDITS

B<libswish3> is inspired by code from
Swish-e (http://www.swish-e.org),
Libxml2 (http://www.xmlsoft.org),
Apache (http://www.apache.org),
Rahul Dhesi (http://www.tug.org/tex-archive/tools/zoo/),
Angel Ortega (http://www.triptico.com/software/unicode.html),
James Henstridge (http://www.jamesh.id.au/articles/libxml-sax/libxml-sax.html),
YoLinux (http://www.yolinux.com/TUTORIALS/GnomeLibXml2.html)
and no doubt many unnamed others.

All mistakes, errors and poor programming choices are, however, those
of the author.

=head1 LICENSE

B<libswish3> is licensed under the GPL.

libswish3 is free software; you can redistribute it and/or
modify it under the terms of the GNU Library General Public
License as published by the Free Software Foundation; either
version 2 of the License, or (at your option) any later version.

libswish3 is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
Library General Public License for more details.

You should have received a copy of the GNU Library General Public
License along with libswish3; see the file COPYING.  If
not, write to the

 Free Software Foundation, Inc.
 59 Temple Place - Suite 330
 Boston, MA 02111-1307, USA

=head1 SEE ALSO

The project homepage: http://dev.swish-e.org/wiki/swish3

swish_lint(1), swish_isw(1), swish_words(1)

=cut