root/swish-e/trunk/pod/CHANGES.pod

=head1 NAME

CHANGES - List of revisions

=head1 OVERVIEW

This document contains list of bug fixes and feature additions to Swish-e.

=head2 Version 2.4.6 - 10 March 2008

=over 4

=item MinWordLength respected in query parser

Clark Vent reported that the query parser was not respecting MinWordLength
settings.  See http://dev.swish-e.org/changeset/2145

=item Patch to file.c.  

The file.c patch was in response to
http://swish-e.org/archive/2007-03/11321.html
although that user never responded about that patch.

=item SWISH_DEBUG_RANK env var now enables rank debugging

Set SWISH_DEBUG_RANK to a true value to enable lots of rank debugging
on stderr.

=item Perl Makefile.PL patched to fix MakeMaker issue

Recent versions of ExtUtils::MakeMaker revealed a bug in Makefile.PL.
Patch from mschwern via RT, report by mpeters.

=item LARGEFILE support detected automatically in configure

jrobinson852@yahoo.com suggest LARGEFILE support be auto-detected since
it is needed so often on Linux systems.

=item New Snowball stemmers

Trygve Falch contributed patches to update
the Snowball stemmers, including new Hungarian and Romanian stemmers.

=item Patched leaks

Anthony Dovgal patched two leaks.  One when there's a failure to
open a file the file name was not freed.

SwishSetSearchLimit() was nulling the search limits when an error was
found in the parameters, but not freeing the existing limits.

=item Leak in SwishResetSearchLimit

Fixed a leak if a limit was set and then reset but not prepared.
Patch provided by Antony Dovgal.

=item New API functions added

Added SwishGetStructure() and SwishGetPhraseDelimiter() functions which return
relevant properties of the search object.
Patch provided by Antony Dovgal.


=back

=head2 Version 2.4.5 - 22 Jan 2007

=over 4

=item Fixed 'deflate' handling in spider.pl

spider.pl was using the wrong method do uncompress HTTP responses that were
'deflate' encoded.  Also decode content based on the document's charset and
encode back to charset before outputting.

=item re-indexing required

The magic numbers in src/swish.h were changed to require re-indexing from
version 2.4.4 indexes. This should have been done in 2.4.4 as well, and anytime
the index format changes. -- karman

=item fixed stemmer bug introduced in 2.4.4

stemmer.c had a mix up in the deprecated stemmer assignments for "Stemmer_en"
and "Stem". Also fixed stemmer.h so that 2.4.3 indexes can be read correctly.
-- karman

=item Now fork/exec to run filters

FileFilter* was using popen to run the filter, which could pass user
data though the shell.  Now uses fork/exec if fork is available which
should be everywhere except Windows.  In windows popen is used but all
parameters are double-quoted. -- moseley

=item fixed signed/unsigned warnings from gcc 4.x

Cleaned up search.c to catch mismatched signedness warnings from newer GCC versions.
This issue pre-existed 2.4.4 but the new wildcard features in search.c made for a lot
more warnings. -- karman

=item Makefile.mingw included in distrib

Modified root Makefile to include the perl/Makefile.mingw file. -- karman

=back

=head2 Version 2.4.4 - 11 Oct 2006

=over 4

=item Version 2.4.4 RC1

Release Candidate 1 for 2.4.4, 2 Oct 2006.

=item quote fix for FileFilter config param

Ludovic Drolez contributed a patch to fix a quoting issue with filenames. This affects
non-Windows builds only.

=item SWISH::Filter now on CPAN

SWISH::Filter is now available on http://cpan.org/. The version in the distribution is
B<not> kept in sync with the CPAN version. Install the CPAN version if you want
the latest and greatest version.

=item SWISH::API updated to 0.04

Added several fixes, including:

=over

=item Perlish method names from mpeters@plusthree.com

=item switched to XSLoader with DynaLoader as fallback

=item added VERSION method to satisfy some versions of MakeMaker

=item Fuzzify() method now actually works as advertised

=back

=item added proximity feature and single character wildcard with '?' instead of '*'

Herman Knoops contributed these patches.
See http://swish-e.org/archive/2006-05/10543.html

Error messages were also changed to better reflect correct use of wildcards.

=item fixed bug when using DoubleMetaphone

Fixed problem reported by Andreas Völter where a query that generated a
two-word query with DoubleMetaphone fuzzy mode was not working.

=item fix sparc64 property issue

Sorithy Seng (pourlassi@gmail.com) submitted a patch against docprop.c to fix
an issue on sparc64 platforms. It is unknown whether this bug affected other 64-bit
architectures.

=item fixed bug when StopWords resulted in no unique words

Added check in db_native.c to check that some words exist before writing index.

=item updates to SWISH-RUN.1

Added doc for -u and -r options.

=item filename only in SWISH::Filters

added fix to SWISH::Filters::pp2html and SWISH::Filters::XLtoHTML to 
save only filename as title without full path

=item Removed Stem and Stemmer_en

The legacy Porter stemmer was removed. This had been deprecated some time ago.
A warning will issue if the old stemmer is indicated in config file, and Stemmer_en1
will be used instead.

=item GPL'd all the source files with the new Swish-e License

After a source code review, the developers decided to put Swish-e under the GPL
with a special exception for linking against libswish-e. See http://swish-e.org/license.html
for the details.

=item Fixed Segfault with updating incremental index

Dobrica Pavlinusic reported a segfaut after updating an index multiple times.
José provided updated worddata.c.  - April 27, 2005

=item Fixed NOT check with incremental indexes

Swish was returning results for deleted files when the NOT operator was used.

=item Fixed bug when using old parsers with zero length input

Thomas Angst reported swish consuming memory when using -S prog
to process large number of empty documents.

When -S prog generated a zero length file the old parsers (e.g. TXT) would
attempt to read in *all* content from the -S prog program into a buffer.
The old parser incorrectly assumed it was reading from a filter and tried to
read to eof().

=item Changes to ParserWarnLevel

The default value for ParserWarnLevel was changed form zero to two.

The ParserWarnLevel controls the error handling of the libxml2 parser. The higher
the setting, the more verbose the output. The change to the default is to report
when libxml2 has problems parsing a document (which often times results in processing
only part of a document).

To get the old behavior, either set ParserWarnLevel to zero in your config file,
or use the new -W command line option to set the ParserWarnLevel at run time.
If ParserWarnLevel is set in the config file, it will override the -W option.

Also, to see UTF-8 to 8859-1 conversion errors set ParserWarnLevel to 3 or more.  Previously,
these warning were issues at ParserWarnLevel of one.

=item Documentation changes

Removed all the target documentation (html, pdf, ps) from cvs.  There's now a separate
cvs module "swish_website" that is used to generate both the website and the html
docs.  If building swish-e from cvs please see the README.cvs file for instructions.

=item Fixed bug in pre-sorted indexes with USE_BTREE

Gunnar Mätzler reported a problem with reading the pre-sorted property index
tables when running with USE_BTREE (--enable-enremental).  Not all entries were
being written to disk.  There was/is a question if the "array" code used for
pre-sorted indexes with USE_BTREE would be slower.  So, added a separate 
define USE_PRESORT_ARRAY to enable that code when USE_BTREE is set.  This allows
using the old integer arrays with USE_BTREE.  Gunnar reported that this is working,
but more testing is needed.  Need to compare speed of the array code vs. the non-array
code, and to verify the workings of USE_PRESORT_ARRAY code.

=item Add strcoll() usage for sorting properties

Andreas Seltenreich provided a patch to use strcoll when sorting properties.
strcoll is locale dependent.

=item Fix incremental indexing when adding back a file

Jose fixed a problem with incremental indexing where a file could not be
added back to the index once removed.

Patch initially provided by Dobrica Pavlinusic:

    http://swish-e.org/Discussion/archive/2004-12/8694.html



=item Documentation correction

A change in the default way the index is compressed was not documented
in 2.4.3.  The change resulted in larger indexes.  See CompressPositions
below and in SWISH-CONFIG.

=item libxml2 UTF-8 conversion failures

Fixed issue where a UTF-8 to Latin1 encoding failure would skip
more input than just the failed character.   Libxml2 passes swish text
that is not null terminated, but the libxml2 functions to skip UTF-8
chars expected a null-terminated string.  Replace libxml2 call with
fixed version.

=back

=head2 Version 2.4.3 December 9, 2004

=over 4

=item New config directive: CompressPositions

This option enables zlib compression for word data in the index.
Previously word data was always compressed but resulted in slower
wildcard searches.  The default now is to not compress the word data,
but results in larger index files.  Set to "YES" to get pre-2.4.3 index
sizes.

[This CHANGES entry was added after 2.4.3 was released]

=item Improved error messsages when using incremental indexing

There was a bit of confusion on how to use incremental indexing (still
experimental) so added better logic for error messages.

Also fixed a logic error when setting the incremental update mode.  Caught by
Paul Loner.

=back

=head2 Version 2.4.3-pr1 - Wed Dec  1 09:52:50 PST 2004

=over 4

=item "Fixed" libxml2's change in UTF8Toisolat1() return value

Bernhard Weisshuhn supplied a patch to parser.c for checking the return value of
UTF8Toisolat1().  Seems that libxml2 now returns the number of characters converted
instead of zero for success.

   http://bugzilla.gnome.org/show_bug.cgi?id=153937

=item Added swish-config and pkg-config

Swish now provides a swish-config script and config file for the pkg-config
utility.  These tools help when building programs that link with the swish-e
library.

The SWISH::API Makefile.PL program uses swish-config to locate the installation
directory of swish-e.  This should make building SWISH::API easier when swish-e
is installed in a non-standard location.

=item Fixed rank bias in merge

Peter van Dijk noticed that MetaNamesRank settings were not being copied to the output
index when merging.

=item Added SwishFuzzy function

SwishFuzzy function (SWISH::API::Fuzzy) lets you stem a word without first searching.
This might be helpful for playing with queries prior to the search.


=item Fixed translate character table

Michael Levy found an error in the table used to translate 8859-1 to
ascii7.  Luckily, it was an upper case translation and the table is only used on lower
case characters.

=item MetaNamesRank documentation

Changed the 'not yet implemented' caveat to 'implemented but experimental'.

=item Added Continuation option to config processing

You can now use continuation lines in the config file:

    IgnoreWords \
        the \
        am \
        is \
        are \
        was

There may not be any characters following the backslash.

=item Fixed Buzzwords (and other word lists entered in the config)

Words entered in config were not converted to lower case before storing in the index.


=item Fixed metaname mapping problem in Merge

Peter Karman found an error when merging indexes where the source indexes had the
same metanames, but listed in a different order in their config files.  Words
would then be indexed under the wrong metaID number in the output index.


=item SWISH::Filters and spider.pl updates

The web spider F<spider.pl> was updated to work better with SWISH::Filter
by default and also make it easier to use the spider default along with
a spider config file.  See spider.pl for details.

SWISH::Filter was updated.  The way filters are created has changed.
If you created your own filters you will need to update them.  Take a look
at SWISH::Filter and the filters included in the distribution.

=item Updates to Documentation

Richard Morin submitted formatting and punctuation dates to the README and
INSTALL docs.

=item Added -R option to support IDF word weighting in ranking. (karman)

Added Inverse Document Frequency calculation to the getrank() routine.
This will allow the relative frequency of a word in relationship to other
words in the query to impact the ranking of documents.

Example: if 'foo' is present twice as often as 'bar' in the collection as a whole,
a search for 'foo bar' will weight documents with 'bar' more heavily (i.e., higher
rank) than those with 'foo'. 

The impact is greatest when OR'ing words in a query rather than
AND'ing them (which is the default).

Also added Rank discussion to the FAQ.


=item Updates to the example scripts

Updated PhraseHighlight.pm as suggested by Bill Schell for an optimization
when all words in a document are highlighted.

Updated search.cgi and PhraseHighlight.pm to use the internal stemmers via
the SWISH::API module as suggested by Jonas Wolf.


=item Leak when using C library

David Windmueller found a memory leak when calling multiple searches
on a swish handle.  The problem was swish loading the pre-sorted
property index on every search, even after the table had been loaded
into memory.

=item Swish.cgi now kills swish-e on time out

The example script F<swish.cgi> uses an alarm (on platforms that support
alarm) to abort processing after some number of seconds, but it was not
killing the child process, swish-e.  Bill Schell submitted a patch to kill
the child when the alarm triggers.

=item The template search.tt was renamed to swish.tt

The template was renamed because it's used by F<swish.cgi>, not by
F<search.cgi>, which was confusing.

=item Updates to the search.cgi

The example script F<search.cgi> was updated to work better with mod_perl
and to use external template files and style sheets.


=item New MS Word Filter

James Job provided the SWISH::Filter::Doc2html filter that uses
the wvWare (http://wvware.sourceforge.net/) program for filtering
MS Word documents.  If both catdoc and wvWare are installed then wvWare
will be used.

wvWare is reported to do a good job at converting MS Word docs
to HTML.  In a few tests it did work well, but other cases it
failed to generate correct output.  It was also much, much slower
than catdoc.  I tested with wvWare 0.7.3 on Debian Linux.  Testing with
both is recommended.

=item Change in way symbolic links are followed

John-Marc Chandonia pointed out that if a symlink is skipped
by FileRules, then the actual file/directory is marked as
"already seen" and cannot be indexed by other links or directly.

Now, files and directories are not marked "already seen" until
after passing FileRules (i.e after a file is actually indexed
or a directory is processed).

=item Could not set SwishSetSort() more than once

David Windmueller found a problem when trying to set the sort
order more than once on an existing search object.  Memory was not
correctly reset after clearing the previous sort values.

=item Access MetaNames and PropertyNames from API

Patch provided by Jamie Herre to access the MetaNames and PropertyNames
via the C API and to test via the testlib program.  Swish::API also updated
to access this data.

=item SwishResultPropertyULong() bug fixed

David Windmueller reported that SwishResultPropertyULong() was 
returning ULONG_MAX on all calls.  This was fixed.

=item Null written to wrong location in file.c

Bill Schell with the help of valgrind found a null written past the end of a
buffer in file.c in the code that supports the old parsers.  This resulted in a
segfault while indexing a large set of XML documents.

=item Fixed problem when indexing very large files

Steve Harris reported a problem when indexing a very large document that 
caused an integer overflow.  José Ruiz updated to used unsigned integers.

=item Bump word position on block tags with HTML2 parser

Peter Karman pointed out the the libxml2 HTML parser was allowing phrase
matches across block level html elements.  Swish now bumps the word
position on these elements.


=back

=head2 Version 2.4.2 - March 09, 2004

=over 4

=item * UseStemming didn't take no for an answer

UseStemming was coded as an alias for FuzzyIndexingMode when Snowball was
compiled in (the default), but "no" doesn't always mean no when the Norwegian
stemmer is available.

=item * Fixed problem building incremental version

Fixed compile problem with building incremental indexing mode.  This is an
experimental option with swish-e to allow adding files to an index.
See configure --help for build option.  Incremental indexes are not
compatible with standard indexes.

=item * Updated build instructions in INSTALL

Added a few comments about use of CPPFLAGS and LDFLAGS.

=item * Updated the index_hypermail.pl

Updated to work with latest version of hypermail (pre-2.1.9).


=item * Time zone in ResultPropertyStr()

Format string for generating date did not include the time zone in location.
Add strftime format string to config.h

=item * Undefined and Blank Properties and (NULL)

Fixed a few problems with printing properties:

1) Using -p and -x showed different results if a bad property value was given:

    $ swish-e -w not dkdk -p badname -H0
    err: Unknown Display property name "badname"
    .
    $ swish-e -w not dkdk -x '<badname>\n' -H0
    (NULL)

Now both return an error.

2) Fixed bug where using a "fmt" string with -x output generated (bad) output
if the result did not have the specified property.

    $ swish-e -w not dkdk -x '<somedate>\n' -H0  # undefined value

    $ swish-e -w not dkdk -x '<somedate fmt="%Y %B %d">\n' -H0
    %Y %B 1075353525

Now nothing is printed if the property does not exist.

3) Updated SWISH::API to croak() on invalid property names, and to return
undefined values for missing properties.

4) Updated swish.cgi and search.cgi to not generate warnings on undefined values
return as properties.  Note that swish.cgi will now die on undefined properties.
Previously would just display (NULL).


=item * Fixed segfault when generating warnings while parsing

Parser.c was incorrectly calling warning() incorrectly.
And -Wall was not catching this!

=item * Added check for internal property names.

Parser was not checking for use of Swish-e reserved property
names.

   <swishrank>foo</swishrank>

This will now generate a warning.

=back

=head2 Version 2.4.1 - December 17, 2003

=over 4

=item * Added new example CGI script

search.cgi is a new skeleton CGI script that uses SWISH::API for searching.
It is installed in the same location as swish.cgi.

=item * Add Fuzzy access to C and Perl interfaces

Added a number of functions to the C API (and SWISH::API)
to access the stemmer used when indexing a given index.

=item * Commas in numbers

Added commas to summary display at end of indexing.

=item * Insert whitespace between tags

Parser.c was updated to flush the text buffer before and after
every (non-inline HTML) tag.

The problem was that:

    foo<tag>bar</tag>baz

would index as a single word "foobarbaz".

=item * DirTree.pl

DirTree.pl was updated to work with SWISH::Filter and to work on Windows.
DirTree.pl is a program to fetch files from the file system and works with
the -S prog input method.

=item * Problem with --enable-incremental option

Fixed configure script to build incremental option.  Note that this is still
experimental.  But testers are welcome.

=item * headers.c bug

Mark Fletcher with the help of valgrind found a bug in headers.c
function SwishIndexHeaderNames used by the C API.

=item * Clarify documentation regarding search order

At the prompting of Doralyn Rossmann updated SEARCH.pod to
try and make the explanation of searching clearer, and to fix an error
in the description of nested searches.

=back

=head2 Version 2.4.0 - October 27, 2003

=over 4

=item * Note: Different Index Format

Swish-e version 2.4.0 has a different index file format from previous
versions of Swish-e.  Upgrading will B<require> reindexing -- version 2.4.0
cannot read indexes created with previous versions.

=back

=head2 Version 2.4.0 (Release Candidate 4)  September 26, 2003

=over 4

=item * robots.txt not closed correctly

When using -S http method robots.txt was not closed and that caused
the (last) .contents file to not be unlinked under Windows.  Windows
seems to think filenames are related to files.

=item * SWISH::Filter and locating programs on Windows

SWISH::Filter now scans $libexecdir in addition to the PATH for programs (such at catdoc and
pdftotext), and also checks for programs by adding the extensions ".exe" and ".bat" to the 
program name.

=item * Install sample templates

The sample templates included with swish.cgi are now installed
in $pkgdatadir (typically /usr/local/share/swish-e).

=back

=head2 Version 2.4.0 (Release Candidate 3)  September 11, 2003

=over 4

=item * Fix parser bug meta=(foo*)

Fixed bug in query parser caused in rc2's (pr2) attempt to catch wildcards
errors. 

=back

=head2 Version 2.4.0 (Release Candidate 2)  September 10, 2003

=over 4

=item * Indexing HTML title

Fixed a problem when these were used in combination:

  MetaNames swishtitle
  MetaNameAlias swishtitle title

That failed to correctly reset the metaname stack and indexed text under
the wrong metaID.

=item * Single Wildcards

Due to the way the query parser "works" a search of

   "foo *"

would result in a search of "foo*".  Now that results in:

   err: Single wildcard not allowed as word 

=item * Fixed search parsing bug

Brad Miele reported that the word "andes" was not being found.  It was being
stemmed to "and" when was then considered an operator.  [moseley]

=item * Add new directive PropertyNamesSortKeyLength

PropertyNamesSortKeyLength sets the sort key length to use when sorting
string properties.  The default is 100 characters.  There was a hard-coded
100 char limit before, but that was a problem where people were not building
from source (Windows).  The value of this is questionable -- it's intended to
limit how much memory is used when sorting while indexing and searching. [moseley]

=item * Fixed sorting issues with multiple indexes and reverse sorting

Reworked much of the sorting code.  Still to do is setting the character sort order. 
[moseley]

=item * Fixed minor memory leak

Fixed leak of not releasing memory of index file name and swish_handle 
destroy, and fixed SwishStemWord to default to the Stemmer_en. [moseley]

Fixed libtest.c example program that was not cleaning up memory after an
error condition.

=item * Replaced Swish-e's Porter Stemmer with Snowball

Swish-e now has support for Snowball stemmers (http://snowball.tartarus.org/).
The stemmers are enabled for an index with FuzzyIndexingMode Stemming_* where "*" can be:

  de, dk, en1, en2, es, fi, fr, it, nl, no, pt, ru, se

In addition, UseStemming yes or FuzzyIndexingMode Stemming_en will use the old stemmer.

=back

=head2 Version 2.4.0 (Release Candidate 1)  May 21, 2003

=over 4

=item * Security Fix: swish.cgi

The swish.cgi script was not correctly escaping HTML when searching by 
the right combination of metanames and highlighting module.  This could
lead to cross-site scripting if indexing un-trusted documents. [moseley]

=item * Added Support for building a Debian Package

To build as a .deb unpack the distribution and chdir then run

   $ fakeroot debian/build binary

Then install the generated .deb file with dpkg -i

=item * Use SWISH::Filter by default with spider.pl

spider.pl is installed in the libexecdir directory as well as the SWISH::Filter modules. 
PDF, MS Word, MP3, and XML documents will be indexed automatically if the required helper
applications (e.g. catdoc, pdftotext) or scripts (e.g. MP3::Tag) are installed.

Swish also knows about libexecdir, so you you specify a relative path with -S prog
swish-e will look for the program in libexecdir.  This is mostly for spider.pl so
indexing only requires:

    IndexDir spider.pl
    SwishProgParameters default http://localhost/index.html

And swish-e will find spider.pl and SWISH::Filter will be used to convert docs.

=item * Fixed Document-Type bug

Document-Type was not being reset after set input from a -S prog program causing
the wrong parser to be used. [moseley]

=item * New Directive: PropertyNamesNoStripChars

Swish replaces all series of low ASCII chars with a single space
character.  This option instructs swish to store all chars in the property. [moseley]

=item * Change HTTP access defaults

Defaults used with -S http access method were changed.


Delay was reduced from one minute between start of each request to five seconds
between requests. 

MaxDepth was changed from five to zero, meaning there is no limit to depth indexed by
default. [moseley]

=item * swishspider location and SpiderDirectory

The swishspider program is now installed in $prefix/lib/swish-e by default.  This can
be changed by the --libexecdir option to configure.  

The SpiderDirectory option now defaults to the value of libexecdir instead of the current
directory. [moseley]


=item * Added libtool and automake support

Replaces the build system with Autotools.  Now builds libswish-e as
a shared library on systems that support shared libraries.
The swish-e binary links against this shared library.
Can also build outside the source tree on platforms with GNU make. [moseley]

=item * Updates to installation

Running "make install" now installs additional files.
Files include the swish-e binary, the libswish-e search library, swish-e.h
header, documentation files, the swishspider program, and Perl modules used for the example
swish.cgi search script. Directories will be created if they do not already exist.
Installation directories can be specified at build time.

=item * Fixed bug when searching at end of inverted index

Swish was not correctly detecting the end of the inverted index
when searching a wildcard word that was past the last word in the index.
Caught by Frank Heasley. [moseley]


=item * Increase sort key length from 50 to 100 characters

The setting MAX_SORT_STRING_LEN in F<src/config.h> sets the max length used 
when sorting in swish-e.  You may reduce this number to save memory while 
sorting, or increase it if you have very long properties to sort.

=item * Remove &quot; entity from -p output

The -p option to print properties was escaping double quotes in properties 
with the &amp;quot; entity.  -x does not do that, so inconsistent.  -p no longer
converts double quotes.  The user should pick a good delimiter with -d or preferably use
the -x method for generating output.

=item * XML parser and Windows

The XML parser was being passed the incorrect buffer length when used on Windows
platform causing the parser to abort with an error.

=item * Version Numbering

SWISH-E versions starting with 2.3.4 use kernel version numbering.  Versions are 
in the form: Major.Minor.Build.  Odd minor versions are development.  Even minor 
versions are releases.  2.3.4 would be a development version.  
2.4.0 would be a release version.  2.3.20 would be the 20th build of 2.3.

=item * Added RPM support

RPMs can be built with:

    ./configure
    make dist

Copy the resulting tarball to RPM's SOURCES directory and then run as a superuser:

    rpmbuild -ba rpm/swish-e.spec


You should have swish-e packages in your RPMS/$arch directory.  [augur]

=item * Changed default perl binary location

Most perl scripts provided with SWISH-E now use /usr/bin/perl by default.
Note that some scripts are generated at build time, so those will look in the
path for the location of the perl binary.

=item * New Feature: MetaNamesRank

MetaNamesRank can be used to adjust the ranking for words based on
the word's MetaName.

=item * New Swish Library API and Perl Module

The Swish-e C library interface was rewritten to provide
better memory management and better separation of data.
Most indexing related code has been removed from the library.
A new header file is provided for the API: swish-e.h.

The Perl module SWISHE was replaced with the SWISH::API module
in the Swish-e distribution.

B<Previous versions of the SWISHE module will not work with this version of Swish-e.>

If you are using the SWISHE module from a previous version of Swish then you must
either rewrite your code to use the new SWISH::API module (highly recommended)
or use the replacement SWISHE module.  The replacement SWISHE module is a thin
interface to the SWISH::API module.  It can be downloaded from

    http://swish-e.org/Download/old/SWISHE-0.03.tar.gz

=item * NoContents not working with libxml2 parser

Corrected problem when using NoContents with binary files and the HTML2 parser.

Trying to index image file names with:

    IndexOnly .gif .jpeg
    NoContents .gif .jpeg

failed to index the path names because the default parser
(HTML2 when libxml2 is linked with swish-e)
was not finding any text in the binary files. [moseley]

=item * Updates to swish.cgi

The example/swish.cgi script can now use the SWISH::API module
for searching an index.  Combined with mod_perl this module
can improve search performance considerably.

The Perl modules used with the swish.cgi script have all been moved into
the SWISH::* namespace.  Hence, files in the F<modules> directory were moved
into the F<modules::SWISH> directory.

=back

=head2 Version 2.2.3 - December 11, 2002

Multiple -L options were ORing instead of ANDing.
Catch by Patrick Mouret. [moseley]

=head2 Version 2.2.2 - November 14, 2002

Pass non- text/* files onto indexing code IF there is a FileFilter
associated with the *extension* of the URL.  Fixes the problem of not
being able to index, say, pdf files by using the FileFilter configuation
option.

Fixed bug where nulls were stripped when using FileFilter with -S prog.
Catch by Greg Fenton. [moseley]

=head2 Version 2.2.1 - September 26, 2002

=over 4

=item * NoContents with -S prog

Failed to use the correct default parser when using the No-Contents header
and libxml2 linked in. [moseley]

=item * Add tests for IRIX and sparc machines

8-byte alignment in mem_zones is is required for these machine [moseley]


=item * Fixed code when removing files

Was not correctly removing words from index when parser aborted [jmruiz]

=item * Merge segfault

Fixed segfault caused by trying to print null dates while merging
duplicate files. [moseley]

=item * Documentation patches

Spelling corrections to the SWISH-CONFIG pod page [Steve Eckert]

=item * Configure corrections

Fixed a zlib test error that used "==" in a test [Steve Eckert]

=item * Updates to VMS build

The VMS build was updated [Jean-François PIÉRONNE]

=item * MANIFEST corrections

Added missing filters and vms build file into MANIFEST [moseley]

=back

=head2 Version 2.2 - September 18, 2002


=over 4

=item * Default parser

Swish-e will now use the HTML2 (libxml2) parser by default if libxml2 is
installed and DefaultContents or IndexContents is not used.

=item * Selecting parsers

Allow HTML*, XML*, and TXT* to automatically select the libxml2-based parsers
if libxml2 is linked with Swish-e, otherwise fallback to the built-in parsers.

=item * SwishSpider and Filters

Filters (FileFilter directive) did not work correctly when spidering
with the -S http method.  A new filter system was developed and now
filtering of documents (e.g. pdf-E<gt>html or MSWord-E<gt>text) is handled
by the src/SwishSpider program.

When indexing with the -S http method only documents of content-type "text/*"
are indexed.  Other documents must be converted to text by using the filter system.

=item * Buffer overflow in xml.c

Fixed bug in xml.c reported by Rodney Barnett when very long words
were indexed. [moseley]

=item * configure script updates

Updated from _WIN32 checks to feature checks using autoconf [moseley, norris]

=item * updates to run on Alpha (Linux 2.4 (Debian 3.0))

Fixed a cast error when calling zlib, and the calls to read/write a packed longs
to disk. [jmruiz, moseley]

=item * COALESCE_BUFFER_MAX_SIZE

Some people were seeing the following error:

    err: Buffer too short in coalesce_word_locations.
    Increase COALESCE_BUFFER_MAX_SIZE in config.h and rebuild.

This was due to indexing binary data or files with very large number of words.
The best solution is to not index binary data or files with a very large number
of words.

Swish-e will now automatically reallocate the buffer as needed.  [jmruiz]


=back

=head2 Version 2.2rc1 - August 29, 2002

Many large changes were made internally in the code, some for performance
reasons, some for feature changes and additions, and some to prepare
for new features in later versions of Swish-e.

=over 4

=item * Documentation!

Documentation is now included in the source distribution as .pod
(perldoc) files, and as HTML files.  In addition, the distribution can now
generate PDF, postscript, and unix man pages from the source .pod files.
See L<README|README> for more information.

=item * Indexing and searching speed

The indexing process has been imporoved.  Depending on a number of
factors, you may see a significant improvement in indexing speed,
especially if upgrading from version 1.x.

Searching speed has also been improved.  Properties are not loaded until
results are displayed, and properties are pre-sorted during indexing to
speed up sorting results by properties while searching.

=item * Properties are written to a sepearte file

Swish-e now stores document properties in a separate file.  This means
there are now two files that make up a Swish-e index.  The default files
are C<index.swish-e> and C<index.swish-e.prop>.

This change frees memory while indexing, allowing larger collections to
be indexed in memory.

=item * Internal data stored as Properties

Pre 2.2 some internal data was stored in fixed locations within the
index, namely the file name, file size, and title.  2.2 introduced new
internal data such as the last modified date, and document summaries.
This data is considered I<meta data> since it is data about a document.

Instead of adding new data to the internal structure of the index file,
it was decided to use the MetaNames and PropertyNames feature of Swish-e
to store this meta information.  This allows for new meta data to be added
at a later time (e.g. Content-type), and provides an easy and customizable
way to print results with the C<-p> switch and the new C<-x> switch.
In addition, search results can now be sorted and limited by properties.

For example, to sort by the rank and title:

    swish-e -w foo -s swishrank desc swishtitle asc


=item * The header display has been slightly reorganized.

If you are parsing output headers in a program then you may need to
adjust your code.  There's a new switch '-H' to control the level of
header output when searching.

=item * Results are now combined when searching more than one index.

Swish-e now merges (and sorts) the results from multiple indexes when
using C<-f> to specify more than one index.  This change effects the way
maxhits (C<-m>) works.  Here's a summary of the way it works for the
different versions.


    1.3.2 - MaxHits returns first N results starting from the first index.
            e.g. maxhits=20; 15 hits Index1, 40 hits Index2
            All 15 from Index1 plus first five from Index2 = 20 hits.

    2.0.0 - MaxHits returns first N results from each index.
            e.g. Maxhits=20; 15 hits Index1, 40 hits Index2
            All 15 from Index1 plus 15 from Index2.

    2.2.0 - Results are merged and first N results are returned.
            e.g. Maxhits=20; 15 hits Index1, 40 hits Index2
            Results are merged from each index and sorted
            (rank is the default sort) and only the first
            20 are returned.


=item * New B<prog> document source indexing method

You can now use -S prog to use an external program to supply documents
to Swish-e.  This external program can be used to spider web servers,
index databases, or to convert any type of document into html, xml,
or text, so it can be indexed by Swish-e.  Examples are given in the
C<prog-bin> directory.

=item * The indexing parser was rewritten to be more logical.

TranslateCharacters now is done before WordCharacters is checked.  For example,

    WordCharacters abcdefghijklmnopqrstuvwxyz
    TranslateCharacters ñ n

Now C<El Niño> will be indexed as El Nino (el and nino), even though C<ñ>
is not listed in WordCharacters.

Previously, stopwords were checked after stemming and soundex conversions,
as well as most of the other word checks (WordCharacters, min/max length
and so on).  This meant that the stopword list probably didn't work as
expected when using stemming.

=item * The search parser was rewritten to be more logical

The search parser was rewritten to correct a number of logic errors.
Swish-e did not differentiate between meta names, Swish-e operators
and search words when parsing the query.  This meant, for example,
that metanames might be broken up by the WordCharacters setting, and
that they could be stemmed.

Swish-e operator characters C<"*()=> can now be searched by escaping
with a backslash.  For example:

    ./swish-e -w 'this\=odd\)word'

will end up searching for the word C<this=odd)word>.  To search for a
backslash character preceed it with a backslash.

Currently, searching for:

    ./swish-e -w 'this\*'

is the same as a wildcard search.  This may be fixed in the future.    

Searching for buzzwords with those characters will still require
backslashing.  This also may change to allow some un-escaped operator
characters, but some will always need to be escaped (e.g. the double-quote
phrase character).

=item * Quotes and Backslash escapes in strings

A bug was fixed in the C<parse_line()> function (in F<string.c>) where
backslashes were not escaping the next character.  C<parse_line()> is used
to parse a string of text into tokens (words).  Normally splitting is done
at whitespace.  You may use quotes (single or double) to define a string
(that might include whitespace) as a single parameter.  The backslash
can also be used to escape the following character when *within* quotes
(e.g. to escape an embedded quote character).

    ReplaceRules append "foo bar"   <- define "foo bar" as a single word
    ReplaceRules append "foo\"bar"  <- escape the quotes
    ReplaceRules append 'foo"bar'   <- same thing


=item * Example C<user.config> file removed.

Previous versions of Swish-e included a configuration file called
C<user.config> which contained examples of all directives.  This has
been replaced by a series of example configuration files located in the
C<conf> directory.  The configuration directives are now described in
L<SWISH-CONFIG|SWISH-CONFIG>.

=item * Ports to Win32 and VMS

David Norris has included the files required to build Swish-e under
Windows.  See C<src/win32>.  A self-extracting Windows version is
available from the Download page of the swish-e.org web site.

Jean-François Piéronne has provided the files required to build Swish-e
under OpenVMS.  See C<src/vms> for more information.

=item * String properties are concatenated

Multiple I<string> properties of the same name in a document are now
concatenated into one property.  A space character is added between
the strings if needed.  A warning will be generated if multiple numeric
or date properties are found in the same document, and the additional
properties will be ignored.

Previously, properties of the same name were added to the index, but
could not be retrieved.

To do: remove the C<next> pointer, and allow user-defined character to
place between properties.

=item * regex type added to ReplaceRules

A more general purpose pattern replacement syntax.


=item * New Parsers

Swish-e's XML parser was replaced with James Clark's expat XML parser
library.

Swish-e can now use Daniel Veillard's libxml2 library for parsing HTML and
XML.  This requires installation of the library before building Swish-e.
See the L<INSTALL|INSTALL> document for information.  libxml2 is not
required, but is strongly recommended for parsing HTML over Swish-e's
internal HTML parser, and provides more features for both HTML and
XML parsing.

=item * Support for zlib

Swish-e can be compiled with zlib.  This is useful for compressing large
properties.  Building Swish-e with zlib is stronly recommended if you
use its C<StoreDescription> feature.

=item * LST type of document no longer supported

LST allowed indexing of files that contained multiple documents.

=item * Temporary files

To improve security Swish-e now uses the C<mkstemp(3)> function to
create temporary files.  Temporary files are used while indexing only.
This may result in some portability issues, but the security issues
were overriding.

(Currently this does not apply to the -S http indexing method.)

C<mkstemp> opens the temporary with O_EXCL|O_CREAT flags.  This prevents
overwriting existing files.  In addition, the name of the file created
is a lot harder to guess by attackers.  The temporary file is created
with only owner permissions.

Please report any portability issues on the Swish-e discussion list.

=item * Temporary file locations

Swish-e now uses the environment variables C<TMPDIR>, C<TMP>, and
C<TEMP> (in that order) to decide where to write temporary files.
The configuration setting of L<TmpDir|SWISH-CONFIG/"item_TmpDir"> will
be used if none of the environment variables are set.  Swish-e uses the
current directory otherwise; there is no default temporary directory.

Since the environment variables override the configuration settings,
a warning will be issued if you set L<TmpDir|SWISH-CONFIG/"item_TmpDir">
in the configuration file and there's also an environment variable set.

Temporary files begin with the letters "swtmp" (which can be changed in
F<config.h>), followed by two or more letters that indicate the type of
temporary file, and some random characters to complete the file name.
If indexing is aborted for some reason you may find these temporary
files left behind.

=item * New Fuzzy indexing method Double Metaphone

Based on Lawrence Philips' Metaphone algorithm, add two
new methods of creating a fuzzy index (in addition to Stemming and Soundex).


=back

Changes to Configuration File Directives.  Please see
L<SWISH-CONFIG|SWISH-CONFIG> for more info.

=over 4

=item * New directives: IndexContents and DefaultContents

The IndexContents directive assigns internal Swish-e document parsers
to files based on their file type.  The DefaultContents directive
assigns a parser to be used on file that are not assigned a parser with
IndexContents.

=item * New directive: UndefinedMetaTags [error|ignore|index|auto]

This describes what to do when a meta tag is found in a document that
is not listed in the MetaNames directive.

=item * New directive: IgnoreTags

Will ignore text with the listed tags.

=item * New directive: SwishProgParameters *list of words*

Passes words listed to the external Swish-e program when running with
C<-S prog> document source method.

=item * New directive: ConvertHTMLEntities [yes|no]

Controls parsing and conversion of HTML entities.

=item * New directive: DontBumpPositionOnMetaTags

The word position is now bumped when a new metatag is found -- this is
to prevent phrases from matching across meta tags.  This directive will
disable this behavior for the listed tags.

This directive works for HTML and XML documents.

=item * Changed directive: IndexComments

This has been changed such that comments are not indexed by default.

=item * Changed directive: IgnoreWords

The builtin list of stopwords has been removed. Use of the SwishDefault
word will generate a warning, and no stop words will be used.  You must
now specify a list of stopwords, or specify a file of stopwords.

A sample file C<stopwords.txt> has been inclu