liblingua-stem-perl 0.84-1 / usr / share / perl5 / Lingua / Stem.pod

=head1 NAME

Lingua::Stem - Stemming of words

=head1 SYNOPSIS

    use Lingua::Stem qw(stem);
    my $stemmmed_words_anon_array   = stem(@words);

    or for the OO inclined,

    use Lingua::Stem;
    my $stemmer = Lingua::Stem->new(-locale => 'EN-UK');
    $stemmer->stem_caching({ -level => 2 });
    my $stemmmed_words_anon_array   = $stemmer->stem(@words);

=head1 DESCRIPTION

This routine applies stemming algorithms to its parameters,
returning the stemmed words as appropriate to the selected
locale.

You can import some or all of the class methods.

use Lingua::Stem qw (stem clear_stem_cache stem_caching
                     add_exceptions delete_exceptions
                     get_exceptions set_locale get_locale
                     :all :locale :exceptions :stem :caching);

 :all        - imports  stem add_exceptions delete_exceptions get_exceptions
               set_locale get_locale
 :stem       - imports  stem
 :caching    - imports  stem_caching clear_stem_cache
 :locale     - imports  set_locale get_locale
 :exceptions - imports  add_exceptions delete_exceptions get_exceptions

Currently supported locales are:

      DA          - Danish
      DE          - German
      EN          - English (also EN-US and EN-UK)
      FR          - French
      GL          - Galician
      IT          - Italian
      NO          - Norwegian
      PT          - Portuguese
      RU          - Russian (also RU-RU and RU-RU.KOI8-R)
      SV          - Swedish

If you have the memory and lots of stemming to do,
I B<strongly> suggest using cache level 2 and processing
lists in 'big chunks' (long lists) for best performance.


=head2 Comparision with Lingua::Stem::Snowball

It functions fairly similarly to the Lingua::Stem::Snowball
suite of stemmers, with the most significant differences being

1) Lingua::Stem is a 'pure perl' (no compiled XS code is needed) suite.
   Lingua::Stem::Snowball is XS based (must be compiled).

2) Lingua::Stem works with Perl 5.6 or later
   Lingua::Stem::Snowball works with Perl 5.8 or later

3) Lingua::Stem has an 'exceptions' system allowing you to override stemming on a 'case by case' basis.
   Lingua::Stem::Snowball does not have an 'exceptions' system.

4) A somewhat different set of supported languages:

 +---------------------------------------------------------------+
 | Language   | ISO code | Lingua::Stem | Lingua::Stem::Snowball |
 |---------------------------------------------------------------|
 | Danish     | da       |      yes     |          yes           |
 | Dutch      | nl       |       no     |          yes           |
 | English    | en       |      yes     |          yes           |
 | Finnish    | fi       |       no     |          yes           |
 | French     | fr       |      yes     |          yes           |
 | Galacian   | gl       |      yes     |           no           |
 | German     | de       |      yes     |          yes           |
 | Italian    | it       |      yes     |          yes           |
 | Norwegian  | no       |      yes     |          yes           |
 | Portuguese | pt       |      yes     |          yes           |
 | Russian    | ru       |      yes     |          yes           |
 | Spanish    | es       |       no     |          yes           |
 | Swedish    | sv       |      yes     |          yes           |
 +---------------------------------------------------------------+

5) Lingua::Stem is faster for 'stem' (circa 30% faster than Lingua::Stem::Snowball)

6) Lingua::Stem::Snowball is faster for 'stem_in_place' (circa 30% faster than Lingua::Stem)

7) Lingua::Stem::Snowball is more consistent with regard to character set issues.

8) Lingua::Stem::Snowball is under active development. Lingua::Stem is currently fairly static.

Some benchmarks using Lingua::Stem 0.82 and Lingua::Stem::Snowball 0.94
gives an idea of how various options impact performance. The dataset was
The Works of Edgar Allen Poe, volumes 1-5 from the Gutenberg Project processed
10 times in a row as single batch of words (processing a long text one word at a time is
very inefficient and drops the performance of Lingua::Stem by about 90%:
So "Don't Do That" ;) )

The benchmarks were run on a 3.06 Ghz P4 with HT on Fedora Core 5 Linux using Perl 5.8.8.

 +------------------------------------------------------------------------+
 | source: collected_works_poe.txt | words: 454691 | unique words: 22802  |
 |------------------------------------------------------------------------|
 | module                          | config        | avg secs | words/sec |
 |------------------------------------------------------------------------|
 | Lingua::Stem 0.82               | no cache      | 1.922    |  236560   |
 | Lingua::Stem 0.82               | cache level 2 | 1.235    |  368292   |
 | Lingua::Stem 0.82               | cachelv2, sip | 0.798    |  569494   |
 | Lingua::Stem::Snowball 0.94     | stem          | 1.622    |  280276   |
 | Lingua::Stem::Snowball 0.94     | stem_in_place | 0.627    |  725129   |
 +------------------------------------------------------------------------+

The script for the benchmark is included in the examples/ directory of this
distribution as benchmark_stemmers.plx.

=head1 CHANGES

 0.84 2010.04.29 - Documentation fixes to the En stemmer and removal
                   of the accidentally included lib/Lingua/test.pl file
                   Thanks goes to Aaron Naiman for bringing the
                   documentation error to my attention and to
                   Alexandr Ciornii and 'kmx' for the pointing out
                   the problem with the test.pl file.

 0.83 2007.06.23 - Disabled Italian locale build tests due to
                   changes in Lingua::Stem::It breaking the tests.

 0.82 2006.07.23 - Added 'stem_in_place' to base package.
                   Tweaks to documentation and build tests.

 0.81 2004.07.26 - Minor documentation tweak. No functional change.

 0.80 2004.07.25 - Added 'RU', 'RU_RU', 'RU_RU.KOI-8' locale.
                   Added support for Lingua::Stem::Ru to
                   Makefile.PL and autoloader.

                   Added documentation stressing use of caching
                   and batches for performance. Added support
                   for '_' as a seperator in the locale strings.
                   Added example benchmark script. Expanded copyright
                   credits.

 0.70 2004.04.26 - Added FR locale and documentation fixes
                   to Lingua::Stem::Gl

 0.61 2003.09.28 - Documentation fixes. No functional changes.

 0.60 2003.04.05 - Added more locales by wrappering various stemming
                   implementations. Documented currently supported
                   list of locales.

 0.50 2000.09.14 - Fixed major implementation error. Starting with
                   version 0.30 I forgot to include rulesets 2,3 and 4
                   for Porter's algorithm. The resulting stemming results
                   were very poor. Thanks go to <csyap@netfision.com>
                   for bringing the problem to my attention.

                   Unfortunately, the fix inherently generates *different*
                   stemming results than 0.30 and 0.40 did. If you
                   need identically broken output - use locale 'en-broken'.

 0.40 2000.08.25 - Added stem caching support as an option. This
                   can provide a large speedup to the operation
                   of the stemmer. Caching is default turned off
                   to maximize compatibility with previous versions.

 0.30 1999.06.24 - Replaced core of 'En' stemmers with code from
                   Jim Richardson <jimr@maths.usyd.edu.au>
                   Aliased 'en-us' and 'en-uk' to 'en'
                   Fixed 'SYNOPSIS' to correct return value
                   type for stemmed words (SYNOPIS error spotted
                   by <Arved_37@chebucto.ns.ca>)

 0.20 1999.06.15 - Changed to '.pm' module, moved into Lingua:: namespace,
                   added OO interface, optionalized the export of routines
                   into the caller's namespace, added named parameter
                   initialization, stemming exceptions, autoloaded
                   locale support and isolated case flattening to
                   localized stemmers prevent i18n problems later.

                   Input and output text are assumed to be in UTF8
                   encoding (no operational impact right now, but
                   will be important when extending the module to
                   non-English).

=head1 METHODS

=over 4

=item new(...);

Returns a new instance of a Lingua::Stem object and, optionally, selection
of the locale to be used for stemming.

Examples:

  # By default the locale is en
  $us_stemmer = Lingua::Stem->new;

  # Turn on the cache
  $us_stemmer->stem_caching({ -level => 2 });

  # Overriding the default for a specific instance
  $uk_stemmer = Lingua::Stem->new({ -locale => 'en-uk' });

  # Overriding the default for a specific instance and changing the default
  $uk_stemmer = Lingua::Stem->new({ -default_locale => 'en-uk' });

=back

=over 4

=item set_locale($locale);

Sets the locale to one of the recognized locales.
locale identifiers are converted to lowercase.

Called as a class method, it changes the default locale for all
subseqently generated object instances.

Called as an instance method, it only changes the locale for
that particular instance.

'croaks' if passed an unknown locale.

Examples:

 # Change default locale
 Lingua::Stem::set_locale('en-uk'); # UK's spellings

 # Change instance locale
 $self->set_locale('en-us');  # US's spellings

=back

=over 4

=item get_locale;

Called as a class method, returns the current default locale.

Example:

 $default_locale = Lingua::Stem::get_locale;

Called as an instance method, returns the locale for the instance

 $instance_locale = $stemmer->get_locale;

=back

=over 4

=item add_exceptions($exceptions_hash_ref);

Exceptions allow overriding the stemming algorithm on a case by case
basis. It is done on an exact match and substitution basis: If a passed
word is identical to the exception it will be replaced by the specified
value. No case adjustments are performed.

Called as a class method, adds exceptions to the default exceptions list
used for subsequently instantations of Lingua::Stem objects.

Example:

 # adding default exceptions
 Lingua::Stem::add_exceptions({ 'emily' => 'emily',
                                'driven' => 'driven',
                            });

Called as an instance method, adds exceptions only to the specific
instance.

 # adding instance exceptions
 $stemmer->add_exceptions({ 'steely' => 'steely' });

The exceptions shortcut the normal stemming - if an exception matches
no further stemming is performed after the substitution.

Adding an exception with the same key value as an already defined
exception replaces the pre-existing exception with the new value.

=back

=over 4

=item delete_exceptions(@exceptions_list);

The mirror of add_exceptions, this allows the _removal_ of exceptions
from either the defaults for the class or from the instance.

 # Deletion of exceptions from class default exceptions
 Lingua::Stem::delete_exceptions('aragorn','frodo','samwise');

 # Deletion of exceptions from instance
 $stemmer->delete_exceptions('smaug','sauron','gollum');

 # Deletion of all class default exceptions
 delete_exceptions;

 # Deletion of all exceptions from instance
 $stemmer->delete_exceptions;

=back

=over 4

=item get_exceptions;

As a class method with no parameters it returns all the default exceptions
as an anonymous hash of 'exception' => 'replace with' pairs.

Example:

 # Returns all class default exceptions
 $exceptions = Lingua::Stem::get_exceptions;

As a class method with parameters, it returns the default exceptions listed
in the parameters as an anonymous hash of 'exception' => 'replace with' pairs.
If a parameter specifies an undefined 'exception', the value is set to undef.

 # Returns class default exceptions for 'emily' and 'george'
 $exceptions = Lingua::Stem::get_exceptions('emily','george');

As an instance method, with no parameters it returns the currently active
exceptions for the instance.

 # Returns all instance exceptions
 $exceptions = $stemmer->get_exceptions;

As an instance method with parameters, it returns the instance exceptions listed
in the parameters as an anonymous hash of 'exception' => 'replace with' pairs.
If a parameter specifies an undefined 'exception', the value is set to undef.

 # Returns instance exceptions for 'lisa' and 'bart'
 $exceptions = $stemmer->get_exceptions('lisa','bart');

=back

=over 4

=item stem(@list);

Called as a class method, it applies the default settings
and stems the list of passed words, returning an anonymous
array with the stemmed words in the same order as the passed
list of words.

Example:

    # Default settings applied
    my $anon_array_of_stemmed_words = Lingua::Stem::stem(@words);

Called as an instance method, it applies the instance's settings
and stems the list of passed words, returning an anonymous
array with the stemmed words in the same order as the passed
list of words.

   # Instance's settings applied
   my $stemmed_words = $stemmer->stem(@words);

The stemmer performs best when handed long lists of words
rather than one word at a time. The cache also provides
a huge speed up if you are processing lots of text.

=back

=over 4

=item stem_in_place(@list);

Stems the passed list of words 'in place'. It returns a reference to the modified list.
This is about 60% faster than the 'stem' method but modifies the original list. This currently
only works for the English locales.

 Example:

  my @words = ( 'a', 'list', 'of', 'words' );
  my $stemmed_list_of_words = stem_in_place(@words);

  # '$stemmed_list_of_words' refers to the @words list
  # after 'stem_in_place' has executed

B<DO NOT> use this method of stemming if you need to keep the original list of words. Its
performance gain derives entirely from the fact it B<does not> make a copy the original list
but instead overwrites the original list.

If you try something like

  my @words_for_stemming = @words;
  my $stemmed_list_of_words = stem_in_place(@words_for_stemming);

thinking you will get a speed boost while keeping the original list, you won't: You wipe out
the speed gain completely with your copying of the original list. You should just use the 'stem'
method instead on the original list of words if you need to keep the original list.

=back

=over 4

=item clear_stem_cache;

Clears the stemming cache for the current locale. Can be called as either
a class method or an instance method.

    $stemmer->clear_stem_cache;

    clear_stem_cache;

=back

=over 4

=item stem_caching ({ -level => 0|1|2 });

Sets stemming cache level for the current locale. Can be called as either
a class method or an instance method.

    $stemmer->stem_caching({ -level => 1 });

    stem_caching({ -level => 1 });

For the sake of maximum compatibility with previous versions,
stem caching is set to '-level => 0' by default.

'-level' definitions

 '0' means 'no caching'. This is the default level.

 '1' means 'cache per run'. This caches stemming results during each
    call to 'stem'.

 '2' means 'cache indefinitely'. This caches stemming results until
    either the process exits or the 'clear_stem_cache' method is called.

stem caching is global to the locale. If you turn on stem caching for one
instance of a locale stemmer, all instances using the same locale will have it
turned on as well.

I B<STRONGLY> suggest turning caching on if you have enough memory and
are processing a lot of data.

=back

=head1 VERSION

 0.84 2008.07.27

=head1 NOTES

It started with the 'Text::Stem' module which has been adapted into
a more general framework and moved into the more
language oriented 'Lingua' namespace and re-organized to support a OOP
interface as well as switch core 'En' locale stemmers.

Version 0.40 added a cache for stemmed words. This can provide up
to a several fold performance improvement.

Organization is such that extending this module to any number
of languages should be direct and simple.

Case flattening is a function of the language, so the 'exceptions'
methods have to be used appropriately to the language. For 'En'
family stemming, use lower case words, only, for exceptions.

=head1 AUTHORS

 Benjamin Franz <snowhare@nihongo.org>
 Jim Richardson  <imr@maths.usyd.edu.au>

=head1 CREDITS

 Jim Richardson             <imr@maths.usyd.edu.au>
 Ulrich Pfeifer             <pfeifer@ls6.informatik.uni-dortmund.de>
 Aldo Calpini               <dada@perl.it>
 xern                       <xern@cpan.org>
 Ask Solem Hoel             <ask@unixmonks.net>
 Dennis Haney               <davh@davh.dk>
 Sébastien Darribere-Pleyt  <sebastien.darribere@lefute.com>
 Aleksandr Guidrevitch      <pillgrim@mail.ru>

=head1 SEE ALSO

 Lingua::Stem::En            Lingua::Stem::En            Lingua::Stem::Da
 Lingua::Stem::De            Lingua::Stem::Gl            Lingua::Stem::No
 Lingua::Stem::Pt            Lingua::Stem::Sv            Lingua::Stem::It
 Lingua::Stem::Fr            Lingua::Stem::Ru            Text::German
 Lingua::PT::Stemmer         Lingua::GL::Stemmer         Lingua::Stem::Snowball::No
 Lingua::Stem::Snowball::Se  Lingua::Stem::Snowball::Da  Lingua::Stem::Snowball::Sv
 Lingua::Stemmer::GL         Lingua::Stem::Snowball

 http://snowball.tartarus.org

=head1 COPYRIGHT

Copyright 1999-2004

Freerun Technologies, Inc (Freerun),
Jim Richardson, University of Sydney <imr@maths.usyd.edu.au>
and Benjamin Franz <snowhare@nihongo.org>. All rights reserved.

Text::German was written and is copyrighted by Ulrich Pfeifer.

Lingua::Stem::Snowball::Da was written and is copyrighted by
Dennis Haney and Ask Solem Hoel.

Lingua::Stem::It was written and is copyrighted by Aldo Calpini.

Lingua::Stem::Snowball::No, Lingua::Stem::Snowball::Se, Lingua::Stem::Snowball::Sv were
written and are copyrighted by Ask Solem Hoel.

Lingua::Stemmer::GL and Lingua::PT::Stemmer were written and are copyrighted by Xern.

Lingua::Stem::Fr was written and is copyrighted by  Aldo Calpini and Sébastien Darribere-Pley.

Lingua::Stem::Ru was written and is copyrighted by Aleksandr Guidrevitch.

This software may be freely copied and distributed under the same
terms and conditions as Perl.

=head1 BUGS

None known.

=head1 TODO

Add more languages. Extend regression tests. Add support for the
Lingua::Stem::Snowball family of stemmers as an alternative core stemming
engine. Extend 'stem_in_place' functionality to non-English stemmers.

=cut