libpar-perl 1.014-1 / usr / share / perl5 / PAR / Tutorial.pod

=head1 NAME

PAR::Tutorial - Cross-Platform Packaging and Deployment with PAR

=head1 SYNOPSIS

This is a tutorial on PAR, first appeared at the 7th Perl Conference.
The HTML version of this tutorial is available online as
L<http://search.cpan.org/perldoc?PAR::Tutorial>

=head1 DESCRIPTION

=head2 On Deploying Perl Applications

 % sshnuke.pl 10.2.2.2 -rootpw="Z1ON0101"
 Perl v5.6.1 required--this is only v5.6.0, stopped at sshnuke.pl line 1.
 BEGIN failed--compilation aborted at sshnuke.pl line 1.

=over 4

=item * Q: "Help! I can't run your program!"

=item * A1: Install Perl & C<perl -MCPAN -e'install(...)'>

=over 4

=item * How do we know which modules are needed?

=item * New versions of CPAN modules may break C<sshnuke.pl>

=back

=item * A2: Install Perl & C<tar zxf my_perllib.tgz>

=over 4

=item * Possibly overwriting existing modules; not cross-platform at all

=back

=item * A3: Use the executable generated by C<perlcc sshnuke.pl>

=over 4

=item * Impossible to debug; C<perlcc> usually does not work anyway

=back

=back



=head2 PAR, the Perl Archive Toolkit

=over 4

=item * Do what JAR (Java Archive) does for Perl

=over 4

=item * Aggregates modules, scripts and other files into a Zip file

=item * Easy to generate, update and extract

=item * Version consistency: solves forward-compatibility problems

=item * Developed by community: C<par@perl.org>

=back

=item * PAR files can be packed into self-contained scripts

=over 4

=item * Automatically scans perl script for dependencies

=item * Bundles all necessary 3rd-party modules with it

=item * Requires only core Perl to run on the target machine

=item * PAR also comes with C<pp>, the Perl Packager:

 % pp -o sshnuke.exe sshnuke.pl	# stand-alone executable!

=back

=back



=head2 Simple Packaging

=over 4

=item * PAR files are just Zip files with modules in it

=item * Any Zip tools can generate them:

 % zip foo.par Hello.pm World.pm	# pack two modules
 % zip -r bar.par lib/		# grab all modules in lib/

=item * To load modules from PAR files:

 use PAR;
 use lib "foo.par";		# the .par part is optional
 use Hello;

=item * This also works:

 use PAR "/home/mylibs/*.par";	# put all of them into @INC
 use Hello;

=back



=head2 PAR Loaders

=over 4

=item * Use C<par.pl> to run files inside a PAR archive:

 % par.pl foo.par		# looks for 'main.pl' by default
 % par.pl foo.par test.pl	# runs script/test.pl in foo.par

=item * Same thing, with the stand-alone C<parl> or C<parl.exe>:

 % parl foo.par			# no perl or PAR.pm needed!
 % parl foo.par test.pl		# ditto

=item * The PAR loader can prepend itself to a PAR file:

=over 4

=item * C<-b> bundles non-core modules needed by C<PAR.pm>:

 % par.pl -b -O./foo.pl foo.par	# self-contained script

=item * C<-B> bundles core modules in addition to C<-b>:

 % parl -B -O./foo.exe foo.par	# self-contained binary

=back

=back



=head2 Dependency Scanning

=over 4

=item * Recursively scan dependencies with C<scandeps.pl>:

 % scandeps.pl sshnuke.pl
 # Legend: [C]ore [X]ternal [S]ubmodule [?]NotOnCPAN
 'Crypt::SSLeay'       => '0', #  X   #
 'Net::HTTP'           => '0', #      #
 'Crypt::SSLeay::X509' => '0', # S    # Crypt::SSLeay
 'Net::HTTP::Methods'  => '0', # S    # Net::HTTP
 'Compress::Zlib'      => '0', #  X   # Net::HTTP::Methods

=item * Scan an one-liner, list all involved files:

 % scandeps.pl -V -e "use Dynaloader;"
 ...
 # auto/DynaLoader/dl_findfile.al [autoload]
 # auto/DynaLoader/extralibs.ld [autoload]
 # auto/File/Glob/Glob.bs [data]
 # auto/File/Glob/Glob.so [shared]
 ...

=back



=head2 Perl Packager: C<pp>

=over 4

=item * Combines scanning, zipping and loader-embedding:

 % pp -o out.exe src.pl		# self-contained .exe
 % out.exe			# runs anywhere on the same OS

=item * Bundle additional modules:

 % pp -o out.exe -M CGI src.pl	# pack CGI + its dependencies, too

=item * Pack one-liners:

 % pp -o out.exe -e 'print "Hi!"'   # turns one-liner into executable
    

=item * Generate PAR files instead of executables:

 % pp -p src.pl			# makes 'source.par'
 % pp -B -p src.pl		# include core modules

=back



=head2 How it works

=over 4

=item * Command-line options are almost identical to C<perlcc>'s

=over 4

=item * Also supports C<gcc>-style long options:

 % pp --gui --verbose --output=out.exe src.pl

=back

=item * Small initial overhead; no runtime overhead

=item * Dependencies are POD-stripped before packing

=item * Loads modules directly into memory on demand

=item * Shared libraries (DLLs) are extracted with File::Temp

=item * Works on Perl 5.6.0 or above

=item * Tested on Win32 (VC++ and MinGW), FreeBSD, NetBSD, Linux, MacOSX, Cygwin, AIX, Solaris, HP-UX, Tru64...

=back



=head2 Aggregating multiple programs

=over 4

=item * A common question:

 > I have used pp to make several standalone applications which work
 > great, the only problem is that for each executable that I make, I am
 > assuming the parl.exe is somehow bundled into the resulting exe.

=item * The obvious workaround:

 You can ship parl.exe by itself, along with .par files built
 by "pp -p", and run those PAR files by associating them to parl.exe.

=item * On platforms that have C<ln>, there is a better solution:

 % pp --output=a.out a.pl b.pl	# two scripts in one!
 % ln a.out b.out		# symlink also works
 % ./a.out			# runs a.pl
 % ./b.out			# runs b.pl

=back



=head2 Cross-platform Packages

=over 4

=item * Of course, there is no cross-platform binary format

=item * Pure-perl PAR packages are cross-platform by default

=over 4

=item * However, XS modules are specific to Perl version and platform

=item * Multiple versions of a XS module can co-exist in a PAR file

=back

=item * Suppose we need C<out.par> on both Win32 and Finix:

 C:\> pp --multiarch --output=out.par src.pl
 ...copy src.pl and out.par to a Finix machine...
 % pp --multiarch --output=out.par src.pl

=item * Now it works on both platforms:

 % parl out.par			# runs src.pl
 % perl -MPAR=out.par -e '...'	# uses modules inside out.par

=back



=head2 The Anatomy of a PAR file

=over 4

=item * Modules can reside in several directories:

 /			# casual packaging only
 /lib/			# standard location
 /arch/			# for creating from blib/ 
 /i386-freebsd/		# i.e. $Config{archname}
 /5.8.0/		# i.e. Perl version number
 /5.8.0/i386-freebsd/	# combination of the two above

=item * Scripts are stored in one of the two locations:

 /			# casual packaging only
 /script/		# standard location

=item * Shared libraries may be architecture- or perl-version-specific:

 /shlib/(5.8.0/)?(i386-freebsd/)?

=item * PAR files may recursively contain other PAR files:
 
 /par/(5.8.0/)?(i386-freebsd/)?

=back



=head2 Special files

=over 4

=item * MANIFEST

=over 4

=item * Index of all files inside PAR

=item * Can be parsed with C<ExtUtils::Manifest>

=back

=item * META.yml

=over 4

=item * Dependency, license, runtime options

=item * Can be parsed with C<YAML>

=back

=item * SIGNATURE

=over 4

=item * OpenPGP-signed digital signature

=item * Can be parsed and verified with C<Module::Signature>

=back

=back



=head2 Advantages over perlcc, PerlApp and Perl2exe

=over 4

=item * This is not meant to be a flame

=over 4

=item * All three maintainers have contributed to PAR directly; I'm grateful

=back

=item * perlcc

=over 4

=item * "The code generated in this way is not guaranteed to work... Use for production purposes is strongly discouraged." (from perldoc perlcc)

=item * I<Guaranteed to not work> is more like it

=back

=item * PerlApp / Perl2exe

=over 4

=item * Expensive: Need to pay for each upgrade

=item * Non-portable: Only available for limited platforms

=item * Proprietary: Cannot extend its features or fix bugs

=item * Obfuscated: Vendor and black-hats can see your code, but you can't

=item * Inflexible: Does not work with existing Perl installations 

=back

=back



=head2 MANIFEST: Best viewed with Mozilla

=over 4

=item * The URL of C<MANIFEST> inside C</home/autrijus/foo.par>:

 jar:file:///home/autrijus/foo.par!/MANIFEST

=item * Open it in a Gecko browser (e.g. Netscape 6+) with Javascript enabled:

=item * No needed to unzip anything; just click on files to view them

=back



=head2 META.yml: Metadata galore

=over 4

=item * Static, machine-readable distribution metadata

=over 4

=item * Supported by C<Module::Build>, C<ExtUtils::MakeMaker>, C<Module::Install>

=back

=item * A typical C<pp>-generated C<META.yml> looks like this:

 build_requires: {}
 conflicts: {}
 dist_name: out.par
 distribution_type: par
 dynamic_config: 0
 generated_by: 'Perl Packager version 0.03'
 license: unknown
 par:
   clean: 0
   signature: ''
   verbatim: 0
   version: 0.68

=item * The C<par:> settings controls its runtime behavior

=back



=head2 SIGNATURE: Signing and verifying packages

=over 4

=item * OpenPGP clear-signed manifest with SHA1 digests

=over 4

=item * Supported by C<Module::Signature>, C<CPANPLUS> and C<Module::Build>

=back

=item * A typical C<SIGNATURE> looks like this:

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1

 SHA1 8a014cd6d0f6775552a01d1e6354a69eb6826046 AUTHORS
 ...
 -----BEGIN PGP SIGNATURE-----
 ...
 -----END PGP SIGNATURE-----

=item * Use C<pp> and C<cpansign> to work with signatures:

 % pp -s -o foo.par bar.pl	# make and sign foo.par from bar.pl
 % cpansign -s foo.par	# sign this PAR file
 % cpansign -v foo.par	# verify this PAR file

=back



=head2 Perl Servlets with Apache::PAR

=over 4

=item * Framework for self-contained Web applications

=over 4

=item * Similar to Java's "Web Application Archive" (WAR) files

=item * Works with mod_perl 1.x or 2.x

=back

=item * A complete web application inside a C<.par> file

=over 4

=item * Apache configuration, static files, Perl modules...

=item * Supports Static, Registry and PerlRun handlers

=item * Can also load all PARs under a directory

=back

=item * One additional special file: C<web.conf>

 Alias /myapp/cgi-perl/ ##PARFILE##/
 <Location /myapp/cgi-perl>
     Options +ExecCGI
     SetHandler perl-script
     PerlHandler Apache::PAR::Registry
 </Location>

=back



=head2 Hon Dah, A-par-che!

=over 4

=item * First, make a C<hondah.par> from an one-liner:

 # use the "web.conf" from the previous slide
 % pp -p -o hondah.par -e 'print "Hon Dah!\n"' \
      --add web.conf
 % chmod a+x hondah.par

=item * Add this to C<httpd.conf>, then restart apache:

 <IfDefine MODPERL2>
 PerlModule Apache2
 </IfDefine>
 PerlAddVar PARInclude /home/autrijus/hondah.par
 PerlModule Apache::PAR

=item * Test it out:

 % GET http://localhost/myapp/cgi-perl/main.pl
 Hon Dah!

=item * Instant one-liner web application that works!

=back



=head2 On-demand library fetching

=over 4

=item * With LWP installed, your can use remote PAR files:

 use PAR;
 use lib 'http://aut.dyndns.org/par/DBI-latest.par';
 use DBI;    # always up to date!

=item * Modules are now cached under C<$ENV{PAR_GLOBAL_TEMP}>

=item * Auto-updates with C<LWP::Simple::mirror>

=over 4

=item * Download only if modified

=item * Safe for offline use after the first time

=item * May use C<SIGNATURE> to prevent DNS-spoofing

=back

=item * Makes large-scale deployment a breeze

=over 4

=item * Upgrades from a central location

=item * No installers needed

=back

=back



=head2 Code Obfuscation

=over 4

=item * Also known as I<source-hiding> techniques

=over 4

=item * It is I<not> encryption

=item * Offered by PerlApp, Perl2Exe, Stunnix...

=back

=item * Usually easy to defeat

=over 4

=item * Take optree dump from memory, feed to C<B::Deparse>

=item * If you just want to stop a casual C<grep>, "deflate" already works

=back

=item * PAR now supports pluggable I<input filters> with C<pp -f>

=over 4

=item * Bundled examples: Bleach, PodStrip and PatchContent

=item * True encryption using C<Crypt::*>

=item * Or even _product activation_ over the internet

=back

=item * Alternatively, just keep core logic in your server and use RPC

=back



=head2 Accessing packed files

=over 4

=item * To get the host archive from a packed program:

 my $zip = PAR::par_handle($0);	# an Archive::Zip object
 my $content = $zip->contents('MANIFEST');

=item * Same thing, but with C<read_file()>:

 my $content = PAR::read_file('MANIFEST');

=item * Loaded PAR files are stored in C<%PAR::LibCache>:

 use PAR '/home/mylibs/*.par';
 while (my ($filename, $zip) = each %PAR::LibCache) {
     print "[$filename - MANIFEST]\n";
     print $zip->contents('MANIFEST');
 }

=back



=head2 Packing GUI applications

=over 4

=item * GUI toolkits often need to link with shared libraries:

 # search for libncurses under library paths and pack it
 % pp -l ncurses curses_app.pl	# same for Tk, Wx, Gtk, Qt...

=item * Use C<pp --gui> on Win32 to eliminate the console window:

 # pack 'src.pl' into a console-less 'out.exe' (Win32 only)
 % pp --gui -o out.exe src.pl

=item * "Can't locate Foo/Widget/Bar.pm in @INC"?

=over 4

=item * Some toolkits (notably Tk) autoloads modules without C<use> or C<require>

=item * Hence C<pp> and C<Module::ScanDeps> may fail to detect them

=item * Tk problems mostly fixed by now, but other toolkits may still break

=item * You can work around it with C<pp -M> or an explicit C<require>

=item * Or better, send a short test-case to C<par@perl.org> so we can fix it

=back

=back



=head2 Precompiled CPAN distributions

=over 4

=item * Installing XS extensions from CPAN was difficult

=over 4

=item * Some platforms do not come with a compiler (Win32, MacOSX...)

=item * Some headers or libraries may be missing

=item * PAR.pm itself used to suffer from both problems

=back

=item * ...but not anymore -- C<Module::Install> to the rescue!

 # same old Makefile.PL, with a few changes
 use inc::Module::Install;	# was "use ExtUtils::MakeMaker;"
 WriteMakefile( ... );		# same as the original
 check_nmake();			# make sure the user have nmake
 par_base('AUTRIJUS');		# your CPAN ID or a URL
 fetch_par() unless can_cc();	# use precompiled PAR only if necessary

=item * Users will not notice anything, except now it works

=over 4

=item * Of course, you still need to type C<make par> and upload the precompiled package

=item * PAR users can also install it directly with C<parl -i>

=back

=back



=head2 Platform-specific Tips

=over 4

=item * Win32 and other icon-savvy platforms

=over 4

=item * Needs 3rd-party tools to add icons to C<pp>-generated executables

=item * PE Header manipulation in Perl -- volunteers wanted!

=back

=item * Linux and other libc-based platforms

=over 4

=item * Try to avoid running C<pp> on a bleeding-edge version of the OS

=item * Older versions with an earlier libc won't work with new ones

=back

=item * Solaris and other zlib-lacking platforms (but not Win32)

=over 4

=item * You need a static-linked C<Compress::Zlib> before installing PAR

=item * In the future, PAR may depend on C<Compress::Zlib::Static> instead

=back

=item * Any platform with limited bandwidth or disk space

=over 4

=item * Use UPX to minimize the executable size

=back

=back



=head2 Thank you!

=over 4

=item * Additional resources

=over 4

=item * Mailing list: C<par@perl.org>

=item * Subscribe: Send a blank email to C<par-subscribe@perl.org>

=item * List archive: L<http://nntp.x.perl.org/group/perl.par>

=item * PAR::Intro: L<http://search.cpan.org/dist/PAR/lib/PAR/Intro.pod>

=item * Apache::PAR: L<http://search.cpan.org/dist/Apache-PAR/>

=item * Module::Install: L<http://search.cpan.org/dist/Module-Install/>

=back

=item * Any questions?

=back



=head2 Bonus Slides: PAR Internals



=head2 Overview of PAR.pm's Implementation

=over 4

=item * Here begins the scary part

=over 4

=item * Grues, Dragons and Jabberwocks abound...

=item * You are going to learn weird things about Perl internals

=back

=item * PAR invokes four areas of Perl arcana:

=over 4

=item * @INC code references

=item * On-the-fly source filtering

=item * Overriding C<DynaLoader::bootstrap()> to handle XS modules

=item * Making self-bootstrapping binary executables

=back

=item * The first two only works on 5.6 or later

=over 4

=item * DynaLoader and C<%INC> are there since Perl 5 was born

=item * PAR currently needs 5.6, but a 5.005 port is possible

=back

=back



=head2 Code References in @INC

=over 4

=item * On 1999-07-19, Ken Fox submitted a patch to P5P

=over 4

=item * To _enable using remote modules_ by putting hooks in @INC

=item * It's accepted to come in Perl 5.6, but undocumented until 5.8

=item * Type C<perldoc -f require> to read the nitty-gritty details

=back

=item * Coderefs in @INC may return a fh, or undef to 'pass':

 push @INC, sub {
     my ($coderef, $filename) = @_;  # $coderef is \&my_sub
     open my $fh, "wget ftp://example.com/$filename |";
     return $fh;	# using remote modules, indeed!
 };

=item * Perl 5.8 let you open a file handle to a string, so we just use that:

        open my $fh, '<', \($zip->memberNamed($filename)->contents);
        return $fh;

=item * But Perl 5.6 does not have that, and I don't want to use temp files...

=back



=head2 Source Filtering without Filter::* Modules

=over 4

=item * ... Undocumented features to the rescue!

=over 4

=item * It turns out that @INC hooks can return B<two> values

=item * The first is still the file handle

=item * The second is a code reference for line-by-line source filtering!

=back

=item * This is how C<Acme::use::strict::with::pride> works:

 # Force all modules used to use strict and warnings
 open my $fh, "<", $filename or return;
 my @lines = ("use strict; use warnings;\n", "#line 1 \"$full\"\n");
 return ($fh, sub {
     return 0 unless @lines;	
     push @lines, $_; $_ = shift @lines; return length $_;
 });

=back



=head2  Source Filtering without Filter::* Modules (cont.)

=over 4

=item * But we don't really have a filehandle for anything

=item * Another undocumented feature saves the day!

=item * We can actually omit the first return value altogether:

 # Return all contents line-by-line from the file inside PAR
 my @lines = split(
     /(?<=\n)/,
     $zip->memberNamed($filename)->contents
 );
 return (sub {
     $_ = shift(@lines);
     return length $_;
 });

=back



=head2 Overriding DynaLoader::bootstrap

=over 4

=item * XS modules have dynamically loaded libraries

=over 4

=item * They cannot be loaded as part of a zip file, so we extract them out

=item * Must intercept DynaLoader's library-finding process

=back

=item * Module names are passed to C<bootstrap> for XS loading

=over 4

=item * During the process, it calls C<dl_findfile> to locate the file

=item * So we install pre-hooks around both functions

=back

=item * Our C<_bootstrap> just checks if the library is in PARs

=over 4

=item * If yes, extract it to a C<File::Temp> temp file

=over 4

=item * The file will be automatically cleaned up when the program ends

=back

=item * It then pass the arguments to the original C<bootstrap>

=item * Finally, our C<dl_findfile> intercepts known filenames and return it

=back

=back



=head2 Anatomy of a Self-Contained PAR executable

=over 4

=item * The par script ($0) itself

=over 4

=item * May be in plain-text or native executable format

=back

=item * Any number of embedded files

=over 4

=item * Typically used to bootstrap PAR's various dependencies

=item * Each section begins with the magic string "FILE"

=item * Length of filename in pack('N') format and the filename (auto/.../)

=item * File length in pack('N') and the file's content (not compressed)

=back

=item * One PAR file

=over 4

=item * Just a regular zip file with the magic string C<"PK\003\004">

=back

=item * Ending section

=over 4

=item * A pack('N') number of the total length of FILE and PAR sections

=item * Finally, there must be a 8-bytes magic string: C<"\012PAR.pm\012">

=back

=back



=head2 Self-Bootstrapping Tricks

=over 4

=item * All we can expect is a working perl interpreter

=over 4

=item * The self-contained script *must not* use any modules at all

=item * But to process PAR files, we need XS modules like Compress::Zlib

=back

=item * Answer: bundle all modules + libraries used by PAR.pm

=over 4

=item * That's what the C<FILE> section in the previous slide is for

=item * Load modules to memory, and write object files to disk

=item * Then use a local C<@INC> hook to load them on demand

=back

=item * Minimizing the amount of temporary files

=over 4

=item * First, try to load PerlIO::scalar and File::Temp

=item * Set up an END hook to unlink all temp files up to this point

=item * Load other bundled files, and look in the compressed PAR section

=item * This can be much easier with a pure-perl C<inflate()>; patches welcome!

=back

=back



=head2 Thank you (again)!

=over 4

=item * Any questions, I<please>?

=back


=cut


=head1 SEE ALSO

L<PAR>, L<pp>, L<par.pl>, L<parl>

L<ex::lib::zip>, L<Acme::use::strict::with::pride>

L<App::Packer>, L<Apache::PAR>, L<CPANPLUS>, L<Module::Install>

=head1 AUTHORS

Audrey Tang E<lt>cpan@audreyt.orgE<gt>

You can write
to the mailing list at E<lt>par@perl.orgE<gt>, or send an empty mail to
E<lt>par-subscribe@perl.orgE<gt> to participate in the discussion.

Please submit bug reports to E<lt>bug-par@rt.cpan.orgE<gt>.

=head1 COPYRIGHT

Copyright 2003, 2004, 2005, 2006 by Audrey Tang E<lt>cpan@audreyt.orgE<gt>.

This document is free documentation; you can redistribute it and/or
modify it under the same terms as Perl itself.

See F<LICENSE>.

=cut