Getting Started with XS

Chapter 9 - Writing C Modules with XS
by?Sam Tregar?
Apress ? 2002
Companion Web Site

The same tool used in Chapter 4, I created a Perl-only module using the following h2xs line:

 h2xs -XA -n Data::Counter 

To create an XS module, I just need to drop the -X option. I'll still keep the -A option to remove support for the Autoloader. The -n option naming the module is always required. To create the skeleton for Gnome::MIME, you use the following:

 h2xs -A -n Gnome::MIME 

This creates all of the same files examined in Table 9-1 for a listing of files created). For the most part, the files are the same as those created for a Perl-only module, with the exception of the module file, MIME.pm, and Makefile.PL.

Table 9-1: Files Generated by h2xs -X -n Gnome::MIME

File	Description
MIME.pm	The module file itself, which contains Perl code and POD documentation
MIME.xs	The XS file itself, which contains XS and C code
Makefile.PL	A script that uses ExtUtils::MakeMaker to generate a Makefile
test.pl	A test script run when a user types "make test" or installs your module with the CPAN module
README	A quick description of your module and how to install it
Changes	A change log in which you can describe differences between versions of your module
MANIFEST	A list of all the files in your distribution

MIME.pm-The Module File

The module file generated for an XS module is mostly the same as that for a Perl-only module with a few additions. First, a new require is specified:

 require DynaLoader; 

This line pulls in the DynaLoader module that allows Perl modules to load shared libraries. Next, DynaLoader is added to the @ISA inheritance array:

 our @ISA = qw(Exporter DynaLoader); 

Inheriting from DynaLoader is the standard way to allow your module to be partially implemented in XS. Later in the file, the bootstrap() subroutine from DynaLoader is called:

 bootstrap Gnome::MIME $VERSION; 

This call to DynaLoader::bootstrap() finds and loads a shared library corresponding to Gnome::MIME. By passing $VERSION as the second argument, DynaLoader will check that the loaded shared library matches this version of the Perl half of the module. This is not a required parameter, but h2xs defaults to including it since it helps prevent a very common class of errors.

Listing 9-1 shows a module file generated for an XS module.

Listing 9-1: MIME.pm Generated by h2xs -A -n Gnome::MIME

 package Gnome::MIME; use 5.006; use strict; use warnings; require Exporter; require DynaLoader; our @ISA = qw(Exporter DynaLoader); # Items to export into callers namespace by default. Note: do not export # names by default without a very good reason. Use EXPORT_OK instead. # Do not simply export all your public functions/methods/constants. # This allows declaration       use Gnome::MIME ':all'; # If you do not need this, moving things directly into @EXPORT or @EXPORT_OK # will save memory. our %EXPORT_TAGS = ( 'all' => [ qw( ) ] ); our @EXPORT_OK = ( @{ $EXPORT_TAGS{'all'} } ); our @EXPORT = qw( ); our $VERSION = '0.01'; bootstrap Gnome::MIME $VERSION; # Preloaded methods go here. 1; __END__ # Below is stub documentation for your module. You better edit it! =head1 NAME Gnome::MIME - Perl extension for blah blah blah =head1 SYNOPSIS  use Gnome::MIME;  blah blah blah =head1 DESCRIPTION Stub documentation for Gnome::MIME, created by h2xs. It looks like the author of the extension was negligent enough to leave the stub unedited. Blah blah blah. =head2 EXPORT None by default. =head1 AUTHOR A. U. Thor, E<lt>[e-mail]<gt> =head1 SEE ALSO L<perl>. =cut 

Aside from the preceding changes, the module file is identical to the normal Perl-only module file generated by h2xs -X. This similarity is more than skin-deep-XS modules often contain a significant portion of their code in Perl, resorting to C only when necessary. I'll demonstrate this style later in this chapter.

Makefile.PL-The Makefile Generator

Like the module file, the Makefile.PL generated for XS modules (see Listing 9-2) is the same as the Makefile.PL in Chapter 4, with a few additional lines.

Listing 9-2: Makefile.PL Generated by h2xs -A -n Gnome::MIME

 use ExtUtils::MakeMaker; # See lib/ExtUtils/MakeMaker.pm for details of how to influence # the contents of the Makefile that is written. WriteMakefile(     'NAME'            => 'Gnome::MIME',     'VERSION_FROM'    => 'MIME.pm', # finds $VERSION     'PREREQ_PM'       => {}, # e.g., Module::Name => 1.1     ($] >= 5.005 ?    ## Add these new keywords supported since 5.005        (ABSTRACT_FROM => 'MIME.pm', # retrieve abstract from module         AUTHOR        => 'A. U. Thor <[e-mail]>') : ()),     'LIBS'            => ["], # e.g., '-lm'     'DEFINE'          => ", # e.g., '-DHAVE_SOMETHING'     # Insert -I. if you add *.h files later:     'INC'             => ", # e.g., '-I/usr/include/other'     # Un-comment this if you add C files to link with later:     # 'OBJECT'        => '$(O_FILES)', # link all the C files too ); 

The added section starts with a new key to include additional libraries to link to:

 'LIBS' => ["], # e.g., '-lm' 

This configuration variable is actually a little more complicated than it seems. You might be tempted to fill it with a list of libraries to link to:

 'LIBS'       => ["-lm", "-lfoo", "-lbar"], # ERROR: only links to -lm 

However, the list assigned to LIBS is actually a list of complete library sets. The first one that "works" on the target platform will be used. This allows you to specify sets that work on different operating systems without having to actually write code to do the testing. If you need to link to three libraries, then all three must be in a single string:

 'LIBS'       => ["-lm -lfoo -lbar"], # link to 3 libraries 

The next new line allows you to add a -D command-line option to the compilation of your module's C code:

 'DEFINE'       => ", # e.g., '-DHAVE_SOMETHING' 

Next is a line to add -I include directories:

 'INC'       => ", # e.g., '-I/usr/include/other' 

and a line to include object files in the build explicitly:

 # Un-comment this if you add C files to link with later: # 'OBJECT'        => '$(O_FILES)', # link all the C files too 

This line is commented out because using it requires you to explicitly list all the object files to be compiled. For example, the functional equivalent of not specifying an OBJECT key is

 'OBJECT' => 'MIME.o' 

or, using the Makefile variables:

 'OBJECT' => '$(O_FILES)' 

MIME.xs-The XS Source File

The new file generated by h2xs is MIME.xs (see Listing 9-3). This is the source file for the XS half of the module.

Listing 9-3: MIME.xs Generated by h2xs -A -n Gnome::MIME

 #include "EXTERN.h" #include "perl.h" #include "XSUB.h" MODULE = Gnome::MIME       PACKAGE = Gnome::MIME 

The MIME.xs file consists of two parts. The first part is a series of #include directives:

 #include "EXTERN.h" #include "perl.h" #include "XSUB.h" 

These are passed straight through to the generated C source by xsubpp. In fact, xsubpp passes through all text until it hits a line that starts with a MODULE directive:

 MODULE = Gnome::MIME       PACKAGE = Gnome::MIME 

After this line, everything must be valid XS code, none of which is generated by h2xs.^[2]

Modifying Makefile.PL

The Makefile.PL generated by h2xs works right away. You can see the XS build process without writing any XS code:

 $ perl Makefile.PL && make Checking if your kit is complete... Looks good Writing Makefile for Gnome::MIME 

As you can see, the Makefile gets built as it normally would. Then the MIME.pm file is copied into blib as usual:

 cp MIME.pm blib/lib/Gnome/MIME.pm 

The first XS-specific step happens next when xsubpp is run on MIME.xs to produce MIME.xs, which is renamed MIME.c:

 /usr/local/bin/perl -I/usr/local/lib/perl5/5.6.1/i686-linux              \  -I/usr/local/lib/perl5/5.6.1 /usr/local/lib/perl5/5.6.1/ExtUtils/xsubpp \  -typemap /usr/local/lib/perl5/5.6.1/ExtUtils/typemap MIME.xs > MIME.xsc \  && mv MIME.xsc MIME.c 

Next MIME.c is compiled into MIME.o by the cc compiler (actually gcc in this case):

 cc -c -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE     \   -D_FILE_OFFSET_BITS=64 -O2   -DVERSION=\"0.01\" -DXS_VERSION=\"0.01\" \   -fpic -I/usr/local/lib/perl5/5.6.1/i686-linux/CORE MIME.c 

After that a bootstrap file is created-MIME.bs. Depending on your platform, the bootstrap file might contain some Perl code used by DynaLoader to load the shared library. You can mostly ignore this as it is not something you should modify.

 Running Mkbootstrap for Gnome::MIME () chmod 644 MIME.bs 

Finally the shared library, MIME.so, is linked using cc again and copied into blib:

 rm -f blib/arch/auto/Gnome/MIME/MIME.so LD_RUN_PATH="" cc  -shared -L/usr/local/lib MIME.o -o                   \   blib/arch/auto/Gnome/MIME/MIME.so chmod 755 blib/arch/auto/Gnome/MIME/MIME.so cp MIME.bs blib/arch/auto/Gnome/MIME/MIME.bs chmod 644 blib/arch/auto/Gnome/MIME/MIME.bs 

Like most XS modules, Gnome::MIME needs to make an addition to Makefile.PL to allow the module to link with extra libraries and find its header files. In this case this information is provided by Gnome itself, via the gnome-config program. First, I'll add a block to check for gnome-config and make sure the Gnome version falls within the supported range:

 # check to make sure we have Gnome 1.2, 1.3 or 1.4 and can use gnome-config my $version = 'gnome-config --version gnome'; unless ($version and $version =~ /1.[234]/) {   print <<END; ###################################################################### Gnome 1.[234].x not found. Please make sure you have Gnome installed and that gnome-config is in your path. Then re-run "perl Makefile.PL". You can find more information about Gnome at http://gnome.org ###################################################################### END  exit 1; } 

Next I'll modify the call to WriteMakefile() to use gnome-config to get the correct LIBS and INC settings:

 WriteMakefile(              NAME          => "Gnome::MIME",              VERSION_FROM  => "MIME.pm",              PREREQ_PM     => {},              ABSTRACT_FROM => "MIME.pm",              AUTHOR        => 'Sam Tregar <[e-mail]>',              LIBS          => ['gnome-config --libs gnome'],              INC           => 'gnome-config --cflags gnome',             ); 

A rebuild shows the effect in the compilation line:

 cc -c -I/usr/include -DNEED_GNOMESUPPORT_H -I/usr/lib/gnome-libs/include    \   -I/usr/include/gtk-1.2 -I/usr/include/glib-1.2 -I/usr/lib/glib/include    \   -I/usr/X11R6/include -fno-strict-aliasing -I/usr/local/include            \   -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2   -DVERSION=\"0.01\"       \   -DXS_VERSION=\"0.01\" -fpic -I/usr/local/lib/perl5/5.6.1/i686-linux/CORE  \   MIME.c 

and the link line:

 LD_RU_PATH="/usr/lib" cc -shared -L/usr/local/lib MIME.o                   \   -o blib/arch/auto/Gnome/MIME/MIME.so   -L/usr/lib -lgnome -lgnomesupport \   -lesd -laudiofile -lm -ldb1 -lglib -ldl 

Although the preceding code is specific to the needs of Gnome::MIME, it demonstrates a general technique for XS modules that link to external libraries. You'll typically need to add some custom Perl code to your Makefile.PL to check that the library you need exists and has the right version. Then you'll need to figure out what to set LIBS and INC to so that your module will compile and link successfully. Sometimes this will be a static string, but it's becoming more and more common for large projects to provide a binary like gnome-config that makes determining these variables easier.

A First XSUB

The fundamental unit of an XS module is an XSUB. An XSUB is simply a definition of a single C function for which the xsubpp compiler will produce a Perl subroutine. The first XSUB I'll provide for Gnome::MIME is a wrapper for the gnome_mime_type() function. This function takes a single parameter, a filename, and returns its MIME type based on an examination of the filename alone. The C signature for gnome_mime_type() is as follows:

 const char * gnome_mime_type(const gchar *filename); 

Here is an XSUB that provides an interface to this function:

 char * gnome_mime_type(filename)      char *filename 

See Listing 9-4 for the new MIME.xs.

Listing 9-4: MIME.xs with First XSUB Added

 #include "EXTERN.h" #include "perl.h" #include "XSUB.h" MODULE = Gnome::MIME PACKAGE = Gnome::MIME char * gnome_mime_type(filename)      char * filename 

After adding this XSUB to the end of MIME.xs, I'll edit my test.pl to read as follows:

 use Test::More 'no_plan'; BEGIN { use_ok('Gnome::MIME'); } # test some simple mime-type recognitions is(Gnome::MIME::gnome_mime_type("foo.gif"),  'image/gif',  "recognizes .gif"); is(Gnome::MIME::gnome_mime_type("foo.jpg"),  'image/jpeg', "recognizes .jpg"); is(Gnome::MIME::gnome_mime_type("foo.html"), 'text/html',  "recognizes .html"); 

Now a normal perl Makefile.PL, make, and make test run will build and test the new module.

XSUB Anatomy

As you can see in the preceding section, an XSUB resembles a C function signature in K&R format. The first line contains the return type of the function. Then comes a line containing the function name and a list of parameters. Finally, a line per parameter specifies the type of each parameter. Here's an annotated version of the previous example (note that XS doesn't actually allow comments inline with XSUBs, so this won't compile):

 char *                    # returns a string gnome_mime_type(filename) # takes a single parameter named filename      char *filename       # filename is a string 

This is the simplest possible type of XSUB-it maps a C function directly to a Perl function with the same parameters and return type. The xsubpp compiler takes this definition and produces C code in the generated MIME.c (see Listing 9-5). This C code makes use of functions in the Perl API introduced in the previous chapter. For example, to translate the incoming SV into a char *, MIME.c contains this line:

 char *    filename = (char *)SvPV(ST(0),PL_na); 

The ST macro provides access to the argument array of an XSUB, but the rest of the call should be familiar.

Listing 9-5: C Function Generated by xsubpp for gnome_mime_type() XSUB

 XS(XS_Gnome__MIME_gnome_mime_type) {    dXSARGS;    if (items != 1)               Perl_croak(aTHX_ "Usage: Gnome::MIME::gnome_mime_type(filename)");    {               char *       filename = (char *)SvPV(ST(0),PL_na);               char *       RETVAL;               dXSTARG;               RETVAL = gnome_mime_type(filename);      sv_setpv(TARG, RETVAL); XSprePUSH; PUSHTARG;    }    XSRETURN(1); } 

^[2]Actually, there is a way to get h2xs to generate some XS code for you. If you install the C::Scan module from CPAN, then you can use the -x option to attempt to auto-generate XS from a C header file. At present this process only succeeds with very simple header files. See the h2xs documentation for more details.