Planning the API | Professional Web APIs with PHP. eBay, Google, PayPal, Amazon, FedEx, Plus Web Feeds

You have doubtlessly heard the old saying "Fail to plan, and you plan to fail." This is most definitely true in this case. Several critical decisions need to be made before development is begun because, after all, as soon as you put your API out there for others to use, it becomes both difficult for you to implement the changes and troublesome to developers utilizing your API that may be affected by your changes.

Security

There are several options when it comes to the level of security (or lack of security) protecting your API. As is usually the case, additional layers of security introduce additional barriers to use, so a balance must be sought between securing the API, while still making it as easy as possible for the target development community to use the API. Keep in mind that for our purposes, security is for both protecting the data on your server as well as possibly protecting the integrity of the developers themselves by enforcing developer registration (and utilizing a developer "token" for web services). Although this discussion doesn't get deep into code, I'll try to highlight some of the important considerations.

An Open API

Under an open API, absolutely no security or authentication methods are used. A query is received from the wild, and the system makes its best effort to respond to it appropriately. This has several advantages and disadvantages.

Advantages:

Absolute minimum barrier to use — By not using encryption or special authentication methods, anyone with access to the Internet should be able to begin working with your API quickly.
Easily distributed code — Login accounts or developer key programs that make use of your API can be widely distributed and used right out of the box.
Less to worry about — If you aren't managing user accounts or development keys, it's one less thing to keep track of, and your code efforts can concentrate solely on developing the API itself.

Disadvantages:

No control — Anyone, anywhere, can use the API, and while this may sound like the goal of web services, it drastically limits your response if abuse requests begin pouring in. If those requests are coming from an application on a single machine, it is easy enough to recognize the requests and block them at the firewall. But should an application that behaves poorly reach wide distribution, you will have a very difficult time dealing with the requests.
No encryption — All requests and responses are visible to anyone between the requesting server and the API server.
Can't contact developers — Because anyone anywhere can access the API without any prior registration, you are left without any method of directly engaging developers using the API. You may want to contact developers in situations where their application is being abusive, when changes are being made to the API that will affect their application, or to seek suggestions on how to improve the API itself.
Abuse — Unfortunately today, systems with little or no security or authentication make prime targets for abuse by some less ethical elements out there. Even if you feel that the risk is minimal, you may end up surprised at what others can take advantage of.

With those elements in mind, the only situations in which a completely open API would be appropriate are ones where the API is used only to request information, never to publish it, and where the information being requested is generated (or cached) in a very CPU nonintensive manner. A perfect example would be the National Weather Service API presented in Chapter 11. It only accepts requests for information, and those requests are easily cached for a full hour on the server. In cases where the API allows information to be pushed to it, some authentication is required to determine who is pushing the information, and when requests can be CPU-intensive, the remote application needs to be identifiable so incoming requests can be throttled if necessary.

HTTP Authentication

Authentication can be passed in the HTTP headers of incoming requests. This is the same type of authentication that is used when your browser creates a small login window when attempting to access a site. The authentication information is Base 64-encoded, so it does look like it is encrypted when transmitted over the wire, but in reality it is not. This encoding only ensures that all characters are valid to be passed in the header and is not intended to provide any level of security.

Advantages:

Easily handled — Because the authentication information is sent in the HTTP headers, it can be handled by some moderately complex routers or gateways. This will allow for hardware-level throttling of abusive clients, or routing based on specific users. On the application side, the authentication will actually be handled by your web server, not your application. Web servers are developed and tested with high performance in mind, so this will likely end up being faster than any attempt to handle authentication in the application itself.
Transparent — Because the web server is handling the authentication, you may choose to completely ignore what user is logged in, and concentrate solely on handling the request. This is obviously only applicable when requests are user-agnostic (every user receives the same response to the same query).
Easy to code — Adding an additional HTTP header is relatively easy in most programming languages. It is also pretty universally available even in shared hosting situations (which may prevent things like SSL requests or external libraries).

Disadvantages:

Authentication is sent in the clear — Base 64 is a two-way algorithm. Anyone who intercepts the request can determine the username and password being used, but they don't even need to; they can just use an identical header themselves.
Username restriction — When using HTTP authentication, the colon (:) cannot be used in the username. A minor restriction, but one to keep in mind.
No encryption — All requests and responses are visible to anyone between the requesting server and the API server.
Slight barrier to use — Those unfamiliar with this method of authentication may shy away from attempting it.

This basic level of authentication is sufficient for many API applications. The presence of some basic authentication allows the API to either be client-aware or client-agnostic, depending on its specific needs, and also allows for throttling or denial to abusive clients. It would be a good idea with this type of authentication to provide some separation between the username and password combo used for the API and the site at large. This way, should the API's authentication information be compromised (by someone with access to the code, or by grabbing it off the wire), the valid user can use their regular information to change the API's credentials.

Server-Side Code

Most of the work is going to be done by the web server, which will handle the authentication from the user. Apache can use a flat file to look up user accounts, but this isn't very practical for an API with a changing user base. Apache can use a Berkeley database instead of a flat file if you use the mod_db or mod_dbm modules. BerkeleyDB is standard under many Linux distributions; under Windows or in situations where it isn't installed, the package can be obtained from www.sleepycat.com. In order to use BerkeleyDB, PHP needs to be configured with the –with–db4 option, and for Apache to support it for authentication, it must be compiled with the --enable-module=auth_db option.

Configuration (either in httpdconf or .htaccess) should look something like this:

 <Directory /www/domains/api.example.com >   AuthName "API Requires Registration"   AuthType Basic   AuthDBUserFile /www/basicAuth/api.example.com/passwords.dat   require valid-user  </Directory>

The Directory parameter sets the directory on the file system that will be protected by the basic authentication. AuthName sets the message that would be shown to users if they attempted to access the page via a browser. AuthType is basic, the appropriate entry for this type of authentication. AuthDBUserFile sets the path to the Berkeley database that users will be authenticated from; note that it is outside the document root — don't be silly and let attackers download your database. Finally, the require statement indicates that users attempting to access the directory must be present in the database.

All of that is useless if users can't be added to the database as required — a feat accomplished by this function:

 function createUser($username, $password) {   $chars = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";   $r1 = rand(1, strlen($chars) - 1);   $r2 = rand(1, strlen($chars) - 1);   $salt = substr($chars, $r1, 1) . substr($chars, $r2, 1);   $saltedPassword = crypt($password, $salt);   $resource = dba_open("/www/basicAuth/api.example.com/passwords.dat", "c", "db4");   if (dba_insert($username, $saltedPassword, $resource))   {     dba_close($resource);     return true;   }else   {     dba_close($resource);     return false;   } }

The essential parts of this function are the opening of the database with the dba_open() function (using the c parameter indicates that the database should be created if it isn't found, and db4 is the database type), inserting the username and encrypted password, and then closing the database connection.

Note

Randomly generated salt is sent to the crypt function with the password to generate an encrypted password resistant to dictionary attacks. Passwords can still be looked up because the crypt function prepends the salt to the encrypted password.

Removing a user from the database is similarly easy:

 function deleteUser($user) {   $resource = dba_open("/www/basicAuth/api.example.com/passwords.dat", "c", "db4");   dba_delete($user, $resource);   dba_close($resource); }

Finally, should you wish to access the user's information from the API script itself, use this:

 $username = $_SERVER['PHP_AUTH_USER']; $password = $_SERVER['PHP_AUTH_PW'];

Both variables are stored in the $_SERVER global array; the password variable is the clear text version of the password. Remember that the user has already been authenticated by the web server by the time the script is executed, so this information is really only going to be used for recordkeeping purposes.

Client-Side Code

In order to access the API, the client application will need to send an additional HTTP header containing their username and password encoded with Base 64.

 $authorization = base64_encode("username:password");

First the username and password must be encoded with the built-in function. The username-colon-password format is specified by the HTTP specification.

 fputs($fp, "Authorization: Basic $authorization\r\n");

During the socket connection, the Authorization header should be sent along with the regular HTTP headers.

Message-Based Authentication

Client credentials can also be passed along with the regular message payload. This is marginally easier to implement on the client side because adding credentials should be no more difficult than adding another parameter to the request. Remember that even if a secure (SSL) endpoint is used, the URL used for the request is still sent in the clear, so if the credentials are passed on the URL (as is the case with a REST request), they will be visible to any and all intermediaries.

Advantages:

Easily handled — Authentication should be checked before any other processing, just like a regular page.
Easy to code — Programmers who wish to access the API need only add an additional parameter.
Easy to track — Configuring your application to track how many calls during a certain time period, and throttle if necessary, should be easy.

Disadvantages:

Credentials in the clear — REST APIs will have their credentials sent in the clear whether or not a secure endpoint is used. Nonsecure endpoints will have credentials sent in the clear for both REST and SOAP APIs.
No encryption — All requests and responses are visible to anyone between the requesting server and the API server.

Message-based authentication is very similar to HTTP authentication in the level of security it provides, the primary difference being the pass off from handling the authentication from the web server to the API application itself. As with HTTP authentication, the API's authentication should be separate from authentication used elsewhere on the site.

Server-Side Code

Once the client's username and password have been made SQL safe, they should be run against your API database. Remember that the authentication information used should be different from authentication used elsewhere on the site.

 function checkUser($username, $password) {   $query = "SELECT `user_level` FROM `users` WHERE `username` = ‘$username' AND     `password` = '$password'";   $results = getAssoc($query, 1);   return $results[‘level']; }

This is just your basic authentication function, nothing fancy.

Client-Side Code

The process the client will go through to pass the authentication information depends on whether SOAP or REST is used, and is discussed in the "REST or SOAP?" section, later in this chapter.

SSL Endpoint

Configuring your web server to present the API over an SSL connection adds protection for both the request and response bodies, while requiring little to no additional coding for the API. Remember that the use of a server certificate only authenticates the server for the client, it does nothing to identify the client itself. It is best used layered with one of the previous two examples.

Advantages:

Encryption — Both request and response bodies are protected from intermediate prying eyes.
Server authenticated — Clients who record the server's SSL certificate can monitor it to ensure it does not change over time (which could indicate a man-in-the-middle attack). Using a certificate signed by a signing authority can also provide a similar level of assurance for the client application.
Easy setup — No additional coding required, just configure the web server.

Disadvantages:

Increased load — Encrypting and decrypting communications is noticeably more CPU-intensive than unencrypted communications. Every request requires additional back and forth communications to set up the secure socket.
No client identification — Using an SSL
Additional client-side requirements — Handling SSL from the client side isn't always trivial, and languages may require additional extensions installed (which may or may not be available in a shared host environment).

SSL is an excellent layer to add to any API. It provides security for the request and response payloads, as well as provides some assurance for the client application as to the identity of the server. It is easily combined with either HTTP authentication or message-based authentication. Some care, however, should be taken not to use SSL unnecessarily, because it has additional requirements for the client application that not all developers may be able to meet, and it places additional CPU load on both sides as messages are encrypted and decrypted.

Server-Side Code

No code is required to support being served over SSL. Just configure the web server to present the API over a secure connection.

Client-Side Code

In order to connect to a secure endpoint, PHP will need to be configured with the --with-openssl option. You will also recognize SSL support because https will appear under the Registered PHP Streams list in the phpinfo() output, and SSL will also show up under Registered Stream Socket Transports.

Client-Side Certificates

The API server can generate a certificate and provide it to the client via a secure channel before any requests are made. This certificate is then used in the authentication process; this confirms the identity of both the client and server before requests are made. Although this method provides the greatest level of security (barring a dedicated VPN connection, which won't be covered here), it also has the most strenuous requirements on both sides: not all modules (say, NuSOAP) can handle client-side certificates.

Advantages:

Identity confirmed — Both the server and the client can be sure of who they are communicating with.
Encryption — Both the request and the response are protected from intermediate prying eyes.

Disadvantages:

Increased load — Encrypting and decrypting communications is noticeably more CPU-intensive than unencrypted communications. Every request requires additional back and forth communications to set up the secure socket.
Additional server requirements — The server must create a unique client certificate for each client that wishes to access the API. These APIs must be created and stored in a secure location and transmitted to the client via a secure channel.

Client-side certificates are as good as it gets in terms of API security; both the server and client are assured of the other's identity. Just like using an SSL endpoint, client-side certificates stack well with either HTTP authentication or message-based authentication. As with all forms of security, this increased level of security comes with a price: additional CPU load for both the client and the server, and the elimination of several useful client-side tools for interacting with the API.

Server-Side Code

Setting up client-side certificates takes some work. It is unreasonable to expect that anyone wishing to connect to your API will have their own certificate, and doing so would place a noticeable financial burden on anyone wishing to connect. It is much easier to become your own Certificate Authority (CA) and issue client certificates to your users yourself. This method is, however, slightly less secure because it is unlikely that your site is as careful about issuing user credentials as the real CAs. That being said, becoming your own CA is still the easiest method, and the method discussed here:

Set up your Certificate Authority. It is a good idea to use a separate machine from your web server for this (particularly any production web server) because the entire process is only as secure as your CA's private keys. Ideally this machine shouldn't be connected to the Internet at all; however, because the API will need to have keys generated regularly, this isn't really feasible. In this case, you can settle for having it inaccessible to the Internet, and preferably only accessible to the machine that will be requesting the new keys for clients.
Generate keys for your users and distribute them over a secure connection. A chain is only as strong as its weakest link: If you send the client keys to the user over an insecure HTTP connection, or even worse, email them, there is really no point in generating them at all.
Configure your web server to accept your generated CA, and require that users attempting to connect to a resource present a certificate signed by you.
Also configure your server to use an SSL certificate, similar to the previous example.

Setting up Your Certificate Authority

OpenSSL is the perfect tool for the job. Source distributions should include two scripts, CA.pl and CA.sh, which will automate this process for you. There are other tools such as OpenCA or TinyCA designed to provide a more user-friendly solution. Because this is something you are only going to be doing once, it seems slightly overkill for this situation.

First, a slight modification should be made to the CA script. There is a line reading $DAYS="-days 365", which indicates that certificates created with the script will be valid for 1 year (365 days). This is far too short for your purposes, so choose a larger value (say 3,650 for 10 years).

Second, when running the script itself, answer the questions as they relate to your firm. The common name field is important — it should be the hostname users will be connecting to when they access your API. Use a secure password for your PEM pass phrase. The script will generate a directory called demoCA at your current location and place the appropriate files there.

Third, create a server certificate. This process has two steps, first creating a Certificate Signing Request (CSR), then getting that CSR signed by the CA you just created. To create the CSR, use the following options with openssl:

 openssl req -new –key server.key –out server.csr

openSSL will ask a series of questions to generate your key. For our purposes (and to avoid needing to make configuration changes), these should match the values you used when creating your CA. To sign the server CSR that was generated, first rename the CSR file (server.csr) to newreq.pem, then use the command

 CA.sh –signreq

This will sign the request. Finally, Apache must be instructed to use this certificate. This configuration should be placed within your httpd.conf file, for the VirtualHost that presents the API.

 SSLEngine On SSLCertificateFile /etc/http/conf/ssl/server.crt SSLCertificateKeyFile /etc/http/conf/ssl/server.key SSLProtocol All –SSLv2 SSLCipherSuite ALL:!EXP:!NULL:!ADH:!LOW

The first three lines turn on the SSL Engine and set the locations for both the certificate and the server's private key. The last two lines prevent SSL from using known broken or insecure protocols.

Fourth, generate a client certificate:

 CA.sh –newreq CA.sh –signreq

Remember to distribute it to the user via a secure channel.

Fifth, configure your web server to require clients attempting to connect to present a certificate, and to ensure that presented certificates were signed (and hence created) by your own CA.

 SSLCACertificateFile /etc/http/conf/ssl/demoCA.crt SSLCARevocationFile /etc/http/conf/ssl/demoCA.crl SSLVerifyClient require SSLVerifyDepth

SSLCACertificateFile should point to wherever you placed the CA Certificate File created earlier, and SSLRevocationFile should point to the .crl file that was generated at the same time as the *.crt. Without this file, you will be unable to revoke certificates from abusive users, or users who no longer wish to make use of your API. SSLVerifyClient instructs the web server that all users must present a certificate; users not presenting a certificate will be denied a connection. Finally, SSLVerifyDepth 1 indicates that all client certificates must have been generated directly by your CA. This prevents other users from creating valid client certificates. Apache will need to be stopped and started again in order for these new certificates to take effect.

Note

For a more in-depth look at SSL and other Apache-related security matters, I would highly recommend Ivan Ristic's book Apache Security.

Client-Side Code

Because a client certificate must be presented, the cURL library will also be used. This function is appropriate for calling both SOAP and REST APIs. Handling the request and response is identical in either case:

 function callAPI($endpoint, $requestBody) {   $ch = curl_init();   curl_setopt($ch, CURLOPT_URL, $endpoint);   curl_setopt($ch, CURLOPT_SSLCERT, "../certs/cert_key_pem-1.txt");   curl_setopt($ch, CURLOPT_POST, TRUE);   curl_setopt($ch, CURLOPT_POSTFIELDS, $requestBody);   ob_start();   curl_exec($ch);   $response = ob_get_clean();   if (curl_error($ch))  {    file_put_contents("/tmp/curl_error_log.txt", curl_errno($ch) . ": ".     curl_error($ch), "a+");    curl_close($ch);    return null;  }else  {     curl_close($ch);     return $response;  } }

This function (which was basically introduced in the PayPal chapter) sets up the connection using the client certificate, and then uses output buffering to capture the response (cURL sends output directly to the browser). On error, the error is recorded to file for later examination; on success, the response is returned to the calling function. It is important for the client certificate to be stored in a secure location outside the document root.

REST or SOAP?

The two primary architectures for APIs are REST and SOAP. When creating your API, you really have three options: REST, SOAP, or both. REST APIs are known for being easy and quick to develop for, but the entire request is sent in the clear regardless of the type of encryption used. SOAP APIs are more complex, requiring more effort to generate the response and handle the request, but allow for greater flexibility by adding namespace support. Providing APIs of both types may sound like an attractive option, but keep in mind that it will double your maintenance, support, and documentation time for the life of the API.

Both API methods have already been introduced at length; this section concentrates on differences to keep in mind when developing an API.

REST APIs

When receiving a REST request, the information will come in via GET. As such, all the information will need to be URL-encoded during transmission; you will likely want to decode it before subjecting it to any further processing (the exception being usernames and passwords, which are generally processed as-is). Different request types should be addressed to different endpoints (URLs); if you want to use a single script to handle all requests, you can either present it to developers in that manner (all requests go to a single endpoint), or configure your web server to map many endpoints to a single script. I would generally suggest the latter; it's inline with the specification and it allows you to make changes later without affecting the external interfaces developers use.

Consider allowing developers to use a web interface to make requests against the API. This can be of great use when attempting to diagnose a problem; developers will be able to quickly determine if the problem is the request or their code. The more tools you can provide to developers in terms of diagnostics, the easier it will be to develop for your site.

SOAP APIs

When the SOAP request comes in, it should first be checked to ensure that it conforms to the format specified by your WSDL document. If you are using a tool such as NuSOAP, this is done for you. In fact, most SOAP APIs use some framework that takes care of a lot of the grunt work when handling the requests. SOAP APIs use a single endpoint for all requests (as a general rule, some large APIs separate disparate functions onto different endpoints), and as a result you will likely either have a large script at that point, or lots of require() calls executed depending on the particular call.

Consider allowing developers to use a web interface where they can paste entire request documents onto a form, and run them against your server. Speaking from direct experience, having something like this available is of great use to developers when trying to diagnose a problem. Providing scripts or functions on your site to allow developers to create requests manually will also be of assistance to developers not using a SOAP framework.

Community Considerations

How the API is presented to the world at large and supported is a great determinant for its success. An example of things done right would be Amazon; it has an API with a large range of functionality. It is supported by an automated registration process, public documentation, development forums, sample code, and perhaps most impressively, regular developer chats where API developers can speak directly with programmers at Amazon responsible for the API. It is my opinion that these community support options have helped bring the API to its current level of popularity. While researching for this book, I have discovered that these features, which sound simple, are not nearly as widespread as they should be.

Automating Registrations

Development projects can be spontaneous and quick. Between end of business on a Friday and start of business on a Monday, projects can start and finish, or (in corporate environments) at least hit the proof of concept stage. If registrations need to be accepted manually because the API is restricted in some manner, consider issuing developers either sandbox or provisional access until the request is either accepted or declined. This is particularly important if it takes several business days for an application to be processed.

Public Documentation

The goal of an API is to allow remote applications to interact with your server. The goal of the team creating the API should be to allow as many people as possible to develop for the API. Hiding documentation behind login prompts, nested menus, or worse (like Non Disclosure Agreements) is counterproductive. Try to use the shortest URL possible for information about your API (api.example.com, for example) so it's easy to access from anywhere, and don't require a login unless the user wishes to access their account.

Development Forums

Developers talk to each other — either they talk to each other on various and disparate forums spread across the Web, or they all find each other on your own forums. The advantages of having all your developers talking in the same place should be obvious: With more eyes looking at the same problems, it's easier for your developers to recognize common problems and resolve them (either through API changes or documentation improvements), and it provides a larger database for searches. Give in-house developers a specific and easily recognizable title or avatar within the forums, so new users can quickly determine which answers are authoritative.

Forums aren't necessarily the only way to accomplish this; use of a mailing list may be an attractive alternative. My personal preference is for forums over mailing lists because it seems to be the easiest way to keep past conversations accessible for new users and Google. It also avoids the constant stream of "How do I get off this list" messages that seem to plague the lists I subscribe to.

Finally, don't underestimate the resource your users represent. I actively participated in the same support forum for a software package for approximately 5 years. It never ceased to amaze me how much effort complete volunteers put into helping fellow users.

Sample Code

In developing your API, you had to test it, so you've already written the code. Why not make it easier on other developers and share it with them? In my experience, development time is split roughly down the middle when interacting with a new API — half of it is spent generating the request and handling the response (namely accessing the data I want), and the other half of it is spent doing stuff with that data. If you can eliminate the first half of that with effective code samples, you have effectively cut your development time in half.

Things to consider before writing the code:

Publish sample code in as many languages as you can.
Consider giving users a way to publish their own sample code, particularly in languages you haven't covered.
Include code that does everything on its own without relying on any external libraries.
Also include code that uses common libraries such as NuSOAP.
Demonstrate both sides of your API if it supports both REST and SOAP.
Include sample requests and responses, at both the HTTP and TCP levels.

Developer Chats

Programmers writing for your API are going to have questions; often these questions can be answered by other development community members, but occasionally not (this is especially true when your API is first released or recently updated). A development presence within the community can be easily accomplished by encouraging participation within the forum or mailing list, but direct communication can be a valuable tool.

Here are a few tips:

Schedule the chats (rather than using one of the "Live Chat" tools). This will help ensure that users attempt to resolve problems on their own first.
Schedule multiple times. Remember those things called time zones? There are people in different ones, who may not be too enthusiastic about waking up at 3:00 a.m. to ask why your API is returning some strange data.
Record the chats, and post them. This information is just as valuable as past forum posts or mailing list archives — try to put them somewhere searchable. Remember to inform users ahead of time that the chat is recorded, and that acceptance of that fact is required for participation (to avoid running afoul of jurisdictions where all parties are required to consent before they can be recorded).
Be prompt. Ensure that staff members who will be participating in the chat treat the commitment as seriously as a meeting with upper management: arrive on time, give the encounter their full attention, and remain for the entire scheduled time. Anything less and your users will quickly lose respect for the process.

Development Sandbox

A development sandbox is a duplicate of the production sandbox, the difference being that actions taken on the sandbox do not have any permanent effect (for example, placing a bid on the sandbox copy of the eBay API does not require [or allow] you to buy the item in question). Every API where an action taken has a permanent effect (sending money, agreeing to purchase an item, and so on) should have a sandbox, so developers can test their programs without fear of accidentally performing an undesired action. It also allows developers to test edge cases (sending a million dollars, bidding on one thousand items at once) to ensure that their code functions properly.

Even in cases where hitting the API does not create any lasting action (for example, running a query against the Google API), but where there are limits placed on the developer (the number of queries allowed per day, for example), a sandbox should be used. This allows for easy separation between the developer's work and the production programs, and prevents either the production or development version of the code from exhausting the other's resources.

Performance Techniques

Websites are designed to be accessed by individuals, and as such tend to rely on the relatively slow speed of the user to avoid any performance bottlenecks. This technique fails miserably with APIs because they are going to be consumed by other servers with high-speed connections, often designed only with their own performance in mind (they won't cache your responses for you, and will instead make exactly the same request time and time again). Designing your API with performance in mind can help keep the server fast even when many requests are being made, and will help ensure that future hardware upgrades can accomplish their desired tasks.

Note

Many websites are either designed poorly or appear to lack any sort of design whatsoever. I've seen a site that required 10 database queries to start the page, then an additional query for every item in their database. With more than 40 items in their database, there were approximately 50 queries being made every time the index page loaded. This technique was failing horribly for a website receiving relatively few hits. It wouldn't have lasted minutes if it was consumed automatically, and I doubt it would have lasted more than a few seconds under the Slashdot effect. All these database queries were basically pointless as well; the firm's inventory changed slowly, so a static page generated once per week from the same script would have functioned just as well for the end user, but would have been several orders of magnitude faster.

Caching Data

Often both websites and APIs request data from the database each and every time a request is made, even though the data used to populate the response changes rarely. This, combined with the database normalization techniques taught since the beginning of time, means that each of those requests is likely making at least one query joining results from multiple tables, possibly multiple queries. If your data isn't changing that often, consider caching the response.

For example, take the fictional Bob's Video website. Every time someone either views detailed information about a movie on his website or requests it through his API, his server runs three queries: one query that finds the movie's full title, plot line, and rating, another query that runs a joined query to retrieve detailed information on each of the cast members, and a final query to determine the film's rental status. This is a gigantic waste of resources; once a movie is released, the only response that will change is its rental status. Yet, each and every time the page is loaded, the data is requested again from the database. It would make far more sense to either use a static page for released films (populating rental status dynamically), or at the very least cache all the film's information and retrieve the rental status dynamically.

Note

You've probably noticed that, because the cache will likely end up in a database, I've really only reduced the query count from 3 to 2. It doesn't look like a drastic improvement, but it is. The joined query looking up detailed information for the cast members is going an order of magnitude slower than a lookup based on a primary key, so there is a big saving there. You can also cache the movie data in a form close to its final web form, saving on all the processing generally needed to go from database to web page. You will need two caches in this example, one for the website and one for the API.

Smarter Use of Database Queries

Although caching data is an excellent method of reducing the number of queries you use, it isn't always appropriate. Just make sure you are getting the most out of each query you run. Many times duplicate data is requested while handling a single request; this often happens when different functions need the same data, but they don't call each other so they don't share their results. Consider either reworking your script to obtain all required data itself, then pass off data to the functions that require it, or creating an abstraction layer with an object that takes care of obtaining information from the database only when required.

Once you're using your database queries to their fullest, begin work on improving the speed of the queries themselves. Never start queries with SELECT * FROM — request only the fields you actually need. Also examine both your queries and your database. Try to ensure that the fields you base your selection on are either primary keys or at least indexed by the database server.

Response Caching

Consider again the case of Bob's Video Store API, which allows users to request information on films. With a small design change (moving rental status to its own query, rather than providing it with each request), many new caching opportunities present themselves. Because the response doesn't change regardless of who requests it, a proxy server can be used server side to handle the response (this is much easier with REST APIs than with SOAP). Setting the appropriate headers for cache life (24 hours for films, and 30 minutes for rental status) will allow the API to shrug off most of its work to the proxy server.

PHP Accelerators

There are a few PHP accelerators available, which can have a drastic effect on the speed of your scripts. Every time a PHP script is executed, it is parsed and compiled into byte code by PHP's scripting engine. Because, generally speaking, the script hasn't changed between executions, this is a huge waste of processing time. PHP accelerators cache the byte code version of the scripts, and execute that copy (being mindful of any changes to the original script). This saves the parse and compilation steps each time the script is executed because your API will be called with great frequency, and changed rarely this can be a significant savings.

It is important to realize how PHP accelerators work to avoid having undue expectations for their results. Consider the parse and compilation time for a script as a fixed cost — every time the script is accessed, regardless of the speed of other resources (databases, for example) or how much processing the finished script requires, this cost must be paid. Caching the byte code copy of the script only saves on that cost; it will not speed your database queries or other CPU-intensive processes.

One of the most prevalent PHP accelerators is from Zend, dutifully titled the Zend PHP [4/5] Accelerator. I found it easy to install and was relatively pleased with its results. Having upgraded to PHP5 shortly after its release, I was unable to test other accelerators that have since become available. One of the other accelerators I did manage to try sigfault'd the calling Apache process on a variety of my scripts, so be sure you test whichever accelerator you use extensively before putting it on the production system.

Error Responses

Error messages really are an essential part of any interface; they can either make or break a programmer's heart. Useful error messages can quickly point a programmer to the problem in their code, while cryptic messages or simple numbers cause premature hair loss. As such, a few moments of planning can save your users hours of frustration.

Error Numbers

While error numbers are generally of little use to the general public, they are great for automatic interpretation. Assigning numbers to the various errors your application will generate will help remote applications log any problems they encounter. If you are creating a SOAP API, there are already various error messages defined by the protocol. When planning error numbers, space them out; for example, invalid requests could be given an error number between 100 and 199, properly formatted requests that contain improperly formatted data could be assigned an error number between 200 and 299, and server faults could be assigned a number between 300 and 399. This way, should you decide to be more descriptive in the future by adding more error types, they can still be grouped logically.

Note

Readers who have been programming long enough to remember when line numbers were entered by the programmer, rather than their present state of GUI fluff, will immediately understand this approach. It worked then, and it still makes sense today.

Error Messages

When returning an error message with an API, or any program, there are two key pieces of information to give: what went wrong and why. Again, with SOAP APIs a lot of this may be handled for you (if you are using a prebuilt SOAP toolkit). With REST APIs, for example, if a request doesn't include a required piece of information, don't just return ERROR: Invalid Request; instead return ERROR: Required element username not present. This way the programmer can quickly see that there is an error, that the error is that a required element was not included, and that the element missing was the username element.

What Services to Offer

Now that you have determined how you want to authenticate your users, how you will support the programmers, and considered both your REST and SOAP options, it's time to think about what services you want to offer via your API.

Generally, the services offered via the API are a subset of the functionality offered through the traditional website, so you probably already have an idea of what you want to offer. With an API there are a few additional considerations that may affect those offerings.

Processor Load

Even with effective caching methodologies, your API will likely offer services that present a noticeable amount of load on the server. Because these services were designed to be consumed automatically, the amount of load generated may quickly rival that of your site, even though only a small percentage of unique people are using the service. Look at the amount of work required to respond to the different types of requests you are considering. Services that will end up creating significant load should either be documented with warnings (this service is quite CPU-intensive, so don't use it unless you actually need to; instead use otherServiceA or otherServiceB to obtain similar results) or left out of the API.

Competitive Usefulness

Providing your entire product line, including pricing information, may sound like a great boon to your customers or resellers, but consider the value of this information to your competitors. There are a few ways to mitigate these concerns: Limit the number of results to any particular query, restrict your API to known parties (resellers), and limit the amount of information provided in the API.

Generally, these concerns are considered a part of doing business, outweighed by the additional visibility that having an API provides, but they remain issues that should be raised when considering different services.