10.1 Directories: Introducing Syndic8

There have always been directories of RSS feeds. From the very beginning, at the My Netscape Network and the HotSauce application before that, people have needed to know where to find the best information. The growth of the standard, however, has not been matched by a growth in directories. Though today's RSS users have many more feed choices, they really are limited to one pure, searchable directory: Jeff Barr's Syndic8 (http://www.syndic8.com), shown in Figure 10-1.

Figure 10-1. A screenshot of Syndic8

10.1.1 Registering Your Feed with Syndic8

The registration process with Syndic8 reflects its community- authored nature and requires the following steps:

1. Submit the URL to feed itself via the Submissions page, http://www.syndic8.com/suggest.php?Mode=data.

2. The new feed is then polled several times to measure reliability and find new items.

3. Once a feed has proven itself to be both reliable and regularly renewed, it is made available for review by a Reviewer.

4. Reviewed and approved feeds are then made available.

10.1.2 Using Syndic8

Before you can access some of the more advanced features the site provides, you must register for a username and password. Technically, you don't need to register to run a search for a site or find RSS feeds, but the other exciting stuff is well worth the effort, and this is your big opportunity to give something back to the RSS community. The content of the Syndic8 databases is enhanced by the volunteer efforts of the Syndic8 community. You should consider joining and giving a few minutes a week to one of the roles offered . You can join by going to http://www.syndic8.com/register.php and filling in the appropriate boxes. You'll find yourself given four options:

Reviewer

I can review syndicated sites for accuracy and approve them.

Evangelist

I can work with nonsyndicated sites to encourage them to syndicate their content.

Scraper

I can do the programming work to scrape a site to syndicate its content.

Fixer

I can work with syndicated sites to help them repair problems with their syndicated information.

Check the ones you feel good about, and get yourself logged on. Now, every time you go back to the site, you can log in and do a little of whatever work needs doing. I encourage you to help out in this way.

Now, back to the searching.

Syndic8 has two methods to search for feeds: via the ordinary text-string search box, which indexes the titles of the feeds, and via the Categories section at the top of the page.

Here we see the whole point of marking up your site with metadata, as described in Chapter 3. Syndic8 classifies feeds with the same hierarchy as the DMOZ Open Directory, but it does so only for feeds that give it the necessary information. So, make sure you mark up your sites before submitting them to Syndic8.

Running a search gives you a list of matching feedsthe small, orange XML buttons link directly to the feed, and the feed title links to the URL given by the link subelement of the channel section of the feed.

10.1.3 Syndic8 API

Syndic8 really came into its own with the introduction of the XML-RPC API. By allowing the database to be queried programmatically by remote applications, Jeff Barr and the other Syndic8 developers produced a facility perfectly suited for inclusion in more ambitious scripts and applications.

The API works using ordinary XML-RPC, with the requests sent to the listener at http://www.syndic8.com/xmlrpc.php.

There are two groups of functions available at Syndic8: the basic set, which anyone can use to query the database in various ways and suggest feeds for inclusion, and the Personal List set, which is available only to users who have the Personal List facility. We will deal with the basic set functions first. These functions are classified according to their various purposes.

The first group finds Syndic8's capabilities:

syndic8.GetFeedCount

Takes : nothing

Returns : a number

This function returns the number of feeds Syndic8 knows about.

syndic8.GetFeedFields

Takes : nothing

Returns : an array of strings, each a name of a field

This function returns all the field names that a feed record can have within the Syndic8 database.

syndic8.GetFeedStates

Takes : nothing

Returns : an array of structs, each a feed state and its description

This function returns all the possible feed states that Syndic8 supports.

syndic8.GetLocationSchemes

Takes : nothing

Returns : an array of strings, each a location scheme

This function returns all the location schemes Syndic8 knows about.

syndic8.GetToolkits

Takes : nothing

Returns : an array of strings, each a toolkit name

This function returns all the toolkits Syndic8 knows about.

The second group deals with the categories the feed author assigned to the feed, as discussed in Chapter 3. You can see the categories in action on the Web at http://www.syndic8.com/feedcat.php.

syndic8.GetCategoryChildren

Takes : nothing

Returns : an array of category names

This function returns the categories that are the immediate children of the given category.

syndic8.GetCategoryRoots

Takes : a string, the name of a category

Returns : a string, the name of the first category within the ontology for the given category

This function returns the topmost category for the given category.

syndic8.GetCategorySchemes

Takes : nothing

Returns : an array of the supported category schemes

This function returns an array of the supported category schemes.

syndic8.GetCategoryTree

Takes : a string, the name of a category scheme

Returns : an array of the categories within the scheme

This function returns an array of all the categories within a scheme.

syndic8.GetFeedsInCategory

Takes : a string, either a category name or a category scheme name

Returns : an array of FeedID s

This function returns an array containing the FeedID s of all the feeds within the named category.

The third group allows simple queries of the database:

syndic8.FindFeeds

Takes : a string pattern to match against all feed fields

Returns : an array of FeedID s of matching feeds

This function matches a string against all the text fields of each feed in the feed list and returns the FeedID s of the matching feeds.

syndic8.FindSites

Takes : a string

Returns : an array of FeedID s of matching feeds

This function matches a string against the SiteURL field of each feed in the feed list (i.e., the content of the link subelement of channel ) and returns the FeedID s of the matching feeds.

syndic8.FindUsers

Takes : a string

Returns : an array of UserID s of matching users

This function matches a string against all text fields of the user list and returns the UserID s of the matching users.

syndic8.GetUserInfo

Takes : a string, the UserID of the user

Returns : a struct containing all the fields in the database pertaining to the user

This returns all known information about a user.

The fourth group allows more complicated database queries:

syndic8.GetChangedFeeds

Takes : an array of structs, each containing an array of strings that denote the fields to check, a string of the search StartDate ( YYYY-MM-DD ) , a string of the search EndDate ( YYYY-MM-DD ) , and an array of strings denoting the fields to return

Returns : an array of structs, each containing the requested fields from feeds with changes in the given date range.

This is a powerful function that returns all the feeds fitting the criteria that are known to have changed within a given date range.

syndic8.GetFeedInfo

Takes : a single scalar or an array of integer FeedID s or string FeedURL s, plus, optionally , an array of field names if you want to limit the result

Returns : an array of all information known about the requested feeds, or just the fields named in the request

The most complicated function in the Syndic8 API, but also the most powerful, this function returns information known about one or more feeds.

syndic8.QueryFeeds

Takes : three strings; the field to match, an operator on the match ( < , > , <= , >= , != , = , like , or regexp ), and the value to match

Returns : an array of FeedID s of matching feeds

This function queries the database for feeds matching a single criterion.

By combining these commands, we can ask Syndic8 for all sorts of information regarding a feed, as well as any feed related to it in any way. It is quite simple, therefore, to produce applications that can query Syndic8 for the URL of a site's RSS feed, plus the URLs of any feeds related to it, selecting only those feeds that have been updated within a certain time frame and are written in a certain language.

These functions are universally availableanyone can write a script to call them. The next set of functions are not. They require the user to have an account with Syndic8, and some require the Personal List option to be turned on, which it is now the default for new users.

The first set of Personal List functions we will look at deal with subscription lists. Syndic8 can store subscription lists for its users. The lists are numbered, beginning with 0. The user can add and remove feeds to any subscription list, then query the list or pass it to another application. If you consider it as a remote blogroll, you have precisely the idea.

syndic8.CreateSubscriptionList

Takes : three strings; the UserID , the user's password, either in cleartext or MD5- hashed , and the list name; also, one Boolean value should be listed as Public.

Returns : the numerical reference for the new list

This function creates a new subscription list for the user.

syndic8.DeleteSubscriptionList

Takes : two strings; the UserID and the user's password, either in cleartext or MD5-hashed, followed by the integer list reference

Returns : 1 on success, on failure

This function deletes the subscription list.

syndic8.GetSubscribed

This function returns a list of all feeds subscribed to by the user, filtered by the given criteria, including feeds subscribed to by category.

syndic8.GetSubscribedFeeds

Takes : two strings; the UserID and the user's password, either in cleartext or MD5-hashed, followed by the integer list reference and, optionally, an array of string field names

Returns : an array of structs; FeedID , DataURL , and Status if field names are not supplied, or requested fields if they are supplied

This function is the same as syndic8.GetSubscribed , except that the returned feeds are only those subscribed to explicitly (i.e., it does not include feeds subscribed to implicity by category).

syndic8.GetSubscribedCategories

Takes : two strings; the UserID and the user's password, either in cleartext or MD5-hashed, followed by the integer list reference

Returns : an array of structures, each a scheme and a category

This function returns the categories subscribed to within a certain subscription list.

syndic8.GetSubscriptionLists

Takes : two strings; the UserID and the user's password, either in cleartext or MD5-hashed

Returns : array of structures, each a list index and a name

This function returns the list of subscription lists for the user.

Subscription lists would be no fun if you couldn't add things to them. The functions return 1 , for a successful result, or an error code. Table 10-1 lists Syndic8's error codes.

Table 10-1. Error codes from Syndic8

Value	Name	Meaning
801	errorQUERY_FAILED	A select query failed due to a database problem.
802	errorNO_FEED	The FeedID supplied to the call is not valid.
803	errorNO_USER	The UserID supplied to the call is not valid.
804	errorINVALID_USER	The user name or password is not valid.
805	errorNO_SCHEME	The categorization or location scheme is invalid.
807	errorNO_SET_CATEGORY	The category could not be set.
808	errorMUST_BE_CATEGORIZER	The function requires that the user have the Categorizer role.
809	errorNO_SET_LOCATION	The location could not be set.
810	errorNO_RETRIEVE_BY_CATEGORY	The feeds could not be retrieved for the category.
811	errorNO_SUBSCRIBE_FEED	The subscription request for the feed could not be completed.
812	errorNO_SUBSCRIBE_CATEGORY	The subscription request for the category could not be completed.
813	errorMUST_HAVE_PERSONAL_LIST	The user must have the PersonalList option set to perform the operation.
814	errorSUGGEST_FAILURE	Autosuggesting the DataURL failed.
815	errorNO_DATA_URL	The nonnumeric FeedID / DataURL given to SubscribeFeed did not appear to be a URL.
816	errorNO_UNSUBSCRIBE_FEED	The unsubscription request for the feed could not be completed.
817	errorNO_UNSUBSCRIBE_CATEGORY	The unsubscription request for the category could not be completed.
818	errorNO_DELETE_LIST	The subscription list could not be deleted.
819	errorDUPLICATE_USERID	The given UserID is already in use.
820	errorDUPLICATE_EMAIL	The given Email is already in use.
821	errorINVALID_SPONSOR	The SponsorID or SponsorPassword is not valid.
822	errorMUST_HAVE_CREATE_USER	The sponsor must have the CreateUser option set to perform the operation.
823	errorBAD_QUERY	The query given to QueryFeeds is defective.
824	errorNO_FIELD	An invalid field name was given to GetFeedInfo .
825	errorNO_LIST	An invalid list identifier was given.
826	errorNO_CREATE_LIST	The subscription list could not be created.
827	errorMUST_BE_EDITOR	The caller must have the Editor role.
828	errorBAD_USERID	The supplied UserID is syntactically invalid.

syndic8.SubscribeFeed

Takes : two strings; the UserID and the user's password, either in cleartext or MD5-hashed, followed by either the integer FeedID or a string URL of the RSS feed itself, and then the number of the list you want to subscribe the feed to and a Boolean value of true or false for "AutoSuggest?"

Returns : a success or error code

This function adds the feed to the user's given subscription list. If a DataURL is given, the AutoSuggest option is true, and the feed is not known by Syndic8, it will be added to the "suggested" list as if by entered by the UserID .

syndic8.UnSubscribeFeed

Takes : two strings; the UserID and the user's password, either in cleartext or MD5-hashed, followed by the integer FeedID and the number of the list you wish to unsubscribe the feed from

Returns : a success or error code

This function removes a feed from a subscription list.

syndic8.SubscribeCategory

Takes : four strings; the UserID , the user's password, either in cleartext or MD5-hashed, the Category scheme, and the category and integer of the subscription list

Returns : a success or error code

This function adds all the feeds within the category to a subscription list.

syndic8.UnSubscribeCategory

Takes : four strings; the UserID , the user's password, either in cleartext or MD5-hashed, the Category scheme, and the category and integer of the subscription list

Returns : a success or error code

This function removes all the feeds within a category from a subscription list.

Finally, the Syndic8 API can be used to add feeds to the Syndic8 database. There are two functions to do this, and they return the new FeedID , plus a special code, as listed in Table 10-2.

syndic8.SuggestDataURL

Takes : a string, the URL of the RSS feed itself

Returns : a struct with the FeedID (newly created or old) and a suggest-status code

This function suggests a feed from its RSS URL.

syndic8.SuggestSiteURL

Takes : a string, the URL of the site itself (i.e., the contents of channel link )

Returns : a struct with the FeedID (newly created or old) and a suggest-status code

This function suggests a feed with its SiteURL .

Table 10-2. Response codes from Syndic8 for feed suggestions

Value	Name	Meaning
1	suggestALREADY_QUEUED	The DataURL is already known, and the site is currently being polled to prepare it for review.
2	suggestALREADY_EVANGELIZING	The SiteURL is already known, and the site is currently being evangelized.
3	suggestALREADY_KNOWN	The DataURL or SiteURL is already known, and the status is neither queued nor evangelizing.
4	suggestACCEPTED	The DataURL or SiteURL was not known. It has been accepted for reviewing or evangelization, as appropriate.
5	suggestERROR	Something went wrong.
6	suggestNO_PREFIX	The URL did not have an http:// prefix.
7	suggestNO_FETCH	The information referred to by the URL could not be fetched .
8	suggestNOT_FEED	The information referred to by the URL did not appear to be a valid feed.
9	suggestNUMERIC_IP	The URL contained a numeric IP address.
10	suggestBLOCKED	The URL is from a domain or subdomain that is not currently accepted into Syndic8.

We will be using the Syndic8 API again in Chapter 12. In the meantime, let's move on to aggregators.