Parsing URLs

Table of contents:

Problem

You want to parse a string representation of a URL into a data structure that articulates the parts of the URL.

Solution

URI.parse TRansforms a string describing a URL into a URI object.^[5] The parts of the URL can be determined by interrogating the URI object.

^[5] The class name is URI, but I use both "URI" and "URL" because they are more or less interchangeable.

	require uri

	URI.parse(https://www.example.com).scheme # => "https"
	URI.parse(http://www.example.com/).host # => "www.example.com"
	URI.parse(http://www.example.com:6060/).port # => 6060
	URI.parse(http://example.com/a/file.html).path # => "/a/file.html"

URI.split transforms a string into an array of URL parts. This is more efficient than URI.parse, but you have to know which parts correspond to which slots in the array:

	URI.split(http://example.com/a/file.html)
	# => ["http", nil, "example.com", nil, nil, "/a/file.html", nil, nil, nil]

Discussion

The URI module contains classes for five of the most popular URI schemas. Each one can store in a structured format the data that makes up a URI for that schema. URI.parse creates an instance of the appropriate class for a particular URLs scheme.

Every URI can be decomposed into a set of components, joined by constant strings. For example: the components for a HTTP URI are the scheme ("http"), the hostname ("www.example.com (http://www.example.com)"), and so on. Each URI schema has its own components, and each of Rubys URI classes stores the names of its components in an ordered array of symbols, called component:

	URI::HTTP.component
	# => [:scheme, :userinfo, :host, :port, :path, :query, :fragment]

	URI::MailTo.component
	# => [:scheme, :to, :headers]

Each of the components of a URI class has a corresponding accessor method, which you can call to get one component of a URI. You can also instantiate a URI class directly (rather than going through URI.parse) by passing in the appropriate component symbols as a map of keyword arguments.

	URI::HTTP.build(:host => example.com, :path => /a/file.html,
	 :fragment => section_3).to_s
	# => "http://example.com/a/file.html#section_3"

The following debugging method iterates over the components handled by the scheme of a given URI object, and prints the corresponding values:

	class URI::Generic
	 def dump
	 component.each do |m|
	 puts "#{m}: #{send(m).inspect}"
	 end
	 end
	end

URI::HTTP and URI::HTTPS are the most commonly encountered subclasses of URI, since most URIs are the URLs to web pages. Both classes provide the same interface.

	url = http://leonardr:pw@www.subdomain.example.com:6060 +
	 /cgi-bin/mycgi.cgi?key1=val1#anchor
	URI.parse(url).dump
	# scheme: "http"
	# userinfo: "leonardr:pw"
	# host: "www.subdomain.example.com"
	# port: 6060
	# path: "/cgi-bin/mycgi.cgi"
	# query: "key1=val1"
	# fragment: "anchor"

A URI::FTP object represents an FTP server, or a path to a file on an FTP server. The typecode component indicates whether the file in question is text, binary, or a directory; it typically won be known unless you create a URI::FTP object and specify one.

	URI::parse(ftp://leonardr:password@ftp.example.com/a/file.txt).dump
	# scheme: "ftp"
	# userinfo: "leonardr:password"
	# host: "ftp.example.com"
	# port: 21
	# path: "/a/file.txt"
	# typecode: nil

A URI::Mailto represents an email address, or even an entire message to be sent to that address. In addition to its component array, this class provides a method (to_mailtext) that formats the URI as an email message.

	uri = URI::parse(mailto:leonardr@example.com?Subject=Hello&body=Hi!)
	uri.dump
	# scheme: "mailto"
	# to: "leonardr@example.com"
	# headers: [["Subject", "Hello"], ["body", "Hi!"]]

	puts uri.to_mailtext
	# To: leonardr@example.com
	# Subject: Hello
	#
	# Hi!

A URI::LDAP object contains a path to an LDAP server or a query against one:

	URI::parse("ldap://ldap.example.com").dump
	# scheme: "ldap"
	# host: "ldap.example.com"
	# port: 389
	# dn: nil
	# attributes: nil
	# scope: nil
	# filter: nil
	# extensions: nil

	URI::parse(ldap://ldap.example.com/o=Alice%20Exeter,c=US?extension).dump
	# scheme: "ldap"
	# host: "ldap.example.com"
	# port: 389
	# dn: "o=Alice%20Exeter,c=US"
	# attributes: "extension"
	# scope: nil
	# filter: nil
	# extensions: nil

The URI::Generic class, superclass of all of the above, is a catch-all class that holds URIs with other schemes, or with no scheme at all. It holds much the same components as URI::HTTP, although theres no guarantee that any of them will be non-nil for a given URI::Generic object.

URI::Generic also exposes two other components not used by any of its built-in sub-classes. The first is opaque, which is the portion of a URL that couldn be parsed (that is, everything after the scheme):

	uri = URI.parse(	ag:example.com,2006,my-tag)
	uri.scheme # => "tag"
	uri.opaque # => "example.com,2006,my-tag"

The second is registry, which is only used for URI schemes whose naming authority is registry-based instead of server-based. Its likely that youll never need to use registry, since almost all URI schemes are server-based (for instance, HTTP, FTP, and LDAP all use the DNS system to designate a host).

To combine the components of a URI object into a string, simply call to_s:

	 uri = URI.parse(http://www.example.com/#anchor)
	 uri.port = 8080
	 uri.to_s # => "http://www.example.com:8080/#anchor"

Problem

Solution

Discussion

See Also