Hack 78. What to Do if Your Government Is Hoarding Geographic Data

Geospatial data is a priceless public resource, but few national governments see it that way. How can you get free and fair use of it?

If you're not based in North America, you're probably flipping through this book thinking, "Almost all these hacks are based on U.S. data! How can I make my own maps of London/Karachi/Accra/Quito"?

This isn't regrettable cultural arrogance on the part of the authors; it is a reflection on the lack of geographic data available in the public domain. Even the world maps and gazetteers built in this book are mostly derived from data published by U.S. intelligence agencies, not from local sources.

National mapping agencies are in a difficult position. They provide an essential public service and are arguably part of the core machinery of government, along with roads, street lamps, and schools: the kinds of services that would not be adequately provided by market forces alone. But mapping agencies are squeezed by commercial pressures; because they have a clear potential revenue model, they are liable to be privatized. Citizens who paid handsomely in taxes for the initial data collection now pay to have it sold back to them piecemeal, without access to or means to contribute to the raw data from which the maps are generated.

Long-established national mapping agencies tend to have a military origin, like Britain's Ordnance Survey and nima.mil, formerly the Defense Mapping Agency. They bring to mind visions of action-men bent over a strategic map spread out over a table, moving tiny flags on a wall-mounted territorial display as the frontline advances and retreats. Maps essentially have their origin as military instruments: a hundred-meter calculation error in the bend of a river or the slope of a hillside could mean death for many thousands, the fall of a city. Without the map to demarcate the territory, how can it be defended? But geographic data is also key to many other more peaceable government functions: road building, public transport planning, pollution monitoring, political boundary (re)drawing, refuse collection, school and hospital catchment-area sizing, just-in-time route planning for emergency services. Location: it's everywhere!

Across the world, centralized, national mapping agencies collect and maintain spatial data as government sub-departments. They are under pressure to realize their commercial potential as a geodata-licensing monopoly, and are having budgets cut or statuses changed as a result. Meanwhile local government departments and community groups are obliged to buy back the maps and processed data from the national agency. They pay millions for proprietary software and consultancy and create their own data in undocumented formats. They can't share the data with each other because they don't have clear legal rights to redistribute it; they can't connect it to their other logistics systems over the Web because the interfaces are closed. The raw geodata underlying the map is not for use by ordinary citizens at all. Academic and nonprofit institutions get a discount.

The European Union faces a particularly challenging set of problems, with 10 new countries and 9 new official languages recently added to the standing 15. It now must deal with 25 "territories," each with different naming conventions for regions, districts, postal codes, and political systems, which need to be accounted for.

There is an exciting prospect, were the underlying map data opened up to free use by these nations, that academics and enthusiasts would solve many of these problems while "scratching their itches." Environmental scientists could predict floods in areas that cross borders, determined linguists could translate metadata models into Welsh from Polish, and hackers could write geocoders for street addresses in different forms to support locative services.

One common argument against freeing national geodata is that national mapping agencies, NMAs, would lose a lot of their revenue model; anyone from a small to medium business, to a local councillor with a copy of Manifold, to an inspired reader of this book could make and maintain his own maps. The mapping agency would be merely left with the task of geodata collection and maintenance. Some of this data needs to stay in the public domain: remote and rural areas, for example, which a purely commercial market (such as cell-tower telephony) would not have an economic incentive to cover. This is surely an argument for keeping geodata collection in the hands of elected authorities, an important source of trust for data distribution.

There are drawbacks to the U.S. model of local free geodata. The mapping agencies, not making commercial offerings, are dependent on federal or state funding, which can fluctuate between administrations. Meanwhile huge companies like Navteq, who can afford to rent time on aerial photography satellites and send fleets of scouts out "ground truthing" with GPS units, are augmenting the free TIGER/Line data with more accurate data and up-to-date feature sets, with no obligation to enrich the public domain with new data.

In truth, these huge companies already have accurate thematic maps and feature databases, and they aren't going into the public domain whether or not state-owned geodata is publicly released. And for the first time since the descrambling of the U.S. GPS signals, with the international offering GALILEO on the way, ordinary citizens have the affordable technology and the free tools outlined in this book to actively contribute to the maps that describe their world. Open geodata would be a millions-saving boon to small businesses and local government departments. The latter would have the means to collect and make sense of metadata about their communities at a local level.

A principled idealist might argue that national mapping data should be released under a GPL or ShareAlike license; http://www.ollivier.co.nz/atlas/freeworldmaps.html makes this case convincingly. But this might prove unacceptable to large businesses and detrimental to some smaller ones focused on mapping. It might also widen the gap between public domain geodata and commercial offerings, in both coverage and intent. That person who controls the map controls how people perceive the world. If you would like to own your map, and the government where you live has a restrictive policy on geodata:

• Formulate a project, a "cool hack," ideally with an educational and civic aspect. Write to named members of your national mapping agency, making the exact scope of what you need clear. If they seem unreceptive, ask if you can come in for a meeting and chat anyway, offering to share your specialist knowledge.
• Get together with other geohackers on the Web working on related projects or ideas. Hold a show-and-tell workshop about your projects and invite some mapping agency representatives.
• If your national mapping agency is still unreceptive, try going upstream: write to the government department that nominally runs or funds your national mapping agency. Stress the fact that data withholding is throttling innovation in the mobile market and masking the true nature of local statistics.
• Look for possible legal recourse; many countries, including Canada, have a Freedom of Information Act that guarantees access to government-held data that is in the national interest, at a reasonable cost. The European Union's Public Sector Information Directive is enforced from July 2005.
• Do it youself! Focus on a small area around you and start making your own shapefiles from GPS traces, annotating local sites of interest in an Open Guide, map the local community wireless network.

7.2.1. If you're in the U.S.

You have nothing to worry about! Under U.S. law, all information published by the federal government must be copyright-free and must be made available at the nominal cost of copying it. In 1997, Bruce Perens purchased the U.S. Census Bureau's TIGER data set on CD-ROM and made the data freely available on his FTP site. This prompted the Census Bureau to publish all its data for free download over the Web. Now the U.S. government is running a "one-stop shop" initiative that includes free data from the National Imagery and Mapping Agency and the U.S. Geological Survey (http://www.geodata.gov/). The latter provides the excellent "seamless" system featured in [Hack #67] .

7.2.1.1 If you're in Canada

Your luck is improving! Partly as a result of grassroots lobbying by GIS industry experts, a government policy study recommended that "Digital geospatial data that are collected or created by any level of government should be made as readily available electronically to the public as possible." Canada now publishes increasing amounts of free data through its "geoconnections" web site, http://cgdi.gc.ca/english/index.html.

7.2.1.2 If you're in Australia or New Zealand

You have ANZLIC (http://www.anzlic.org.au/), the Spatial Information Council that covers Australia and New Zealand and publishes metadata standards based on ISO 19115, which in turn is converging with the OGC standards. Geoscience Australia is at http://www.ga.gov.au/, and its Australian Spatial Data Directory (ASDD) is a one-stop data-directory service for Australia. There are free topographic and digital elevation models available, as well as a selection of feature and geoscience data sets that are deemed "fundamental." There's not much in the way of demographic data

The Australian government geodata policy states that "Fundamental spatial data will be provided free of charge over the Internet, and at no more than the marginal cost of transfer for packaged products" (see http://www.osdm.gov.au/osdm/policy.html).

7.2.1.3 If you are in Denmark

You are lucky! The Danish government has the most liberal data policy in the EU. http://www.geodata-info.dk/ is a central portal service. http://dk.space.frot.org/ is an interesting semantic web service based on the complete, public-domain set of Danish addresses.

7.2.1.4 If you're elsewhere in the EU

You need deep pockets. Most EU National Mapping Agencies operate on a partial cost-recovery-by-user payment model. At the time of writing, there is a European Commission-proposed directive aiming to establish a common European Spatial Data Infrastructure that covers intellectual property and licensing policy.

At http://www.ec-gis.org/inspire/ are a series of enlightening documents assessing the "state of play" in GIS infrastructure in the then 15 EU and 10 accession countries. Included are URLs and contact details for the individual national mapping agencies. Most are part-government, part-privately funded and are under pressure to "pay for" the costs of their activities, even though many of the licensing fees are "false profits," coming from other branches of publicly funded activity, such as local government, environmental monitoring, and city planning.

7.2.1.5 If you're in India

The Geological Survey of India (http://www.gsi.gov.in/aboutgsi.htm) was founded in 1851. They publish hardcopy maps, carry out geological surveys, and have digitization of their databases at hand. Their Web presence is a bit scattershot, but there must be an amazing amount of potential data and depth in there. There is currently no available geodata or digital mapping provided by the Indian national mapping agency. It seems ironic that the UK's Ordnance Survey now outsources its digital map drawing to Delhi.

7.2.1.6 If you're in Japan

Oddly for a country so renowned for its obsession with mobile devices, their mapping agency (http://www.gsi.go.jp/) seems very old-fashioned. However, the Japanese mapping agency is working hard to organize data-sharing efforts throughout Asia and indeed the rest of the world. But at the time of writing, no raw geodata is available for Japan itself.

7.2.1.7 If you are in the Asia/Pacific area

Check out the Permanent Committee on GIS Infrastructure for Asia and the Pacific (http://www.pcgiap.org). This intermapping-agency institution, hosted on the Web by the Japanese national mapping agency, seems more concerned with the social and organizational aspects of mapping than the very technical. It encompasses almost all of central and eastern Asia and the Pacific. Their problem set is even more complicated than Europe; many countries use obscure or outdated ellipsoid projections and have different character sets.

7.2.1.8 If you are in South America

The Mexico-hosted Instituto Panamericano de Geografía E Historia has existed in some form since 1928 and has a dense organizational structure. On its list of activities you can find conferences on remote sensing alongside "Estudios de Filosofia Practica e Historia de las Ideas."

Interestingly, many South American national mapping agencies are still overtly military concerns. Others integrate national mapping with national statistics. Brazil's national mapping agency has a lot of arbitrary census data and maps files free for download at http://www.ibge.gov.br/.

Otherwise, map offerings are mostly conventional, paper-based cartographic products. Chile and Argentina both offer proprietary digital GIS packages at high cost.

UNESCO maintains an excellent current index of world mapping agencies, with web sites and contact email addresses where available, and phone and street address in most cases (see http://whc.unesco.org/map-agencies.htm).

7.2.1.9 If you're in Africa

National mapping coverage of Africa is patchy. Though most countries have a geological survey or government cartographical body, many in Sub-Saharan Africa don't have an online presence at all. Most data is to be found in the hands of uncoordinated relief agencies and NGOs. South Africa runs a modern, commercial national mapping agency with DEM, with topographic data for sale, and looks best positioned to lead intracontinental efforts. Try the UNESCO list (http://whc.unesco.org/map-agencies.htm) to obtain national mapping agency contact information.

Another tantalizing possibility has arisen with the availability of one-meter resolution commercial satellite imagery, such as those available from the IKONOS satellite. Although the images themselves cannot be redistributed, the license that comes with their purchase usually explicitly permits the redistribution of new derivative works, such as vector layers of road and rail networks extracted from the imagery (either by hand or through some automated process). With such imagery typically costing less than $1,000 for a minimum purchase of 50 km2, hackers and other interested parties in urban areas outside the U.S. can in theory take up a collection for the price of the imagery, use it to build a vector data set of their home cities, and then release the resulting street data under an open licence (e.g., Creative Commons), thereby completely sidestepping the NMAs. An example of one such ongoing effort is the London Free Map (http://uo.space.frot.org/), which hopes not only to produce a comprehensive, free vector map of London but also to develop Open Source tools to help repeat the process elsewhere.