Section 6.15. Conclusion

6.16.2. Objective #2: Reunite the World's Nations

The would-be GRASS ninja is not daunted at this juncture. Surely, the ninja trainee must concur, GRASS must have some facility for merging polygons. We simply tell GRASS to merge all of the adjacent shapes belonging to the same country in a new vector layer, and we are done.

If a ninja-in-training may be permitted one flaw, perhaps that flaw is overconfidence. After a bit of poking in the GRASS documentation, we were forced to conclude that GRASS had no such tool, per se. GRASS has a tool, v.extract which extracts vector features into a new layer, possibly based on an SQL WHERE query against the attribute tableand even has an option to "dissolve" the borders of adjacent polygons in the process. GRASS also has a tool called v.patch , which will patch multiple vector layers into a single one. The problem is that v.extract must be run separately for each of the 251 countries or country-like territories in the admin layer and then v.patch must patch each country into a new international borders layer. Worse, the attribute data that identifies each country by name and FIPS code must somehow be preserved in the process. To attempt the task by hand would be tedious even to contemplate.

However, you may have gathered that the ninja is second only to MacGyver in sheer resourcefulness. If there is a means at hand, the ninja must find some way to exploit it. First, we need a list of the countries and assorted territories involved. The FIPS code will suffice for this purpose:

GRASS:~/hacks/world >

echo "select fips_cntry from admin"  
db.select  sort  uniq >fips.list

We're obliged to use the hoary Unix sort and uniq commands to extract the list of unique countries, because GRASS's rudimentary SQL parser doesn't support DISTINCT . Now, for each country in fips.list , we must v.extract it out of admin , and v.patch it into our new layer, which we will call borders . We could feed the list of countries to a bit of Perl and have the Perl script generate the necessary GRASS commands for us. That bit of Perl might look as follows :

while (<>) {
    chomp;
    print qq/v.extract -d in=admin out=tmp new=$. where="FIPS_CNTRY = '$_'"\n/;
    print qq/v.patch -a in=tmp out=borders\n/;
}

The -d option to v.extract instructs it to dissolve internal borders between shapes. v.extract then writes the shapes for each country to a layer called tmp . In the process, the shapes are assigned a new category value equal to $ ., which is Perl for "the current line number of the input file." Recall that fips.list is alphabetized. As a result, all the shapes from the first country from our fips.list file (Aruba) will receive the category #1 in the new layer, the second country (Antigua and Barbuda) #2, the third (Afghanistan) #3, and so on. This will turn out to be important later.

Next, the -a option to v.patch tells it to append the input layer to the output layer rather than overwriting the output layer, which is the default. So each call to v.patch will successively add each country to the borders layer, until they're all in there. We use this wodge of Perl as follows, and then run the script it generates:

GRASS:~/hacks/world >

perl gen_extract.pl <fips.list >extract.sh

GRASS:~/hacks/world >

less extract.sh

v.extract -d in=admin out=tmp new=1 where="FIPS_CNTRY = 'AA'"
v.patch -a in=tmp out=borders
v.extract -d in=admin out=tmp new=2 where="FIPS_CNTRY = 'AC'"
v.patch -a in=tmp out=borders
v.extract -d in=admin out=tmp new=3 where="FIPS_CNTRY = 'AF'"
v.patch -a in=tmp out=borders
v.extract -d in=admin out=tmp new=4 where="FIPS_CNTRY = 'AG'"
v.patch -a in=tmp out=borders
...
GRASS:~/hacks/world >

sh ./extract.sh

And then we sit back and wait, while GRASS extracts, merges, and patches every polygon belonging to each of the world's 251 countries or country-like territories. Fortunately, waiting comes easy to the would-be ninja. The would-be ninja is not even fazed by the irritating console beep emitted by v.extract each time it runs, warning that the tmp layer is being overwritten. (The true would-be ninja would have thought to silence his handiwork by adding a print qq/g.remove vect=tmp\n/ to the top of the loop in gen_extract.pl , but never mind that.)

Some minutes later, GRASS completes its labors without complaint; however d.vect borders type=area fails to reveal the goods. Any landmass completely within a single countrysuch as the British Isleshows up, but any landmass spanning more than one countrylike, say, mainland North Americahas simply disappeared. However, d.vect borders type=boundary shows that the vectors are all there. What gives?

It turns out that, had the ninja trainee paid close attention, as every ninja must, he would have noted the following warning emitted by each execution of v.patch :

Patch complete. 1 files patched.
Intersections at borders will have to be snapped.
Lines common between files will have to be edited.
The header information also may have to be edited.

A message like this from GRASS is a sure sign that the vector layer in question needs a run through v.clean to fix what ails it and reassure GRASS that the polygons contained therein really are polygons. The following does the trick:

GRASS:~/hacks/world >

v.clean in=borders out=borders2 tool=rmdupl,snap,bpol

v.clean needs a few minutes to work its magic (this is 13 megs of vector data, after all) and it won't overwrite the original layer, so we have to create a new layer called borders2 . A quick call to d.vect reveals, as shown in Figure 6-53, that the borders2 layer does indeed contain a vector map of the world's international boundaries.

Figure 6-53. Part of a vector map of national boundaries successfully extracted from the first-level administrative boundaries layer

Having verified for ourselves that this is indeed the case, we can engage in a bit of cleanup work, as a proper ninja always covers her tracks:

GRASS:~/hacks/world >

g.remove vect=borders

GRASS:~/hacks/world >

g.copy vect=borders2,borders

GRASS:~/hacks/world >

g.remove vect=borders2,tmp

However, if we now run v.info on the new borders layer, to see what sort of attribute data made it through the extraction process, we are brought up a bit short:

GRASS:~/hacks/world >

v.info -c borders

Database connection for map <borders> is not defined in DB file

The answer is that no attribute data made it through. In fact, GRASS is saying that the borders layer doesn't even have an attribute table. It is as though the ninjas have successfully infiltrated the tower, only to find that the villain has absconded with the princess. Honor demands that we give chase!

6.16.3. Objective #3: Recover the Missing Plans

Fortunately, if you'll recall, in our current set up, the vector-layer attribute tables are just DBase .dbf files stored in the filesystem. It would be awful nice if we could just pipe a CREATE TABLE command to db.execute to create the data table for the borders layer, but GRASS's DBF driver isn't nearly that cooperative. So we have to resort to some other means of creating a .dbf file.

The admin layer already has all the data we need, and, if you'll recall, we took care to assign category numbers to each country that correspond to its place in an alphabetized list of FIPS country codes. We can generate a pipe character ( ) delimited text file containing all the attribute data we need for our borders layer with a single line of shell, as follows:

GRASS:~/hacks/world >

echo "select fips_cntry, cntry_name 
from admin"  db.select  sort  uniq  perl -ne 'print "$.$_"' > borders.txt

Now, that's one heck of a shell command, but let's parse it bit by bit. The db.select command will dump the FIPS country code and the country name of each feature, separated by a pipe, in the admin layer. When the list goes through sort and then uniq , we will get a list of unique country codes and names , sorted alphabetically by code. You may recognize our old friend $ . in the subsequent perl command, which simply prints out each line of input with the current line number plus a pipe character added to the front of the line. The borders.txt file ends up looking like this:

1AAAruba
2ACAntigua and Barbuda
3AFAfghanistan
4AGAlgeria
...

All it needs is a header line, which we can add by loading it up in our favorite text editor, and making it look like this:

CATFIPSNAME
1AAAruba
2ACAntigua and Barbuda
3AFAfghanistan
4AGAlgeria
...

The upshot is that the category numbers that we gave each country's features in the extraction step are now matched up to that country's name and FIPS code in the current step. GRASS likes the column containing the category numbers to be called CAT , but that's not a problem at all. The only step that remains is to get this attribute data into an actual .dbf file and then tell GRASS about it.

There are a bunch of options out there, butremember what we said about ninjas and resourcefulnessthe easiest one turned out to be a bloody spreadsheet program, because a spreadsheet is all a .dbf file is, really, and all modern spreadsheet programs know how to read and write them. You could very easily rely on OpenOffice for this, but we just used Microsoft Excel because it happened to be on the machine we were using at the time. Load borders.txt , set the delimiter to the pipe character, import all columns, and then save in DBase IV format to borders.dbf . We had to do this twice, because DBF is a fixed-width format, and MS Excel decided to chop each column at the width that it defaulted to on import (i.e., the width of the column name), which truncated some of the country names. The fix was to select each column in Excel in turn, go to Format Column AutoFit Selection, and then save the whole thing in .dbf again. No problem. No doubt if we had used more suitable tools for dealing with DBFsuch as, say, Perl's DBD::XBase this wouldn't have been an issue.

Next, we need to put the file where GRASS can find it, which is usually $GISDBASE/$LOCATION_NAME/$MAPSET/dbf/ . Our database directory is /home/sderle/grass , our current location is called global , and our current mapset sderle , so, by default, GRASS looks for the DBF file in /home/sderle/grass/global/sderle/dbf/ , although we could put it somewhere else if we really wanted to. We choose to go with the default, copy borders.dbf into that directory, and then inform GRASS using v.db.connect :

GRASS:~/hacks/world >

v.db.connect borders driver=dbf table=borders

WARNING: The table <borders> is now part of vector map <borders> and may be
         deleted or overwritten by GRASS modules.

Our work here is done. v.info -c borders gives us the output we expect:

GRASS:~/hacks/world >

v.info -c borders

Displaying column type for database connection of field 1:
INTEGERCAT
CHARACTERFIPS
CHARACTERNAME

Ordinarily, we might verify that the attribute data was correctly assigned by running d.what.vect and clicking around on the X display monitor. Another, somewhat more roundabout, way of testing this would be to pick any country (say, Spain), zoom in on that country, use d.vect to display only that country, and see what comes out:

GRASS:~/hacks/world >

d.erase

GRASS:~/hacks/world >

d.vect borders type=area where="NAME = 'Spain'"

If all of Spain, and only Spain, is what shows up, then we can be more or less confident that the attribute data has come through correctly. Figure 6-54 suggests that this is indeed the case. The princess is saved, the villain in ruinsin other words, success!

Figure 6-54. Spain, by way of testing the attribute data from our vector map of the world

6.16.4. The Path to Ninjahood

The path to becoming a true ninja of any kind is fraught with danger and pitfalls. Following that path demands flexibility, ingenuity, and clarity of mind. When it comes to mastering GRASS, one must always bear in mind the Unix tools philosophy: the idea that an operating system, or a GIS, is best made up of a myriad of small, interconnectable tools, each of which should do one thing very well. GRASS's integration with the Unix shell environmentand all of the tools that come with itmeans that, in the hands of a capable ninja, GRASS is the hacker's GIS, and easily the equal of any fancy, flashy, graphical GIS environment you could care to stack it up against. Remember the example of the ninja, and you too will in short order be using GRASS to make neat work of your cartographic and geographic problems.