Eliminating Duplicates with uniq


Eliminating Duplicates with uniq

If you've sorted files using the handy-dandy sort command, you might end up with results that have duplicates in them. Heck, you might have files with duplicates. At any rate, here's how to find and work with them. As Code Listing 6.18 shows, you can get rid of duplicate lines by using the uniq command (short for "unique") in conjunction with sort.

Tips

  • uniq finds only identical, adjacent (sorted) lines. For example, if you have both Jones and jones in your address book, uniq won't identify either entry because they differ in capitalization.

  • You can also use the -d flag to specify that you want to see only the duplicate lines. For example, say you want to see all of the people who are in both your carpool file and your nightout file. You'd just use sort carpool nightout | uniq -d

  • You can sort and eliminate duplicates in one step with sort -u address.book.


To Eliminate Duplicates with uniq:

  • sort address.book | uniq

    At the shell prompt, type sort and the filename, then type | uniq to pipe the output to uniq. The output of uniq will not contain any duplicated entries (Code Listing 6.18).

Code Listing 6.18. Use sort with uniq to eliminate duplicates.

[ejr@hobbes manipulate]$ more long.address.book Schmidt, Sven, 1 Circle Drive, Denver, CO, 80221, 555-555-8382 Feldman, Fester, RR1, Billings, MT 62832, 285-555-0281 Brown, John, 1453 South Street, Tulsa, OK, 74114, 918-555-1234 Smith, Sally, 452 Center Ave., Salt Lake City, UT, 84000, 801-555-8982 Jones, Kelly, 14 Main Street, Santa Clara, CA, 95051, 408-555-7253 Schmidt, Swen, 1 Circle Drive, Denver, CO, 80221, 555-555-8382 Feldman, Fester, RR1, Billings, MT 62832, 285-555-0281 Brown, Jonathon, 1453 South Street, Tulsa, OK, 74114, 918-555-1234 Smith, Sally, 452 Center Ave., Salt Lake City, UT, 84000, 801-555-8982 Jones, Kelly, 14 Main Street, Santa Clara, CA, 95051, 408-555-7253 [ejr@hobbes manipulate]$ sort long.address.book | uniq Brown, John, 1453 South Street, Tulsa, OK, 74114, 918-555-1234 Brown, Jonathon, 1453 South Street, Tulsa, OK, 74114, 918-555-1234 Feldman, Fester, RR1, Billings, MT 62832, 285-555-0281 Jones, Kelly, 14 Main Street, Santa Clara, CA, 95051, 408-555-7253 Schmidt, Sven, 1 Circle Drive, Denver, CO, 80221, 555-555-8382 Smith, Sally, 452 Center Ave., Salt Lake City, UT, 84000, 801-555-8982 [ejr@hobbes manipulate]$ 




Unix(c) Visual Quickstart Guide
UNIX, Third Edition
ISBN: 0321442458
EAN: 2147483647
Year: 2006
Pages: 251

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net