7.3. Other Tools
When trying to find out what is going on with your server, many times you have to use an external tool. There are tools to help you debug from the client's perspective, from the server's perspective, and finally from the network's perspective.
7.3.1. Client Debugging
When you are not sure if the problem is in the client or the server, simply using one of these tools can really help you rule out the client.
FireBug A very nice tool that lets you view nearly everything about the requests in FireFox.
curl A good command-line tool that you can use to snap headers and data from a server and try out quick parameters to see if what you're getting is really what a browser should see. Try -I to see the returned headers (a nifty way to verify which Web server is getting hit, Mongrel or your front-facing one).
RFuzz An HTTP client with fuzzing and client testing capabilities that Zed built to help test Mongrel (and also to see just what kinds of crap well-known HTTP servers and Web sites would accept. You'd be amazed). It is an excellent tool for thrashing your own application to find holes, and it happens to be a tight little HTTP client should you want one. It is as compliant an HTTP 1.1 client as Zed could build.
 Fuzzing is a technique where you try to break applications by handing them mountains of random data. It works amazingly well.
httperf As well as being able to do performance testing, httperf can help you spot stability problems while under heavy loads, and simulate user sessions.
7.3.2. Server Debugging
If you know the problem is with the server, you may still not know what component is responsible. It could be Mongrel, the fronting Web server, load balancers, the operating system, or a bad network configuration. These tools help you test servers in various ways.
Lsof "list of open files" tells you which processes have which files, sockets, domain sockets, etc. open on your server. It will also tell you what state each socket is in and which process owns it. It's kind of slow and only really works right when run as root, but it's a great way to find out who's leaving files open and what files those are. The main benefit of lsof over something like netstat is that lsof works the same everywhere.
netstat Although usually very specific to each particular OS and even different OS versions, netstat will dump tons of different information about your system's network configuration, including routing, packet transfer statistics, open ports, and who's currently connected to which processes.
strace You gotta be desperate to turn this on, but if you see a process that's hung or eating all of your CPU, then running strace (also known as truss on some systems) against it will print out the system calls the program is making. Many times this is all you need to figure out what's causing the problem.
7.3.3. Network Debugging
When you have no evidence that the client is causing the problem, and you are convinced the server is configured correctly, then you either blame green Pygmy elves or you break out various network debugging tools.
traceroute It's an old one but good one. Don't waste hours or weeks trying to find out why a server is slow before you run this tool and make sure the pipes are all running well.
mtr A nicer little graphical version of traceroute is "Matt's Traceroute". Works as a GUI tool and a curses interface. It also will calculate better statistics to show you where links in the routing chain are most likely slow or dropping packets.
 Not Matt Pelletier, a different Matt (there are a few of us around).
tcpdump If you're not sure about the traffic going to your various servers then you can have tcpdump print out information about packets being transmitted on the network. This is a quick way to spot problems with routing, connections, etc. since tcpdump only logs what it sees.
Wireshark (Ethereal) The absolute best network analysis tool out there. It's changed names, but Wireshark is an incredibly good graphical tool for analyzing network traffic. Think of it as a super fantastic tcpdump. You can capture packet traces, save them to files, use filters to reduce the information, follow TCP streams to see full conversations between peers, and decode entire protocols.