Deploying multiple database servers seems like an obvious idea, at first. Not only can you configure your Web servers to use more than one source for their data, you also eliminate a single point of failure. Should one database server go down, your Web site stays up.
The problem, however, is that databases are not a read-only device, as are Web servers. Requests made to them are frequently to update, delete, or insert data, not just query it.
All is not lost, however. As long as whatever appliance you are using is "conscious'' of this fact and can examine each request to determine whether it is a read or a write, load-balancing database servers can indeed be achieved.
This packet examination is crucial, however; a write request must be relayed to all servers, not just one. Should any server be unable to receive that write request, it must be buffered and replayed to that server as soon as it comes back online, lest each server's database get out of sync.
Bigger-name databases such as Oracle and SQL Server have had a solution for some years now. Thankfully, PostgreSQL and MySQL have caught up, too. Getting replication set up is a huge topic, but these Web sites have the detail you need: http://gborg.postgresql.org/project/slony1/projdisplay.php for PostgreSQL, and http://dev.mysql.com/doc/mysql/en/Replication.html for MySQL.
Hooking all this equipment up might seem to be the simple part, but a bit of planning in this department can save you grief and heartache later.
Consider the traffic going across each interface and try to plan around that; for example, a database-heavy Web application will have enormous amounts of traffic going to and from the database server, traffic that is completely unrelated to the Web request being made by the client Web browser. With this in mind, you may wish to consider having a separate interface on each Web server linking into the database server.
Such a configuration is common; you have a core switch into which all your Web servers link and into which your firewall's internal interface will link, too. Your database switch is linked to the secondary NIC of each Web server and to the NIC of each database server you use.
As a result, traffic between Web servers and database servers doesn't interfere with true Web traffic a performance increase. If you are using a high-end switch, you can even use a VLAN to separate, say, all 24 ports into two virtual switches of 12 ports each one for the Web servers and firewall; one for the database servers and secondary Web server NICs.
A side-effect of this configuration is yet another layer of protection for your databases; with no exposed interface to the Internet, they are yet another step away from being compromised.
Having redundant Web servers and database servers in place is a good idea to ensure a high-availability application. But using redundancy in other areas can actually prevent the loss of data perhaps even more important from a commercial point of view.
With this in mind, if you are given the option when specifying servers, always request a RAID configuration of SCSI disks or, in a pinch, IDE RAID.
Furthermore, do not be tempted to opt for maximum disk space with a RAID 0 stripe. Instead, go for RAID 5, which gives you as much disk space as you have disks, less the capacity of one disk. This gives you redundancy in that if one disk fails, the others should kick in and your server will stay up. Simply take out the faulty disk, replace it, and the RAID controller will bring it up to speed. There's also a big performance increase to be had from using RAID highly recommended.
RAID is an enormous topic; for further reading, check out http://www.uni-mainz.de/~neuffer/scsi/what_is_raid.html.
While the data resilience theme is still in mind, you'd be wisely counseled to devise a coherent backup policy for your database servers and any data directories used by your Web servers. A tape backup is sufficient, but ensure that your data center is briefed to rotate the tapes on a daily basis. Talk to your data center about the various tape rotation services and other backup facilities they can provide, and choose one most appropriate to your project and budget.
Often, your ISP will offer to perform basic maintenance on your servers. It is very much up to you and the relationship with your client as to whether you opt for this or decide to take care of it yourself.
If you do decide to bear the burden, then you should devise a weekly action plan for server maintenance. This should be a list of tasks to perform weekly to ensure that your servers are clean, efficient, and up-to-date. Tasks for your action plan will include:
Kernel version is each server running the latest version?
Apache and PHP versions is each server running the latest version?
PHP dependencies libxml, libxslt, expat are they all up-to-date?
Disk space does each server have enough space on each partition? Are any log files growing out of control and needing trimming? Apache, in particular, will not allow you to exceed 2GB in size for any given log file and will crash quite spectacularly if you do.
Finally, you should include in your systems architecture a coherent plan for securing your entire network against unwanted intruders and attackers.
A firewall is absolutely essential. There is absolutely no excuse whatsoever for directly exposing servers on the Internet, no matter how secure or up-to-date they may be. You are simply asking for trouble.
Opt for an appliance firewall, if possible, rather than simply a Linux or BSD box with firewall software loaded. The less sophisticated its inner workings, the less chance it has to be compromised. A good choice would be one of the Cisco PIX series of firewalls. For a small site, the $300 PIX-501 will be more than adequate.
Don't be tempted to simply enable some kind of on-installation firewall on each server. This is inadequate for proper protection. A separate appliance is required.
One final hint: don't forget to enable outbound Internet access via NAT for your Web servers. If your application needs to send e-mail, your Web servers will need to be able to make outbound SMTP connections on port 25.