The greatest test tool for rootkit installation is anti-rootkit software.
ProcessGuard (www.diamondcs.com.au/processguard) and AntiHook (www.infoprocess.com.au/AntiHook.php) are great for installation testing. F-Secure Blacklight (www.f-secure.com/blacklight) is also recommended, though Blacklight can be used to test much more than just installation techniques.
Keep in mind that even the best rootkit will eventually be detected. Moreover, clever users will always be able to find ways to get around even the most persistent software. Taking a statistical approach to this problem will help. If your goal is to develop Digital Rights Management software to prevent the proliferation of freely shared music, filtering 80% of all music file transfers might be a reasonable goal. In the end, you will run into the point of diminishing returns, whereby the effort you put into prevention is not worth the additional prevention provided.