| Evaluating integrators is just as important as evaluating tools. Evaluating integrators is more difficult than evaluating tools because it's pretty much impossible to demo an implementation process. You will have to rely a lot more on references and other indirect checks. However, the evaluation process is basically the same as what was recommended to evaluate tools: -
You need a requirements list ; that is, you need to define what you expect from the integrator versus what you will provide through your internal resources. -
You need an effective reference checking strategy to evaluate the various points on your requirements list. -
You need a scoring mechanism to evaluate the various points on your requirements list. Your task is to confirm that the integrator has been successful with projects similar to yours in the past, and that the staff that will be assigned to your project will be the same or of the same caliber as staff that participated in past projects so that you can expect the same level of success. Therefore the first step is to be very clear about what your project requirements are so you can identify similar projects and benchmark against them. Integrator Requirements List Building a requirements list for an integrator is very similar to building the requirements list for the tool: you need to define what you want and need, you need to capture it in reasonable detail, and you need to identify the high-priority items. The project team creates the requirement list for the integrator selection just like for the tool requirements list, although usually only a small subgroup is involved. The IT owner usually plays a key role in creating the integrator requirements, especially if the project is mostly focused on technical issues. The integrator requirements include five categories: corporate fit, methodology, technical expertise, experience, and price. They are discussed here together with many suggestions on how to approach each area with the integrator candidates. Corporate Fit Although you don't need to be quite as demanding about the corporate fit with the integrator as you would about the fit with the tool vendor, you do need to check its financial health and strategic focus, especially for larger implementations . Make sure that the integrator will be able to sustain your implementation comfortably through the rollout stage and a little beyond, just to be sure. For instance, the integrator should have a long- term focus on the tool you have chosen . Methodology To be successful, integrators must have a system that is logical and repeatable and that encompasses your requirements, whether they are technical or business-based. -
Does the integrator use a proven methodology to drive projects to a successful conclusion? There should be identifiable stages for defining, executing, and testing with tangible milestones, together with an acknowledgement that the reality is not always as well structured as the theory. Many integrators incorporate at least some so-called Rapid Application Development (RAD) methods , whereby multiple prototypes are shared with the users and refined until a positive and complete outcome is achieved. Such methods accelerate development by making sure developers get feedback early on. -
How does the integrator minimize project risk? For any implementation project you want to identify the high-risk areas early on and get them working and tested first (or at least as soon as possible) so that a backup strategy can be defined and executed in case there is a problem. It's always interesting to contrast the points of view of several integrators on what they think the high-risk areas are. Ask the tool vendor, too! Ask many questions about testing. There should be both unit testing (does it work for one person) and load testing (does it work for bunches of people). -
How does the integrator elicit and maintain support from the business owners ? There should be mechanisms in place to ensure good communication both on a regular basis and on an exception basis. Look for a mix of formal and informal communication techniques. Chapter 9 includes practical suggestions so you may want to read ahead for inspiration. -
Can the integrator handle process definition and implementation? Is the integrator familiar with best practices in the areas that you are considering? Can the integrator handle detailed task analysis if that's the level of detail you need? This is usually an easy item to evaluate. A short conversation with the integrator staffers should tell you whether they have heard about consultative selling, issue routing, or whatever other business process area you are concerned about. Make sure that the integrator is not blindly wedded to a particular way to approach the business process, unless you are yourself convinced it's the only way to go. -
Can the integrator help you with change management? Change management is a part of each CRM project since by definition the CRM tool will be a change. However you may have particularly high requirements in this area if there will be significant process changes, if there is internal opposition to the project, or if there are lots of other changes happening at the same time. Integrators that flatly state that you should be handling all change management issues have a purely technical approach to implementation and may not be right for you. Thoroughly review this particular point when checking references if you have any concerns about it. -
How does the integrator measure success? Look for meaningful business metrics being defined at the start of each project and measured throughout. Metrics that start and stop with the project (for instance "deliver the project on time and on budget") are good but not sufficient. -
How does the integrator handle post-deployment issues? Most projects need tidying up after deployment, beyond the expected handholding required for the rollout, even if the project was successful overall. Experienced integrators plan for that so that required hardware adjustments or software changes can be handled properly. Beware abrupt departures on the day of the rollout. Will you need long-term support? Some, but not all integrators offer full-blown support services. -
Is documentation developed? What kind? How is knowledge transfer handled for system administrators? Since most integrators do not provide long-term support, it's essential to transfer knowledge to the IT staffers who will administer and support the system after implementation. Integrators usually create some written documentation, which ranges from sketchy to robust, but it's best if the integrator has some way of getting the system administrators meaningfully involved and briefed along the way rather than relying on a hasty discussion at the very end of the project. -
How does the integrator handle end- user training? Some integrators don't do it at all, which is fine, but in that case they need to build in time and a process to provide information to the curriculum developers. Technical Expertise Methodology isn't everything; the integrators also need to have the technical resources that you need. -
What is the background of your project managers? How do they approach their work? What are typical items that the project manager handles and which items doesn't he or she handle? In many ways, good project management is the key to a successful project, so make time to understand what project managers do and who they are. Ideally , you want to meet the project manager that will be assigned to your project ahead of time. Sadly, project managers are usually assigned late in the game so you won't really know who the project manager will be until he or she starts. You should be able to make a pretty good assessment based on the project managers you meet during the evaluation, however. Make sure you understand whether project managers manage the entire project or just the technical side. Probe both their project management skills and their communication skills. -
What technical resources are available to the integrator? Are the resources employees or subcontractors ? What skills and experience are present on the team? If you find it difficult to determine who the project manager will be ahead of time, you will probably find it impossible to find out who any of the other individuals will be, so here again you will have to make do with overall impressions . Integrators often use subcontractors so probe about the type and quality of subcontractors used. If an integrator subcontracts out all work of a particular type, question why they choose not to do that type of work in-house. Specifically probe for the technical areas that are required for your project. If you are integrating with SAP, look for SAP expertise. If you are doing a wireless project, look for experience in that area. Don't assume anything and ask about network, database, operating system, etc. that match your configuration. -
How long have employees been with the integrator? Skilled resources have many appealing choices when it comes to employment, and consulting is a tough way of life, in particular because of the travel requirements. If the integrator you are considering is managing to keep employees for a long time it's a good sign that things are going well there. Also, long-tenured employees have more experience and that should be good for your project, too! Experience A good methodology and a good set of skills are critical ingredients , but can the integrator combine them into successful projects? Look for integrators who have recently completed projects using the same tool, with the same level of technical complexity (customizations and integrations). They should have worked with a comparable number of users and a similar distribution of those users. Consider whether they have done an implementation for an organization in a similar industry, that is, retail if you are doing retail, selling complex products to Fortune 500 companies if that's what you do, etc. Ask the integrator for examples of similar projects. If the integrator suggests projects that are widely different from your requirements, this may be a sign that the integrator doesn't understand your business and you need to keep looking. The next section, about checking references, goes into more details about how to gauge similarities between projects. Price Price is an important element of selecting an integrator. Because integrator pricing is less flexible than tool pricing, you should bring up pricing earlier in the evaluation process. -
What is your target price? There's no sense in evaluating an integrator that's way outside your price range. Although no serious integrator will give you a firm quote early in the game, you should be able to get ballpark estimates once you have defined the tool and a high-level strategy. -
How does the integrator determine pricing? Some integrators offer fixed-price but many do not. If you have a preference, better find out early on. We'll talk about price negotiation strategies in the major section Evaluating the Integrator The process for evaluating the integrator is similar to the process for evaluating the tool. In particular, you may choose to use an RFP. There are some differences, however. Many items on your requirements list will require checking references since they are not as tangible as the items on the tool requirement list. (You should be able to eliminate completely unsuitable candidates before the reference check.) Also, since the consulting business has a heavy communication component, trust your instincts when it comes to communicating and customer service. If a particular integrator candidate is not delivering in that area before the sale, you can be sure that things won't improve appreciably after the sale. Checking Integrators' References Selecting References Selecting references for the integrator is very similar to selecting references for the tool. Actually, it would be ideal to use the same references for both, since it would save you time and also make for stronger, all-around references. Ask the tool references who did the implementation for them: you may get good ideas, or at least interesting feedback. Look for references that have gone through similar projects reasonably recently. "Similar" means that the projects should be: -
For the same tools . The main reason to go outside to staff an implementation project (beyond the requirement for bodies) is to benefit from the expertise of the individuals you are bringing in. Therefore, for me, successful implementation experience with the very tool or tools you are selecting is not negotiable. -
For roughly the same technical complexity . It's not easy to gauge the technical complexity of a project before you start it, but as a rough estimate you can compare the level of customizations and the number and type of integrations. It's also very useful to consider projects for companies with similar business models since that has important consequences for CRM systems. -
With equivalent business consulting requirements . For instance, if you need process definition assistance, check that the integrator has successful experience there. If you want the integrator to handle end-user training, they should present you with a reference for whom they handled it. -
For about the same number of users , and, ideally a similar distribution of users . Deploying a system for 20 users is completely different than for 300, both in the infrastructure requirements and in the level of customizations that will be required. A mismatch is just as bad if the integrator usually works with much larger projects. I was once a part of a tool implementation for a smallish company that decided to work with what was then one of the "Big Five" consulting companies. The integrator had plenty of experience with similar projects, but all of it was gained while working with much larger organizations. Although the integrator deployed a very much scaled-down team for that particular project, their approach almost overwhelmed the company's staff with an overly formal and thorough methodology. The project was successful in the end, but it was much more painful than it could have been. -
In production . The proof is in the pudding. It can be useful to talk to references that are still in the implementation process, in particular to gauge how best to work with the integrator during the implementation, but you must talk to references that are in production to confirm that the process is indeed successful. Make sure that reference has been in production through business peaks such as quarter ends or the holiday season , depending on your business model. Reasonably recent implementations are best, since the tool won't have changed too much since them. Integrators should be able to supply you with references. If the integrator cannot come up with production references for the same tool, complexity, and number of users, it's a red flag and you should select another integrator. Asking the Right Questions When you talk to the references you will want to check the items from the requirements list that you could not confirm directly. You also want to confirm all aspects of the implementation projects they went through with the integrator. The questions below give you a structure for organizing the questions. I like to ask the same questions to the integrator and to the reference and I compare their answers. If the integrator tells you that the project was on budget, but the reference disagrees, check what the size of the difference might have been. Perceptions are important for consulting work so any gaps between the answers are potential flags for you. -
Describe the project the integrator worked on for you. This is a calibrating question to make sure that the project characteristics are reasonable matches for yours in terms of technical complexity, user base, and the types of services the integrator provided. Was it a team effort with the internal team? Was the integrator asked to lead a process definition effort? If the project is significantly different from yours, thank the reference and check with someone else. -
When did the project take place? Check that the project is now in full production but still reasonably recent. Several months of successful production are ideal as long as they include a business peak. Don't waste time checking references with an account that is not in production yet. -
Did you participate in the implementation? You want to talk to someone who was on the implementation team, preferably the business owner or the IT owner ”ideally both. If the reference no longer works for the organization where the implementation took place, go through the reference check but ask to talk to someone within the organization to confirm that the tool is still in place and functioning properly. Don't bother checking references with someone who did not participate actively in the implementation. Such an individual will not be able to respond to most of your questions anyway. -
Describe the overall implementation process. Listen carefully for signs of an orderly process that matches what the integrator described to you. If you hear something quite different from the integrator's description, go back to the integrator. It could be that it is now using a new process (which may be a good thing, although you should decide whether you want to be a guinea pig) or it could be that the neatly described process is for pre-sales use only. The latter is a potential red flag. -
Was the project on time? Accept that all projects are a little late, and the delays are not always caused by any problems on the side of the integrator. I would not worry about a small delay, say 10% of the total length, and I would not worry about even a large discrepancy if there is a "good" reason for it, such as the customer putting a temporary halt to the project for financial or other business reasons. On the other hand, delays caused by unforeseen, last minute technical issues are a red flag. Should the integrator have predicted the problem and tested that particular part of the project earlier? Also probe delays caused by political issues, since a good integrator should be able to skillfully manage the client and to anticipate when the business owners have not bought into the project, or the executive sponsor is flagging, both common causes of project delays. Probe how well the integrator handled the delay. Were efforts made to mitigate the delay? To work around issues? To communicate with the client to explain the delay and the action plan? -
Was the project within budget? Here again, expect some overage and remember that clients often choose to add features midway through projects. Find out whether the integrator pushed back on scope creep. It's usually better if large, newly discovered requirements are scheduled for a second phase. It shows that the integrator did a decent job of gathering requirements in the first place so that the last-minute candidates are not absolutely required for the initial rollout. It also demonstrates that the integrator has the skill and strength to manage to an orderly process. -
How was the communication process? What worked and what did not work? Most implementation projects are long and suffer at least one major problem, as we will see in the next chapter. Therefore, good communication is essential. If the report is particularly positive, find out who the project manager was and try to get the same individual to work on your project. -
What worked very well? Why did it work well? Note any characteristics that are meaningful to you. For instance, if the references are praising the benefits of a kickoff workshop while you don't think you can bring your team together for such an exercise, it may be the proof that you just have to make it happen. As another example, if the references liked a very hands-off approach but you feel you need close ties with the integrator, you may want to rethink your approach or find another integrator who would provide a more hands-on approach. -
What did not go so well? What would you do differently next time? Most references don't like to say anything negative, so gently probe until you get an answer. Pay attention to statements such as "The integrator wanted to do X but we did not, and in the end we should have followed their suggestion." Was the integrator clear enough in presenting arguments? Did the integrator simply fold under pressure? You want an integrator with a bit of a backbone. If an item is critical, a good integrator should know to push back even if the client is persistent. Figure out how much pushback the integrator managed. It's a red flag to hear something like "We wanted to do X, but the integrator refused and then we had to do it over." -
Anything you did not implement that you wish you had? What was the involvement of the integrator in this particular decision? Try to determine if the integrator should have foreseen the situation, and how the situation was handled if the gap was discovered prior to the end of the project. -
Anything you did do that you now think was useless? What was the involvement of the integrator in this particular decision? If anything pops up here that you are thinking of doing yourself, make a note of it. Although each implementation is unique, you may be able to take something off the list, especially if the reference is very similar to you. Again question whether the integrator should have provided better advice on the decision. -
Did you hit any problems during the implementation? What were they? What did you do to overcome them? Unless the project being referenced is extremely simple, insist on an answer to this question. I have yet to encounter a single implementation where there was no problem whatsoever. Most implementations encounter at least one large problem, so the question is not whether a problem occurred, but rather whether it should have been forecasted, and how well it was handled. Did the integrator take an active role in finding a solution, even if the problem was not something it created? Was the integrator creative? -
Anyone particularly good on the team? Anyone weaker than the rest? Take note of the individuals that are particularly recommended and try to get them assigned to your project. Be warned , it's a difficult task since the stronger people tend to be in great demand. If the stars have moved on, find out why, since the strong people tend to abandon ship first when there's trouble aboard. Specially request that weaker individuals not be assigned to your project, at least if their weaknesses are relevant to your project. Be sure to ask about subcontractors as well. -
Did you participate in selecting the integrator? What criteria did you use for the selection? Who were the other candidates? Perhaps you are considering the same candidates and, if so, you could learn much from how the references ended up selecting one over the other. You may even end up making the opposite decision they made because your requirements are different. -
Do you think you got good value from the integrator? Why or why not? Comparing prices with the references is not that useful since each project is different. Talking about value is more helpful since it focuses on the outcome. Explore where the value could have been greater and adapt your project accordingly . -
Would you use the integrator again on a new project? Why or why not? This is a classic reference-checking question, but always useful, especially when you hear the reasons for the choice.  |
