Have your business applications given you performance headaches at some point during their lifecycle? Maybe you resolved the problems and the application was able to keep servicing your customers and performing its business function for you.But no doubt there were other times when you tried to solve the performance problems internally, but sadly-and expensively-you didn’t succeed. To keep future hassles away, you probably (a) added more hardware or, worse yet, (b) started a server recycle program to minimize the damage to your production.As a result, the performance is still sub-optimum, your company is losing money, and your customer satisfaction keeps sinking. To get your application back on track, you’ve decided it’s time to bring in a professional performance consultant.You’ve made a smart decision. But now you face the challenge of finding the right person. You must find someone who can quickly identify the performance problems, resolve them, and implement a plan to bring stability to the application.But that’s just for starters. You also need someone who’ll work well with your team and can put standard operating procedures in place to stabilize the current environment and prevent future problems. Plus, this person must be able to transition the best practices to your team clearly and effectively.As you can imagine, it’s tough to find all these qualifications in one person. Performance consultants are not all created equal. When the time comes to look for your professional performance consultant, here are the 10 key things to keep in mind:1) SOLVABILITY: The performance troubles you’re facing are common, and cost many companies money. The good news is these problems are solvable. But companies often have a consultant look at an application and he or she decides the problems are just too complex or aren’t solvable.For example, one company had a software pricing engine that was critical to the business: if the pricing engine was down, the company lost revenue. The CIO admitted that to remedy these problems, they’d have to add more servers. The standing architect said he’d need 240 new servers to handle the load volume.I came in, did some testing and found that the performance troubles stemmed from the amount of memory the current application was using. My options in this case were limited: I couldn’t rewrite code or change architecture, but I was able to change the Java Virtual Machine for the poorly performing application. This alternative JVM was more forgiving on memory consumption. With new settings, the company was able to scale back from buying the 240 new servers to only 10. Imagine the cost savings!RULE: Performance problems are common and above all solvable. Sometimes the creative solutions are the ones that offer the most cost effective results.2) FUNDAMENTALS: A performance consultant must possess a certain amount of common knowledge. Ultimately, you’re looking for a specialist in identifying and resolving performance issues. This person must be well-rounded technically. Here are some basic qualifications to look for:a. JAVA: the foundation of the application server. Don’t hire someone to identify and solve Java Application Server issues if they don’t have a strong foundation in JAVA. At a minimum, they must understand threads, know how Java uses memory, and be able to read stack traces and write test cases.b. Networking: includes Load Balancers, Network Interface Cards (NICS) on the boxes, firewalls and anything that’s responsible for routing traffic to your applications. Your consultant must be up to speed on these.c. N-Tier: These architectural designs add complexity to any system Java Application Servers. Having solid experience with other n-tier applications will help the consultant look at your big picture.d. J2EE Specifications and Standards: having someone who understands the J2EE specifications is a must.e. Operating Systems: make sure the consultant has experience with your platform. If you’re UNIX, make sure they have Unix skills.f. Database: At a minimum, your consultant should understand Structured Query Language (SQL), and be able to identify long-running queries with your database version.RULE: Understanding basic fundamentals will enable your consultant to adapt to your unique environment quicker and cure your ills in a shorter time.3) SKILL SET: Systems are so complex that it’s understandable why companies bring in the wrong consultants to find and fix performance headaches. The problems could range from a code bug, a vendor code or a tuning issue to technical architecture troubles and sometimes an application architecture issue. Consultants have varying levels of skill and expertise. Some are strong in architecture and others excel in systems. They could have a development background or something totally different.Before you start interviewing consultants, consider nailing down the skill set you’re looking for. I once had a customer who hired a consultant with a strong architecture background to come in and look at the company’s performance problems. After doing some analysis, the consultant told them the architecture for the application was wrong and that they’d have to re-architect their systems. I took a look at the same system and found the trouble: a combination of application settings and a bug in the underlying operating system. Once I fixed those, the system performed much better and didn’t need re-architecting.If you hire a specialist in a particular area, expect the recommendations you get to reflect their field of expertise. Understand what you’re looking for before you open your door to any consultant.RULE: Identify the focal point of where the problems are, and then hire the best generalist to help you find where they are.4) BIG-PICTURE THINKER: One of the advantages of bringing in a consultant is seeing your operation from a new and different point of view. Plus, it’s a great way to learn the latest about what other companies are doing or new trends that are changing your industry. These insights can enable you to see the big picture and how your environment(s) compare with others’. A big-picture thinker can help you understand how other companies are managing their Java Application Servers.The true big-picture thinker can look at your environment and help make recommendations beyond your current performance difficulties. You’ll discover solutions for problems related to architecture, systems selection, load-testing capacity and other aspects of the system, such as its technical architecture. This lets you to plan for the future with your systems. It helps you act proactively rather than reactively.RULE: Find a consultant who’s a big-picture thinker, someone who can see beyond your immediate needs and give you insightful recommendations for the best-practices approach to running your applications.5) GOOD BEDSIDE MANNER: One of the most valuable assets of a top consultant is excellent people skills. Production problems put extreme stress on people, so when you’re bringing in a consultant, it’s vital to find one who can communicate effectively and build instant rapport with your team. Getting a handle quickly on what’s happening with your system requires good communication skills all around. This means the ability not only to communicate with others but to interact well with them and build a positive relationship with your team. You need someone who can come in and quickly add value to your team.I’ve lost count of how many times I’ve heard someone complain that a consultant simply came in, asked some questions and produced a report. This benefits the company hardly at all, and will leave a team working to bring stability to the system feeling alienated.RULE: You want someone technical who can diagnose your situation, but most important, be able to communicate those findings well to the team they’re working with.6) SELECTIVITY: It’s better to hire no consultant than a bad one. Hiring the wrong person for the job can cause you more problems than solutions. A good rule of thumb: if you have any reservations about the consultant, then follow your instinct and don’t hire them.Sure, you’re under pressure to clear up the trouble, but think about who you’re bringing in to do it before you give the green light. Not only are you bringing in a consultant to tackle performance problems, but you’re also going to need one or more dedicated resources to assist. If you adopt the recommendations of someone you were wary of to start with, you might end up with more headaches than if you hadn’t brought anyone in.RULE: Make sure the consultant you choose is technically qualified to address your particular problems. Be picky and get the best consultant.7) DOMAIN EXPERTISE: Besides the ability to troubleshoot, customers usually look for a specific application skill. It’s actually a mistake to narrow your focus only to such skills. The engineers might focus on a specific application, but what you really need is a different skill set to help turn the situation around. You need someone who can think outside of the application to bring light to the system as a whole, not just the application in question.You probably haven’t found the application skill set for the problems you’re having, or the performance-troubleshooting skills, either. Application specialists might have the business logic understanding, but most of them are light on the troubleshooting aspect.RULE: You’ve probably identified the right skill set(s) to handle technical or business problems with your application. Consider looking for a consultant who can augment your internal knowledge.8) MANAGING EXCEPTIONS: Suppose you’ve narrowed your field of consultants and have found the perfect one for the job. You’re ready to bring this person on board. Make sure you’ve both explained and documented your expectations with the consultant before starting the engagement. You must identify, up front, the deliverables and artifacts that you expect before the work starts. If a report or a recommendation is due, make sure you negotiate this up front. Have regular checkpoints during the engagement to ensure you’re getting what you expect.If you expect any deliverables, bear in mind that this usually adds time to an engagement. Allocate time during the engagement, with checkpoints, to finalize reports and deliverables.RULE: Manage the expectations up front by determining exactly what the consultant will do. Specify verbally and in writing any deliverables and checkpoints that the consultant must meet during the engagement.9) SOFTWARE SOLUTIONS: Customers are always buying expensive software, hoping this will solve the performance problems on their systems. Be wary of consultants who want to install software they say will cure the ills in your environment. The truth is, installing a software package usually adds overhead to the system and opens the door to increased instability.If a consultant needs to install software to help identify the issue, limit the footprint to one server in the cluster or to a non-production environment first. In some cases, you do need to install software. Consultants might also need to install tools to find the root cause of a problem. Those tools are only as helpful as the person who’s going to analyze the data. Installing the software won’t necessarily lead to the correct action.RULE: Software packages add overhead and can actually worsen instability. Remember, once you let a consultant install the software, you’ll l also need someone who can interpret the data and take action on them.10) PROCESS: Solving problems in any environment will give you some immediate relief from performance headaches. There are usually multiple reasons for performance problems. Imagine you resolve an issue and send the consultant home, having new confidence in your system. Then two months later you launch a large marketing effort and find yourself with your site down again. Or imagine you brought the consultant in during non-seasonal loads and are now in the middle of your busy season.Solving the problem is not the most crucial goal of the engagement. The most crucial goal is to understand how the problem was solved. What tools did the consultant use? Is this something you can build into your best practices to minimize performance troubles early in the development lifecycle? And how can you put a process in place for continual improvement?RULE: Create a process to improve your system continually. Fixing a problem one time only sets you up for your next fire drill.SUMMARYFirst, find a strong technical professional who has superior communication skills and can understand the big picture, in terms of what n-tier application architecture looks like.Second, look for a consultant who isn’t afraid of challenges, has worked through similar problems in the past, and has a well-rounded track record of experience.And third, as I stressed above, it’s often better to bring in no consultant than the wrong consultant.