Working with Scientists

Working with scientists can frequently be challenging, and requires a fair amount of patience. Here are some things to consider:

Scientists who use HPC and the people who build and maintain HPC systems have a symbiotic relationship. It's easy to get caught up in the "bigger, faster, better" cycle of building clusters and lose sight of the goal, which is to advance science. Without the scientists, there's no need to have the cluster at all.

Funding for research has been dwindling for decades. The scientists who are still around are the ones who have aggressively pursued funding, with aggressive proposals and aggressive schedules. For a modern-day scientist, that aggressive pursuit is what keeps them employed, and keeps their students and HPC administrators employed. Have some patience with them and give them a little latitude. They're aggressive for a reason, and that reason keeps you employed.

Approach every interaction with your user base with "How can I help?" instead of by pushing back. Everyone is busy and overloaded with work, including your users. Take the time to work with your user base; you can often solve multiple problems or requests at once if you can collect them together to find some commonalities.

Try to get your users to focus on problems, not solutions. I've frequently been approached by users who tell me "We need to buy a Banana 9000 as soon as possible or I can't do my work!" This is usually because their colleagues have been using one for their research, and your users decide that they need the same thing in order to keep up. If you can get them to focus on the problem instead of a solution, you can respond with "Well, the Banana 9000 used to be the standard for the problem that you're trying to solve, but the new Orange 3500 can solve the same problem in half the time for a fraction of the cost." You may still lose the battle (they control the funding, after all), but at least you've given them options and demonstrated that you're keeping up to date on new technology.

Meet with your users regularly, and form an advisory panel. Your users are a lot easier to work with when they know what's going on, both good and bad. Tell them what changes are happening, solicit their feedback on what's most important to work on, what projects they have coming up, etc. Keep an internal web site with an overview of what changes are happening now, and what is planned for the future. Show them metrics from your computational resources. Show them the number of jobs that have run, average run times, cluster utilization, how the utilization is growing over time, storage capacity, etc. Your users want to know what's going on, and informing them helps them to understand your workload.

Your advisory panel is a smaller group of people who can represent the user base as a whole. This group should have representatives from all concerned parties (projects, disciplines, research domains, whatever unit works for your organization). It's significantly easier to reach a consensus with 5 people than it is with 50. The added benefit is that when people complain or question the direction that things are moving, you have a group with the authority to make decisions to back up what you're doing. Keep your advisory panel informed and happy.

Hopefully, you're sufficiently humble now. With all that said, you should be the expert in HPC in your organization. Many of your users will keep track of what's going on in the industry and with other research facilities. You should strive to know more than they do, and not be taken by surprise when they bring up an HPC topic. That may be an impossible goal, but it is still what you should work toward. Your users are experts in their field, and you should be an expert in yours. You should know better than your users do how to help them the best.

A few years ago, Andrew McAfee (MIT scientist) was on the talk circuit, promoting his book The Second Machine Age. As part of that talk, he covered the topic of HiPPOs vs. Geeks in terms of decision making. In short, a HiPPO is the "Highest Paid Person's Opinion," and Geeks are... well, the people who actually work with the technology, benchmark it, and make recommendations based on hard data. There conclusion was that while HiPPOs traditionally make decisions based on their "gut" feeling, Geeks typically make better decisions based on real data. A study of this effect produced remarkable results, showing that decisions based on real data are usually "better" (based on rather complicated definitions of "better") than decisions made based on the HiPPO's gut feeling (although, this won't surprise any of the Geeks). This study has been repeated many times, with similar results (see this paper on PubMed for example). This is a worthwhile concept to gently introduce to your users and management.