Managing a modern HPC cluster
Bright Computing VP of marketing Lionel Gibbons explains how the use of cluster management software can
help to manage data centre resources H
igh-performance computing (HPC) has seen the introduction of many new technologies in recent years – from heterogeneous computing to the
cloud, big data, and even deep learning. All of these changes can be challenging, particularly for systems administrators accustomed to traditional HPC clusters. Te task of managing a data centre has
become much more important – and much more complex. Fortunately, if you are responsible for managing such a data centre, there is some help to be found in excellent management tools.
Reining in the challenges of cluster management A major piece of the cutting-edge data centre technology puzzle is the cluster. Te term is used to mean different things so, for the sake of this discussion, let’s define a cluster as a collection of computers that are linked through a network that acts as a single, much more powerful machine. Clusters are modular with individual servers (nodes) that are independent, fully-functioning units that can be added or subtracted, upgraded, shared, assigned, and rearranged as needed. Clusters are redundant by nature and, generally speaking, if one node in a cluster fails the rest of the nodes can share the load and continue operating. Why are clusters moving to the data centre?
For many users today, this is because a clustered infrastructure can increase efficiency, enabling new types of workloads to be run effectively at scale. Examples of the types of workloads running on clustered infrastructure in modern data centres include: 1. High-performance computing (HPC) – once the exclusive domain to research labs, HPC is finding its way into corporate data centres where it can be applied to solving all kinds of business problems, not just scientific simulations and calculations.
22 SCIENTIFIC COMPUTING WORLD
2. Cloud computing – perhaps one of the most powerful resources of modern data and information systems, cloud computing creates an environment where resources can be allocated to different applications dynamically. Tese resources may be corporate-owned in a private cloud, or they may be in a public cloud, like Amazon Web Services (AWS).
3. Big data analytics – the uses for big data analytics are diverse: growth algorithms, demand forecasting, clickstream analysers, recommendation engines, fraud detection and more, each benefiting from the technology behind big data analytics. Tese include Hadoop, Spark, and Cassandra.
4. Deep learning – a practical incarnation of artificial intelligence that is increasingly being used to go a step beyond big data analysis. Deep learning is improving video processing, natural language processing, translation, and a host of other practical applications.
Because these are mission-critical workloads, effective cluster design and management is crucial and has to scale to enterprise IT levels. Production applications demand reliable, scalable infrastructure to run on. As more scale- out architectures move to the data centre, the importance of cluster management will continue to grow.
Bigger, better hardware is becoming the norm As the needs of consumers change with time, so does HPC. We continue to see an increasing need for large scale (even exascale in some cases) computing and data-centric processing with high I/O performance metrics. At the same time, tight budgets are driving requirements for better energy efficiency and encouraging multi-use cluster hardware. One way to make the most of the compute
footprint is to adopt accelerators. According to some predictions, more than half of the systems
installed this year will include accelerators, such as GPUs. Additionally, servers with big data capabilities are becoming more affordable making them the standard choice for general- purpose servers and storage clusters.
New parameters, new atmospheres: HPC in the cloud Te term HPC escapes a one-size-fits-all, cut and dried definition. HPC leverages the aggregation of computing resources to solve problems that can’t be handled by a single machine. Of course, the aggregation of computing
resources can take many forms, and each scenario will be a little different from the next. Some only involve traditional HPC workload managers. Others will include Hadoop or Spark for big data. Still, others will require cloud-based servers or a mix of on-premise and cloud-based servers. Te question then becomes: how does your
organisation aggregate these resources in a manner that is both flexible and efficient?
Invest in good software to manage your clustered infrastructure If you are running a modern data centre that incorporates clustered infrastructure, an effective management solution can be a tremendous investment. Let’s take a look at how it can help. Consider the effort involved in deploying,
configuring, monitoring, and managing all of the servers in your clusters. Compare the work involved in setting up something like a Hadoop cluster (with tens or even hundreds of servers), and you’ll begin to see the value that automated provisioning and installing brings. With the right tools, you can deploy a cluster from bare metal, reliably and quickly. A centralised management console that
enables you to monitor and manage your clustered infrastructure is a tremendous time
@scwmagazine l
www.scientific-computing.com
Page 1 |
Page 2 |
Page 3 |
Page 4 |
Page 5 |
Page 6 |
Page 7 |
Page 8 |
Page 9 |
Page 10 |
Page 11 |
Page 12 |
Page 13 |
Page 14 |
Page 15 |
Page 16 |
Page 17 |
Page 18 |
Page 19 |
Page 20 |
Page 21 |
Page 22 |
Page 23 |
Page 24 |
Page 25 |
Page 26 |
Page 27 |
Page 28 |
Page 29 |
Page 30 |
Page 31 |
Page 32
