In this video from the 2013 HPC User Forum, Burak Yenier presents: The HPC Experiement – Paving the way to HPC as a Service.
For the 2nd Round of the HPC experiment, we will apply the cloud computing service model to workloads on remote Cluster Computing resources in the areas of HPC, Computer Aided Engineering, and the Life Sciences.
In this video from Moabcon 2013, Robert Clyde and Chad Harrington from Adaptive Computing discuss the company’s recent announcement that Adaptive has been named as a “Cool Vendor” in Cloud Management by Gartner.
We believe to be recognized as a ‘Cool Vendor’ by Gartner for our cloud management technology is confirmation of our pioneering work in policy-based optimization for this space,” said Robert Clyde, CEO of Adaptive Computing. “Our Moab Cloud Suite allows enterprise IT leaders and cloud architects to maximize cloud return on investment through cost savings in capacity and management complexity. Moab’s ability to perform ongoing service optimization ensures organizations achieve both agility and service performance with their private cloud.”
Trevor Clarke at The Sydney Morning Herald, writes that Animal Logic realizes the potential the cloud can provide when faced with the need for additional resources in rendering visual effects.
Despite this computing power, in the last nine months the company’s workload – it has been working on The Great Gatsby, Walking with Dinosaurs for the BBC and also Iron Man 3 – has meant it needed additional resources, which it sourced from service provider Steam Engine.
Glenn K. Lockwood from SDSC writes that setting up a working set of EC2 instances that have the necessary configuration to run MPI applications can be quite daunting, so he has posted a guide.
Most guides online are kind of unhelpful in that they try to illustrate some proof of concept in how easy it is to get a fully configured cluster-in-the-cloud setup using some sort of provisioning toolchain. They gloss over the basics of exactly how to start these instances up and what to expect as far as their connectivity. Fortunately it only took me a morning to get MPI up and running, and for the benefit of anyone else who just wants to get MPI applications running on EC2 with as little effort as possible, here are my notes.
Over at GigaOm, David Meyer writes that European IaaS provider CloudSigma has abandoned magnetic disks for solid-state storage. After a pilot test of SolidFire’s all-SSD storage system, CloudSigma now feels confident enough to offer a service-level agreement for performance, as well as uptime.
According to CloudSigma COO Bernino Lind, the shift to SSD is a major help when it comes to handling HPC workloads, such as those of Helix Nebula users CERN, the European Space Agency (ESA) and the European Molecular Biology Laboratory (EMBL):
They want to go to opex instead of capex, but the problem is there is no-one really who does public infrastructure-as-a-service which works well enough for HPC. There is contention — variable performance on compute power and, even worse, really variable performance on IOPS [Input/Output Operations Per Second]. When you have a lot of I/O operations, then you get all over the spectrum from having a couple of hundred to having 1,000 and it just goes up and down. It means that, once you run a large big data setup, you get iowaits and your entire stack normally just stops and waits.” Lind pointed out that, while aggregated spinning-disk setups will only allow up to 10,000 IOPS, one SSD will allow 100,000-1.5 million IOPS. That mitigates that particular contention problem. “There should be a law that public IaaS shouldn’t run on magnetic disks,” he said. “The customer buys something that works sometimes and doesn’t work other times – it shouldn’t be possible to sell something that has that as a quality.”
Big data science emerges as a new paradigm for scientific discovery that reflects the increasing value of observational, experimental and computer-generated data in virtually all domains, from physics to the humanities and social sciences. Addressing this new paradigm, the EUDAT project is a European data initiative that brings together a unique consortium of 25 partners — including research communities, national data and high performance computing (HPC) centers, technology providers, and funding agencies — from 13 countries. EUDAT aims to build a sustainable cross-disciplinary and cross-national data infrastructure that provides a set of shared services for accessing and preserving research data. The design and deployment of these services is being coordinated by multi-disciplinary task forces comprising representatives from research communities and data centers.”
Depending on the application of the user’s system, it may be necessary to modify the default configuration of the network adapters and the system/chipset configuration. This slide deck describes common tuning parameters, settings & procedures that can improve performance of the network adapter. Different Server & NIC vendors may have different recommendations for the values to be set – but the general tuning approach should be similar. For the hands-on demo we will utilize Mellanox ConnectX adapters – thus we will implement the recommended settings issued by Mellanox.
Today Univa released the latest version of Univa Grid Engine. With cross-platform support, Release 8.1.4. of Univa Grid Engine includes a number of customer-driven enhancements:
Improved Load collection tool for Intel Xeon Phi coprocessors
Extended memory usage metrics for Multi-Threaded applications
Scheduler performance enhancements ensuring maximum number of jobs running in the cluster while improving system responsiveness
Interactive Univa Grid Engine jobs can now set their memory affinity
Our latest Univa Grid Engine version 8.1.4 has been completely customer driven and is the largest update of the last 10 months,” said Fritz Ferstl, CTO Univa Corporation and father of Grid Engine. “We are leading the industry right now in converged infrastructures supporting Big Data and Big Compute, and our customers rely on Univa Grid Engine to manage mission-critical applications – so we make sure to always stay close to them in order to support their needs.”