Virtualization, the Cloud, and HPC

Virtualization has been proven to be a viable architectural approach that addresses the many challenges of today’s datacenters. Now HPC is adopting virtualization in new and exciting ways that don’t compromise on performance.

By creating a virtualized infrastructure, the IT organization ensures that:

  • HPC Virtualization Guide - CoverDepartments, principal investigators and other key stakeholders receive the HPC resources they need when they need them Clusters and cluster nodes can be sized to meet specific application requirements
  • Different operating systems and software stacks can be hosted simultaneously on the same infrastructure and adjusted dynamically
  • IT can make efficient use of the underlying host hardware even though individual user jobs may only require a small number of CPUs
  • Hardware can be shared while providing fault and security separation between users
  • Policies can be enacted that allow high priority jobs to receive a higher “fairshare” of the underlying resources

For HPC environments, wrapping the virtualization infrastructure in a secure private cloud provides the most value to both the end users and the IT organization.

This approach enables self-provisioning, allowing researchers and engineers to instantiate the resources they need for a particular project without waiting for IT to create the resource for them. To instantiate a virtual HPC cluster, the user applies a previously defined blueprint that specifies the required virtual machine (VM) attributes, the number of machines involved, and the needed software – including the operating system and middleware. Users can fully customize the VMs to meet their specifications. The blueprint also allows the centralized IT organization to enforce corporate IT mandates – for example, security and data protection policies.

This solution assumes that end users will continue to run their familiar HPC batch schedulers within their virtual HPC clusters. At the same time, the VMware Distributed Resource Scheduler (DRS) and other components are able to dynamically manage the placement and priority of virtual machines on the underlying physical resources.

What the end user sees is an HPC cluster that looks just like a standard bare metal cluster running a standard job scheduler – there is no indication that they are interacting with virtual machines. This allows multiple engineering or research clusters to be instantiated on the same physical infrastructure – all available through a private cloud.

Underneath it all, virtualization is handling load balancing, protection, network services and all the other fundamentals that allow for multi-tenancy on the physical hardware while still delivering high performance. Cloud automation provides policy-based governance and logical application modeling to make sure that multi-vendor, multi-cloud services are delivered at the right size and service level for the task that needs to be performed.
Virtualization and cloud automation are fundamental attributes of a software defined data center, which allows IT to create private clouds that deliver agility and economies of scale while maintaining data sovereignty and governance.
Protecting Applications
Virtualization allows the adoption of advanced resiliency practices such as using telemetry from the underlying system to predict impending hardware failures and then proactively migrating the workload to another host to avoid application interruption. For example, the system would detect a potential fan failure or an increase in the rate of soft memory errors and take action to make sure the workload continues despite incipient system problems.

This approach should also reduce the need for frequent checkpointing and restoration, resulting in increased overall job throughput.

To learn more, download the insideHPC Guide to Virtualization, the Cloud, and HPC.

Interview on the Cloud and HPC

Michael Resch, HLRS

Michael Resch, HLRS

Over at International Science Grid This Week, Michael Resch from HLRS in Germany discusses his upcoming ISC Cloud’14 keynote and why he has changed his mind about the Cloud and HPC.

I was rather skeptical about ‘the cloud’ when it first came up. To me, it looked very much like yet another buzzword. However, over recent years I have seen industry and academia alike put some flesh on the bones and there are a variety of activities going on that deserve our support. Pressure from the commercial side, particularly from cloud providers like Amazon that can compete with large national centers, has forced the computing community to focus on what we can do to make concepts like cloud more productive for the end users.

Resch notes that government leaders are starting to move away from the mindset of just building the fastest LINPACK machines to focus on what we can do to make concepts like cloud more productive for the end users.

Read the Full Story.

Newvem Startup Analytics Help You Know Your Cloud

In this slidecast, Zev Laderman from Newvem presents: Newvem – Know Your Cloud.

Newvem is the leader in enhancing cloud usage effectiveness through data analytics. Newvem’s suite of tools utilize both cloud data analysis and crowd sourcing to enable DevOps, IT Managers and other Cloud Stakeholders to get to the bottom of their cloud faster by operating more efficient, secure, and cost effective clouds.”

Read the Full Story * Download the MP3 * Subscribe on iTunes


Mellanox Powers Highly Efficient Virtual Desktop Infrastructure

imgresToday Mellanox announced a collaborative solution using VMware Virtual SAN, a hyper-converged storage product, Mellanox’s 10GbE interconnect and Micron’s solid-state drives (SSDs) that enables a highly efficient Virtual Desktop Infrastructure (VDI) deployment.

The solution consists of 3 servers running VMware vSphere and VMware Virtual SAN, each with one Mellanox ConnectX-3 10GbE NIC, two Micron 1.4TB P420m PCIe solid-state drives and six hard disk drives (HDDs). Combined, the three servers support 360 Virtual Desktops with less than 5 minutes boot time, significantly reducing the total cost per user.

VMware’s Virtual SAN aggregates locally-attached disks in a vSphere cluster to create a storage solution that can rapidly be provisioned from VMware’s vCenter™. This simple model saves both time and money while delivering high-performance with SSD caching.

We are happy to see the growing adoption of VMware Virtual SAN,” said Alberto Farronato, director, product marketing at VMware. “We are pleased that Mellanox and Micron have recognized the advantages that Virtual SAN enables and have integrated it into their joint VDI solution.”

Mellanox’s scale-out 10/40/56GbE products enable users to benefit from a far more scalable, lower latency, and virtualized fabric with lower overall fabric costs and power consumption. These benefits yield greater efficiencies, and simplifies and streamlines storage management compared to traditional Ethernet fabrics.

Visit Mellanox Technologies at booth #2023 or Micron at booth #929 to learn about the benefits of using this flash-based storage solution.

AMAX to Showcase CloudMax OpenStack Solution at VMworld

Amax logoToday AMAX announced that the company will showcase its award-winning CloudMax Converged Cloud Infrastructure Solution at VMworld 2014, to be held August 24-27th at the Moscone Center in San Francisco.

CloudMax, recent winner of Intel’s Server Innovation Award for Best Data Center Product, is a total turnkey private cloud solution featuring OpenStack integrated with either traditional server or Facebook-inspired Open Compute (OCP) platforms, Open Networking/SDN support capabilities, and multi-hypervisor integration including VMware’s vSphere ESXi, Citrix’s Xen, Windows Hyper-V and KVM. Easily deployed within days and provisioned in minutes, CloudMax is the simple and flexible way to bring up, maintain and scale your private cloud. For customers who need further assistance with setting up their environment or further customization, AMAX offers professional services ranging from feature development to onsite installation for a full plug and play experience.

CloudMax can also be deployed as a cloud platform compatible with cloud operating systems outside of the CloudMax™ software. For customers who are using VMware’s vSphere and are looking for a more cost-effective way to scale their infrastructure, CloudMax’s hardware platform is also validated with vSphere and can be deployed as a vSphere platform with vSphere loaded in place of the CloudMax software, giving customers the ultimate flexibility in how they want to deploy and scale their cloud.

Private clouds work best as converged total solutions with built-in modularity,” said James Huang, Product Marketing Manager, AMAX. “Whether deployed as a complete converged solution based on OpenStack to run multiple hypervisors, or as a platform to extend an existing vSphere infrastructure, CloudMax™ is a fully-integrated yet customizable private cloud solution designed with agility, scalability, operational cost efficiency and compatibility in mind.”

AMAX will be showcasing the OCP Open Rack version of the CloudMax infrastructure along with demos of the CloudMax OpenStack software featuring a centralized management GUI for automated machine discovery, allocation, configuration and total management of the OpenStack cloud environment.

Network Design for the Modern Data Center

imagesAs new technologies evolve, Datacenter infrastructure is becoming more complex. It has led to incompatible framework and consoles throughout network, storage and server. If you are looking for more simplicity and flexibility then you can go for modular design, which also lets IT architects make changes to building blocks when required.

New traffic patterns call for new designs

It’s time you let go of the conventional tree structure to keep up with data center traffic. Any-to-any storage/server mesh means traffic doesn’t have to move from north to south before moving from east to west. Specialized storage switches like Fiber channels can be used by companies in order to link storage devices and servers. To make the most of efficiencies of scale, you can think of consolidating your storage networks with that of your data center. It also ensures that number of siloed networks that require a lot of maintenance can be reduced too.


Manual provisioning of data center is becoming tricky, but automating it with the tools can be difficult as well. That’s because errors creep up on account of network complexity. Three or four-tier network structure has a lot of potential errors that just can’t be accounted for. Assessing how traffic flows in each switch and how it can make a difference to packet delay and loss might be a tough ask as well. There are new challenges with cloud computing and virtualization too, which is why your network should evolve and rise up to them. Otherwise you are faced with more problems than solutions.

Commodity Hardware has its benefits

Low-cost commodity hardware running distributed software came into the picture as Google enhanced its web search and cloud services. This strategy ensures that you can scale fast without having to make huge investments. Data centers of the yore have to shell huge amounts to upgrade their software after every few years. But commodity hardware offers them the same advantages as cloud providers. You have a distributed software layer, which means resources from all clusters of commodity nodes can be abstracted. As a result you get an aggregate capacity that’s better than the most powerful monolithic approaches.

Enhanced flexibility

Compartmentalizing unique technology capabilities into various silos only makes things harder for management. Another problem with that is in every separate silo you have to manage the scale out operations individually. These are just some of the issues with regular data centers, which are not easily scalable besides being more rigid as well. In today’s times there have been several advancements in technology and it has necessitated frequent updating of skills amongst siloed teams. In fact it’s imperative for them in order to manage their responsibilities. That’s why silo based infrastructure is becoming increasingly difficult to manage.

Hybrid Clouds

Public clouds have their merits because they offer storage, resources and Internet-accessible compute for different users. That’s the reason they have become integral to businesses’ IT strategy. You can pick applications that work well in public clouds; for example, infrastructure as a service. It’s particularly true for applications with unpredictable demands that work better because of the global elasticity they get. Their ability to offer self-service resources means public clouds work well for developers of applications who require quick access to computing and storage abilities too.

Focus on service continuity

User expectations have changed with consumerization and your disaster strategies cannot only be reactive. Unauthorized cloud-based services seem tempting in case there are any interruptions to contend with. That’s why it’s important for admin to make sure there is 100% availability, which can ensure continuity in service. Focusing on recovery when problems arise is just not good enough. That leads to data centers that have to be re-architected. You also have to keep round trip times low and have a lot of bandwidth as well. Applications architectures can be passed on through different sites and data centers, which allow them to scale globally. Moreover their up-time is increased and they perform more efficiently.

End users can be empowered

Data centers have to be more reliable today and modernizing them will ensure that you can keep up with the demands of consumerization. Compute-intensive VDI systems and existing virtualized enterprise applications can also be dealt with a lot better.

Let Software Drive

Data centers today have to have latest software capabilities, but the problem is that they are rigid and often run by field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). It means admin can have new services without adding hardware, which offers flexibility while you save costs. Scalability, up-times are improved too.


Enterprises have to stay competitive to adapt to latest changes in business environment. Data computing and storage capacity should be increased while having the provision of adding new capabilities easily.

About the author

Ramya Raju is a freelance writer/web designer from India. His web site is

A3Cube Develops Military-Grade Network Interface Controller

a3cubeThis week A3CUBE announced that it has teamed with electronic manufacturing firm AirBorn Inc. to develop an “unbreakable” network interface controller called the RONNIEE RIO.

We leveraged our experience in the defense and aerospace industries to develop a connector where extreme speed does not come at the expense of availability and robustness,” said Emad Soubh, Director of New Product Development of AirBorn. “Working with A3CUBE, we were able to produce a high-density, high-bandwidth, high-reliability interconnect adapter card ready for the challenges of the next-generation data center, cloud computing platforms, analytics, storage and converged architectures.”

RONNIEE RIO is available in a low-profile PCIe card form factor designed to guarantee maximum availability, military-grade reliability and low-latency capabilities of peer-to-peer operations, RDMA and remote CPUs load and store operations.

OSC Powers Welding Simulation in the Cloud

awesimThe DOE has announced a grant to Engineering Mechanics Corporation of Columbus (Emc2) and its partners for $1 million to develop a cloud-based tool that will simulate welding processes employed in the manufacture of metallic products.

Small and mid-sized manufacturing firms need improved weld-fabrication processes to ensure improved quality at lower costs to remain globally competitive,” said Frederick “Bud” Brust, Ph.D., senior research leader at Emc2. “A sophisticated high performance computing based tool will be accessible to SMM firms on a supercomputer center to easily permit use of these tools at affordable prices.”

Emc2 seeks to adapt a welding design software package known as Virtual Fabrication Technology, or VFT, to a more accessible “app” format through the Ohio Supercomputer Center’s (OSC) AweSim program. VFT is a mathematics-based computational tool developed in conjunction with Caterpillar in the late 1990s, with improvements since then to permit small- and medium-sized manufacturers (SMM) to take advantage of the benefits of high performance computing. VFT allows designers to “predict weld-induced distortion and residual-stress states so that weld-design strategies can be implemented to significantly reduce or eliminate these undesirable states prior to fabrication.”

For this project, Emc2 has partnered with members of the AweSim program, a $6.4 million initiative led by the Ohio Supercomputing Center. The program kicked off last summer when the Ohio Third Frontier Commission awarded the public/private partnership a $3 million Innovation Platform Program grant to design and deploy easy-to-use advanced “manufacturing apps.” Client partners are funding the balance of the three-year project.

Ubercloud Announces Technical Computing Marketplace and Appstore

In this video from ISC’14, Wolfgang Gentzsch announces the new Ubercloud Technical Computing Marketplace and Appstore.

Two years ago the UberCloud started its free HPC Experiment which today has attracted more than 2,000 organizations and individuals from computational fluid dynamics, finite element material analysis, multi-physics, chemistry, life sciences, biology, finance, big data, and the HPC community. Today, UberCloud announces the addition of its Marketplace and Appstore, where engineers and scientists can discover, try and buy the computing power and expertise they need for their computational and data-intensive tasks.”

Read the Full Story.

In related news, the Second Ubercloud Compendium of Case Studies is now available for download.

Video: What’s New in Moab 8.0 for Cloud & Big Workflow

In this video, Trev Harmon from Adaptive Computing presents: HPC, Cloud & Big Workflow: What’s New in Moab 8.0.

Adaptive Computing powers many of the world’s largest private/hybrid cloud, and technical computing environments with Moab, its award-winning optimization and scheduling middleware software. Moab enables large enterprises, academic institutions and governments to perform simulations and analyze data faster, more accurately and most cost effectively with its Technical Computing, Cloud and Big Data solutions for Big Workflow applications. With dynamic scheduling, provisioning and management of multi-step/multi-application services across HPC, Cloud and Big Data environments, Moab inspires users to pursue game-changing endeavors. Adaptive Computing’s Moab speeds the time to discovery, allowing the business to gain a competitive advantage.

HP is Looking for a Cloud Deployment Engineer – HP Helion in our Job of the Week

Hewlett-Packard in Seattle, WA is looking for a Cloud Deployment Engineer – HP Helion in our Job of the Week.

“HP has been a consistent leader in defining the next generation computing platforms, and we are doing this again with #HPHelion and the next-generation open source computing platform. More than 2,000 enterprise customers and 37% of the Fortune 100 use our cloud solutions today and serve as the foundation for HP Helion. HP is the #1 private cloud provider in the world. We are a leader in OpenStack and Cloud Foundry foundations and communities. HP is committed to investing billions of dollars in HP Helion products, services, technologies and programs, and the global reach of HP Helion, over the next three years.

Are you paying too much for your job ads? Not only do we offer ads for a fraction of what the other guys charge, our inside-Cloud Job Board is powered by SimplyHIred, the world’s largest job search engine.

FusionStorm Boosts Its Cloud Computing Offering with Peak’s Enterprise Cloud and NetApp Storage Platform

Peak announces that it will bring its world-class Infrastructure-as-a Service (IaaS) capabilities to FusionStorm to give the IT solution provider a more robust cloud solution.

“FusionStorm specializes in helping our clients connect their employees, partners and customers by building solid foundations for their IT infrastructure,” states Mike Souza, Senior VP of Sales and Marketing for FusionStorm. “We know that the cloud has become a key component in delivering resources, increased capabilities and flexibility to our clients, and Peak’s cloud is unmatched in terms of technology and their hands-on expert support at all levels. Peak’s cloud also allows us to easily integrate a NetApp storage component, which is key for our customers.”

Read the Press Release.

AgileIT is Looking for a Senior Microsoft Cloud Engineer in our Job of the Week II

AgileIT in San Diego, CA is looking for a Senior Microsoft Cloud Engineer in our Job of the Week II.

“Our San Diego network support team is looking for a Microsoft Solutions Engineer (Office 365, Azure, Hyper-V, and Powershell) to assist and support our customer projects.  Join the expert Agile IT team and work in a highly dynamic environment with leading Microsoft architects. Under the direction of project managers and Principal Architects, you will perform customers engagements remotely from our San Diego network operations center (near Sea World).

Are you paying too much for your job ads? Not only do we offer ads for a fraction of what the other guys charge, our inside-Cloud Job Board is powered by SimplyHIred, the world’s largest job search engine.

Global Music on Demand Market to Grow Says a New Research Report at

Sandler Research has issued a report detailing the surge in desire for on demand music from various devices. The report says a big portion of the increase is due to the escalation of offerings in music from social media platforms.

“Cloud computing provides consumers with legal access to music online and is rapidly being adopted by firms because it enhances portability, enabling subscribed users to access digital content stored in remote servers irrespective of the devices used. Listening to music, initially, was only possible using devices such as iPods. But the current scenario is witnessing consumers switching to an access-based approach, which consists of subscribing to digital music libraries stored in the cloud.

Please read the Press Release.

OpenStack Delivers the Future of Cloud with Robust Open Source Features

Over at Talkin’ Cloud, David Darrough reports that leading open source cloud solutions provider, OpenStack, is showing the way into a cloud-based future with superb offerings in storage, compute and networking.

“The goal of the OpenStack Foundation is to serve developers, users, and the entire ecosystem by providing a set of shared resources to grow the footprint of public and private OpenStack clouds, enable technology vendors targeting the platform and assist developers in producing the best cloud software in the industry.

Read the Full Story.

Informatica is Looking for a Senior Sales Operations Analyst – Cloud in our Job of the Week II

Informatica in Redwood City, CA is looking for a Senior Sales Operations Analyst – Cloud in our Job of the Week II.

“Reporting into the Chief-of-Staff for Cloud & Subscription Sales, you will be joining a fast moving, and the fastest growing business at Informatica. You will need to be agile, energetic, thoughtful and collaborative. You will be empowered to grow the depth and breadth of your experience and “write your own job description” and “create your own role”. This position has two major focus areas, where you will divide your attention and energy on.

Are you paying too much for your job ads? Not only do we offer ads for a fraction of what the other guys charge, our inside-Cloud Job Board is powered by SimplyHIred, the world’s largest job search engine.

Virtustream Takes a New Approach to Take on the Giants of Cloud Computing

Over at Network World, Brandon Butler, reports that Virtustream–while not as big as Microsoft and AWS–has carved out a nice niche in the Infrastructure-as-a-Service (IaaS) world and takes a unique consultancy approach.

“Co-CEO and CTO Kevin Reid describes it like this: You can’t just walk into a bank and deposit $100,000; the financial institution would ask questions, making sure the money is not laundered or gained from some illicit activity. Similarly, Virtustream doesn’t just allow customers to swipe a credit card and get access to hundreds of thousands of virtual machines holding sensitive data of its large enterprise customers. “We want to know our customers,” says Reid, who used to manage a consulting firm that was bought by Capgemini before working at Virtustream. “We run more of what could be considered a community cloud, or a country club cloud. None of the workloads in our cloud are unknown to us – we know where they came from.”

Read the Full Story.

Guest Post: Virtualizing Datacenter Storage with EMC ViPR

The way that we store data has changed and the need for a single solution that lets you pool all of your datacenter storage resources into a self-sustaining storage mechanism has presented itself. EMC ViPR lets you achieve the goal of grabbing virtual datacenter hard disk on demand. This free flow of storage space allows utilities and datacenter applications to function without restriction. It is important to note that just because a datacenter is virtualized, it doesn’t mean that the corresponding cloud is a private cloud.

How does Software Defined Storage work?

Storage solutions such as ViPR pool together storage resources and allow them to be accessed abstractly from within the network. A provisioning controller exists to handle requests while a self-service panel is available for users to request additional storage resources. A reporting component gives administrators up to the minute data on what is being using and how it is being used. The last component of software defined storage is the automation component, which can be setup with triggers that allow for more storage resources to be made available as needed.

Key features of using Software Defined Storage such as EMC ViPR

Change on Demand: Do you manage a dynamic pool of server resources? Perhaps these resources have needs that change rapidly? Using a software defined storage system would allow your organization to provide storage space on a scale that matches your business model.

APIs: APIs can help your existing tools interact cohesively with a software defined storage solution. EMC ViPR provides robust APIs that allow developers to specifically code using the framework provided. This allows organizations to flawlessly integrate with the Storage as a Service platform.

Diverse Storage Options: Regardless if you currently use file, block or object storage, a solution like EMC ViPR can help you pool these resources into an on-demand solution. This centralized approach helps administrators easily add storage into the Storage as a Service solution from a diverse array of vendors. This storage can then be queried when needed.

Can Storage as a Service really help?

EMC ViPR allows organizations to keep their existing infrastructure in place. Vendors such as EMC realize that organizations sometimes have millions invested in private cloud datacenter solutions. Instead of reinventing the wheel, EMC ViPR allows you to keep your existing infrastructure. These solutions allow you to abstract server storage resources and use them as needed within your environment. EMC ViPR doesn’t require you to tailor your existing infrastructure. The dynamic private datacenter storage solution is designed to compliment and enhance what you have already built out in your datacenter.

Author Bio:

Natalie Lehrer is a senior contributor for CloudWedge. In her spare time, Natalie enjoys exploring all things cloud and is a music enthusiast. Follow Natalie’s daily posts on Twitter: @Cloudwedge, or on Facebook.