HPC

Data in a Flash, Part II: Using NVMe Drives and Creating an NVMe over Fabrics Network

Petros Koutoupis — Mon, 20 May 2019 11:00:00 +0000

By design, NVMe drives are intended to provide local access to the machines they are plugged in to; however, the NVMe over Fabric specification seeks to address this very limitation by enabling remote network access to that same device.

This article puts into practice what you learned in Part I and shows how to use NVMe drives in a Linux environment. But, before continuing, you first need to make sure that your physical (or virtual) machine is up to date. Once you verify that to be the case, make sure you're able to see all connected NVMe devices:


$ cat /proc/partitions |grep -e nvme -e major
major minor  #blocks  name
 259        0 3907018584 nvme2n1
 259        1 3907018584 nvme3n1
 259        2 3907018584 nvme0n1
 259        3 3907018584 nvme1n1

Those devices also will appear in sysfs:


$ ls /sys/block/|grep nvme
nvme0n1
nvme1n1
nvme2n1
nvme3n1

If you don't see any connected NVMe devices, make sure the kernel module is loaded:


petros@ubu-nvme1:~$ lsmod|grep nvme
nvme                   32768  0
nvme_core              61440  1 nvme

Next, install the drive management utility called nvme-cli. This utility is defined and maintained by the very same NVM Express committee that defined the NVMe specification. The nvme-cli source code is hosted on GitHub. Fortunately, some operating systems offer this package in their internal repositories. Installing it on the latest Ubuntu looks something like this:


petros@ubu-nvme1:~$ sudo add-apt-repository universe
petros@ubu-nvme1:~$ sudo apt update && sudo apt install
 ↪nvme-cli

Using this utility, you're able to list more details of all connected NVMe drives (note: the tabular output below has been reformatted and truncated to better fit here):

Go to Full Article

Data in a Flash, Part I: the Evolution of Disk Storage and an Introduction to NVMe

Petros Koutoupis — Mon, 29 Apr 2019 11:30:00 +0000

by Petros Koutoupis

NVMe drives have paved the way for computing at stellar speeds, but the technology didn't suddenly appear overnight. It was through an evolutionary process that we now rely on the very performant SSD for our primary storage tier.

Solid State Drives (SSDs) have taken the computer industry by storm in recent years. The technology is impressive with its high-speed capabilities. It promises low-latency access to sometimes critical data while increasing overall performance, at least when compared to what is now becoming the legacy Hard Disk Drive (HDD). With each passing year, SSD market shares continue to climb, replacing the HDD in many sectors. The effects of this are seen in personal, mobile and server computing.

IBM first unleashed the HDD into the computing world in 1956. By the 1960s, the HDD became the dominant secondary storage device for general-purpose computers (emphasis on secondary storage device, memory being the first). Capacity and performance were the primary characteristics defining the HDD. In many ways, those characteristics continue to define the technology—although, not in the most positive ways (more details on that shortly).

The first IBM-manufactured hard drive, the 350 RAMAC, was as large as two medium-sized refrigerators with a total capacity of 3.75MB on a stack of 50 disks. Modern HDD technology has produced disk drives with volumes as high as 16TB, specifically with the more recent Shingled Magnetic Recording (SMR) technology coupled with helium—yes, that's the same chemical element abbreviated as He in the periodic table. The sealed helium gas increases the potential speed of the drive while creating less drag and turbulence. Being less dense than air, it also allows more platters to be stacked in the same space used by 2.5" and 3.5" conventional disk drives.

Figure 1. A lineup of Standard HDDs throughout Their History and across All Form Factors (by Paul R. Potts—Provided by Author, CC BY-SA 3.0 us, https://commons.wikimedia.org/w/index.php?curid=4676174)

A disk drive's performance typically is calculated by the time required to move the drive's heads to a specific track or cylinder and the time it takes for the requested sector to move under the head—that is, the latency. Performance is also measured at the rate by which the data is transmitted.

Being a mechanical device, an HDD does not perform nearly as fast as memory. A lot of moving components add to latency times and decrease the overall speed by which you can access data (for both read and write operations).

Go to Full Article

The High-Performance Computing Issue

Bryan Lunduke — Fri, 30 Nov 2018 15:43:31 +0000

by Bryan Lunduke

Since the dawn of computing, hardware engineers have had one goal that's stood out above all the rest: speed.

Sure, computers have many other important qualities (size, power consumption, price and so on), but nothing captures our attention like the never-ending quest for faster hardware (and software to power it). Faster drives. Faster RAM. Faster processors. Speed, speed and more speed. [Insert manly grunting sounds here.]

What's the first thing that happens when a new CPU is released? Benchmarks to compare it against the last batch of processors.

What happens when a graphics card is unveiled? Reviewers quickly load up whatever the most graphically demanding video game is and see just how it stacks up to the competition in frame-rate and resolution. Power and speed captures the attention of everyone from software engineers to gamers alike.

Nowhere is this never-ending quest for speed more apparent than in the high-performance computing (HPC) space. Built to handle some of the most computationally demanding work ever conceived by man, these supercomputers are growing faster by the day—and Linux is right there, powering just about all of them.

In this issue of Linux Journal, we take a stroll through the history of supercomputers, from its beginnings (long before Linux was a gleam in Linus Torvalds' eye...heck, long before Linus Torvalds was gleam in his parents' eyes) all the way to the present day where Linux absolutely dominates the Supercomputer and HPC world.

Then we take a deep dive into one of the most critical components of computing (affecting both desktop and supercomputers alike): storage.

Petros Koutoupis, Senior Platform Architect on IBM's Cloud Object Storage, creator of RapidDisk (Linux kernel modules for RAM drives and caching) and LJ Editor at Large, gives an overview of the history of computer storage leading up to the current, ultra-fast SSD and NVMe drives.

Once you're up to speed (see what I did there?) on NVMe storage, Petros then gives a detailed—step-by-step—walk-through of how to best utilize NVMe drives with Linux, including how to set up your system to have remote access to NVMe resources over a network, which is just plain cool.

Taking a break from talking about the fastest computers the Universe has ever known, let's turn our attention to a task that almost every single one of us tackles at least occasionally.

Photography.

Professional photographer Carlos Echenique provides an answer to the age-old question: is it possible for a professional photographer to use a FOSS-based workflow? (Spoiler: the answer is yes.)

Go to Full Article

Linux and Supercomputers

Bryan Lunduke — Thu, 29 Nov 2018 13:00:00 +0000

by Bryan Lunduke

As we sit here, in the year Two Thousand and Eighteen (better known as "the future, where the robots live"), our beloved Linux is the undisputed king of supercomputing. Of the top 500 supercomputers in the world, approximately zero of them don't run Linux (give or take...zero).

The most complicated, powerful computers in the world—performing the most intense processing tasks ever devised by man—all rely on Linux. This is an amazing feat for the little Free Software Kernel That Could, and one heck of a great bragging point for Linux enthusiasts and developers across the globe.

But it wasn't always this way.

In fact, Linux wasn't even a blip on the supercomputing radar until the late 1990s. And, it took another decade for Linux to gain the dominant position in the fabled "Top 500" list of most powerful computers on the planet.

A Long, Strange Road

To understand how we got to this mind-blowingly amazing place in computing history, we need to go back to the beginning of "big, powerful computers"—or at least, much closer to it: the early 1950s.

Tony Bennett and Perry Como ruled the airwaves, The Day The Earth Stood Still was in theaters, I Love Lucy made its television debut, and holy moly, does that feel like a long time ago.

In this time, which we've established was a long, long time ago, a gentleman named Seymour Cray—whom I assume commuted to work on his penny-farthing and rather enjoyed a rousing game of hoop and stick—designed a machine for the Armed Forces Security Agency, which, only a few years before (in 1949), was created to handle cryptographic and electronic intelligence activities for the United States military. This new agency needed a more powerful machine, and Cray was just the man (hoop and stick or not) to build it.

Figure 1. Seymour Cray, Father of the Supercomputer (from http://www.startribune.com/minnesota-history-seymour-cray-s-mind-worked-at-super-computer-speed/289683511

This resulted in a machine known as the Atlas II.

Weighing a svelte 19 tons, the Atlas II was a groundbreaking powerhouse—one of the first computers to use Random Access Memory (aka "RAM") in the form of 36 Williams Tubes (Cathode Ray Tubes, like the ones in old CRT TVs and monitors, capable of storing 1024 bits of data each).

Go to Full Article

ONNX: the Open Neural Network Exchange Format

Braddock Gaskill — Wed, 25 Apr 2018 14:19:00 +0000

by Braddock Gaskill

An open-source battle is being waged for the soul of artificial intelligence. It is being fought by industry titans, universities and communities of machine-learning researchers world-wide. This article chronicles one small skirmish in that fight: a standardized file format for neural networks. At stake is the open exchange of data among a multitude of tools instead of competing monolithic frameworks.

The good news is that the battleground is Free and Open. None of the big players are pushing closed-source solutions. Whether it is Keras and Tensorflow backed by Google, MXNet by Apache endorsed by Amazon, or Caffe2 or PyTorch supported by Facebook, all solutions are open-source software.

Unfortunately, while these projects are open, they are not interoperable. Each framework constitutes a complete stack that until recently could not interface in any way with any other framework. A new industry-backed standard, the Open Neural Network Exchange format, could change that.

Now, imagine a world where you can train a neural network in Keras, run the trained model through the NNVM optimizing compiler and deploy it to production on MXNet. And imagine that is just one of countless combinations of interoperable deep learning tools, including visualizations, performance profilers and optimizers. Researchers and DevOps no longer need to compromise on a single toolchain that provides a mediocre modeling environment and so-so deployment performance.

What is required is a standardized format that can express any machine-learning model and store trained parameters and weights, readable and writable by a suite of independently developed software.

Enter the Open Neural Network Exchange Format (ONNX).

The Vision

To understand the drastic need for interoperability with a standard like ONNX, we first must understand the ridiculous requirements we have for existing monolithic frameworks.

A casual user of a deep learning framework may think of it as a language for specifying a neural network. For example, I want 100 input neurons, three fully connected layers each with 50 ReLU outputs, and a softmax on the output. My framework of choice has a domain language to specify this (like Caffe) or bindings to a language like Python with a clear API.

However, the specification of the network architecture is only the tip of the iceberg. Once a network structure is defined, the framework still has a great deal of complex work to do to make it run on your CPU or GPU cluster.

Go to Full Article

PSSC Labs' PowerServe HPC Servers and PowerWulf HPC Clusters

James Gray — Mon, 16 Oct 2017 14:43:17 +0000

by James Gray

In its quest to provide customers the latest and best computing solutions that deliver relentless performance with the absolute lowest TCO, PSSC Labs has supercharged two server solutions with next-generation processing power.

The breakthrough technology of Intel's new Xeon Scalable Processors has been integrated into PSSC Labs' PowerServe HPC line of servers and the PowerWulf line of HPC clusters, a move that guarantees performance capable of handling cutting-edge computing tasks, such as real-time analytics, virtualized infrastructure and high-performance computing.

Besides the advanced architecture, the new processors offer a diverse suite of platform innovations for enhanced application performance including Intel AVX-512, Intel Mesh Architecture, Intel QuickAssist, Intel Optane SSDs and Intel Omni-Path Fabric.

Both PSSC Labs solutions are designed for reliable, flexible, HPC solutions targeted at government, academic and commercial environments. Some examples of sectors that will benefit from the new performance include design and engineering, life and physical sciences, financial services and machine/deep learning.

Go to Full Article

LINBIT's DRBD Top

James Gray — Wed, 11 Oct 2017 14:19:54 +0000

by James Gray

Many proprietary high-availability (HA) software providers require users to pay extra for system-management capabilities. Bucking this convention and driving down costs is LINBIT, whose DRBD HA software solution, part of the Linux kernel since 2009, powers thousands of digital enterprises.

The cost savings originate from LINBIT's DRBD Top, a new software tool to simplify the management of the LINBIT DRBD application. Via DRBD Top's unified graphical interface, administrators can navigate their DRBD resources conveniently without typing multiple commands.

Available on GitHub, DRBD Top provides critical status, assessment and troubleshooting capabilities for administrators who manage HA clusters, especially those with greater than two nodes.

Go to Full Article

JMR SiloStor NVMe SSD Drives

James Gray — Wed, 09 Aug 2017 15:50:57 +0000

by James Gray

Compute-intensive workflows are the environments in which the newly developed JMR SiloStor NVMe family of SSD drives is designed to show its colors. Ideal for HPC, data centers, genome research, content creation, CGI/animation, codec processing and gaming, among others, the SiloStor drive family comes in three NVMe/PCIe configurations: single-drive module, x4 PCIe connectivity in 512GB/1TB/2TB capacities; dual-drive, x8 connectivity in 1TB/2TB/4TB capacities; and quad-drive module, x8 connectivity, available in 2TB/4TB/8TB capacities. The dual- and quad-drive cards incorporate a PCIe switch, and the drives can be striped (on a single card) for additional performance.

All SiloStor designs incorporate active heatsink coolers on the drive modules themselves, maintaining low operating temperatures even during intensive sequential write operations. Key performance metrics include <1 mS average access time of <1 mS, 2 million hours MTBF, 1,200 TBW minimum endurance, 90,000/70,000 IOPS random 4K read/write speed and 4,000/3,000 MB/sequential read/write speed.

Go to Full Article

SUSE Linux Enterprise High Availability Extension

James Gray — Wed, 29 Mar 2017 15:25:00 +0000

by James Gray

Historically, data replication has been available only piecemeal through proprietary vendors. In a quest to remediate history, SUSE and partner LINBIT announced a solution that promises to change the economics of data replication. The two companies' collaborative effort is the headliner in the updated SUSE Linux Enterprise High Availability Extension, which now includes LINBIT's integrated geo-clustering technology.

Providing a new capability to replicate data across unlimited distances, LINBIT has enhanced SUSE's high availability solution that is built on open-source software and runs on commodity hardware. The LINBIT solution guards against failures or disasters by providing policy-driven mechanisms for customer applications, and data, to continue operations in another geographically dispersed data center.

SUSE Linux Enterprise High Availability Extension is an integrated suite of open-source clustering technologies—from LINBIT and others—that enables customers to eliminate single points of failure, thus helping to maintain business continuity, enable compliance, protect data integrity, maintain isolation for multiple tenants and reduce unplanned downtime for mission-critical workloads.

Go to Full Article

Three EU Industries That Need HPC Now

Ted Schmidt — Sat, 25 Mar 2017 06:03:14 +0000

by Ted Schmidt

The success of High Performance Computing (HPC) relies in no small part on the OpenPOWER Foundation, which was founded in 2013. The reason this open ecosystem is so important is that it provided members open access to the IBM POWER8 technology, which resulted in huge advances in innovation. One of those innovations came in the form of the NVIDIA GPU accelerator, which not only provides improved graphics capabilities, but also assumes some of the computational load stemming from simulations. IBM POWER8 servers are already capable of clock speeds of more than 4GHz and of providing 96 simultaneous threads. Include NVIDIA Tesla GPU Accelerators, and the result is an HPC system that is extremely fast, which ends up solving some tricky problems in three key industries.

Auto Industry Product Design and Testing
The EU auto industry faces increasing pressures to provide more fuel-efficient and safer vehicles, while at the same time providing new products like reliable, electric vehicles and even self-driving vehicles. Although the European Automobile Manufacturer’s Association (ACEA) continues to predict growth in the EU market, margins of 1-3% keep EU auto manufacturers looking for ways to gain efficiencies and cut costs.

One place the answers to these challenges can be found is in the ability to consume and make sense of data from a multitude of sources, and do it quickly and effectively. Fuel efficiency, for instance, is a product of data from not only engine components, but also braking systems, batteries, tires and the external environment as well. Self-driving cars must handle even more complex datasets, and they must do it in a reliable and safe way.

Go to Full Article