Virtualization

Userspace Networking with DPDK

Rami Rosen — Mon, 23 Apr 2018 12:07:07 +0000

DPDK is a fully open-source project that operates in userspace. It's a multi-vendor and multi-architecture project, and it aims at achieving high I/O performance and reaching high packet processing rates, which are some of the most important features in the networking arena. It was created by Intel in 2010 and moved to the Linux Foundation in April 2017. This move positioned it as one of the most dominant and most important open-source Linux projects. DPDK was created for the telecom/datacom infrastructure, but today, it's used almost everywhere, including the cloud, data centers, appliances, containers and more. In this article, I present a high-level overview of the project and discuss features that were released in DPDK 17.08 (August 2017).

Undoubtedly, a lot of effort in many networking projects is geared toward achieving high speed and high performance. Several factors contribute to achieving this goal with DPDK. One is that DPDK is a userspace application that bypasses the heavy layers of the Linux kernel networking stack and talks directly to the network hardware. Another factor is usage of memory hugepages. By using hugepages (of 2MB or 1GB in size), a smaller number of memory pages is needed than when using standard memory pages (which in many platforms are 4k in size). As a result, the number of Translation Lookaside Buffers (TLBs) misses is reduced significantly, and performance is increased. Yet another factor is that low-level optimizations are done in the code, some of them related to memory cache line alignment, aiming at achieving optimal cache use, prefetching and so on. (Delving into the technical details of those optimizations is outside the scope of this article.)

DPDK has gained popularity in recent years, and it's used in many open-source projects. Many Linux distributions (Fedora, Ubuntu and others) have included DPDK support in their packaging systems as well.

The core DPDK ingredients are libraries and drivers, also known as Poll Mode Drivers (PMDs). There are more than 35 libraries at the time of this writing. These libraries abstract away the low-level implementation details, which provides flexibility as each vendor implements its own low-level layers.

Go to Full Article

FOSS Project Spotlight: Ravada

Francesc Guasch — Fri, 13 Apr 2018 18:48:24 +0000

by Francesc Guasch

Ravada is an open-source project that allows users to connect to a virtual desktop.

Currently, it supports KVM, but its back end has been designed and implemented in order to allow future hypervisors to be added to the framework. The client's only requirements are a web-browser and a remote viewer supporting the spice protocol.

Ravada's main features include:

KVM back end.
LDAP and SQL authentication.
Kiosk mode.
Remote access for Windows and Linux.
Light and fast virtual machine clones for each user.
Instant clone creation.
USB redirection.
Easy and customizable end-user interface (i18n, l10n).
Administration from a web browser.

It's very easy to install and use. Following the documentation, virtual machines can be deployed in minutes. It's an early release, but it's already used in production. The project is open source, and you can download the code from GitHub. Contributions welcome!

Go to Full Article

Kodiak Data's MemCloud

James Gray — Fri, 04 Aug 2017 15:30:56 +0000

by James Gray

Scientists working with big data regularly confront the high cost of acquiring the computational power needed to push the boundaries and innovate in data science. In an effort to bridge the Big Data infrastructure chasm, Kodiak Data—a leader in cluster virtualization technology—presents MemCloud, an innovative IaaS solution that accelerates the entire big data-deployment chain.

MemCloud is also "the first memory-speed cloud infrastructure solution for big data scientists and software developers" that provides big data analytic clusters "at up to one-fifth the cost and five times the performance of typical leading cloud hosting services". MemCloud is built on Kodiak Data's Virtual Cluster Infrastructure platform, "the only solution capable of in-software provisioning of compute, networking, storage and data at the cluster level within minutes".

Besides the hosted cloud service option, MemCloud also is available as a compact on-premises appliance for private clouds, an industry first, asserts Kodiak.

Go to Full Article

CloudBees, Inc.'s CloudBees Jenkins Enterprise

James Gray — Mon, 05 Jun 2017 10:35:52 +0000

by James Gray

Modern IT departments are adopting continuous delivery (CD) and automating software pipelines to accelerate and scale their software development and delivery across environments. This means that CD platforms are now business-critical and need to be scalable, secure, stable and reliable. To address these requirements, CloudBees, Inc., developed CloudBees Jenkins Enterprise, a new unified CD platform offering that allows enterprises to achieve CD for their entire software portfolio, securely and in a manner that handles a large volume of projects.

CloudBees Jenkins Enterprise is based on Docker container technology and architected from the ground up to allow for scale and failure backup. It can run anywhere, including public cloud or private cloud, virtual environments (VMware vSphere), Red Hat Enterprise Linux and from legacy to container-based technology.

CloudBees warns that most continuous delivery solutions focus on a specific technology stack or process, leading to the deployment of multiple competing or overlapping solutions lacking holistic support of the enterprise-wide IT delivery processes. In contrast, CloudBees Jenkins Enterprise leverages Jenkins' 1,200+ extensions and is the only solution on the market that makes it possible for enterprises to have a unified view across their entire software portfolio. As a result, it is possible to evaluate how teams are progressing in their adoption of DevOps, continuous delivery and ultimately time-to-market goals.

Go to Full Article

FinTech and SAP HANA

Ted Schmidt — Fri, 24 Mar 2017 12:39:10 +0000

by Ted Schmidt

During the past few years, FinTech has emerged as a startup realm that is growing astronomically. In 2015, the EU alone experienced a growth of investment in FinTech of more than 215%, according to a report by Accenture. This new and disruptive technology scene is not only introducing new sectors, such as mobile payments and crowdsourcing, but it is completely disrupting legacy sectors, such as funds transfers and loans, areas traditionally held by big banks.

Although it may be more convenient to split the dinner check with a friend on your mobile device or to receive funding for a new venture in a matter of weeks versus the more traditional months it might take to secure a bank loan, when 40% of the London workforce is composed of financial and technology services, what are the implications of this emerging and highly disruptive technology?

To start, small-to-medium enterprises now can offer financial services at a much smaller price than ever before, which creates a virtual cornucopia of opportunities. Since throttling access to financial data is not a viable, long-term strategy for banks, they ultimately will be forced into making a choice of whether to buy these startups or build their own in-house solutions to compete. Either way, massive amounts of data are being generated, and big banks need to figure out a way to manage both customer and regulatory demands.

Given these challenges, a model for big banks planning for a competitive future, Commerzbank, one of the largest banks in Germany, recently launched a FinTech incubator, which was the first of its kind in continental Europe. Not only does the incubator offer capital to FinTech startups, but it also offers access to Commerzbank customers (private and corporate), venture capital, banking expertise and IT infrastructure.

Along these same lines of innovative thinking, Commerzbank recently implemented SAP HANA to help accelerate its closing processes and ensure regulatory compliance. With the bulk of its balance sheet now running on SAP Bank Analyzer and SAP Accounting for Financial Instruments (both running on SAP HANA), Commerzbank has digitized all its lines of business, including investment, commercial and retail banking, and at the same time is able to close its books much faster.

Go to Full Article

The Tiny Internet Project, Part III

John S. Tonello — Thu, 22 Dec 2016 16:43:18 +0000

by John S. Tonello

In a previous article, I introduced the Tiny Internet Project, a self-contained Linux project that shows you how to build key pieces of the internet on a single computer using virtualization software, a router and free open-source applications. In the second installment, I explained how to set up the host server using Proxmox and build a first basic Ubuntu 14.04 virtual machine. In this third installment, you'll learn how to set up an Ubuntu mirror, a DNS server, a mail server and a web server.

As you finished with Part II, you hopefully had just booted a raw Ubuntu 14.04 server VM. Now, I'll describe how to customize that VM with some user accounts and software, keeping it fairly generic, but ready to become a template for most everything else you'll build.

Initially, you'll do all your work from the Proxmox web interface on your Proxmox server: https://10.128.1.2:8006.

Log in and start the Ubuntu VM you made, which probably was named "100 (ubuntu)". Wait a moment for it to boot, and click the Proxmox Console button to launch what is essentially a web-based terminal.

Figure 1. Ubuntu Installation Screen—Selecting Your Language

When the shell opens, you'll see the Ubuntu installation screens. Select your language and choose "Install Ubuntu Server" from the action list. You'll be prompted again for language choices and keyboard layouts; choose the ones that suit your needs. The installer will detect your network and prompt you to enter a hostname.

Figure 2. Entering a Hostname

Since you'll be making this VM a template, give the machine a generic hostname like "ubuntu". That way, if you later deploy a different type of server (say, ArchLinux), you'll easily be able to tell them apart.

When you're asked to create a user name, choose something that follows a naming convention you can use for all future users, such as your first initial and your full last name. Then when you need to figure out user names (and email addresses) later, you won't have to guess.

Figure 3. Selecting a User Name

Go to Full Article

The Tiny Internet Project, Part II

John S. Tonello — Thu, 17 Nov 2016 12:04:15 +0000

by John S. Tonello

In the May 2016 issue (also available here), I introduced the idea of the Tiny Internet Project, a self-contained Linux project that shows how to build the key pieces of the public internet on a single computer using one or two old computers, a router and a bunch of Linux software. In this second part, you'll learn how to build the virtual-machine host—using Proxmox—and deploy your first server. In Part III (coming soon), you'll build a template to make DNS and email hosts, a website and a Linux distribution mirror.

You'll need two separate computers for this project. This first is your "administration PC". It's any desktop or laptop with network access, a graphical browser (like Firefox or Chrome) and at least one USB port. The second machine will become your virtual machine host.

The central idea is to build a sort of internet-in-a-box with common Linux servers and use the setup to teach young people or newcomers about Linux. Using virtualization software, you'll deploy several servers that will handle domain names, email, web pages and more—all on a single piece of hardware. You don't need anything new or fancy. I built the prototype on a six-year-old Velocity Micro desktop with an Intel i3 processor, 8GB of RAM, two network cards and a couple 1TB drives.

Choosing Your Hardware

First off, you need a computer that supports virtualization—meaning its BIOS, 64-bit-capable CPU and motherboard allow you to share all the system's resources with virtual machines that will run on it. You may have experimented with VirtualBox or even free versions of VMware's ESXi software. The idea here is the same: place a number of virtual servers on a single physical machine and share all of that physical machine's resources, including memory, CPU and drives.

The Linux version is called KVM for Kernel-based Virtual Machine. For this project, you'll use a pre-compiled version called Proxmox, which comes with everything you need.

To see if the computer you have in mind can become a Proxmox server, you'll need to check whether it supports virtualization. Graphical tools are available for Windows, Linux or Mac OS X, and if you've got a machine with no operating system installed, you can boot it from a USB or CD drive using any flavor of Linux to test it. See the Resources section at the end of this article to learn how to do that.

On Windows, you can learn a lot from the main Computer properties. Right-click on Computer (on the desktop or Start menu), and look at the lower part of the window, which will look something like Figure 1.

Figure 1. Windows Computer Properties

Go to Full Article

Installing and Running a Headless Virtualization Server

Petros Koutoupis — Tue, 25 Oct 2016 11:06:30 +0000

by Petros Koutoupis

In recent years, hardware virtualization has become commonplace in the computing industry and more available to end users. The idea behind it is a noble one. Why invest in allocating more server hardware and not utilize it to its full potential, when instead you can consolidate it all onto one or a few servers and share their resources?

In this article, I focus on full hardware virtualization. This refers to the creation of a virtual machine with enough simulated hardware to act like a real computer with its own operating system. In turn, this virtual machine, or virtual guest, is enabled and running on top of a host machine. The software that controls virtualization is called the hypervisor. The hypervisor runs on the host machine.

Virtualization typically is used to consolidate hardware servers, reduce energy consumption, simplify server management and ease migration/recovery (in the situation of a failure). It sometimes is accomplished with hardware that needs to be accessed remotely, as the hardware may reside in what may not be a local data center. With that in mind, the hardware's operating system also may be configured without a graphical environment (meaning one that also includes a lack of attached monitor and input devices). This commonly is referred to as running headless. This article assumes that you have Secure Shell or SSH access to what will become the host machine.

Preparation

Prior to configuring the virtual machines, install the appropriate packages from your distribution's package repository. These packages include the libvirt library that provides a simple virtualization API, client-side utilities to access the library and manage the virtual machines, a Python virtual machine installer utility and the userspace components to the kernel-based virtual machine (KVM).

Note: libvirt is an open-source API, dæmon and management utility for managing platform virtualization. KVM is a virtualization infrastructure that turns the Linux kernel into a hypervisor. It requires a processor with hardware virtualization extensions.

Depending on your distribution, the package names may differ. On Red Hat/CentOS, they are labeled as libvirt, libvirt-client, python-virtinst and qemu-kvm. On Debian/Ubuntu, they are labeled as libvirt, libvirt-bin, python-virtinst and qemu-kvm.

Enable the libvirt dæmon to run on all appropriate runlevels at system boot up and manually start the service:


$ sudo /sbin/chkconfig libvirtd on
$ sudo /sbin/service libvirtd start
Starting libvirtd daemon:                   [  OK  ]

View the status of the dæmon and validate that it is running:

Go to Full Article

The Tiny Internet Project, Part I

John S. Tonello — Thu, 29 Sep 2016 10:00:00 +0000

by John S. Tonello

As LJ readers well know, Linux drives many of the technologies we use every day, from smart TVs to Web servers. Linux is everywhere—except most homes and classrooms.

That's a problem if we want to help breed the next generation of engineers and computer scientists. In fact, if teenagers (or any other group of curious individuals) want to learn about Linux, they often must rely on a geeky friend or parent willing to show them the way.

This three-part series seeks to change that by offering a way for anyone to learn about Linux by building what is essentially a tiny, self-contained Internet. Using old equipment and free software, you'll build a private network (with your own domain name), build Web sites, set up an e-mail server, install and use a database, and set up a Linux distro mirror.

If you like to learn by doing, but you're intimidated by the thick Linux texts you find at the bookstore, this Tiny Internet Project is for you. If you're a teacher interested in bringing Linux to the classroom, this is a great way to do it.

At the core of the project is a Proxmox KVM environment. KVM, or kernel-based virtual machine, is an open-source alternative to often costly VM technology like VMware and Hyper-V. You'll use Proxmox to host several Ubuntu 14.04 servers (or other Linux flavors), connect them over a private network and learn a lot about Linux along the way.

Figure 1. Proxmox

The Tiny Internet Project assumes you have some basic computer skills (Windows, Mac or Linux), that you have a couple computers lying around and that you have some time to tinker. The project can be done in whole or in part, depending on your interests and needs. It's particularly designed for educators who want to introduce school-aged kids to Linux.

What You'll Be Building

You'll be using open-source software for everything in this project, so everything you need will be free to download and use. You'll also take advantage of virtualization technology, which will enable you to deploy a bunch of virtual machines. In all, you'll deploy:

The Proxmox server to host all your virtual machines.
Two DNS servers, a primary and a secondary.
An e-mail server.
One or more Web servers.
An Ubuntu 14.04 repository mirror.

Go to Full Article

Vagrant Simplified

Shawn Powers — Fri, 25 Sep 2015 17:30:13 +0000

by Shawn Powers

I admit it, some tools confuse me. I know they must be amazing, because programs don't get popular by being dumb (well, reality TV, but that's another story). I have the same sort of confusion with Vagrant that I have with Wine, Docker, Chef and countless other amazing tools people constantly rave about. So in this article, I'm going to break down Vagrant into its simplest form.

Don't get me wrong, I could follow along with the tutorials and get a virtual machine running by typing the magic vagrant up command. The thing is, I really don't like magic when it comes to computers. I like to know what is happening, why it's happening and where to look when things go wrong. Ultimately that's my goal, to be able to fix it when it breaks. Without an understanding of how things truly work, it gets really scary when the magic button quits working.

What It Is

Simply put, Vagrant is a front end to an underlying virtualization program. By default, the back-end program is VirtualBox, although Vagrant can work with other underlying virtualization systems. This realization was important for me, because the line between what had to be inside VirtualBox and what Vagrant actually did on its own was murky. If you use Vagrant, you don't ever need to start VirtualBox—truly. It won't hurt anything if you do start it, but Vagrant uses VirtualBox more like a tool than a system.

Another reason this is important is because it means there is no intermingled dependencies between Vagrant and VirtualBox. By that I mean you can take your Vagrantfiles to another computer, and it will work just fine. It simply will use the copy of VirtualBox you have installed on the new computer and work exactly the same.

When Does It Make Sense to Use Vagrant?

Much like brushing your teeth with a hairbrush doesn't make much sense, using Vagrant for setting up your permanent data center might not be the best idea. Sure, you could use it, but Vagrant really excels at building VMs very fast and destroying them when you're finished. In fact, most people use Vagrant for one of two things: creating a development environment to test their code and creating temporary servers on demand when the workload requires it.

Go to Full Article