AWS

CloudWatch Is of the Devil, but I Must Use It

Corey Quinn — Wed, 31 Oct 2018 11:30:00 +0000

Let's talk about Amazon CloudWatch.

For those fortunate enough to not be stuck in the weeds of Amazon Web Services (AWS), CloudWatch is, and I quote from the official AWS description, "a monitoring and management service built for developers, system operators, site reliability engineers (SRE), and IT managers." This is all well and good, except for the part where there isn't a single named constituency who enjoys working with the product. Allow me to dispense some monitoring heresy.

Better, let me describe this in the context of the 14 Amazon Leadership Principles that reportedly guide every decision Amazon makes. When you take a hard look at CloudWatch's complete failure across all 14 Leadership Principles, you wonder how this product ever made it out the door in its current state.

"Frugality"

I'll start with billing. Normally left for the tail end of articles like this, the CloudWatch billing paradigm is so terrible, I'm leading with it instead. You get billed per metric, per month. You get billed per thousand metrics you request to view via the API. You get billed per dashboard per month. You get billed per alarm per month. You get charged for logs based upon data volume ingested, data volume stored and "vended logs" that get published natively by AWS services on behalf of the customer. And, you get billed per custom event. All of this can be summed up best as "nobody on the planet understands how your CloudWatch metrics and logs get billed", and it leads to scenarios where monitoring vendors can inadvertently cost you thousands of dollars by polling CloudWatch too frequently. When the AWS charges are larger than what you're paying your monitoring vendor, it's not a wonderful feeling.

"Invent and Simplify"

CloudWatch Logs, CloudWatch Events, Custom Metrics, Vended Logs and Custom Dashboards all mean different things internally to CloudWatch from what you'd expect, compared to metrics solutions that actually make some fathomable level of sense. There are, thus, multiple services that do very different things, all operating under the "CloudWatch" moniker. For example, it's not particularly intuitive to most people that scheduling a Lambda function to invoke once an hour requires a custom CloudWatch Event. It feels overly complicated, incredibly confusing, and very quickly, you find yourself in a situation where you're having to build complex relationships to monitor things that are themselves far simpler.

Go to Full Article

Kubernetes, Four Years Later, and Amazon Redefining Container Orchestration

Petros Koutoupis — Wed, 06 Jun 2018 19:35:09 +0000

by Petros Koutoupis

Well, here we are. Kubernetes turns four years old this month—technically, on June 7, 2018—the very same platform that brings users and data center administrators scalable container technologies. Its popularity has skyrocketed since its initial introduction by Google. Celebrating the project’s birthday is not the only thing making the headlines today. Amazon recently announced the general availability of its Elastic Container Services for Kubernetes (EKS), accessible via Amazon Web Services (AWS).

Once upon a time, it wasn’t a simple task to orchestrate and manage containers in the cloud. Up until this recent EKS announcement, it was up to the administrator to spin up a virtual machine through an Elastic Cloud Compute (EC2) service, run Kubernetes on top of a traditional Linux server installation in EC2 and rely on other AWS moving components to host the container image registry. The entire process was very involved. Not any more!

The excitement doesn’t end there. Companies like Heptio (co-founded by the folks who gave us Kubernetes, Craig McLuckie and Joe Beda) have positioned themselves to enhance the user experience around the Kubernetes technology by producing products and services to simplify and scale the Kubernetes ecosystem. The Heptio Kubernetes Subscriptions (HKS) package offerings help users run Kubernetes in AWS EKS, EC2 or on-premises.

Visit Amazon's EKS product page and Heptio's company website to learn more.

Go to Full Article

Everything You Need to Know about the Cloud and Cloud Computing, Part II: Using the Cloud

Petros Koutoupis — Tue, 15 May 2018 11:45:00 +0000

by Petros Koutoupis

How to get started with AWS, install Apache, create an EFS volume and much more.

The cloud is here to stay, regardless of how you access data day to day. Whether you are uploading and sharing new photos with friends in your social-media account or updating documents and spreadsheets alongside your peers in your office or school, chances are you're connecting to the cloud in some form or another.

In the first part of this series, I explored what makes up the cloud and how it functions when all of its separate moving pieces come together. In this article, building from Part I's foundations, I cover using the cloud through some actual examples.

Getting Started with AWS

For the purposes of this article, I'm focusing on a few of the top offerings provided by Amazon Web Services (AWS). Please know that I hold no affiliation to or with Amazon, nor am I stating that Amazon offerings exceed those of its competitors.

If you haven't already, be sure to register an account. But before you do, understand that charges may apply. Amazon, may provide a free tier of offerings for a limited time, typically a year, to newly registered users. In most cases, the limitations to these offerings are far less than ideal for modern use cases. It is a pay-as-you go model, and you'll be charged only as long as the instance or service continues to be active.

As soon as you are registered and logged in from within your web browser, you'll be greeted by a fairly straightforward dashboard.

Figure 1. The AWS Main Dashboard of services and resources.

Compute

At first, companies leveraging cloud compute applied a straight copy-and-paste of their very own data centers for deploying standard web/application/database servers. The model was the same. There is nothing wrong with that approach. The transition for most converting from on-premises to the cloud would have been somewhat seamless—at least from the perspective of the user accessing those resources. The only real difference being that it was just in a different data center and without the headache of maintaining the infrastructure supporting it.

In the world of AWS, virtual compute servers are managed under the Elastic Cloud Computing (EC2) stack, from whole virtual instances to containers and more. Let's begin an example EC2 experiment by navigating to the EC2 dashboard.

Figure 2. The Elastic Cloud Computing Dashboard

Go to Full Article

Everything You Need to Know about the Cloud and Cloud Computing, Part I

Petros Koutoupis — Wed, 09 May 2018 13:30:00 +0000

by Petros Koutoupis

An in-depth breakdown of the technologies involved in making up the cloud and a survey of cloud-service providers.

The cloud has become synonymous with all things data storage. It additionally equates to the many web-centric services accessing that same back-end data storage. But the term also has evolved to mean so much more.

Cloud computing provides more simplified access to server, storage, database and application resources, with users provisioning and using the minimum set of requirements they see fit to host their application needs. In the past decade alone, the paradigm shift toward a wider and more accessible network has forced both hardware vendors and service providers to rethink their strategies and cater to a new model of storing information and serving application resources. As time continues to pass, more individuals and businesses are connecting themselves to this greater world of computing.

What Is the Cloud?

Far too often, the idea of the "cloud" is confused with the general internet. Although it's true that various components making up the cloud can be accessible via the internet, they are not one and the same. In its most general terms, cloud computing enables companies, service providers and individuals to provision the appropriate amount of computing resources dynamically (compute nodes, block or object storage and so on) for their needs. These application services are accessed over a network—and not necessarily a public network. Three distinct types of cloud deployments exist: public, private and a hybrid of both.

The public cloud differentiates itself from the private cloud in that the private cloud typically is deployed in the data center and under the proprietary network using its cloud computing technologies—that is, it is developed for and maintained by the organization it serves. Resources for a private cloud deployment are acquired via normal hardware purchasing means and through traditional hardware sales channels. This is not the case for the public cloud. Resources for the public cloud are provisioned dynamically to its user as requested and may be offered under a pay-per-usage model or for free.

Some of the world's leading public cloud offering platforms include:

Amazon Web Services (AWS)
Microsoft Azure
Google Cloud Platform
IBM Cloud (formerly SoftLayer)

As the name implies, the hybrid model allows for seamless access and transitioning between both public and private deployments, all managed under a single framework.

For those who prefer either to host their workload internally or partially on the public cloud—sometimes motivated by security, data sovereignty or compliance—private and hybrid cloud offerings continue to provide the same amount of service but all within your control.

Go to Full Article

The Agony and the Ecstasy of Cloud Billing

Corey Quinn — Tue, 17 Apr 2018 14:40:00 +0000

by Corey Quinn

Cloud billing is inherently complex; it's not just you.

Back in the mists of antiquity when I started reading Linux Journal, figuring out what an infrastructure was going to cost was (although still obnoxious in some ways) straightforward. You'd sign leases with colocation providers, buy hardware that you'd depreciate on a schedule and strike a deal in blood with a bandwidth provider, and you were more or less set until something significant happened to your scale.

In today's brave new cloud world, all of that goes out the window. The public cloud providers give with one hand ("Have a full copy of any environment you want, paid by the hour!"), while taking with the other ("A single Linux instance will cost you $X per hour, $Y per GB transferred per month, and $Z for the attached storage; we simplify this pricing into what we like to call 'We Make It Up As We Go Along'").

In my day job, I'm a consultant who focuses purely on analyzing and reducing the Amazon Web Services (AWS) bill. As a result, I've seen a lot of environments doing different things: cloud-native shops spinning things up without governance, large enterprises transitioning into the public cloud with legacy applications that don't exactly support that model without some serious tweaking, and cloud migration projects that somehow lost their way severely enough that they were declared acceptable as they were, and the "multi-cloud" label was slapped on to them. Throughout all of this, some themes definitely have emerged that I find that people don't intuitively grasp at first. To wit:

It's relatively straightforward to do the basic arithmetic to figure out what a current data center would cost to put into the cloud as is—generally it's a lot! If you do a 1:1 mapping of your existing data center into the cloudy equivalents, it invariably will cost more; that's a given. The real cost savings arise when you start to take advantage of cloud capabilities—your web server farm doesn't need to have 50 instances at all times. If that's your burst load, maybe you can scale that in when traffic is low to five instances or so? Only once you fall into a pattern (and your applications support it!) of paying only for what you need when you need it do the cost savings of cloud become apparent.

Go to Full Article

Simple Cloud Hardening

Kyle Rankin — Tue, 10 Apr 2018 15:30:00 +0000

by Kyle Rankin

Apply a few basic hardening principles to secure your cloud environment.

I've written about simple server-hardening techniques in the past. Those articles were inspired in part by the Linux Hardening in Hostile Networks book I was writing at the time, and the idea was to distill the many different hardening steps you might want to perform on a server into a few simple steps that everyone should do. In this article, I take the same approach only with a specific focus on hardening cloud infrastructure. I'm most familiar with AWS, so my hardening steps are geared toward that platform and use AWS terminology (such as Security Groups and VPC), but as I'm not a fan of vendor lock-in, I try to include steps that are general enough that you should be able to adapt them to other providers.

New Accounts Are (Relatively) Free; Use Them

One of the big advantages with cloud infrastructure is the ability to compartmentalize your infrastructure. If you have a bunch of servers racked in the same rack, it might be difficult, but on cloud infrastructures, you can take advantage of the technology to isolate one customer from another to isolate one of your infrastructure types from the others. Although this doesn't come completely for free (it adds some extra overhead when you set things up), it's worth it for the strong isolation it provides between environments.

One of the first security measures you should put in place is separating each of your environments into its own high-level account. AWS allows you to generate a number of different accounts and connect them to a central billing account. This means you can isolate your development, staging and production environments (plus any others you may create) completely into their own individual accounts that have their own networks, their own credentials and their own roles totally isolated from the others. With each environment separated into its own account, you limit the damage attackers can do if they compromise one infrastructure to just that account. You also make it easier to see how much each environment costs by itself.

In a traditional infrastructure where dev and production are together, it is much easier to create accidental dependencies between those two environments and have a mistake in one affect the other. Splitting environments into separate accounts protects them from each other, and that independence helps you identify any legitimate links that environments need to have with each other. Once you have identified those links, it's much easier to set up firewall rules or other restrictions between those accounts, just like you would if you wanted your infrastructure to talk to a third party.

Go to Full Article

FOSS Project Spotlight: CloudMapper, an AWS Visualization Tool

Scott Piper — Thu, 29 Mar 2018 16:18:52 +0000

by Scott Piper

Duo Security has released CloudMapper, an open-source tool for visualizing Amazon Web Services (AWS) cloud environments.

When working with AWS, it's common to have a number of separate accounts run by different teams for different projects. Gaining an understanding of how those accounts are configured is best accomplished by visually displaying the resources of the account and how these resources can communicate. This complements a traditional asset inventory.

Duo built CloudMapper to generate interactive network diagrams of AWS accounts and released it as open source on Github.

See a demo here.

Figure 1. Screenshot of CloudMapper Visualizing a Demo Account

Using CloudMapper, you can quickly answer a number of questions, such as:

Which resources are publicly exposed?
What resources can communicate internally with which other resources?
Do you have a robust architecture in the event of an availability zone failure?
How many regions is this account using? How "big" is this account? How complex is it?

CloudMapper allows engineers to double-check their understanding of what they've built, quickly understand other environments and present that information to other stakeholders.

How It Works

There are three steps to getting up and running with CloudMapper:

Collect information about an AWS account via a shell script that uses the AWS CLI.
Convert that data into a format usable by the web browser.
Run a simple web server to view the collected data in your browser.

The first step of collecting information only requires the privileges to describe and list information about an account. This can be done with the AWS SecurityAudit policy. If you don't have direct access to the account, someone who does can run this script and send you the bundle of files it creates.

The second step of converting these cached files into something for the web browser display is where most of the logic is. This is where the Security Groups are analyzed to determine what network pathways exist, and parent/child relationships are created between nodes, such as EC2 instances, and compound node structures, such as subnets, availability zones, VPCs, regions and accounts.

Go to Full Article

DivvyCloud Platform for VMware Cloud on AWS

James Gray — Wed, 08 Nov 2017 14:03:52 +0000

by James Gray

DivvyCloud's unique niche in the IT ecosystem is helping organizations automate and manage their multi-cloud infrastructure at scale. The latest innovation from the company is the DivvyCloud Platform for VMware Cloud on AWS, a solution enabling consistent policy enforcement and automation of cloud best practices to customers of VMware Cloud on Amazon Web Services (AWS).

VMware Cloud on AWS unites VMware's enterprise-class Software-Defined Data Center (SDDC) software together with the elastic, bare-metal infrastructure from AWS, which results in a consistent operating model and application mobility for the private and public cloud.

DivvyCloud maintains that its software is unique in the marketplace, due to its ability to track real-time changes across clouds and take customer-defined, autonomous action to fix problems and ensure policy compliance. Standard automation bots proactively address myriad security, cost and compliance challenges commonly faced by any organization adopting or expanding cloud-based infrastructure. The visibility and policy automation afforded by the DivvyCloud solution remediate security, cost and compliance issues, adds the firm.

Go to Full Article

AWS Quickstart for Kubernetes

Craig McLuckie — Wed, 28 Jun 2017 16:27:03 +0000

by Craig McLuckie

Kubernetes is an open-source cluster manager that makes it easy to run Docker and other containers in production environments of all types (on-premises or in the public cloud). What is now an open community project came from development and operations patterns pioneered at Google to manage complex systems at internet scale.

AWS Quick Starts are a simple and convenient way to deploy popular open-source software solutions on Amazon’s infrastructure. While the current Quick Start is appropriate for development workflows and small team use, we are committed to continuing our work with the Amazon solutions architects to ensure that it captures operations and architectural best practices. It should be easy to get started now, and achieve long term operational sustainability as the Quick Start grows.

Our hope is that you will be able to use the CloudFormation template and written guide to get going quickly with Kubernetes. Or, wire the Quick Start template into CloudFormation templates you already have, bringing Cloud Native Computing elements on Amazon’s infrastructure to your existing solutions.

It is also worth mentioning that the AWS Quick Start represents our first upstream-friendly, supported configuration. At Heptio we are working hard to make Kubernetes more accessible to developers everywhere, and to provide quality support and services to Kubernetes users who want a clean, friendly, supported configuration of the upstream open-source project.

You can expect to see us put the work into maintaining and enhancing this Quick Start. We also view it as a way to help other key members of the Kubernetes ecosystem deliver value on the Amazon platform. We believe it will “take a village” to bring the full potential of Cloud Native Computing to the enterprise, so we are passionate about helping our partners realize the full potential of their technology on a convenient Kubernetes base.

Working with our friends at Tigera, we have integrated Project Calico into the AWS Quick Start so you have production-ready, secure networking right out of the box. Check out their Calico for Kubernetes guide here.

Go to Full Article

Puppet's Cloud Discovery: Know What's Running in Your Cloud

John S. Tonello — Wed, 07 Jun 2017 12:11:19 +0000

by John S. Tonello

The promise of automation always has been its ability to manage a wide range of tasks across all your systems, whether they're in your own data center or somewhere in the cloud. But in order to automate, you need to know what you have, and that's getting harder these days.

We've all come across orphaned cloud VMs and instances, perhaps spun up for a quick test by a developer, created as a bit of shadow IT or merely forgotten during the press of the latest product release. Regardless of why they were created and forgotten, these instances pose quite a few risks to your time, security and budget. After all, the meter's pretty much always running on cloud instances, orphaned or not.

With AWS, Google Compute Engine, Azure and others, it's easier than ever to spin up new instances and just as easy to lose track of them. It's also easy to lose track of what's running on them, if you ever fully knew at all. Automating those services and packages would help, but it's hard to automate what you don't know you have.

Puppet's new Cloud Discovery seeks to change that by giving you a clear picture of what's running on all your AWS instances—even the ones you didn't know you had. It moves beyond the cloud infrastructure and instances to show you their workloads and everything from packages and firewall rules to users and OS variants.

The idea is the more you know, the better able you are manage it with automation tools like Puppet. But if you haven't gotten around to using Puppet, don't worry. You don't need it to use Cloud Discovery. In fact, you don't need to install anything.

You start by creating a Cloud Discovery account and adding your AWS EC2 credentials (for as many accounts as you want), and the tool authenticates with the cloud service. In its first iteration, due out in Q3, Cloud Discovery will work with Amazon Web Services, but Puppet plans to add support for Google, Azure, VMware and others after that.

Once connected, Cloud Discovery uses the AWS API to gather information about your instances, including each private and public hostname, when it was launched and the listed security groups. It does this initial scan across your cloud account quickly and in real time.

From the result list, you can choose which instances to deep-scan—perhaps just instances in a certain IP range or just the instances in a certain AWS region, like west-2. When it's released, the folks at Puppet aim to give Cloud Discovery users a number of choices based on data points offered up by the AWS API.

Go to Full Article