Stacks on Stacks

The Serverless Ecosystem Blog by Stackery.

How Do You Actually Create a Diverse Team in Tech?
Nicole Blystone

Nicole Blystone | September 19, 2018

How Do You Actually Create a Diverse Team in Tech?

It’s easy to say that you value diversity and inclusivity, and that it’s important that people from all walks of life of represented and heard from in your work environment. The perspectives a diverse team can bring to the table, along with the variety of tools and solutions, is appealing, and it feels good to be able to say that you respect and learn from those different ways of seeing the world. But it’s one thing to say it, it’s another thing to mean it, and it’s yet a third thing to actually demonstrate it.

So how do you walk the walk of an inclusive work environment rather than just talking the talk?

It’s not as simple as saying, “We are going to hire a diverse team!” While that sentiment is great, it doesn’t do anything to ensure that you’ll get a group of candidates that are actually from many different walks of life. Fortunately, there are lots of ways to reach diverse recruits in the age of the internet. Most major cities have groups of women, people of color, and other underrepresented communities who run organizations geared toward empowering professionals in their particular fields.

In Portland, for example, we have PDX Women in Tech. Their organization advocates for women to enter careers in the tech field, offers the ability to post jobs through their website, and helps connect women to jobs in a field that is often male-dominated. There are many other resources for getting job postings to diverse candidates, including hiring groups like Hire <Div>ersity, that can work as strategic partnerships to encourage diverse hiring practices.

More than just getting the word out through many channels, it’s important to make sure to discuss things like unconscious bias with your hiring team. Whether we mean to or not, it can be easy to form opinions about a potential recruit in advance based on things like appearance, age, race, gender, and many other factors. The process of judging a candidate can begin with something as simple as seeing their name before you even have a chance to meet with them. Making sure that any team members meeting with a candidate are aware of unconscious bias can help mitigate some of the damage these stereotypes can cause.

Conversations about the importance of diversity, including making sure that the team is aware of unconscious bias, set an important foundation for ensuring that everyone is on the same page about hiring goals. Transparency around diversity goals helps ensure that the existing team understand the importance (and benefits of) having people from a variety of backgrounds and representations join the company. Making inclusivity a regular part of the conversations, and making sure it is something that can be talked about opening, helps promote the idea of within.

Finally, one of the most important things a company can do to make sure that they are walking the walk of inclusivity and diversity is this: make your company a safe place for people from many backgrounds to be valued and to grow.

Improving the Workplace

There are lots of ways to create a safe place for employees. Create clear and open communication channels, and encourage people to use them (because, once again, transparency is important). Managers should work to make sure that employees have a voice in their work and in the decision making around it. Empowered employees feel a sense of ownership in their work and in the company itself. Allow for flexible schedules so that people from many different backgrounds have the freedom arrange their schedules around their beliefs and their needs, whether those are cultural, medical, or otherwise. Give staff time to get to know one another outside of their work, whether that’s team lunches, happy hours, off-site meetings, or team building activities, and be sensitive to whether or not everyone will be able to participate in those activities. Be mindful, patient, and open to conversations around diversity.

Hiring a diverse team isn’t as simple as putting a line in an employee handbook stating that inclusion is a company value. It takes work, and it’s not something that happens overnight. Luckily, it’s work that creates so much value in terms of the ideas that can be brought to the table, the perspectives that might otherwise get missed, and the great pleasure of getting to know people who have lived lives different from your own.

Disaster Recovery in a Serverless World - Part 2
Apurva Jantrania

Apurva Jantrania | September 17, 2018

Disaster Recovery in a Serverless World - Part 2

This is part two of a multi-part blog series. In the previous post, we covered Disaster Recovery planning when building serverless applications. In this post, we’ll discuss the systems engineering needed for an automated solution in the AWS cloud.

As I started looking into implementing Stackery’s automated backup solution, my goal was simple: In order to support a disaster recovery plan, we needed to have a system that automatically creates backups of our database to a different account and to a different region. This seemed like a straightforward task, but I was surprised to find that there was no documentation on how to do this in an automated, scalable solution - all existing documentation I could find only discussed partial solutions and were all done manually via the AWS Console. Yuck.

I hope that this post will make help fill that void and help you understand how to implement an automated solution for your own disaster recovery solution. This post does get a bit long so if that’s not your thing, see the tl;dr.

The Initial Plan

AWS RDS has automated backups which seemed like the perfect platform to base this automation upon. Furthermore, RDS even emits events that seem ideal for using to kick off a lambda function that will then copy the snapshot to the disaster recovery account.

Discoveries

The first issue I discovered was that AWS does not allow you to share automated snapshots - AWS requires that you first make a manual copy of the snapshot before you can share it with another account. I initially thought that this wouldn’t be a major issue - I can easily make my lambda function first kick off a manual copy. According to the RDS Events documentation, there is an event RDS-EVENT-0042 that would fire when a manual snapshot was created. I could then use that event to then share the newly created manual snapshot to the disaster recovery account.

This leads to the second issue - while RDS will emit events for snapshots that are created manually, it does not emit events for snapshots that are copied manually. The AWS docs aren’t clear about this and it’s an unfortunate feature gap. This means that I have to fall back to a timer based lambda function that will search for and share the latest available snapshot.

Final Implementation Details

While this ended up more complicated than initially envisioned, Stackery still makes it easy to add all the needed pieces for fully automated backups. My implementation ended up looking like this:

The DB Event Subscription resource is a CloudFormation Resource in which contains a small snippet of CloudFormation that subscribes the DB Events topic to the RDS database

Function 1 - dbBackupHandler

This function will receive the events from the RDS database via the DB Events topic. It then creates a copy of the snapshot with an ID that identifies the snapshot as an automated disaster recovery snapshot

const AWS = require('aws-sdk');
const rds = new AWS.RDS();

const DR_KEY = 'dr-snapshot';
const ENV = process.env.ENV;

module.exports = async message => {
  // Only run DB Backups on Production and Staging
  if (!['production', 'staging'].includes(ENV)) {
    return {};
  }

  let records = message.Records;
  for (let i = 0; i < records.length; i++) {
    let record = records[i];

    if (record.EventSource === 'aws:sns') {
      let msg = JSON.parse(record.Sns.Message);
      if (msg['Event Source'] === 'db-snapshot' && msg['Event Message'] === 'Automated snapshot created') {
        let snapshotId = msg['Source ID'];
        let targetSnapshotId = `${snapshotId}-${DR_KEY}`.replace('rds:', '');

        let params = {
          SourceDBSnapshotIdentifier: snapshotId,
          TargetDBSnapshotIdentifier: targetSnapshotId
        };

        try {
          await rds.copyDBSnapshot(params).promise();
        } catch (error) {
          if (error.code === 'DBSnapshotAlreadyExists') {
            console.log(`Manual copy ${targetSnapshotId} already exists`);
          } else {
            throw error;
          }
        }
      }
    }
  }

  return {};
};

A couple of things to note:

  • I’m leveraging Stackery Environments in this function - I have used Stackery to define process.env.ENV based on the environment the stack is deployed to
  • Automatic RDS snapshots have an id that begins with ‘rds:’. However, snapshots created by the user cannot have a ‘:’ in the ID.
  • To make future steps easier, I append dr-snapshot to the id of the snapshot that is created

Function 2 - shareDatabaseSnapshot

This function runs every few minutes and shares any disaster recovery snapshots to the disaster recovery account

const AWS = require('aws-sdk');
const rds = new AWS.RDS();

const DR_KEY = 'dr-snapshot';
const DR_ACCOUNT_ID = process.env.DR_ACCOUNT_ID;
const ENV = process.env.ENV;

module.exports = async message => {
  // Only run on Production and Staging
  if (!['production', 'staging'].includes(ENV)) {
    return {};
  }

  // Get latest snapshot
  let snapshot = await getLatestManualSnapshot();

  if (!snapshot) {
    return {};
  }

  // See if snapshot is already shared with the Disaster Recovery Account
  let data = await rds.describeDBSnapshotAttributes({ DBSnapshotIdentifier: snapshot.DBSnapshotIdentifier }).promise();
  let attributes = data.DBSnapshotAttributesResult.DBSnapshotAttributes;

  let isShared = attributes.find(attribute => {
    return attribute.AttributeName === 'restore' && attribute.AttributeValues.includes(DR_ACCOUNT_ID);
  });

  if (!isShared) {
    // Share Snapshot with Disaster Recovery Account
    let params = {
      DBSnapshotIdentifier: snapshot.DBSnapshotIdentifier,
      AttributeName: 'restore',
      ValuesToAdd: [DR_ACCOUNT_ID]
    };
    await rds.modifyDBSnapshotAttribute(params).promise();
  }

  return {};
};

async function getLatestManualSnapshot (latest = undefined, marker = undefined) {
  let result = await rds.describeDBSnapshots({ Marker: marker }).promise();

  result.DBSnapshots.forEach(snapshot => {
    if (snapshot.SnapshotType === 'manual' && snapshot.Status === 'available' && snapshot.DBSnapshotIdentifier.includes(DR_KEY)) {
      if (!latest || new Date(snapshot.SnapshotCreateTime) > new Date(latest.SnapshotCreateTime)) {
        latest = snapshot;
      }
    }
  });

  if (result.Marker) {
    return getLatestManualSnapshot(latest, result.Marker);
  }

  return latest;
}
  • Once again, I’m leveraging Stackery Environments to populate the ENV and DR_ACCOUNT_ID environment variables.
  • When sharing a snapshot with another AWS account, the AttributeName should be set to restore (see the AWS RDS SDK)

Function 3 - copyDatabaseSnapshot

This function will run in the Disaster Recovery account and is responsible for detecting snapshots that are shared with it and making a local copy in the correct region - in this example, it will make a copy in us-east-1.

const AWS = require('aws-sdk');
const rds = new AWS.RDS();

const sourceRDS = new AWS.RDS({ region: 'us-west-2' });
const targetRDS = new AWS.RDS({ region: 'us-east-1' });

const DR_KEY = 'dr-snapshot';
const ENV = process.env.ENV;

module.exports = async message => {
  // Only Production_DR and Staging_DR are Disaster Recovery Targets
  if (!['production_dr', 'staging_dr'].includes(ENV)) {
    return {};
  }

  let [shared, local] = await Promise.all([getSourceSnapshots(), getTargetSnapshots()]);

  for (let i = 0; i < shared.length; i++) {
    let snapshot = shared[i];
    let fullSnapshotId = snapshot.DBSnapshotIdentifier;
    let snapshotId = getCleanSnapshotId(fullSnapshotId);
    if (!snapshotExists(local, snapshotId)) {
      let targetId = snapshotId;

      let params = {
        SourceDBSnapshotIdentifier: fullSnapshotId,
        TargetDBSnapshotIdentifier: targetId
      };
      await rds.copyDBSnapshot(params).promise();
    }
  }

  return {};
};

// Get snapshots that are shared to this account
async function getSourceSnapshots () {
  return getSnapshots(sourceRDS, 'shared');
}

// Get snapshots that have already been created in this account
async function getTargetSnapshots () {
  return getSnapshots(targetRDS, 'manual');
}

async function getSnapshots (rds, typeFilter, snapshots = [], marker = undefined) {
  let params = {
    IncludeShared: true,
    Marker: marker
  };

  let result = await rds.describeDBSnapshots(params).promise();

  result.DBSnapshots.forEach(snapshot => {
    if (snapshot.SnapshotType === typeFilter && snapshot.DBSnapshotIdentifier.includes(DR_KEY)) {
      snapshots.push(snapshot);
    }
  });

  if (result.Marker) {
    return getSnapshots(rds, typeFilter, snapshots, result.Marker);
  }

  return snapshots;
}

// Check to see if the snapshot `snapshotId` is in the list of `snapshots`
function snapshotExists (snapshots, snapshotId) {
  for (let i = 0; i < snapshots.length; i++) {
    let snapshot = snapshots[i];
    if (getCleanSnapshotId(snapshot.DBSnapshotIdentifier) === snapshotId) {
      return true;
    }
  }
  return false;
}

// Cleanup the IDs from automatic backups that are prepended with `rds:`
function getCleanSnapshotId (snapshotId) {
  let result = snapshotId.match(/:([a-zA-Z0-9-]+)$/);

  if (!result) {
    return snapshotId;
  } else {
    return result[1];
  }
}
  • Once again, leveraging Stackery Environments to populate ENV, I ensure this function only runs in the Disaster Recovery accounts

TL;DR - How Automated Backups Should Be Done

  1. Have a function that will manually create an RDS snapshot using a timer and lambda. Use a timer that makes sense for your use case
    • Don’t bother trying to leverage the daily automated snapshot provided by AWS RDS.
  2. Have a second function, that monitors for the successful creation of the snapshot from the first function and shares it to your disaster recovery account.

  3. Have a third function that will operate in your disaster recovery account that will monitor for snapshots shared to the account, and then create a copy of the snapshot that will be owned by the disaster recovery account, and in the correct region.
Empathy Through Observation: A User Testing Reality Check
Anna Yovandich

Anna Yovandich | September 13, 2018

Empathy Through Observation: A User Testing Reality Check

Engineers, by the nature of their work, cannot objectively experience their product like a legitimate user. While busy cranking out new features and pushing the product forward, it’s common to accrue some technical debt in the codebase. However, sprints to the finish line are likely increasing debt in an arguably more critical area: usability.

Usability is the measure of ease-of-use and learnability. In product design, it’s the glue that keeps people coming back and compels them to tell others about their indispensable discovery. Without it, they’ll give up, move on, and may very well feel insulted on the way out - all in the blink of an eye, pound of a key, or in the quiet defeat of not knowing what the hell to do next. It’s easy, on a product team, to build and pave our workflows with blinders on (makes sense on my machine!). We know how the thing works so well that our empathy becomes lost in the assumption that everyone will find their way through our tried and not-so-tested paths.

Empathy - the ability to understand and share feelings - is increasingly obscured when we don’t check in with the people for whom we’re building these tools. Examining new perspectives and understanding different personas are key to detaching from our own well worn assumptions. By observing user behavior in a few test sessions, interaction patterns begin to surface that expose design flaws varying from simple fix to total redo: words that go to waste, workflows that frustrate, and features that intimidate. These are a few common challenges facing product usability that engineers may fail to notice, or inadvertently train themselves to ignore.

Words That Go to Waste

Words often go unread. The more words there are, the less likely someone is to read them. When features and functionality are explained left and right, users feel fatigued. Verbose dialog causes people to turn the other way, assume it’s complicated, and disconnect. Maybe they’ll just ignore that feature and carry on with vague dissatisfaction, or maybe they’re one step closer to signing out forever. Describing a feature isn’t inherently bad, but if it comes with an instruction manual it’s likely to be ignored. If the message isn’t short and sweet, save it for the docs.

Workflows That Frustrate

Many times, users are forced through a set of instructions that might not make sense to them, in order to serve needs of the app (e.g. data gathering and 3rd party integration). Users forced into a prerequisite workflow - requiring them to perform actions before they can explore - may result in frustration as the first impression or ultimately, a rage quit. Instead, maximize functionality by minimizing restriction. How far can a person get in the experience just by signing in? Reducing barriers enables experimentation and learning. Isolating constraints to an atomic level - the moment of need - liberates the user experience and showcases what the product can do.

Features That Intimidate

Features require discovery and learning. Ideally, that occurs intuitively and without cognitive awareness. One of the toughest pills to swallow during usability testing is finding that robust functionality is cryptic, intimidating, or elusive. A test participant waves their mouse over an entire area: “I don’t know what thiiisss does.” It’s likely doing too much or lacks useful context. The single responsibility principle that shapes sturdy application development can provide a solid construct for product design as well. When a feature is overloaded with functionality, consider ways to split it into smaller parts, provide clear context, and support intuitive discovery.

These are some common pitfalls we found through our own user testing that will help inform design decisions going forward. Usability testing isn’t an end-all to holistic product design but it is a necessary practice for gaining insights into pain points, drop offs, and blind spots. In addition to revealing design flaws, these tests are imperative in reminding us that people will use an app much differently than those who build it.

How to Write 200 Lines of YAML in 1 Minute
Anna Spysz

Anna Spysz | September 11, 2018

How to Write 200 Lines of YAML in 1 Minute

Last month, our CTO Chase wrote about why you should stop YAML engineering. I completely agree with his thesis, though for slightly different reasons. As a new developer, I’ve grasped that it’s crucial to learn and do just what you need and nothing more - at least when you’re just getting started in your career.

Now, I’m all about learning for learning’s sake - I have two now-useless liberal arts degrees that prove that. However, when it comes to being a new developer, it’s very easy to get overwhelmed by all of the languages and frameworks out there, and get caught in paralysis as you jump from tutorial to tutorial and end up not learning anything very well. I’ve certainly been there - and then I decided to just get good at the tools I’m actually using for work, and learn everything else as I need it.

Which is what brings us to YAML - short for “YAML Ain’t Markup Language”. I started out as a Python developer. When I needed to, I learned JavaScript. When my JavaScript needed some support, I learned a couple of front-end frameworks, and as much Node.js as I needed to write and understand what my serverless functions were doing. As I got deeper into serverless architecture, it seemed like learning YAML was the next step - but if it didn’t have to be, why learn it? If I can produce 200+ lines of working YAML without actually writing a single line of it, in much less time than it would take me to write it myself (not counting the hours it would take to learn a new markup language), then that seems like the obvious solution.

So if a tool allows me to develop serverless apps without having to learn YAML, I’m all for that. Luckily, that’s exactly what Stackery does, as you can see in the video below:

Creating Your First AWS RDS Database with Stackery
Toby Fee

Toby Fee | September 10, 2018

Creating Your First AWS RDS Database with Stackery

Once you’ve built your first Lambda, you’ll need a datastore.

AWS does have an official instruction guide to help with this, but the official AWS instruction guide is extensive. You need to set up at least four resources before you can deploy your first RDS table, and by the time their 2,000-word guide is done you still haven’t even selected your DB format.

Thankfully there’s a much easier way: The deployment service Stackery, can get your RDS up and running and hooked up to Lambdas in just a few minutes.

Getting Started with RDS

AWS’s Relational Database Service (RDS) is a good choice if you’re familiar with SQL databases already or just prefer the reliable classic of tables over NoSQL.

At first glance, RDS looks like it’s basically just database hosting from Amazon, but in reality, it’s very different from their traditional server options hosted on EC2. When looking the tooling closely, it’s obvious RDS is as different from a traditional DB as a Lambda is different from a web server.

A few key differences:

  1. No OS management — While you can select the exact DB type and version (e.g., PostgreSQL, MySQL, MariaDB, Oracle, Microsoft SQL Server, Aurora) the operating system level is fully virtualized
  2. AWS’s usual restrictive access policies — Centrally, this is a positive thing: Amazon forces you to be in tight control of what services can access each other. As a result most beginners often end up stumped when, after trying to stand up their RDS, they want to connect to it from their laptop.
  3. Indescribably smooth uptime management — With the concept of an Availability Zone (AZ) you can create multiple instances that Amazon can automatically use for failover

(Seen here: two availability zones in action)

  1. And finally, backups and scaling are simpler — Because if virtualization doesn’t make scaling and backups easier, what the heck is it even for, right?

Deploy an RDS Table

Start by creating a new Stack in the Stackery UI and deleting the default nodes. We’ll add one node for our database and a second for a lambda to access DB data. You’ll need to set a DB name. Note that this is not the database name for DB requests, just the name it’ll be referred to in the Stackery UI. Set a root password here as well. Be sure to record the password somewhere for later use in this tutorial.

For the moment, our Lambda function will be fine with the default settings, but we’ll edit it later to retrieve records from the database.

This tutorial doesn’t cover populating the database in detail, since the most likely scenario is that you’ll have test data you’d like to populate. Let’s set up our Lambda to load records from the account DB.

To get access to edit your Lambda’s code: commit this stack in the Stackery UI (no need to deploy yet), then click the commit ID to go to the stack code in Github.

Next, clone the repository. Then you can edit it in whatever editor you prefer.

Let’s populate the Lambda with some code to access records from the DB:

const knex = require('knex');
cont dbInfo = {
  development: {
    client: 'mysql',
    connection: {
      host: 'localhost',
      user: 'root',
      password: 'mySecretPW',
      database: 'accounts'
    }
  },

  production: {
    client: 'mysql',
    connection: {
      host,
      user: 'root',
      password: process.env.DB_PASSWORD,
      database: 'accounts'
    }
  }
};

const connectionName = process.env.CONNECTION || 'development';
const connection = dbInfo[connectionName];

/**
 * Fetch list of accounts from database and respond with an array of account
 * names.
 */
module.exports = async message => {
  const client = knex(connection);

  try {
    const records = await client('accounts').select('name');
    return records.map(record => record.name);
  } finally {
    client.destroy();
  }
};

A few things to note about this code:

  • As is the default for new Lambda Resources in Stackery, this is written in Node 8, with support for finally{}.
  • This code relies on inline connection info, but you’ll more likely want to share that info between multiple functions — See how to share code modules between Javascript functions in a separate guide.
  • If you’ve written code to access a database from a web app, you might be surprised to discover we’re creating and destroying a client for every request. Since we’re deploying this code to a serverless environment, it’s critical that we don’t max out our connections on smaller RDS instances. It’s quite easy for multiple requests to our Lambda to result in a point of 65 simultaneous connections, which is the limit on a db.t2.micro instance — We’ve previously discussed database connection usage in a serverless context on our blog.

Push the new Lambda code up to the GitHub repository, and refresh the Stackery UI to show your updates.

The last step is to give the Lambda function the database password. It expects to find it as a config variable for this Lambda — which you can set from the Stackery UI — but this isn’t an ideal place to put the password, since it will be visible in both the Stackery UI and the GitHub repo for this stack.

Instead, set two environment variables CONNECTION to production and DB_PASSWORD to ${config.dbPassword}, so the password will be populated from the environment config.

You can access ‘environments’ from the menu at the top right. Finally, set any environment variables as standard Javascript Object Notation (JSON).

Now your stack should be ready to commit, prepare, and deploy!

Why You Should Stop YAML Engineering
Chase Douglas

Chase Douglas | August 30, 2018

Why You Should Stop YAML Engineering

Here’s some sample JavaScript code:

let foo = 5;

Do you know register held the value 5 when executing that code? Do you even know what a register is or what are plausible answers to the question?

The correct answer to this question is: No, and I don’t care. Registers are locations in a CPU that can hold numerical values. Your computing device of choice calculates values using registers millions of times each second.

Way back when computers were young people programmed them using assembly languages. The languages directly tell CPUs what to do: load values into registers from memory, calculate other values in the registers, save values back to memory, etc.

Between 1943 and 1945, Konrad Zuse developed the first (or at least one of the first) high-level programming languages: Plankalkül. It wasn’t pretty by modern standards, but its thesis could be boiled down to: you will program more efficiently and effectively using logical formulas. A more blunt corollary might be: Stop programming CPU registers!

A Brief History Of Computing Abstractions

Software engineering is built upon the practice of developing higher-order abstractions for common tasks. Allow me to evince this pattern:

1837 — Charles Babbage: Stop computing by pencil and paper! (Analytical Engine)

1938 — Conrad Zuse: Stop computing by hand! (Z1, first electrical computer)

1950 — UNIVAC: Stop repeating yourself! (UNIVAC 1101, first computer with programs stored in memory)

1985 — Bjarne Stroustrup: Stop treating everything like primitives! (C++ language, introduced object-oriented-programming to the C language)

1995 — James Gosling: Stop tracking memory! (Java language makes garbage collection mainstream)

2006 — Amazon: Stop managing data centers! (AWS EC2 makes virtual servers easy)

2009 — Ryan Dahl: Stop futzing with threads! (Node.js introduces event/callback-based semantics)

2011 — Amazon: Stop provisioning resources manually! (AWS CloudFormation makes Infrastructure-As-Code easier)

2014 — Amazon: Stop managing servers! (AWS Lambda makes services “functional”)

Software engineers are always finding ways to make whatever “annoying thing” they have to deal with go away. They do this by building abstractions. It’s now time to build another abstraction.

2018: Stop YAML Engineering!

The combination of infrastructure-as-code and serverless apps means developers are inundated with large, complex infrastructure templates, mostly written in YAML.

It’s time we learn from the past. We are humans who work at a logical, higher-order abstraction level. Infrastructure-as-code is meant to be consumed by computers. We need to abstract away the compiling of our logic into infrastructure code.

That’s what Stackery does. It turns this…

…into Serverless Application Model (SAM) YAML without asking the user to do anything more than drag and wire resources in a canvas. Stackery’s own backend is over 2,000 lines of YAML. We wouldn’t be able to manage it all without a tool that helps us both maintain and diagram the relationships between resources. It even performs this feat in both directions: you can take your existing SAM applications, import them into Stackery, and instantly visualize and extend the infrastructure architecture.

You may be a principal architect leading adoption of serverless in your organization. You may be perfectly capable of holding thousands of lines of infrastructure configuration in your head. But can everyone on your team do the same? Would your organization benefit from the automatic visualization and relationship/dependency management of a higher-order infrastructure-as-code abstraction?

As can be seen throughout the history of software engineering, the industry will move (and already is moving) to abstract away this lower level of engineering. Check out Stackery if you want to stay ahead of the curve.

GitLab + Stackery = Serverless CI/CD <3
Sam Goldstein

Sam Goldstein | August 28, 2018

GitLab + Stackery = Serverless CI/CD <3

GitLab is a git hosting solution which features a built-in CI/CD pipeline that automates the delivery process. We’ve seen more and more serverless development teams asking how they can integrate their GitLab with Stackery. I am happy to announce that today Stackery features full support for GitLab source code hosting and serverless CI/CD deployments.

By linking your Stackery account with GitLab you can quickly develop and deploy serverless apps. Stackery helps generate AWS SAM YAML infrastructure-as-code templates and manage their Lambda source code, integrating directly with GitLab’s source code hosting. However the bigger payoff is taking full advantage of Stackery’s serverless deployment automation which is intentionally simple to integrate into GitLab’s CI/CD release automation. Stackery’s CLI deployment tool is a cross-compiled Go binary with no external dependencies. It’s just one step to download and bundle it in your repo and you then it’s simple to invoke it from your GitLab project’s .gitlab-ci.yml.

Here’s a basic example showing how to integrate Stackery into you .gitlab-ci.yml

stages:
  - test
  - build
  - deploy

test:
  stage: test
  script: echo "Running tests"

build:
  stage: build
  script: echo "Building the app"

deploy_staging:
  stage: deploy
  script:
    - stackery deploy --stack-name "myStack" --env-name "staging" --git-ref "$CI_COMMIT_SHA"
  environment:
    name: staging
    url: https://staging.example.com
  only:
  - master

By integrating Stackery and GitLab you can take advantage of a number of interesting features to take your serverless deployment automation to the next level. For example:

  • GitLab pipeline security can be used to provide automated production change control for serverless applications.
  • GitLab’s environments and deployments are straight-forward to integrate with stackery deploy and can be used to orchestrate sophisticated CI/CD pipelines across multiple AWS accounts and environments.
  • Serverless Deploy from Chat is great. You know you’re doing it right when you’re deploying serverless SAM applications by chat. 💬🦊λ 🙌

We hope you enjoy this new GitLab integration.

AWS Serverless Application Model YAML Templates: An Introduction
Toby Fee

Toby Fee | August 21, 2018

AWS Serverless Application Model YAML Templates: An Introduction

If you’ve never worked with YAML before, you’ll probably find the basic template format daunting. YAML — which somehow doesn’t stand for Yet Another Markdown Language, but instead YAML Ain’t Markdown Language — is a simplified way of storing data structures. YAML is popular with Rails users and sees some popularity with all higher order language programmers (JS, Python, etc…).

Clean, Simple Formatting

YAML is popular with languages that tend to see only three types of data:

  • Scalars: numbers and strings, the ‘text’ of your data
  • Heaps: e.g. objects, dictionaries, or any other term for a mapping of key-value pairs
  • Lists: i.e. arrays or an ordered sequence of the other two data types

YAML primarily tries to produce configuration documents that are readable without any special interpretation and formatting. If you look at even a short snippet of YAML from the middle of a SAM configuration file the key-value pairs are pretty readable, and you can even guess how lists might be used:

   DynamoDBTable:
     Type: AWS::DynamoDB::Table
     Properties:
       AttributeDefinitions:
         - AttributeName: id
           AttributeType: S

Indentation is used for structure, colons separate key-value pairs, and dashes are used to create “bullet” lists.

This simple, clean formatting also brings us to the most common complaint about YAML, it is whitespace-dependent, and missing a single tab can mess up the interpretation of the whole document.

YAML Basics

Always remember that YAML Lint is your friend. I still use yamllint.com every time I write YAML, turning on a display for invisible characters and purpose built linters for your IDE are all well and good, but that final check of YAML is still crucial.

YAML vs. XML

If you’re familiar with XML you may well have already interacted with systems that use either YAML or XML for configuration, or noticed that they are used by competing data storage systems. YAML is instantly readable, XML is not. On the other hand, the X in XML stands for “extensible” and while YAML really only has three ways to store data, the markup available in XML is limitless.

YAML vs. JSON

JavaScript Object Notation (JSON) and YAML look more similar and share design goals. While JSON is designed to be a bit easier to compile programmatically, the biggest difference for everyday use is that YAML has a method for referencing other items in the same JSON file. This might seem pretty minor but when standing up multiple API pathways in a SAM file, the ability to say TABLE_NAME: !Ref Table is a huge convenience.

The official specifications for YAML are well written and have some general guidance on best practices, implementation differences, and the design goals of YAML.

AWS Serverless Application Model

AWS CloudFormation templates are a standardized specification for describing, documenting, and deploying components of a serverless application Let’s look at one of the shortest possible SAM files:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Resources:
    MyFirstFunction:
        Type: AWS::Serverless::Function
        Properties:
           Handler: index.handler
           Runtime: nodejs4.3
           CodeUri: s3://bucketName/codepackage.zip

With the Handler property set to index.handler, the code package at the CodeUri will be opened, CloudFormation will look for a file (“index”) and a function or module (“handler”). Both the AWSTemplateFormatVersion and Transform should be the same for all your Sam files, and the Type and Runtime properties are self-explanatory.

Events

While the template above will create a Lambda, it’ll be fundamentally incomplete since it needs something to trigger a Lambda in order to run. The Events property defines these triggers.

In general you’d use SAM files when you want to define multiple interlinked pieces of your application. This example (leaving off the top three lines that will be the same in every SAM file) grabs events from a DynamoDB table:

  TableAlerter:
    Type: AWS::Serverless::Function
    Properties:
      Handler: index.handler
      Runtime: nodejs6.10
      Events:
        Stream:
          Type: DynamoDB
          Properties:
            Stream: !GetAtt DynamoDBTable.StreamArn
            BatchSize: 100
            StartingPosition: TRIM_HORIZON  

Bringing it All Together

Most of the time SAM is the best choice because SAM can stand up an interlinked set of resources. So our SAM file will have more than one key under Resources.

Let’s stand up a table for the lambda above to read from:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  TableAlerter:
    Type: AWS::Serverless::Function
    Properties:
      Handler: index.handler
      Runtime: nodejs6.10
      Events:
        Stream:
          Type: DynamoDB
          Properties:
            Stream: !GetAtt DynamoDBTable.StreamArn
            BatchSize: 100
            StartingPosition: TRIM_HORIZON   

  Alerts:
    Type: AWS::DynamoDB::Table
    Properties:
      AttributeDefinitions:
        - AttributeName: id
          AttributeType: S
      KeySchema:
        - AttributeName: id
          KeyType: HASH
      ProvisionedThroughput:
        ReadCapacityUnits: 5
        WriteCapacityUnits: 5

At this point it might be helpful to use anchors from the YAML specification to share config information or try the AWS SAM system for creating and sharing environment variables.

The official AWS documentation on SAM isn’t particularly instructive, with just a few examples and some tutorial references. However the full specification is laid out in the AWSLabs GitHub project documentation.

Once you’ve mastered the basics, or if you’re feeling overwhelmed by the tool, you may want to use a service to create and deploy your stack via CloudFormation. Stackery does just that, and recently announced that Stackery now builds SAM templates natively.

Get the Serverless Development Toolkit for Teams

Sign up now for a 60-day free trial. Contact one of our product experts to get started building amazing serverless applications today.

To Top