AWS Lambda Layers

Serverless developers frequently import libraries and dependencies into their AWS Lambda functions. While you can zip these dependencies as part of the build and deployment process, in many cases it’s easier to use layers instead. You can configure your Lambda function to pull in additional code and content in the form of layers. A layer is a .zip file archive that contains libraries, a custom runtime, or other dependencies. With layers, you can use libraries in your function without needing to include them in your deployment package. Layers let you keep your deployment package small, which makes development easier. You can avoid errors that can occur when you install and package dependencies with your function code. You can create layers, or use layers that AWS or an AWS customer has published. Layers support resource-based policies for granting layer usage permissions to specific AWS accounts, AWS Organizations, or all accounts.

Layers are extracted to the /opt directory in the function execution environment. Each runtime looks for libraries in a different location under /opt, depending on the language. Structure your layer so that function code can access libraries without additional configuration. You can also use the AWS Serverless Application Model (AWS SAM) to manage layers and your function's layer configuration.

AWS CloudShell

AWS CloudShell is a browser-based shell that makes it easy to securely manage, explore, and interact with your AWS resources. CloudShell is pre-authenticated with your console credentials. Common development and operations tools are pre-installed, so no local installation or configuration is required. With CloudShell, you can quickly run scripts with the AWS Command Line Interface (AWS CLI), experiment with AWS service APIs using the AWS SDKs, or use a range of other tools to be productive. You can use CloudShell right from your browser and at no additional cost.

Benefits

No extra credentials to manage
CloudShell inherits the credentials of the user signed in to the AWS Management Console, so there's no need to spend extra effort managing credentials locally.

Always up to date
CloudShell provides a fully managed Amazon Linux 2 environment that has the latest versions of popular tools already installed and updated. You don't need to patch the environment or update the pre-installed tools.

No cost
CloudShell provides you with a browser-based shell to run scripts and commands. It includes 1 GB of persistent storage per Region at no extra cost to you. You only pay for the AWS resources you use with CloudShell to create and run your applications.

Customizable
With 1 GB of persistent storage per Region, you can store scripts, files, configuration preferences, and additional tools in your home directory. You can begin working immediately, without customizing your environment every time you use the shell.

Link: https://aws.amazon.com/cloudshell/

AWS Fargate

AWS Fargate is a compute engine for Amazon ECS that allows you to run containers without having to manage servers or clusters. With AWS Fargate, you no longer have to provision, configure, and scale clusters of virtual machines to run containers. This removes the need to choose server types, decide when to scale your clusters, or optimize cluster packing. AWS Fargate removes the need for you to interact with or think about servers or clusters. Fargate lets you focus on designing and building your applications instead of managing the infrastructure that runs them.

Amazon ECS has two modes: 

• Fargate launch type
• EC2 launch type

With Fargate launch type, all you have to do is package your application in containers, specify the CPU and memory requirements, define networking and IAM policies, and launch the application. EC2 launch type allows you to have server-level, more granular control over the infrastructure that runs your container applications. 

With EC2 launch type, you can use Amazon ECS to manage a cluster of servers and schedule placement of containers on the servers. Amazon ECS keeps track of all the CPU, memory and other resources in your cluster, and also finds the best server for a container to run on based on your specified resource requirements. You are responsible for provisioning, patching, and scaling clusters of servers. You can decide which type of server to use, which applications and how many containers to run in a cluster to optimize utilization, and when you should add or remove servers from a cluster. EC2 launch type gives you more control of your server clusters and provides a broader range of customization options, which might be required to support some specific applications or possible compliance and government requirements.

Link: https://aws.amazon.com/fargate/

Queues - RabbitMQ vs SQS

RabbitMQ
RabbitMQ supports powerful message routing via exchange. This is very important when we need to run the same job on a specific server, group of servers or all servers. Our application sends one message and exchange will route it. RabbitMQ also has vhosts so that multiple applications can share the same RabbitMQ server but be isolated from each other (we can create unique logins for separate applications to access their vhosts). RabbitMQ can be setup in clusters for redundancy / failover and will acknowledge receipt of messages.
RabbitMQ has a powerful GUI which is accessible http://localhost:15672/. Hosting for RabbitMQ offers fewer choices and is more expensive.

AWS SQS
AWS SQS takes care of managing our queues. AWS SDK offers low level access but shoryuken is a higher level integration (shoryuken author acknowledges sidekiq as inspiration). Retrying failed jobs will happen automatically unless the messages is explicitly deleted. We can only delay jobs for 15 minutes (or we get The maximum allowed delay is 15 minutes (RuntimeError)). Recurring jobs are not supported but there are workarounds with AWS lambda and CloudWatch. AWS SQS UI is decent and we can use AWS SDK to access data directly. SQS has other interesting features such as long polling, batch operations and dead letter queues. SQS also has FIFO queues which guarantee he order in which messages are sent and received (and does not allow dupes). However, FIFO queues only allow 300 TPS (much less that regular SQS). Shoryuken works with standard and FIFO queues.
Hosting - easy to setup and cheap (pay for what you use) but obviously only available on AWS. SQS is a great choice when we need to run LOTS of jobs or when we do not care about more advanced options such as scheduling.

	RabbitMQ	Amazon SQS
Description	RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.	Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use.
Pros	• It's fast and it works with good metrics/monitoring • Ease of configuration • I like the admin interface • Easy to set-up and start with • Intuitive work through python • Standard protocols • Durable • Written primarily in Erlang • Simply superb • Completeness of messaging patterns	• Easy to use, reliable • Low cost • Simple • Doesn't need to maintain • Delayed delivery upto 12 hours • Has a max message size (currently 256K) • Delayed delivery upto 15 mins only
Cons	• Needs Erlang runtime. Need ops good with Erlang runtime • Too complicated cluster/HA config and management • Configuration must be done first, not by your code	• SQS has guaranteed delivery, but messages can be delivered more than once.

AWS Service : Parameter Store

Parameter store provides a centralized store to manage your configuration data, whether plain-text data such as database strings or secrets such as passwords, encrypted through AWS KMS. With Parameter store, your critical information stays within your environment, saving you the manual overhead of storing and managing it in configuration files. Parameters can be easily re-used across your AWS configuration and automation workflows without having to type them in plain-text, improving your security posture. Parameters can be easily referenced across AWS services such as Amazon ECS and AWS Lambda, as well as other EC2 Systems Manager capabilities such as Run Command, State Manager, and Automation.

Through integration with AWS Identity and Access Management, you can provide access control to specific parameters, letting you provide access to the data only to the users who need them and on which resources they can be used. AWS Key Management Service (KMS) integration helps you encrypt your sensitive information and protect the security of your keys. Additionally, all calls to the parameter store are recorded with AWS CloudTrail so that they can be audited.

Parameter Store offers the following benefits and features.

• Use a secure, scalable, hosted secrets management service (No servers to manage).
• Improve your security posture by separating your data from your code.
• Store configuration data and secure strings in hierarchies and track versions.
• Control and audit access at granular levels.
• Configure change notifications and trigger automated actions.
• Tag parameters individually, and then secure access from different levels, including operational, parameter, EC2 tag, or path levels.
• Reference parameters across AWS services such as Amazon EC2, Amazon EC2 Container Service, AWS Lambda, AWS CloudFormation, AWS CodeBuild, AWS CodeDeploy, and other Systems Manager capabilities.
• Configure integration with AWS KMS, Amazon SNS, Amazon CloudWatch, and AWS CloudTrail for encryption, notification, monitoring, and audit capabilities.

Jenkins - Automation Tool

Jenkins is free software released under MIT License. Jenkins is an open source automation server written in Java. Jenkins helps to automate the non-human part of software development process, with continuous integration and facilitating technical aspects of continuous delivery. It is a server-based system that runs in servlet containers such as Apache Tomcat. It supports version control tools, including AccuRev, CVS, Subversion, Git, Mercurial, Perforce, ClearCase and RTC, and can execute Apache Ant, Apache Maven and sbt based projects as well as arbitrary shell scripts and Windows batch commands.

Features:

• Continuous Integration and Continuous Delivery - As an extensible automation server, Jenkins can be used as a simple CI server or turned into the continuous delivery hub for any project.
• Easy installation - Jenkins is a self-contained Java-based program, ready to run out-of-the-box, with packages for Windows, Mac OS X and other Unix-like operating systems.
• Easy configuration - Jenkins can be easily set up and configured via its web interface, which includes on-the-fly error checks and built-in help.
• Plugins - With hundreds of plugins in the Update Center, Jenkins integrates with practically every tool in the continuous integration and continuous delivery toolchain.
• Extensible - Jenkins can be extended via its plugin architecture, providing nearly infinite possibilities for what Jenkins can do.
• Distributed - Jenkins can easily distribute work across multiple machines, helping drive builds, tests and deployments across multiple platforms faster.

Amazon CloudWatch

Amazon CloudWatch monitors your Amazon Web Services (AWS) resources and the applications you run on AWS in real-time.You can use CloudWatch to collect and track metrics, which are the variables you want to measure for your resources and applications. CloudWatch alarms send notifications or automatically make changes to the resources you are monitoring based on rules that you define.

Amazon CloudWatch Architecture

Metrics
A metric is the fundamental concept in CloudWatch and represents a time-ordered set of data points. These data points can be either your custom metrics or metrics from other services in AWS. You or AWS products publish metric data points into CloudWatch and you retrieve statistics about those data points as an ordered set of time-series data. You can use metrics to calculate statistics and present the data graphically in the CloudWatch console. Metrics exist only in the region in which they are created.

Namespaces
CloudWatch namespaces are containers for metrics. Metrics in different namespaces are isolated from each other, so that metrics from different applications are not mistakenly aggregated into the same statistics.

Dimensions
A dimension is a name/value pair that helps you to uniquely identify a metric. Every metric has specific characteristics that describe it, and you can think of dimensions as categories for those characteristics. Dimensions help you design a structure for your statistics plan. Because dimensions are part of the unique identifier for a metric, whenever you add a unique name/value pair to one of your metrics, you are creating a new metric.

DynamoDB Data Model

In Amazon DynamoDB, a table is a collection of items and each item is a collection of attributes. In a relational database, a table has a predefined schema such as the table name, primary key, list of its column names and their data types. All records stored in the table must have the same set of columns. In contrast, DynamoDB only requires that a table has a primary key, but does not require you to define all of the attribute names and data types in advance. Individual items in a DynamoDB table can have any number of attributes, although there is a limit of 400 KB on the item size. An item size is the sum of lengths of its attribute names and values (binary and UTF-8 lengths).

Each attribute in an item is a name-value pair. An attribute can be a scalar (single-valued), a JSON document, or a set. For example, consider storing a catalog of products in DynamoDB.You can create a table, ProductCatalog, with the Id attribute as its primary key. The primary key uniquely identifies each item, so that no two products in the table can have the same Id.

Primary Key

When you create a table, in addition to the table name, you must specify the primary key of the table. The primary key uniquely identifies each item in the table, so that no two items can have the same key. DynamoDB supports two different kinds of primary keys: 

• Partition Key – A simple primary key, composed of one attribute known as the partition key. DynamoDB uses the partition key's value as input to an internal hash function; the output from the hash function determines the partition where the item will be stored. No two items in a table can have the same partition key value.

• Partition Key and Sort Key – A composite primary key, composed of two attributes. The first attribute is the partition key, and the second attribute is the sort key. DynamoDB uses the partition key value as input to an internal hash function; the output from the hash function determines the partition where the item will be stored. All items with the same partition key are stored together, in sorted order by sort key value. It is possible for two items to have the same partition key value, but those two items must have different sort key values.

Secondary Indexes

When you create a table with a composite primary key (partition key and sort key), you can optionally define one or more secondary indexes on that table. A secondary index lets you query the data in the table using an alternate key, in addition to queries against the primary key.

DynamoDB Data Types

Amazon DynamoDB supports the following data types:

• Scalar types – Number, String, Binary, Boolean, and Null.

• Document types – List and Map.

• Set types – String Set, Number Set, and Binary Set.

Item Distribution

DynamoDB stores data in partitions. A partition is an allocation of storage for a table, backed by solid state drives (SSDs) and automatically replicated across three facilities within an AWS region. Partition management is handled entirely by DynamoDB—customers never need to manage partitions themselves. If your storage requirements exceed a partition's capacity, DynamoDB allocates additional partitions automatically.

AWS CodeStar

AWS CodeStar is a cloud-based service for creating, managing, and working with software development projects on AWS. You can quickly develop, build, and deploy applications on AWS with an AWS CodeStar project. An AWS CodeStar project creates and integrates AWS services for your project development toolchain. Depending on your choice of AWS CodeStar project template, that toolchain might include source control, build, deployment, virtual servers or serverless resources, and more. AWS CodeStar also manages the permissions required for project users (called team members). By adding users as team members to an AWS CodeStar project, project owners can quickly and simply grant each team member role-appropriate access to a project and its resources.

AWS CodeStar provides a unified user interface, enabling you to easily manage your software development activities in one place. With AWS CodeStar, you can set up your entire continuous delivery toolchain in minutes, allowing you to start releasing code faster. AWS CodeStar makes it easy for your whole team to work together securely, allowing you to easily manage access and add owners, contributors, and viewers to your projects. Each AWS CodeStar project comes with a project management dashboard, including an integrated issue tracking capability powered by Atlassian JIRA Software. With the AWS CodeStar project dashboard, you can easily track progress across your entire software development process, from your backlog of work items to teams’ recent code deployments.

Amazon DynamoDB Accelerator (DAX)

DynamoDB Accelerator (DAX) is a new caching service for DynamoDB.

Amazon DynamoDB Accelerator (DAX) is a fully managed, highly available, in-memory cache for DynamoDB that delivers up to a 10x performance improvement – from milliseconds to microseconds – even at millions of requests per second. DAX does all the heavy lifting required to add in-memory acceleration to your DynamoDB tables, without requiring developers to manage cache invalidation, data population, or cluster management.

Amazon DynamoDB is designed for scale and performance. In most cases, the DynamoDB response times can be measured in single-digit milliseconds. However, there are certain use cases that require response times in microseconds. For these use cases, DynamoDB Accelerator (DAX) delivers fast response times for accessing eventually consistent data.

DAX is a DynamoDB-compatible caching service that enables you to benefit from fast in-memory performance for demanding applications. DAX addresses three core scenarios:

• As an in-memory cache, DAX reduces the response times of eventually-consistent read workloads by an order of magnitude, from single-digit milliseconds to microseconds.
• DAX reduces operational and application complexity by providing a managed service that is API-compatible with Amazon DynamoDB, and thus requires only minimal functional changes to use with an existing application.
• For read-heavy or bursty workloads, DAX provides increased throughput and potential operational cost savings by reducing the need to over-provision read capacity units. This is especially beneficial for applications that require repeated reads for individual keys.

Amazon DynamoDB

Amazon DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability. DynamoDB lets you offload the administrative burdens of operating and scaling a distributed database, so that you don't have to worry about hardware provisioning, setup and configuration, replication, software patching, or cluster scaling.

With DynamoDB, you can create database tables that can store and retrieve any amount of data, and serve any level of request traffic.You can scale up or scale down your tables' throughput capacity without downtime or performance degradation, and use the AWS Management Console to monitor resource utilization and performance metrics.

DynamoDB automatically spreads the data and traffic for your tables over a sufficient number of servers to handle your throughput and storage requirements, while maintaining consistent and fast performance. All of your data is stored on solid state disks (SSDs) and automatically replicated across multiple Availability Zones in an AWS region, providing built-in high availability and data durability.

Tables, Items, and Attributes
In Amazon DynamoDB, a table is a collection of items and each item is a collection of attributes. Each attribute in an item is a name-value pair. An attribute can be a scalar (single-valued), a JSON document, or a set.

Primary Key
When you create a table, in addition to the table name, you must specify the primary key of the table. The primary key uniquely identifies each item in the table, so that no two items can have the same key.
DynamoDB supports two different kinds of primary keys:

• Partition Key – A simple primary key, composed of one attribute known as the partition key. DynamoDB uses the partition key's value as input to an internal hash function; the output from the hash function determines the partition where the item will be stored. No two items in a table can have the same partition key value.
• Partition Key and Sort Key – A composite primary key, composed of two attributes. The first attribute is the partition key, and the second attribute is the sort key. DynamoDB uses the partition key value as input to an internal hash function; the output from the hash function determines the partition where the item will be stored. All items with the same partition key are stored together, in sorted order by sort key value. It is possible for two items to have the same partition key value, but those two items must have different sort key values.

Secondary Indexes
DynamoDB supports two kinds of secondary indexes:

• Global secondary index – an index with a partition key and sort key that can be different from those on the table.
• Local secondary index – an index that has the same partition key as the table, but a different sort key.

DynamoDB Data Types
Amazon DynamoDB supports the following data types:

• Scalar types – Number, String, Binary, Boolean, and Null.
• Document types – List and Map.
• Set types – String Set, Number Set, and Binary Set.

AWS Step Functions

Step function is a new AWS service. AWS Step Functions makes it easy to coordinate the components of distributed applications and microservices using visual workflows. Building applications from individual components that each perform a discrete function lets you scale and change applications quickly. Step Functions is a reliable way to coordinate components and step through the functions of your application. Step Functions provides a graphical console to arrange and visualize the components of your application as a series of steps. This makes it simple to build and run multi-step applications. Step Functions automatically triggers and tracks each step, and retries when there are errors, so your application executes in order and as expected. Step Functions logs the state of each step, so when things do go wrong, you can diagnose and debug problems quickly. You can change and add steps without even writing code, so you can easily evolve your application and innovate faster.

AWS Step Functions manages the operations and underlying infrastructure for you to help ensure your application is available at any scale.

With regards to pricing you only pay for the transitions, that is the steps between tasks.

Custom Authorization HTTP Request in AWS NodeJS Lambda function

Below is the sample code for AWS NodeJS lambda function. This will show how to make an HTTP request with Custom Authorization using "request" node module.

request v2.79.0 - Simplified HTTP request client.

Request package is designed to be the simplest way possible to make http calls. It supports HTTPS and follows redirects by default.

/// Module to test HTTP POST Request /// Created By: Sohan Fegade /// Created Date: 17-Feb-2017 var request = require('request'); var buffer = require('buffer'); console.log('Loading http-test-module'); exports.handler = function (event, context, callback) {     if (event != null) {         console.log('event = ' + JSON.stringify(event));         var username = 'sampleuser';         var password = 'samplepassword';                 var auth = 'Basic ' + buffer.Buffer(username + ':' + password).toString('base64');                 var options = {             url: 'https://www.sample-auth-url.com/oauth/token',             headers: {                 'Content-Type': 'application/json',                 'Authorization': auth             },             json: {                 'type': 'test',                 'input': 'test',                 'field': 'test',                 'scope': 'urn:scope-api',                 'token-type': 'auth-token'             }         }         request.post(options, function (error, response, body) {             if (!error && response.statusCode === 200) {                console.log('Access Token : ' + body.access-token);             }                             context.callbackWaitsForEmptyEventLoop = false;             callback(null, 'SUCCESS');        });     }     else {         console.log('No event object');     } };

 

AWS Route 53

Amazon Route 53 (Route 53) is part of Amazon.com's cloud computing platform, Amazon Web Services (AWS). Route 53 provides a scalable and highly available Domain Name System (DNS). The name is a reference to TCP or UDP port 53, where DNS server requests are addressed. In addition to being able to route users to various AWS services, including EC2 instances, Route 53 also enables AWS customers to route users to non-AWS infrastructure. Route 53's servers are distributed throughout the world. Amazon Route 53 supports full, end-to-end DNS resolution over IPv6. Recursive DNS resolvers on IPv6 networks can use either IPv4 or IPv6 transport to send DNS queries to Amazon Route 53.

One of the key features of Route 53 is programmatic access to the service that allows customers to modify DNS records via web service calls. Combined with other features in AWS, this allows a developer to programmatically bring up a machine and point to components that have been created via other service calls such as those to create new S3 buckets or EC2 instances.

AWS supposedly named the service Route 53 because all DNS requests are handled through port 53, and the "route" piece resembles the historic "Route 66" of the USA. And Route is basically for routing any traffic to Amazon DNS.

Developer Guide for AWS Lambda using Node.JS

Developing Simple Lambda function using Node.js

Pre Requisite:
• AWS Account
• Visual Studio
• Node.JS tool

Step 1: Create Lambda function in Visual Studio
Step 1.1 Open Visual Studio (Run as Administrator)
Add Lambda Project in Visual Studio
File -> Add New Project

Create new project as “Hello World”.
On Next screen select – Create Simple project

Click Finish.
Below screen will appear after click on Finish.

By default some files will be added in project which we can be found in Solution explorer.

1. _testdriver.js  : This is a utility file to help invoke and debug the lambda function. It is not included as part of the bundle upload to Lambda.
2. _sampleEvent : This is JSON file use to set Event sample event data to test / debug
3. App.js : Main program file in which we will write code (Application logic)
4. npm : Node Package module. Utility to install NPM package
5. node_modules : Repository for node.js packages. Folder contains supporting node module files. By default it contains “aws-sdk” module. When add new module it will automatically get updated and store supporting file

Open App.JS file, file contains below code:

exports.handler : start execution of your Lambda function

Step 1.2 – Update _sampleEvent.json file

Step 1.3 – Update app.js file with below code

Step 1.4 – Run Application (Press F5), below screen should appear which contains Event information from _sampleEvent.JSON file

This complete our Hello world program.

Step 2: Deploy Lambda function in AWS console
Step 2.1: Open AWS console and find Lambda under Compute service.

Step 2.2: Screen will show List of Lambda functions. Click on Create Lambda function button

Step 2.3: Screen will show list of Blueprint use to create Lambda function. We can select existing blueprint or Skip this part and move next.
Click on Skip button on screen (Bottom right corner)

Step 2.4: Configure Lambda function.

Configure function - Set Name,Description,Runtime.
Set Lambda Function code details as shown below

Code entry type: Upload ZIP file
Handler: Main JavaScript file, in our case it is app.js. Update Handler Textbox as app.handler.
Role: Set role as lambda_basic_execution

Step 2.5: Create Zip folder contains code and Node modules
To create ZIP file open Physical location of Lambda code (HelloWorld Lambda Developed in VS2013)

Select below files/folder for ZIP,
1.    Node_modules
2.    App.js

Step 2.6:  Set Advanced settings.

Review Lambda function

Click on Create Function button. This will create your First Lambda Function.

Step 3: Configure Test Event and Test Lambda function
Step 3.1:  To set Test event click on Action button and select “Configure test event” option.

Select event template and set data. This is same data which we used to test in debug mode. You can find this in _sampleEvent.json file. Click Save button (Bottom Right corner)

Click on Test button to test Lambda function. Scroll down and you will see below content on screen.

Highlighted section in Blue is output generated by Context object.
Section highlighted in yellow is output log.

Step 4: See Test result logs in CloudWatch
Step 4.1: To check logs in click on Monitor Tab. Click “View logs in CloudWatch”

You can see details logs on new screen.