AWS Fargate

AWS Fargate is a compute engine for Amazon ECS that allows you to run containers without having to manage servers or clusters. With AWS Fargate, you no longer have to provision, configure, and scale clusters of virtual machines to run containers. This removes the need to choose server types, decide when to scale your clusters, or optimize cluster packing. AWS Fargate removes the need for you to interact with or think about servers or clusters. Fargate lets you focus on designing and building your applications instead of managing the infrastructure that runs them.

Amazon ECS has two modes: 

• Fargate launch type
• EC2 launch type

With Fargate launch type, all you have to do is package your application in containers, specify the CPU and memory requirements, define networking and IAM policies, and launch the application. EC2 launch type allows you to have server-level, more granular control over the infrastructure that runs your container applications. 

With EC2 launch type, you can use Amazon ECS to manage a cluster of servers and schedule placement of containers on the servers. Amazon ECS keeps track of all the CPU, memory and other resources in your cluster, and also finds the best server for a container to run on based on your specified resource requirements. You are responsible for provisioning, patching, and scaling clusters of servers. You can decide which type of server to use, which applications and how many containers to run in a cluster to optimize utilization, and when you should add or remove servers from a cluster. EC2 launch type gives you more control of your server clusters and provides a broader range of customization options, which might be required to support some specific applications or possible compliance and government requirements.

Link: https://aws.amazon.com/fargate/

Snowflake : Data Encryption

Snowflake provides best-in-class key management, which is entirely transparent to customers.

End-to-end encryption (E2EE) is a form of communication in which no one but end users can read the data. In Snowflake, this means that only a customer and the runtime components can read the data. No third parties, including Snowflake’s cloud computing platform or any ISP, can see data in the clear. E2EE minimizes the attack surface. In the event of a security breach of the cloud platform, the data is protected because it is always encrypted, regardless of whether the breach exposes access credentials indirectly or data files directly, whether by an internal or external attacker.

Client-side encryption provides a secure system for managing data in cloud storage. Client-side encryption means that a user encrypts stored data before loading it into Snowflake. The cloud storage service only stores the encrypted version of the data and never includes data in the clear.

Document Link: https://docs.snowflake.net/manuals/user-guide/security-encryption.html

Sharing Data Securely in Snowflake

Secure Data Sharing enables sharing selected objects (tables, secure views, and secure UDFs) in a database in your account with other Snowflake accounts. The principle participants in any data sharing relationship are the provider and one or more consumers. Snowflake enables the sharing of databases through shares, which are created by data providers and "imported" by data consumers. All database objects shared between accounts are read-only (i.e. the objects cannot be modified or deleted, including adding or modifying table data).
With Secure Data Sharing, no actual data is copied or transferred between accounts. All sharing is accomplished through Snowflake’s unique services layer and metadata store. This is an important concept because it means that shared data does not take up any storage in a consumer account and, therefore, does not contribute to the consumer’s monthly data storage charges. The only charges to consumers are for the compute resources (i.e. virtual warehouses) used to query the shared data.
In addition, because no data is copied or exchanged, Secure Data Sharing setup is quick and easy for providers and access to the shared data is instantaneous for consumers:

• The provider creates a share of a database in their account and grants access to specific objects (i.e. tables, secure views, and secure UDFs) in the database. One or more accounts are then added to the share, which can include your own accounts (if you have multiple Snowflake accounts).
• On the consumer side, a read-only database is created from the share. Access to this database is configurable using the same, standard role-based access control that Snowflake provides for all objects in the system.

Shares are named, first-class Snowflake objects that encapsulate all of the information required to share a database. Each share consists of:

• The privileges that grants access to the database and the schema containing the objects to share.
• The privileges that grant access to the specific objects (tables, secure views, and secure UDFs).
• The consumer accounts with which the database and its objects are shared.

Once a database is created (in a consumer account) from a share, all the shared objects are accessible to users in the consumer account. Shares are secure, configurable, and controlled 100% by the provider account:

• New objects added to a share become immediately available to all consumers, providing real-time access to shared data.
• Access to a share (or any of the objects in a share) can be revoked at any time.

Link: https://docs.snowflake.net/manuals/user-guide/data-sharing-intro.html

Snowflake Architecture

Snowflake’s architecture is a hybrid of traditional shared-disk database architectures and shared-nothing database architectures. Similar to shared-disk architectures, Snowflake uses a central data repository for persisted data that is accessible from all compute nodes in the data warehouse. But similar to shared-nothing architectures, Snowflake processes queries using MPP (massively parallel processing) compute clusters where each node in the cluster stores a portion of the entire data set locally. This approach offers the data management simplicity of a shared-disk architecture, but with the performance and scale-out benefits of a shared-nothing architecture.

Snowflake’s unique architecture consists of three key layers:

• Database Storage
• Query Processing
• Cloud Services

Database Storage When data is loaded into Snowflake, Snowflake reorganizes that data into its internal optimized, compressed, columnar format. Snowflake stores this optimized data in cloud storage. Snowflake manages all aspects of how this data is stored — the organization, file size, structure, compression, metadata, statistics, and other aspects of data storage are handled by Snowflake. The data objects stored by Snowflake are not directly visible nor accessible by customers; they are only accessible through SQL query operations run using Snowflake.

Query Processing Query execution is performed in the processing layer. Snowflake processes queries using “virtual warehouses”. Each virtual warehouse is an MPP compute cluster composed of multiple compute nodes allocated by Snowflake from a cloud provider. Each virtual warehouse is an independent compute cluster that does not share compute resources with other virtual warehouses. As a result, each virtual warehouse has no impact on the performance of other virtual warehouses.

Cloud Services The cloud services layer is a collection of services that coordinate activities across Snowflake. These services tie together all of the different components of Snowflake in order to process user requests, from login to query dispatch. The cloud services layer also runs on compute instances provisioned by Snowflake from the cloud provider. Among the services in this layer:

• Authentication
• Infrastructure management
• Metadata management
• Query parsing and optimization
• Access control

For more information: https://docs.snowflake.net/manuals/user-guide-intro.html

Big Data: Spark

Apache Spark is an open-source distributed general-purpose cluster-computing framework. Apache Spark has as its architectural foundation the resilient distributed dataset (RDD), a read-only multiset of data items distributed over a cluster of machines, that is maintained in a fault-tolerant way.

Spark uses cluster computing for its computational (analytics) power as well as its storage. This means it can use resources from many computer processors linked together for its analytics. It's a scalable solution meaning that if more oomph is needed, you can simply introduce more processors into the system. With distributed storage, the huge datasets gathered for Big Data analysis can be stored across many smaller individual physical hard discs. This speeds up read/write operations, because the "head" which reads information from the discs has less physical distance to travel over the disc surface. As with processing power, more storage can be added when needed, and the fact it uses commonly available commodity hardware (any standard computer hard discs) keeps down infrastructure costs.

Unlike Hadoop, Spark does not come with its own file system - instead it can be integrated with many file systems including Hadoop's HDFS, MongoDB and Amazon's S3 system.
Another element of the framework is Spark Streaming, which allows applications to be developed which perform analytics on streaming, real-time data - such as automatically analyzing video or social media data - on-the-fly, in real-time.

In fast changing industries such as marketing, real-time analytics has huge advantages, for example ads can be served based on a user's behavior at a particular time, rather than on historical behavior, increasing the chance of prompting an impulse purchase.

Snowflake : Cloud Data Warehouse

Snowflake allows corporate users to store and analyze data using cloud-based hardware and software. The data is stored in Amazon S3. It is built for speed, even with the most intense workloads. Its patented architecture separates compute from storage so you can scale up and down on the fly, without delay or disruption. You get the performance you need exactly when you need it.

Snowflake is a fully columnar database with vectorized execution, making it capable of addressing even the most demanding analytic workloads. Snowflake’s adaptive optimization ensures queries automatically get the best performance possible – no indexes, distribution keys or tuning parameters to manage. Snowflake can support unlimited concurrency with its unique multi-cluster, shared data architecture. This allows multiple compute clusters to operate simultaneously on the same data without degrading performance. Snowflake can even scale automatically to handle varying concurrency demands with its multi-cluster virtual warehouse feature, transparently adding compute resources during peak load periods and scaling down when loads subside.

Snowflake’s mission is to enable every organization to be data-driven with instant elasticity, secure data sharing and per-second pricing, across multiple clouds. Snowflake combines the power of data warehousing, the flexibility of big data platforms and the elasticity of the cloud at a fraction of the cost of traditional solutions.

AWS Service : Parameter Store

Parameter store provides a centralized store to manage your configuration data, whether plain-text data such as database strings or secrets such as passwords, encrypted through AWS KMS. With Parameter store, your critical information stays within your environment, saving you the manual overhead of storing and managing it in configuration files. Parameters can be easily re-used across your AWS configuration and automation workflows without having to type them in plain-text, improving your security posture. Parameters can be easily referenced across AWS services such as Amazon ECS and AWS Lambda, as well as other EC2 Systems Manager capabilities such as Run Command, State Manager, and Automation.

Through integration with AWS Identity and Access Management, you can provide access control to specific parameters, letting you provide access to the data only to the users who need them and on which resources they can be used. AWS Key Management Service (KMS) integration helps you encrypt your sensitive information and protect the security of your keys. Additionally, all calls to the parameter store are recorded with AWS CloudTrail so that they can be audited.

Parameter Store offers the following benefits and features.

• Use a secure, scalable, hosted secrets management service (No servers to manage).
• Improve your security posture by separating your data from your code.
• Store configuration data and secure strings in hierarchies and track versions.
• Control and audit access at granular levels.
• Configure change notifications and trigger automated actions.
• Tag parameters individually, and then secure access from different levels, including operational, parameter, EC2 tag, or path levels.
• Reference parameters across AWS services such as Amazon EC2, Amazon EC2 Container Service, AWS Lambda, AWS CloudFormation, AWS CodeBuild, AWS CodeDeploy, and other Systems Manager capabilities.
• Configure integration with AWS KMS, Amazon SNS, Amazon CloudWatch, and AWS CloudTrail for encryption, notification, monitoring, and audit capabilities.

What is Docker?

Docker is a software platform that allows you to build, test, and deploy applications quickly. Docker packages software into standardized units called containers that have everything the software needs to run including libraries, system tools, code, and runtime. Using Docker, you can quickly deploy and scale applications into any environment and know your code will run. Amazon ECS uses Docker images in task definitions to launch containers on EC2 instances in your clusters.

Running Docker on AWS provides developers and admins a highly reliable, low-cost way to build, ship, and run distributed applications at any scale. AWS supports both Docker licensing models: open source Docker Community Edition (CE) and subscription-based Docker Enterprise Edition (EE).

Docker is available on many different operating systems, including most modern Linux distributions, like Ubuntu, and even Mac OSX and Windows.

Docker Benefits
Ship More Software Faster
Docker users on average ship software 7x more frequently than non-Docker users. Docker enables developers to ship isolated services as often as needed by eliminating the headaches of software dependencies.
Improve Developer Productivity
Docker reduces the time spent setting up new environments or troubleshooting differences between environments.
Seamlessly Move Applications
Docker-based applications can be seamlessly moved from local development machines to production deployments on AWS.
Standardize Application Operations
Small containerized applications make it easy to deploy, identify issues, and roll back for remediation.

Docker Use Cases
Continuous Integration & Delivery
Accelerate application delivery by standardizing environments and removing conflicts between language stacks and versions.
Data Processing
Provide big data processing as a service. Package data and analytics packages into portable containers that can be executed by non-technical users
Containers as a Service
Build and ship distributed applications with content and infrastructure that is IT-managed and secured.

AWS Step Functions

Step function is a new AWS service. AWS Step Functions makes it easy to coordinate the components of distributed applications and microservices using visual workflows. Building applications from individual components that each perform a discrete function lets you scale and change applications quickly. Step Functions is a reliable way to coordinate components and step through the functions of your application. Step Functions provides a graphical console to arrange and visualize the components of your application as a series of steps. This makes it simple to build and run multi-step applications. Step Functions automatically triggers and tracks each step, and retries when there are errors, so your application executes in order and as expected. Step Functions logs the state of each step, so when things do go wrong, you can diagnose and debug problems quickly. You can change and add steps without even writing code, so you can easily evolve your application and innovate faster.

AWS Step Functions manages the operations and underlying infrastructure for you to help ensure your application is available at any scale.

With regards to pricing you only pay for the transitions, that is the steps between tasks.

Puppet on the AWS Cloud

Puppet is a declarative, model-based configuration management solution from Puppet Labs that lets you define the state of your IT infrastructure, and automatically enforces that desired state on your systems. This Quick Start automates the deployment of a Puppet master and Puppet agents from scratch, using AWS CloudFormation templates.

Puppet Enterprise, comprises a commercially supported version of its open-source configuration management tool, Puppet. Puppet IT automation software uses Puppet's declarative language to manage various stages of the IT infrastructure lifecycle, including the provisioning, patching, configuration, and management of operating system and application components across enterprise data centers and cloud infrastructures.

Built as cross-platform software, Puppet and Puppet Enterprise operate on Linux distributions, including RHEL (and clones such as CentOS and Oracle Linux), Fedora, Debian, Mandriva, Ubuntu, and SUSE, as well as on multiple Unix systems (Solaris, BSD, Mac OS X, AIX, HP-UX), and has Microsoft Windows support. It is a model-driven solution that requires limited programming knowledge to use.

Puppet is a server management application that can use ServiceNow configuration item (CI) data to bring computers into a desired state by managing files, services, or packages installed on physical or virtual machines. ServiceNow can interact with Puppet systems that run Linux. ServiceNow identifies a Puppet Master, which controls Puppet nodes, and uses a standalone utility to discover the components in the Puppet environment. ServiceNow uses information about server CIs from the Puppet Master to classify those servers as Puppet nodes. Puppet then evaluates a node's current state and modifies the node to achieve the desired state.

Note: A group of configuration items is called a configuration template in ServiceNow Provider and a node definition in Puppet.

Chef

Chef is the platform for automating your infrastructure on Amazon Web Services. Chef is a configuration management tool written in Ruby and Erlang. It uses a pure-Ruby, domain-specific language (DSL) for writing system configuration "recipes". Chef is used to streamline the task of configuring and maintaining a company's servers, and can integrate with cloud-based platforms such as Internap, Amazon EC2, Google Cloud Platform, OpenStack, SoftLayer, Microsoft Azure and Rackspace to automatically provision and configure new machines. Chef contains solutions for both small and large scale systems, with features and pricing for the respective ranges.

The user writes "recipes" that describe how Chef manages server applications and utilities (such as Apache HTTP Server, MySQL, or Hadoop) and how they are to be configured. These recipes (which can be grouped together as a "cookbook" for easier management) describe a series of resources that should be in a particular state: packages that should be installed, services that should be running, or files that should be written. These various resources can be configured to specific versions of software to run and can ensure that software is installed in the correct order based on dependencies. Chef makes sure each resource is properly configured and corrects any resources that are not in the desired state.

Chef can run in client/server mode, or in a standalone configuration named "chef-solo". In client/server mode, the Chef client sends various attributes about the node to the Chef server. The server uses Solr to index these attributes and provides an API for clients to query this information. Chef recipes can query these attributes and use the resulting data to help configure the node.

Chef allows you to define, create, and manage your entire application stack on AWS. With a single recipe you can manage and orchestrate a multi-tier application that relies on a variety of AWS services such as Amazon Virtual Private Cloud (Amazon VPC), Amazon Elastic Compute Cloud (Amazon EC2), elastic load balancing (ELB), Amazon Simple Storage Service (Amazon S3), and Amazon Relational Database Service (Amazon RDS) relational database service (RDS). Because Chef recipes describe resources in the order of their execution, you can ensure that, for example, an RDS database is created and populated before enabling the Amazon EC2 instances that connect to it.

Website: chef.io

Google Cloud Platform

Google Cloud Platform is a cloud computing service by Google that offers hosting on the same supporting infrastructure that Google uses internally for end-user products like Google Search and YouTube. Cloud Platform provides developer products to build a range of programs from simple websites to complex applications. 

Google Cloud Platform is a part of a suite of enterprise services from Google for Work and provides a set of modular cloud-based services with a host of development tools. For example, hosting and computing, cloud storage, data storage, translations APIs and prediction APIs.

The core cloud computing services in the Google Cloud Platform include:

• Google Compute Engine: An infrastructure as a service (IaaS) offering that provides users with virtual machine (VM) instances for workload hosting.
• Google App Engine: A platform as a service (PaaS) offering that gives software developers access to Google's scalable hosting. Developers can also use a software developer kit (SDK) to develop software products that run on App Engine.
• Google Cloud Storage: A cloud storage platform designed to store large, unstructured data sets. Google also offers database storage options including Cloud Datastore for NoSQL non-relational storage, Cloud SQL for MySQL fully-relational storage and Google's native Cloud Bigtable database.
• Google Container Engine: A management and orchestration system for Docker containers that run within Google's public cloud. Google Container Engine is based on the Google Kubernetes container orchestration engine.

Website: cloud.google.com

Source: http://searchcloudcomputing.techtarget.com/definition/Google-Cloud-Platform, https://en.wikipedia.org/wiki/Google_Cloud_Platform

Hybrid Cloud Storage

Cloud Storage is a form of  networked online storage where data is stored in virtualized pools of storage. The data center operators, in the background, virtualize the resources according to the requirements of the customer and expose them as storage pools, which the customers can themselves use to store files or data objects. Physically, the resource may span across multiple servers and multiple locations. Cloud storage services may be accessed through a web service application programming interface (API) or by applications that utilize the API, such as cloud desktop storage, a cloud storage gateway or Web-based content management systems.

Hybrid cloud storage is a type of cloud storage model that derives and combines the functionality of public and private cloud storage models to provide storage services. This storage technique uses internal and external cloud applications, infrastructure and storage systems to form integrated storage architecture.

Hybrid cloud storage overcomes the problems of managing data and storage by integrating on-premises storage with cloud storage services. In this architecture, on-premises storage uses the capacity on internal SSDs and HDDs, as well on the expanded storage resources that are provided by cloud storage. A key element of the architecture is that the distance over which data is stored is extended far beyond the on-premises data center, thereby providing disaster protection.

Cloud Storage Provider's on-premise storage components vary widely in architecture, functionality, cloud integration and form factors.  There are huge differences between what are called cloud storage gateways and cloud-integrated storage.  Gateways provide an access point from on-premises to cloud data  transfers, whereas cloud-integrated storage is an automated solution to some of IT’s biggest problems such as managing data growth, backup and recovery, archiving data and disaster preparation.

Cloud-integrated storage (CiS), like Microsoft’s recent StorSimple acquisition, is a full-featured on-premises storage system that integrates with the cloud (i.e. Windows Azure Storage). Enterprises can create a complete hybrid cloud storage architecture where they store their data locally and protect it with snapshots on-premises and in the cloud, and where dormant data can be seamlessly tiered to the cloud to make room for new data locally. This way, CiS gives IT an automated  “safety valve” to prevent running out of disk capacity, while also providing rapid disaster recovery via cloud snapshots.

Cloud computing

In past few years cloud computing infiltrated computer industries, the cloud has been variously hailed as the savior of IT and reviled as a risky and chaotic environment. “The cloud” is simply a business model for the creation and delivery of compute resources. The model’s reliance on shared resources and virtualization allows cloud users to achieve levels of economy and scalability that would be difficult in a traditional data center. It is a colloquial expression used to describe a variety of different types of computing concepts that involve a large number of computers connected through a real-time communication network such as the Internet. Cloud computing relies on sharing of resources to achieve coherence and economies of scale similar to a utility (like the electricity grid) over a network. At the foundation of cloud computing is the broader concept of converged infrastructure and shared services.

Common cloud options include:

• Public cloud, in which multiple companies share physical servers and networking resources hosted in a provider’s data center.
• Private cloud, in which companies do not share resources (although efficiencies may be realized by hosting multiple virtual applications from the same company on a single physical server). Private clouds can be located either in a provider’s data center or in the company’s own on-premises data center.
• Hybrid cloud, in which virtualized applications can be moved among private and public cloud environments.

Common cloud benefits include:

• Scalable, on-demand resources: The ability to launch a cloud application in minutes, without having to purchase and configure hardware, enables enterprises to significantly cut their time to market. By taking advantage of cloud options for “bursting” during peak work periods, enterprises can also cost-effectively improve application performance and availability.
• Budget-friendly: Cloud computing services require no capital investment, instead tapping into the operating budget. As many companies tighten up their processes for approval of capital expenditures, a service can be easier and faster to approve and deploy.
• Utility pricing: The pay-per-use model that characterizes most cloud services appeals to enterprises that want to avoid overinvesting. It also can shorten the time to recoup the investment.

Few cloud computing providers:

Google	Oracle	Amazon Web Services	Microsoft
Rackspace	Salesforce	VMware	Joyent
Citrix	Bluelock	CenturyLink/Savvis	Verizon/Terremark