Snowflake Architecture

Snowflake’s architecture is a hybrid of traditional shared-disk database architectures and shared-nothing database architectures. Similar to shared-disk architectures, Snowflake uses a central data repository for persisted data that is accessible from all compute nodes in the data warehouse. But similar to shared-nothing architectures, Snowflake processes queries using MPP (massively parallel processing) compute clusters where each node in the cluster stores a portion of the entire data set locally. This approach offers the data management simplicity of a shared-disk architecture, but with the performance and scale-out benefits of a shared-nothing architecture.

Snowflake’s unique architecture consists of three key layers:

• Database Storage
• Query Processing
• Cloud Services

Database Storage When data is loaded into Snowflake, Snowflake reorganizes that data into its internal optimized, compressed, columnar format. Snowflake stores this optimized data in cloud storage. Snowflake manages all aspects of how this data is stored — the organization, file size, structure, compression, metadata, statistics, and other aspects of data storage are handled by Snowflake. The data objects stored by Snowflake are not directly visible nor accessible by customers; they are only accessible through SQL query operations run using Snowflake.

Query Processing Query execution is performed in the processing layer. Snowflake processes queries using “virtual warehouses”. Each virtual warehouse is an MPP compute cluster composed of multiple compute nodes allocated by Snowflake from a cloud provider. Each virtual warehouse is an independent compute cluster that does not share compute resources with other virtual warehouses. As a result, each virtual warehouse has no impact on the performance of other virtual warehouses.

Cloud Services The cloud services layer is a collection of services that coordinate activities across Snowflake. These services tie together all of the different components of Snowflake in order to process user requests, from login to query dispatch. The cloud services layer also runs on compute instances provisioned by Snowflake from the cloud provider. Among the services in this layer:

• Authentication
• Infrastructure management
• Metadata management
• Query parsing and optimization
• Access control

For more information: https://docs.snowflake.net/manuals/user-guide-intro.html

.NET Core 3

.NET Core is a free and open-source managed computer software framework for the Windows, macOS and Linux operating systems. It consists of CoreCLR, a complete runtime implementation of CLR, the virtual machine that manages the execution of .NET programs. CoreCLR comes with an improved just-in-time compiler, called RyuJIT. .NET Core also includes CoreFX, which is a partial fork of FCL. While .NET Core shares a subset of .NET Framework APIs, it comes with its own API that is not part of .NET Framework. Further, .NET Core contains CoreRT, the .NET Native runtime optimized to be integrated into AOT compiled native binaries. A variant of the .NET Core library is used for UWP. .NET Core's command-line interface offers an execution entry point for operating systems and provides developer services like compilation and package management.

.NET Core 3.0 supports for Windows Desktop applications, specifically Windows Forms, Windows Presentation Framework (WPF), and UWP XAML. There are many benefits that .NET Core will bring for desktop applications. Some of them are mentioned below.

• Performance improvements and other runtime updates
• Easy to use or test a new version of .NET Core
• Enables both machine-global and application-local deployment
• Support for the .NET Core CLI tools and SDK-style projects in Visual Studio

New features in .NET Core 3.0

• Side-by-side and App-local Deployment: The .NET Core deployment model is one the biggest benefits that Windows desktop developers will experience with .NET Core 3. In short, you can install .NET Core in pretty much any way you want. It comes with a lot of deployment flexibility. Deployment of .NET Core desktop applications can either use a global install of the .NET Core runtime (similar to how .NET Framework is deployed), or side-by-side deployment so that each application uses its own version of the runtime.
• Easily convert existing Desktop applications to .Net Core 3: The conversion of existing desktop application to .NET Core 3.0 will be pretty straightforward.
• Improvement to Project Files: The .Net Core has adopted the SDK based project structure as it offers many advantages like:
  
  • Much smaller and cleaner project files
  • Much friendlier to source control (fewer changes and smaller diffs)
  • Edit project files in Visual Studio without unloading
  • NuGet is part of the build and responsive to changes like target framework update
  • Supports multi-targeting
• Continue to support Controls, NuGet Packages, and Existing Assembly References: Desktop applications often have many dependencies, maybe from a control vendor, from NuGet or binaries that don’t have a source any more. .NET Core 3.0 will continue to support dependencies as-is without requiring developers to rewrite functionalities.

Splunk

Over the last decade, due to growing number of machines in the IT infrastructure and use of IoT devides there is an exponential growth in machine data. This machine data has a lot of valuable information that can drive efficiency, productivity and visibility for the business. Splunk was founded in 2003 for one purpose: To Make Sense Of Machine Generated Log Data. Splunk  captures, indexes, and correlates real-time data in a searchable repository from which it can generate graphs, reports, alerts, dashboards, and visualizations. Splunk's mission is to make machine data accessible across an organization by identifying data patterns, providing metrics, diagnosing problems, and providing intelligence for business operations. Splunk is a horizontal technology used for application management, security and compliance, as well as business and Web analytics.

Splunk's core collects and analyzes high volumes of machine-generated data. It uses a standard API to connect directly to applications and devices. It was developed in response to the demand for comprehensible and actionable data reporting for executives outside a company's IT department.

Splunk Enterprise Security (ES) is a security information and event management (SIEM) solution that provides insight into machine data generated from security technologies such as network, endpoint, access, malware, vulnerability and identity information. Its a premium application that is licensed independently from Splunk core.

Splunkbase is a community hosted by Splunk where users can go to find apps and add-ons for Splunk which can improve the functionality and usefulness of Splunk, as well as provide a quick and easy interface for specific use-cases and/or vendor products. Splunk apps and add-ons can be developed by anyone, including Splunk themselves.

Real time processing is Splunk’s biggest selling point, the other benefits with implementing Splunk are:

• Your input data can be in any format for e.g. .csv, or json or other formats
• You can configure Splunk to give Alerts / Events notification at the onset of a machine state
• You can accurately predict the resources needed for scaling up the infrastructure
• You can create knowledge objects for Operational Intelligence

Big Data: Spark

Apache Spark is an open-source distributed general-purpose cluster-computing framework. Apache Spark has as its architectural foundation the resilient distributed dataset (RDD), a read-only multiset of data items distributed over a cluster of machines, that is maintained in a fault-tolerant way.

Spark uses cluster computing for its computational (analytics) power as well as its storage. This means it can use resources from many computer processors linked together for its analytics. It's a scalable solution meaning that if more oomph is needed, you can simply introduce more processors into the system. With distributed storage, the huge datasets gathered for Big Data analysis can be stored across many smaller individual physical hard discs. This speeds up read/write operations, because the "head" which reads information from the discs has less physical distance to travel over the disc surface. As with processing power, more storage can be added when needed, and the fact it uses commonly available commodity hardware (any standard computer hard discs) keeps down infrastructure costs.

Unlike Hadoop, Spark does not come with its own file system - instead it can be integrated with many file systems including Hadoop's HDFS, MongoDB and Amazon's S3 system.
Another element of the framework is Spark Streaming, which allows applications to be developed which perform analytics on streaming, real-time data - such as automatically analyzing video or social media data - on-the-fly, in real-time.

In fast changing industries such as marketing, real-time analytics has huge advantages, for example ads can be served based on a user's behavior at a particular time, rather than on historical behavior, increasing the chance of prompting an impulse purchase.

Snowflake : Cloud Data Warehouse

Snowflake allows corporate users to store and analyze data using cloud-based hardware and software. The data is stored in Amazon S3. It is built for speed, even with the most intense workloads. Its patented architecture separates compute from storage so you can scale up and down on the fly, without delay or disruption. You get the performance you need exactly when you need it.

Snowflake is a fully columnar database with vectorized execution, making it capable of addressing even the most demanding analytic workloads. Snowflake’s adaptive optimization ensures queries automatically get the best performance possible – no indexes, distribution keys or tuning parameters to manage. Snowflake can support unlimited concurrency with its unique multi-cluster, shared data architecture. This allows multiple compute clusters to operate simultaneously on the same data without degrading performance. Snowflake can even scale automatically to handle varying concurrency demands with its multi-cluster virtual warehouse feature, transparently adding compute resources during peak load periods and scaling down when loads subside.

Snowflake’s mission is to enable every organization to be data-driven with instant elasticity, secure data sharing and per-second pricing, across multiple clouds. Snowflake combines the power of data warehousing, the flexibility of big data platforms and the elasticity of the cloud at a fraction of the cost of traditional solutions.

Jest : Javascript Unit Testing Framework

Jest is delightful JavaScript Testing framework. It has below features:

• Easy Setup: Complete and easy to set-up JavaScript testing solution. Works out of the box for any React project.
• Instant Feedback: Fast interactive watch mode runs only test files related to changed files and is optimized to give signal quickly.
• Snapshot Testing: Capture snapshots of React trees or other serializable values to simplify testing and to analyze how state changes over time.
• Zero configuration testing platform: Jest is used by Facebook to test all JavaScript code including React applications. Jest is already configured when you use create-react-app or react-native init to create your React and React Native projects. Place your tests in a __tests__ folder, or name your test files with a .spec.js or .test.js extension. Whatever you prefer, Jest will find and run your tests.
• Fast and sandboxed: Jest parallelizes test runs across workers to maximize performance. Console messages are buffered and printed together with test results. Sandboxed test files and automatic global state resets for every test so no two tests conflict with each other.
• Built-in code coverage reports: Easily create code coverage reports using --coverage. No additional setup or libraries needed! Jest can collect code coverage information from entire projects, including untested files.
• Powerful mocking library: Jest has powerful mocking library for functions and modules. Mock React Native components using jest-react-native.
• Works with TypeScript: Jest works with any compile-to-JavaScript language and integrates seamlessly with Babel and with TypeScript through ts-jest.

Queues - RabbitMQ vs SQS

RabbitMQ
RabbitMQ supports powerful message routing via exchange. This is very important when we need to run the same job on a specific server, group of servers or all servers. Our application sends one message and exchange will route it. RabbitMQ also has vhosts so that multiple applications can share the same RabbitMQ server but be isolated from each other (we can create unique logins for separate applications to access their vhosts). RabbitMQ can be setup in clusters for redundancy / failover and will acknowledge receipt of messages.
RabbitMQ has a powerful GUI which is accessible http://localhost:15672/. Hosting for RabbitMQ offers fewer choices and is more expensive.

AWS SQS
AWS SQS takes care of managing our queues. AWS SDK offers low level access but shoryuken is a higher level integration (shoryuken author acknowledges sidekiq as inspiration). Retrying failed jobs will happen automatically unless the messages is explicitly deleted. We can only delay jobs for 15 minutes (or we get The maximum allowed delay is 15 minutes (RuntimeError)). Recurring jobs are not supported but there are workarounds with AWS lambda and CloudWatch. AWS SQS UI is decent and we can use AWS SDK to access data directly. SQS has other interesting features such as long polling, batch operations and dead letter queues. SQS also has FIFO queues which guarantee he order in which messages are sent and received (and does not allow dupes). However, FIFO queues only allow 300 TPS (much less that regular SQS). Shoryuken works with standard and FIFO queues.
Hosting - easy to setup and cheap (pay for what you use) but obviously only available on AWS. SQS is a great choice when we need to run LOTS of jobs or when we do not care about more advanced options such as scheduling.

	RabbitMQ	Amazon SQS
Description	RabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.	Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use.
Pros	• It's fast and it works with good metrics/monitoring • Ease of configuration • I like the admin interface • Easy to set-up and start with • Intuitive work through python • Standard protocols • Durable • Written primarily in Erlang • Simply superb • Completeness of messaging patterns	• Easy to use, reliable • Low cost • Simple • Doesn't need to maintain • Delayed delivery upto 12 hours • Has a max message size (currently 256K) • Delayed delivery upto 15 mins only
Cons	• Needs Erlang runtime. Need ops good with Erlang runtime • Too complicated cluster/HA config and management • Configuration must be done first, not by your code	• SQS has guaranteed delivery, but messages can be delivered more than once.