Sunday, October 21, 2018

Snowflake : Cloud Data Warehouse

Snowflake allows corporate users to store and analyze data using cloud-based hardware and software. The data is stored in Amazon S3. It is built for speed, even with the most intense workloads. Its patented architecture separates compute from storage so you can scale up and down on the fly, without delay or disruption. You get the performance you need exactly when you need it.
Snowflake is a fully columnar database with vectorized execution, making it capable of addressing even the most demanding analytic workloads. Snowflake’s adaptive optimization ensures queries automatically get the best performance possible – no indexes, distribution keys or tuning parameters to manage. Snowflake can support unlimited concurrency with its unique multi-cluster, shared data architecture. This allows multiple compute clusters to operate simultaneously on the same data without degrading performance. Snowflake can even scale automatically to handle varying concurrency demands with its multi-cluster virtual warehouse feature, transparently adding compute resources during peak load periods and scaling down when loads subside.
Snowflake’s mission is to enable every organization to be data-driven with instant elasticity, secure data sharing and per-second pricing, across multiple clouds. Snowflake combines the power of data warehousing, the flexibility of big data platforms and the elasticity of the cloud at a fraction of the cost of traditional solutions.

Wednesday, April 25, 2018

Jest : Javascript Unit Testing Framework

Jest is delightful JavaScript Testing framework. It has below features:

  • Easy Setup: Complete and easy to set-up JavaScript testing solution. Works out of the box for any React project.
  • Instant Feedback: Fast interactive watch mode runs only test files related to changed files and is optimized to give signal quickly.
  • Snapshot Testing: Capture snapshots of React trees or other serializable values to simplify testing and to analyze how state changes over time.
  • Zero configuration testing platform: Jest is used by Facebook to test all JavaScript code including React applications. Jest is already configured when you use create-react-app or react-native init to create your React and React Native projects. Place your tests in a __tests__ folder, or name your test files with a .spec.js or .test.js extension. Whatever you prefer, Jest will find and run your tests.
  • Fast and sandboxed: Jest parallelizes test runs across workers to maximize performance. Console messages are buffered and printed together with test results. Sandboxed test files and automatic global state resets for every test so no two tests conflict with each other.
  • Built-in code coverage reports: Easily create code coverage reports using --coverage. No additional setup or libraries needed! Jest can collect code coverage information from entire projects, including untested files.
  • Powerful mocking library: Jest has powerful mocking library for functions and modules. Mock React Native components using jest-react-native.
  • Works with TypeScript: Jest works with any compile-to-JavaScript language and integrates seamlessly with Babel and with TypeScript through ts-jest.

Monday, April 16, 2018

Queues - RabbitMQ vs SQS

RabbitMQ
RabbitMQ supports powerful message routing via exchange. This is very important when we need to run the same job on a specific server, group of servers or all servers. Our application sends one message and exchange will route it. RabbitMQ also has vhosts so that multiple applications can share the same RabbitMQ server but be isolated from each other (we can create unique logins for separate applications to access their vhosts). RabbitMQ can be setup in clusters for redundancy / failover and will acknowledge receipt of messages.
RabbitMQ has a powerful GUI which is accessible http://localhost:15672/. Hosting for RabbitMQ offers fewer choices and is more expensive.

AWS SQS
AWS SQS takes care of managing our queues. AWS SDK offers low level access but shoryuken is a higher level integration (shoryuken author acknowledges sidekiq as inspiration). Retrying failed jobs will happen automatically unless the messages is explicitly deleted. We can only delay jobs for 15 minutes (or we get The maximum allowed delay is 15 minutes (RuntimeError)). Recurring jobs are not supported but there are workarounds with AWS lambda and CloudWatch. AWS SQS UI is decent and we can use AWS SDK to access data directly. SQS has other interesting features such as long polling, batch operations and dead letter queues. SQS also has FIFO queues which guarantee he order in which messages are sent and received (and does not allow dupes). However, FIFO queues only allow 300 TPS (much less that regular SQS). Shoryuken works with standard and FIFO queues.
Hosting - easy to setup and cheap (pay for what you use) but obviously only available on AWS. SQS is a great choice when we need to run LOTS of jobs or when we do not care about more advanced options such as scheduling.


RabbitMQ
Amazon SQS
DescriptionRabbitMQ gives your applications a common platform to send and receive messages, and your messages a safe place to live until received.Transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be always available. With SQS, you can offload the administrative burden of operating and scaling a highly available messaging cluster, while paying a low price for only what you use.
Pros• It's fast and it works with good metrics/monitoring
• Ease of configuration
• I like the admin interface
• Easy to set-up and start with
• Intuitive work through python
• Standard protocols
• Durable
• Written primarily in Erlang
• Simply superb
• Completeness of messaging patterns
• Easy to use, reliable
• Low cost
• Simple
• Doesn't need to maintain
• Delayed delivery upto 12 hours
• Has a max message size (currently 256K)
• Delayed delivery upto 15 mins only
Cons• Needs Erlang runtime. Need ops good with Erlang runtime
• Too complicated cluster/HA config and management
• Configuration must be done first, not by your code
• SQS has guaranteed delivery, but messages can be delivered more than once.

Friday, March 16, 2018

NoSQL : CAP

Distributed
– Sharding: splittng data over servers “horizontally”
– Replication

Lower-level than RDBMS/SQL
– Simpler ad hoc APIs
– But you build the application (programming not querying)
– Operations simple and cheap

Different flavours (for different scenarios)
– Different CAP emphasis
– Different scalability profiles
– Different query functionality
– Different data models
NoSQL: CAP (not ACID)
Consistency: each client always has the same view of the data.
Availability: all clients can always read and write.
Partition tolerance: the system works well across physical network partitions.

Source: AWS NoSQL DynamoDB

Monday, March 12, 2018

Node Version Manager (NVM)

Node Version Manager (NVM) is a neat little bash script that allows you to manage multiple versions of Node.js on the same box. A version manager really helps to test our applications under different versions of the related software. nvm is a tool that allows you to download and install Node.js. You don't need nvm unless you you want to keep multiple versions of Node.js installed on your system or if you'd like to upgrade your current version.
Command to check NVM version:
nvm --version.

Bash command to install NVM:
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.26.1/install.sh | bash
How to use NVM:
1. Install Node.js
If you have not already installed Node.js on your machine, it is time to install the latest or a specific version of Node.js using NVM. Below command will install version 0.12.7 for 64-bit Windows by running the command:
$ nvm install 0.12.7 64

2. List the available Node.js versions
In order to list all the Node.js versions installed on your machine, simply run the command:
$ nvm list

3. Use a specific Node.js version
Below command used to set version by running the command:
$ nvm use 4.2.1 64
As Node.js is still a go-to solution, many versions of it are released and new versions will be released in the future. That is where testing an application with various Node.js version comes handy.

Wednesday, February 28, 2018

Microservices

Microservices is a variant of the service-oriented architecture (SOA) architectural style that structures an application as a collection of loosely coupled services. In a microservices architecture, services should be fine-grained and the protocols should be lightweight. The benefit of decomposing an application into different smaller services is that it improves modularity and makes the application easier to understand, develop and test.
Microservice architecture, or simply microservices, is a distinctive method of developing software systems that has grown in popularity in recent years.  In fact, even though there isn’t a whole lot out there on what it is and how to do it, for many developers it has become a preferred way of creating enterprise applications.  Thanks to its scalability, this architectural method is considered particularly ideal when you have to enable support for a range of platforms and devices—spanning web, mobile, Internet of Things, and wearables—or simply when you’re not sure what kind of devices you’ll need to support in an increasingly cloudy future.
Microservices Pros and Cons:
Among their advantages for software development, microservices:
    Are easily deployed
    Require less production time
    Can scale quickly
    Can be reused among different projects
    Work well with containers, such as Docker
    Complement cloud activities

However, there are also drawbacks with microservices, such as:
    Potentially too granular
    Latency during heavy use
    Testing can be complex

Tuesday, February 27, 2018

SSL : Secure Sockets Layer

What is SSL? SSL stands for Secure Sockets Layer, an encryption technology that was originally created by Netscape in the 1990s. SSL creates an encrypted connection between your web server and your visitors' web browser allowing for private information to be transmitted without the problems of eavesdropping, data tampering, and message forgery.
To enable SSL on a website, you will need to get an SSL Certificate that identifies you and install it on your web server. When a web browser is using an SSL certificate it usually displays a padlock icon but it may also display a green address bar. Once you have installed an SSL Certificate, you can access a site securely by changing the URL from http:// to https://. If SSL is properly deployed, the information transmitted between the web browser and the web server (whether it is contact or credit card information), is encrypted and only seen by the organization that owns the website.
SSL vs. TLS
SSL and TLS generally mean the same thing. TLS 1.0 was created by RFC 2246 in January 1999 as the next version of SSL 3.0. Most people are familiar with the term SSL so that is usually the term that is used when the system is using the newer TLS protocol.
What is a certificate authority (CA)?
A certificate authority is an entity which issues digital certificates to organizations or people after validating them. Certification authorities have to keep detailed records of what has been issued and the information used to issue it, and are audited regularly to make sure that they are following defined procedures. Every certification authority provides a Certification Practice Statement (CPS) that defines the procedures that will be used to verify applications. There are many commercial CAs that charge for their services (VeriSign). Institutions and governments may have their own CAs, and there are also free Certificate Authorities.