AMQP Essentials

WHAT IS AMQP?
AMQP (Advanced Message Queuing Protocol) is a binary transfer protocol that was made for enterprise applications and server-to-server communication (e.g., for financial businesses), but today it can be very useful in the Internet of Things world, thanks to the following primary features. AMQP is binary and avoids a lot of the useless data sent on the wire when using a text-based protocol like HTTP; because of this, it can be considered compact, too. Thanks to its multiplexed nature, only one connection (over a reliable stream transport protocol) is needed to allow separated data flows between the two peers; and of course it’s symmetric and provides both a client-server communication style and peer-to-peer exchange. Finally, it’s secure and reliable, providing three different levels of QoS (Quality of Service).
The last ratified version of AMQP (1.0) is the only one standardized by OASIS (since 2012/10) and ISO/IEC (since 2014/05), and it’s totally broker-model agnostic, as it doesn’t define any requirements on broker internals (this is the main difference with previous “not standard” versions like 0.9.1); the protocol is focused on how the data is transferred on the wire.

AMQP ARCHITECTURE
AMQP has a layered model defined in the following way from a bottom-up perspective:
• TRANSPORT/FRAMING:
Defines the connection behavior  and the security layer between peers on top of an underlying network transport protocol (TCP, for example). It also adds the framing protocol and how the exchanged data is formatted and encoded.
• MESSAGING:
Provides messaging capabilities at application level on top of the previous layer defining the message
entity as built of one or more frames.

Regarding the network transport layer, AMQP isn’t strongly tied to TCP, and as such can be used with any reliable stream transport protocol; so, for example, SCTP (Stream Control Transmission Protocol) and pipes are suitable.

AMQP COMMUNICATIONS
All the AMQP concepts—from connection, session, and link to performatives and messages—fit together to define how the communication happens between two peers. The main steps involved are:
• OPEN/CLOSE a connection (respectively after opening a network connection and before closing it) using “open” and “close” performatives
• BEGIN/END a session inside the connection thanks to “begin” and “end” performatives
• ATTACH/DETACH a link inside the session using “attach” and “detach” performatives
• SEND/RECEIVE messages with flow control thanks to “transfer, ” “disposition, ” and “flow” performatives

Best Wishes for New Year 2016

A good beginning makes a good end

yield Keyword Explained

When you use the yield keyword in a statement, you indicate that the method, operator, or get accessor in which it appears is an iterator. Using yield to define an iterator removes the need for an explicit extra class when you implement the IEnumerable and IEnumerator pattern for a custom collection type. The yield keyword was introduced in C# 2.0 (it is not currently available in VB.NET) and is useful for iteration. In iterator blocks it can be used to get the next value.

“Yield keyword helps us to do custom stateful iteration over .NET collections.” There are two scenarios where “yield” keyword is useful:

• Customized iteration through a collection without creating a temporary collection
• Stateful iteration

It is a contextual keyword used in iterator methods in C#. Basically, it has two use cases:

• yield return obj; returns the next item in the sequence.
• yield break; stops returning sequence elements (this happens automatically if control reaches the end of the iterator method body).

When you use the "yield return" keyphrase, .NET is wiring up a whole bunch of plumbing code for you, but for now you can pretend it's magic. When you start to loop in the calling code, this function actually gets called over and over again, but each time it resumes execution where it left off.
Typical Implementation

1. Caller calls function
2. Function executes and returns list
3. Caller uses list

Yield Implementation

1. Caller calls function
2. Caller requests item
3. Next item returned
4. Goto step #2

Although the execution of the yield implementation is a little more complicated, what we end up with is an implementation that "pulls" items one at a time instead of having to build an entire list before returning to the client.
Example:

public class PowersOf2
{
    static void Main()
    {
        // Display powers of 2 up to the exponent of 8:
        foreach (int i in Power(2, 8))
        {
            Console.Write("{0} ", i);
        }
    }

    public static System.Collections.Generic.IEnumerable<int> Power(int number, int exponent)
    {
        int result = 1;

        for (int i = 0; i < exponent; i++)
        {
            result = result * number;
            yield return result;
        }
    }
    // Output: 2 4 8 16 32 64 128 256
}

RabbitMQ : Tackling the broker SPOF

Though RabbitMQ brokers are extremely stable, a crash is always possible. Losing an instance altogether due to a virtual instance glitch is a likely possibility that can't be ignored. Therefore, it is essential to tackle the broker single point of failure (SPOF) before something bad happens, to prevent losing data or annoying users.The RabbitMQ can easily be configured to run in an active/active deployment, where several brokers are engaged in a cluster to act as a single highly-available AMQP middleware. The active/active aspect is essential, because it means that no manual fail-over operation is needed if one broker goes down.

Mirroring queues

With clustering, the configuration gets synchronized across all RabbitMQ nodes. This means that clients can now connect to one node or the other and find the exchanges and queues they're expecting. However, there is one thing that is not carried over the cluster by default: the messages themselves. By default, queue data is local to a particular node. When a queue is mirrored, its instances across the network organize themselves around one master and several slaves. All interaction (message queuing and dequeuing) happens with the master; the slaves receive the updates via synchronization over the cluster. If you interact with a node that hosts a slave queue, the interaction would actually be forwarded across the cluster to the master and then synchronized back to the slave.

Activating queue mirroring is done via a policy that is applied to each queue concerned.

Federating brokers

If you think beyond the notion of a single centralized enterprise resource and instead think in terms of distributed components, the idea of creating a topology of RabbitMQ brokers will emerge. RabbitMQ offers the following two plugins that allow the connection of brokers:

• The shovel plugin, which connects queues in one broker to exchanges in another broker

• The federation plugin, which connects queues to queues or exchanges to exchanges across brokers 

Both plugins ensure a reliable delivery of messages across brokers; if messages can't be routed to the target broker, they'll remain safely accumulated. Neither require brokers to be clustered, which simplifies setup and management (RabbitMQ and Erlang versions can mismatch). The federation plugin needs to be installed on all RabbitMQ nodes that will engage in the topology.

Keep in mind...

• RabbitMQ relies on Erlang's clustering feature, which allows several Erlang nodes to communicate with each other locally or over the network.
• The Erlang cluster requires a so-called security cookie as a means of cross-node authentication. 
• If RabbitMQ instances are firewalled from each other, its needed to open specific ports on top of the one used by AMQP (5672); otherwise, the cluster will not work.
• Make sure the same version of Erlang is used by all the RabbitMQ nodes that engage in a cluster; otherwise, the join_cluster command will fail with an OTP version mismatch error.
• Similarly, the same major/minor version of RabbitMQ should be used across nodes, but patch versions can differ; this means that versions 3.2.1 and 3.2.0 can be used in the same cluster, but not 3.2.1 and 3.1.0.
• In a federation, only the node where messages converge needs to be manually configured; its upstream nodes get automatically configured for the topology. Conversely with shovels, each source node needs to be manually configured to send to a destination node, which itself is unaware of the fact that it's engaged in a particular topology.

WS-Policy

WS-Policy is a specification that allows web services to use XML to advertise their policies (on security, quality of service, etc.) and for web service consumers to specify their policy requirements. WS-Policy is a W3C recommendation as of September 2007. WS-Policy represents a set of specifications that describe the capabilities and constraints of the security (and other business) policies on intermediaries and end points (for example, required security tokens, supported encryption algorithms, and privacy rules) and how to associate policies with services and end points.

• To integrate software systems with web services
• Need a way to express web services characteristics
• Without this standard, developers need docs
• Provides a flexible and extensible grammar for expressing the capabilities, requirements, and general characteristics of Web Service entities
• Defines a model to express these properties as policies
• Provide the mechanisms needed to enable Web Services applications to specify policies

WS-Policy specifies:

• An XML-based structure called a policy expression containing policy information
• Grammar elements to indicate how the contained policy assertions apply

Terminology:

• Policy: refers to the set of information being expressed as policy assertions
• Policy Assertion: represents an individual preference, requirement, capability, etc.
• Policy Expression: set of one or more policy assertions
• Policy Subject: an entity to which a policy expression can be bound

Policy Namespaces:

• WS-Policy schema defines all constructs that can used in a policy expression

Prefix	Description	Namespace
wsp	WS-Policy, WS-PolicyAssertions, and WS_PolicyAttachment	http://schemas.xmlsoap.org/ws/2002/12/policy
wsse	WS-SecurityPolicy	http://schemas.xmlsoap.org/ws/2002/12/secext
wsu	WS utility schema	http://schemas.xmlsoap.org/ws/2002/07/utility
msp	WSE 2.0 policy schema	http://schemas.microsoft.com/wse/2003/06/policy

MongoDB Fundamentals

MongoDB is a document-oriented DBMS. It uses JSON-like objects comprising the data model, rather than RDBMS tables. Significantly, MongoDB supports neither joins nor transactions. However, it features secondary indexes, an expressive query language, atomic writes on a per-document level, and fully-consistent reads. Operationally, MongoDB features master-slave replication with automated failover and built-in horizontal scaling via automated range-based partitioning.
Instead of tables, a MongoDB database stores its data in collections, which are the rough equivalent of RDBMS tables. A collection holds one or more documents, which corresponds to a record or a row in a relational database table, and each document has one or more fields, which corresponds to a column in a relational database table.

MongoDB Storage
A storage engine is the part of a database that is responsible for managing how data is stored on disk. Many databases support multiple storage engines, where different engines perform better for specific workloads. For example, one storage engine might offer better performance for read-heavy workloads, and another might support a higher-throughput for write operations. MMAPv1 is the default storage engine in 3.0. With multiple storage engines, you can decide which storage engine is best for your application.

MMAPv1 Storage Engine
A memory-mapped file is a file with data that the operating system places in memory by way of the mmap() system call. mmap() thus maps the file to a region of virtual memory. Memory-mapped files are the critical piece of the MMAPv1 storage engine in MongoDB. By using memory mapped files, MongoDB can treat the contents of its data files as if they were in memory. This provides MongoDB with an extremely fast and simple method for accessing and manipulating data.
MongoDB uses memory mapped files for managing and interacting with all data. Memory mapping assigns files to a block of virtual memory with a direct byte-for-byte correlation. MongoDB memory maps data files to memory as it accesses documents. Unaccessed data is not mapped to memory. Once mapped, the relationship between file and memory allows MongoDB to interact with the data in the file as if it were memory.

Database Properties

Atomicity and Transactions
In MongoDB, a write operation is atomic on the level of a single document, even if the operation modifies multiple embedded documents within a single document. When a single write operation modifies multiple documents, the modification of each document is atomic, but the operation as a whole is not atomic and other operations may interleave. However, we can isolate a single write operation that affects multiple documents using the $isolated operator.
Get more information at Concurrency.

Consistency
MongoDB is consistent by default: reads and writes are issued to the primary member of a replica set. Applications can optionally read from secondary replicas, where data is eventually consistent by default. Reads from secondaries can be useful in scenarios where it is acceptable for data to be slightly out of date, such as some reporting applications. Applications can also read from the closest copy of the data (as measured by ping distance) when latency is more important than consistency.
Get more information at Consistency.

SOA : Service-Oriented Architecture

What is Service-Oriented Architecture?
Service-Oriented Architecture (SOA) is an architectural style. Applications built using an SOA style deliver functionality as services that can be used or reused when building applications or integrating within the enterprise or trading partners. It is a method of design, deployment, and management of both applications and the software infrastructure where:

• All software is organized into business services that are network accessible and executable.
• Service interfaces are based on public standards for interoperability.

Key Characteristics of SOA

• Uses open standards to integrate software assets as services
• Standardizes interactions of services
• Services become building blocks that form business flows
• Services can be reused by other applications
• Quality of service, security and performance are specified
• Software infrastructure is responsible for managing
• Services are catalogued and discoverable
• Data are catalogued and discoverable
• Protocols use only industry standards

What is an Enterprise Service Bus (ESB)?

• An enterprise service bus is an infrastructure used for building compound applications
• The enterprise service bus is the glue that holds the compound application together
• The enterprise service bus is an emerging style for integrating enterprise applications in an implementation-independent fashion
• An enterprise service bus can be thought of as an abstraction layer on top of an Enterprise Messaging System

Key Characteristics of an ESB

• Streamlines development
• Supports multiple binding strategies
• Performs data transformation
• Intelligent routing
• Real time monitoring
• Exception handling
• Service security