Analytics, Apache Spark, Bigdata, Kafka, Messaging System

In-depth Kafka Message queue principles of high-reliability

At present many open source distributed processing systems such as Cloudera, Apache Storm, Spark and others support the integration with Kafka. Kafka is increasingly being favored by many internet shops and they use Kafka as one of its core messaging engines. The reliability of the Kafka message can be imagined as a commercial-grade messaging middleware solution. In this article, we will understand Kakfa storage mechanism, replication principle, synchronization principle, and durability assurance to analyze its reliability. As shown in the figure above, a typical Kafka architecture includes several Producers (which can be server logs, business data, page views generated by […]

Bigdata, Messaging System

ZeroMQ Part-1

  Programs like people need to communicate, and for them we have the UDP, TCP, HTTP, IPX, WebSocket protocol to connect and other related applications. But the underlying protocol is difficult to achieve. We need a high level of abstraction, scalable, and easy to use things, it is ZeroMQ(ØMQ). ØMQ gives us advanced levels of availability and speed. ØMQ is a neat messaging library that allows us to build our own messaging infrastructure. It can help build framework that scales where services could be handled by different applications. I am inclined to python programming and luckily pyzmq provides python bindings […]