Arun ShankarKafka is an open-source project, which is licensed to the Apache Foundation and is amongst the top three most popular open-source projects of all time. Kafka enables event streaming, called data in motion. It enables a lot of companies to get started into some sense of scale and Confluent completes Kafka with a set of features around governance, security, support. Confluent also provides thought leadership and guidance on how to build a central nervous system around Kafka.
Think of Kafka as an engine and Confluent is the car. The engine by itself, is not much use; and the car without an engine does not move. So, they are complimentary to each other. Some companies want to build their own car, but the majority of companies want to buy the finished product.
There are two types of available platforms. There is Confluent platform, which is a self-managed platform and companies have to implement and operate that platform by themselves. They can choose to deploy that in their own datacentres or in public or private clouds and responsibility of management is on the customer.
The second platform is a fully managed Kafka in the cloud, delivered as a SaaS and guaranteed through SLAs. The Confluent Cloud platform is fully managed and is like a self-driving autonomous car.
To get to a centralised nervous system data platform, you need to build more and more use cases
Kafka is a transport mechanism for data, it is software that sits between other software. Within Kafka, there is compute and storage, which is very different to messaging – messaging is not a storage tool. Kafka has ability to process data as it is moving.
What that means is in the analogy of transferring money, the bank can, as you are requesting the bank to transfer money for you, it can process whether this is a fraudulent or verified transaction. And that is happening in real time.
Kafka enables real time processing of data. Data can be stored within Kafka just as data can be stored within the database. That is something most people only discover as a second step because people will see Kafka initially as a message.
You need to add a democracy to data and make data decentralised and available, and that takes a bit of time
Kafka answers the needs of the modern-day company that is trying to be digital first. The challenge with building a complex IT environment is that generally you have to connect an application to a database, and that is a very tight coupling.
If you change anything on the database or the application side, you need to evolve that connection, that coupling. And if you think about companies that have several hundreds, and in some cases thousands of databases, applications located in different areas, that adds a lot of complexity.
To simplify that architecture, Kafka decouples the sources of data to the seats of data. Also, there are scalability challenges with modern kind of transport mechanisms for data, while Kafka is scalable. What that means is the more messages that you have, the more events that are flying through Kafka, Kafka can actually handle that.
With Kafka, it is almost instantaneous in real time. So that is the real benefit that drives that convenience and great customer experience.
To get to a centralised nervous system data platform, you need to build more and more use cases and you need to add a democracy to data and make data decentralised and available, for whoever in the company needs it. And that takes a bit of time.
There is some impact on organisation and processes as Kafka is brought into the organisation and Confluent provides expertise around how to build that within the organisation.
Kafka enables real time processing of data. Data can be stored within Kafka just as data can be stored within the database. The challenge with building a complex IT environment is that generally you have to connect an application to a database, and that is a very tight coupling. If you change anything on the database or the application side, you need to evolve that coupling. To simplify that architecture, Kafka decouples the sources of data to the seats of data.
Kafka is an open-source project for event streaming, called data in motion, and enables companies to get started into some sense of scale.
