Upstash Kafka with Decodable

Decodable is a platform which enables developers to build data pipelines using SQL. It is built on Apache Flink under the hood to provide a seamless experience, while abstracting away the underlying complexity. In this post, we will show how to connect an Upstash Kafka topic to Decodable to streamline messages from Kafka to Decodable.

​

Upstash Kafka Setup

Create a Kafka cluster using Upstash Console or Upstash CLI by following Getting Started.

​

Decodable Setup

Just like Upstash, Decodable is a managed service that means you do not need to host or provision anything. You can easily register for free and start using it.

After creating your account, click on Connections and New Connection. Select Apache Kafka. Then:

• Select Source as connection type.
• Select SASL_SSL as security protocol and SCRAM-SHA-256 as SASL mechanism.
• Enter your topic, SASL username, SASL password. You can find all those from Upstash console.
• Value format should be JSON.

In the next step, click on New Stream and give a name to it.

In the schema screen, add country, city, region and url with string type.

Give a name to your connection and click Create Connection. In the next screen click on Start.

​

Test the Setup

Now, let’s some events to our Kafka topic. Go to Upstash console, click on your cluster then Topics, click mytopic. Select Messages tab then click Produce a new message. Send a message in JSON format like the below:

{
  "country": "US",
  "city": "San Jose",
  "region": "CA",
  "url": "https://upstash.com"
}

Now, go back to Decodable console, click Streams and select the one you have created. Then click Run Preview. You should see something like:

​

Links

Decodable documentation

Decodable console

Upstash console