Skip to main content

REST API for Kafka

Introduction​

Alongside Kafka clients using TCP connections, Upstash provides a REST API to access Kafka topics over HTTP. REST API is essential for more restricted environments (mobile, edge etc) but also significantly lightweight when compared to native Kafka clients. With the REST API you don't need to manage Kafka clients and connections yourself.

Get Started​

If you do not have a Kafka cluster and/or topic already, follow these steps to create one.

In the cluster details section of the Upstash Console, scroll down the REST API section. You will see two basic REST API snippets there; the first one is to produce a message to a topic and the second one is to consume messages from a topic using Kafka consumer group mechanism.

  • Producer
curl https://tops-stingray-7863-eu1-rest-kafka.upstash.io/produce/$TOPIC/$MESSAGE \
  -u {{ UPSTASH_KAFKA_REST_USERNAME }}:{{ UPSTASH_KAFKA_REST_PASSWORD }}
  • Consumer
curl https://tops-stingray-7863-eu1-rest-kafka.upstash.io/consume/$GROUP_NAME/$GROUP_INSTANCE_NAME/$TOPIC \
  -u {{ UPSTASH_KAFKA_REST_USERNAME }}:{{ UPSTASH_KAFKA_REST_PASSWORD }}

Upstash Kafka REST API uses HTTP Basic Authentication scheme. You should copy the UPSTASH_KAFKA_REST_USERNAME and UPSTASH_KAFKA_REST_PASSWORD from the console and replace then in the code snippets shown above.

Produce​

To produce a message just replace the $TOPIC variable with a topic name which you've created before and replace the $MESSAGE with the message you want to send to the Kafka topic. 

curl https://tops-stingray-7863-eu1-rest-kafka.upstash.io/produce/mytopic/hello_kafka -u myuser:mypass

This will send the message to the mytopic Kafka topic and return the metadata related to the message as a JSON, like:

{
  "topic" : "mytopic",
  "partition" : 0,
  "offset" : 0,
  "timestamp" : 1637743323016
}

For more info and options about producer API please see REST Producer API section.

Consume​

To consume messages from the topic, replace the $TOPIC variable with a topic name which you've created before, replace the $GROUP_NAME with a meaningful name to be used as the Kafka consumer group id, and $GROUP_INSTANCE_NAME with a name for Kafka consumer instance id.

curl https://tops-stingray-7863-eu1-rest-kafka.upstash.io/consume/mygroup/mygroup_instance0/mytopic -u myuser:mypass

This will consume some messages from the mytopic using Kafka consumer group mechanism and return the messages as a JSON array, like:

[ {
  "topic" : "mytopic",
  "partition" : 0,
  "offset" : 5,
  "timestamp" : 1637745824883,
  "key" : "",
  "value" : "hello-world",
  "headers" : [ ]
}, {
  "topic" : "mytopic",
  "partition" : 0,
  "offset" : 6,
  "timestamp" : 1637745829327,
  "key" : "",
  "value" : "hello-kafka",
  "headers" : [ ]
}, {
  "topic" : "mytopic",
  "partition" : 0,
  "offset" : 7,
  "timestamp" : 1637745834756,
  "key" : "",
  "value" : "hello-upstash",
  "headers" : [ ]
} ]

For more info and options about consumer API please see REST Consumer APIs section.

Responses​

Each API returns a JSON response and they have their own specific structures. When the API call fails for a reason (illegal argument, unauthorized access, invalid API etc), a common error JSON message returned. Its structure is as following:

{error: String, status: Int}

error field contains the error message which explains the cause and status field shows the HTTP status code for the error.

Next​

Apart from the basic usages explained in this section, there are three categories for the REST API: