Build a system to send ARTIK Cloud data to Amazon Kinesis Streams

ARTIK Cloud subscription and notification services are securely integrated with Amazon Kinesis, the real-time data ingestion framework of Amazon Web Services (AWS). Together with AWS tools such as Amazon Machine Learning and Amazon EMR, this allows you to enhance applications with custom analytics or machine learning.

Let’s build a sample system to utilize this integration. Devices send data to ARTIK Cloud and then the data is read from AWS Kinesis Stream. By completing this sample, you will learn how to:

Find the source code at GitHub.

In this article:

High-level system view

As illustrated above, the sample system has the following functionalities:

  1. Subscribe AWS Kinesis Stream to ARTIK Cloud
  2. Devices send data to ARTIK Cloud
  3. ARTIK Cloud pushes device data to Amazon Kinesis Streams
  4. Read device data from Kinesis Stream

We build two applications to perform 1 and 4, and use the Device Simulator (instead of a real device) to do item 2. We are set for item 3, since ARTIK Cloud is integrated with Amazon Kinesis Streams.

Installation and setup

Now let’s set up your system to play with.

  1. Set up at ARTIK Cloud
  2. Set up at Amazon AWS 
  3. Download two JAR files: Subscription app and Stream reader app, contained in this zip.

Instead of downloading the JAR files for step 3, you can follow the Java project setup instructions to build the apps from the source code.  

After finishing setup, you should have the following values:

  • ARTIK Cloud user ID and access token
  • Amazon AWS access key and secret
  • Amazon Kinesis Stream name and region

These will come in later.

Demo

The demo shows that devices send data to ARTIK Cloud and that this data is read from the Kinesis Stream.

  1. Run the subscription app to subscribe your Kinesis Stream to ARTIK Cloud. Open a terminal and run the command at the directory where the JAR file is located:
    java -jar sub-kinesis-to-akc-x.x.jar -u ARTIKCLOUD_UID -t ARTIKCLOUD_TOKEN -k AWS_KEY -s AWS_SECRET -r KINESIS_STREAM_REGION -n KINESIS_STREAM_NAME 

    Remember that you got the required input arguments in the setup phase. From the output, you should see the subscription succeed with a response like the following:

    response code : 200
    response : {"data":{"id":"230117430e6e47","aid":"b6951bf387b84f63","messageType":"message","uid":"240","description":"This is a subscription to user devices","subscriptionType":"awsKinesis","awsKey":"ACRWQ","awsRegion":"us-west-1","awsKinesisStreamName":"akcstream","status":"ACTIVE","createdOn":1489701092932,"modifiedOn":1489701092932}} 

    id in the response JSON is the subscription ID. You will need this ID when deleting this subscription.

  2. Start the Device Simulator to send data to ARTIK Cloud on the behalf of the device. Log into My ARTIK Cloud, navigate to DEVICES, click “Show Simulator”, pick a device, and configure the data fields used in the simulation. In the following screenshot, I have enabled the simulation of Demo Fire Sensor on my account. 

  3. Now click “Start Simulation” to send data to ARTIK Cloud.  You should see the real-time data on the CHARTS page:  

     

    You can simulate sending data from multiple devices. All data sent to ARTIK Cloud from your account are being pushed to your Kinesis Stream.

  4. Start the Kinesis Stream reader app to read device data. Run the following command at the directory where the JAR file is located:

    java -jar read-stream-x.x.jar -k AWS_KEY -s AWS_SECRET -r KINESIS_STREAM_REGION -n KINESIS_STREAM_NAME 

    From the terminal printout, you should see that the stream shard iterator has been initialized and the app is receiving messages from the stream as follows:

    Kinesis record: {SequenceNumber: 495,ApproximateArrivalTimestamp: Thu Mar 16 15:16:58 PDT 2017,Data: java.nio.HeapByteBuffer[pos=0 lim=264 cap=264],PartitionKey: 51f1e30f37f6401f9ac71f24a67eb08f}
    ARTIK Cloud message: {"uid":"240bc","cts":1489702616991,"data":{"temp":3300.747802734375,"onFire":true},"mid":"51f1e30f37f6401f9ac71f24a67eb08f","mv":1,"sdid":"4deb","sdtid":"dtce45703593274ba0b4feedb83bc152d8","ts":1489702616991}
    Kinesis record: {SequenceNumber: 496,ApproximateArrivalTimestamp: Thu Mar 16 15:16:59 PDT 2017,Data: java.nio.HeapByteBuffer[pos=0 lim=261 cap=261],PartitionKey: 70db081db5164aaf9bc4a4db4973f820}
    ARTIK Cloud message: {"uid":"240b","cts":1489702618409,"data":{"temp":6352.900390625,"onFire":true},"mid":"70db081db5164aaf9bc4a4db4973f820","mv":1,"sdid":"4debdd60ea9f4b1499c7c292b81f001f","sdtid":"dtce45703593274ba0b4feedb83bc152d8","ts":1489702618409}
    Received 9 records. sleep for 2s ...
    Comparing messages in ARTIK Cloud’s Data Logs and the printout of the reader app, you should see one-to-one mapping. If you have multiple devices sending data to ARTIK Cloud, you will see all of them in the printout of the app.

  5. After you are done, you can delete the subscription:
    java -jar sub-kinesis-to-akc-x.x.jar -del ARTIKCLOUD_SUBSCRIPTION_ID -t ARTIKCLOUD_TOKEN
    Remember that you got your subscription ID at Step 1. You should delete the subscription if you start over.

Implementation

Subscribe AWS Kinesis

Subscribing a Kinesis Stream to ARTIK Cloud is straightforward: make a POST request to ARTIK Cloud with a valid access token. The following is an example of the request body: 

{
"messageType": "message",
"uid": "5beb4e30",
"subscriptionType": "awsKinesis",
"awsKey": “ABC”,
"awsSecret": “XYZ”,
"awsRegion": "us-west-1",
"awsKinesisStreamName": “stream1”,
"description": "This is a subscription to user’s devices"
}
On success, the response will contain the subscription ID and related info. Refer to the developer documentation for various subscribing options. 

To remove the subscription, simply make the HTTP DELETE request with a valid access token.

Consume data from Kinesis Stream

The implementation of the Stream reader follows the AWS documentation: Developing Streams Consumer using Kinesis Streams API with the AWS SDK for Java.

From sample to service

For simplicity, the sample uses a user token as an ARTIK Cloud access token. This restricts the sample system to a use case where the device owner and the application developer are the same. In a real-world scenario, application developers (this means YOU!) and device owners (who are also end users of the application) are separate.

We suggest using the three types of tokens to implement a real-world system. The system should have at least two components: backend servers and a mobile/web application. All these components use the same ARTIK Cloud application ID.

  1. In the mobile or web application, end users log into ARTIK Cloud and grant the application permission to read data from their devices. The application implements the login flow to obtain user tokens.
  2. After the permission is granted, the backend server obtains an application token and then uses that token to subscribe your Kinesis Stream to ARTIK Cloud.
  3. Devices belonging to the end users use device tokens to send data to ARTIK Cloud.

Interested in trying out Samsung ARTIK Cloud? Sign up for a developer account and connect devices and services while bypassing silos. We look forward to seeing your IoT solutions!