Feeding Apache Spark Streaming from Amazon SQS?

amazon-sqs, apache-spark

Solution

There's a github project called spark-sql-receiver. It's been uploaded to the maven repository with the groupId of com.github.imapi artifactId of spark-sqs-receiver_2.10. It's currently on version 1.0.1. By the looks of the github project, it's being actively maintained as well. The following is some sample code shamelessly copied from the project's README.md file:

ssc.receiverStream(new SQSReceiver("sample")
      .credentials(<key>, <secret>)
      .at(Regions.US_EAST_1)
      .withTimeout(2))

Problem

Spark can be fed in many ways as it is explained in the documentation (like Kafka, Flume, Twitter, ZeroMQ, Kinesis or plain old TCP sockets). Does anybody know how to feed Spark Streaming from Amazon SQS?

Original source