How to copy data in bulk from Kinesis -> Redshift
amazon-kinesis, amazon-redshift, amazon-s3, amazon-web-services
Solution
It is now possible to do so without user-code via a new managed service called Kinesis Firehose. It manages the desired buffering intervals, temp uploads to s3, upload to Redshift, error handling and auto throughput management.
Problem
When i read about AWS data pipeline the idea immediately struck - produce statistics to kinesis and create a job in pipeline that will consume data from kinesis and COPY it to redshift every hour. All in one go. But it seems there is no node in pipeline that can consume kinesis. So now i have two possible plans of action: - Create instance where Kinesis's data will be consumed and sent to S3 split by hours. Pipeline will copy from there to Redshift. - Consume from Kinesis and produce COPY directly to Redshift on the spot. What should I do? Is there no way to connect Kinesis to redshift using AWS services only, without custom code?