Happy to Announce Our New Discussion Forum | Join Now

DynamoDB+SQS vs Kinesis+Redshift in case of regular log vs emergency


#1

The department of transportation for a major metropolitan area has placed sensors on roads at key locations around the city. The goal is to analyze the flow of traffic and notifications from emergency services to identify potential issues and to help planners correct trouble spots.
A data engineer needs a scalable and fault-tolerant solution that allows planners to respond to issues within 30 seconds of their occurrence.
Which solution should the data engineer choose?
A. Collect the sensor data with Amazon Kinesis Firehose and store it in Amazon Redshift for analysis. Collect emergency services events with Amazon SQS and store in Amazon DynamoDB for analysis.
B. Collect the sensor data with Amazon SQS and store in Amazon DynamoDB for analysis. Collect emergency services events with Amazon Kinesis Firehose and store in Amazon Redshift for analysis.
C. Collect both sensor data and emergency services events with Amazon Kinesis Streams and use DynamoDB for analysis.
D. Collect both sensor data and emergency services events with Amazon Kinesis Firehose and use Amazon Redshift for analysis.


#2

Option D seems to be an optimal solution. Amazon Kinesis Data Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon Elasticsearch Service (Amazon ES), and Splunk.

Amazon Redshift is a fully managed, petabyte-scale data warehouse service in the cloud.

The first step to create a data warehouse is to launch a set of nodes, called an Amazon Redshift cluster. After you provision your cluster, you can upload your data set and then perform data analysis queries. Regardless of the size of the data set, Amazon Redshift offers fast query performance using the same SQL-based tools and business intelligence applications that you use today.


#3

@devip I think the question is more focus on separating the usage (regular / historical cases versus ongoing, time sensitive info). Putting option D means we have to wait till the data change (I remember it’s about 5GB) and load from S3 to Redshit. Plus, Redshift does poorly while handling upsert case, which happens in emergency service event all the time.


#4

notifications from emergency services = sqs


#5

Hello Guys, I bring to your consideration my approach waiting for feedback. These questions spot me in a scenario where I need to validate the content of data and in case of an emergency event trigger right after an emergency alert for the people in charge. Ok so having said that let’s move forward with the answer options. Well, the Kinesis FH answers for me are not suitable because FH spends a time for flushing process so every certain time of period is flushing itself based on size or seconds went by. So, whereas this flush process is ongoing I might lose data because my FH is not listening. So, the answer A, B and D are pulled off for me. I used to work for a project where we had to move from Kinesis FH to Kinesis DS because of that short period of time when FH flushes the buffer and is not catching messages.

On the other hand, Kinesis DS is all the entire time listen up :slight_smile: . my remaining solution so is C but then this answer is storing directly in Dynamo which is quite possible with FH or the KCL. Eitherways is not mentioning an operation on the middle like a lambda function or perhaps within my customized pipeline for KCL I might process the payload catching an urgent event and trigger a notification with SNS.

I believe that this question is missing details on the answers or perhaps I don’t get enough idea of how the possibilities serve may fulfil the right answer for me none of these may work. The ones that mention an SQS that would be the ideal also involve as a collector a Kinesis FH and at that point, this is not any more reliable. So the last option, if I have to decide of what there is available, might be the SQS even though any option guarantees 30 seconds of notice when the event arise.

thanks so much.


#6

Option A looks most appropriate . Here we are dealing typically with sensor(real time) for which Kinesis firehorse would be good fit and other one is notification service(adhoc frequency), similar to event based model , for which SQS looks suitable