About the S3 uploads, they cannot go direct to AWS servers. It has to go through the Instagram servers since the upload needs to be authenticated AND the S3 connection secrets can/should not be kept on the client end.
Hey Ravi! The diagram simplifies this a bit so that there aren’t too many arrows going everywhere. But yes, you are absolutely correct! The application server is involved in image uploads for the reasons you mentioned.
We don't want clients to upload images to the application servers, so the servers can then put them into your object store. This will just burn resources and slow things down. Instead, the application server will be used to create signed URL requests so that the client can upload directly to S3.
I am curious on why they needed to query Redis for the User ID instead of retrieving it in along with the Picture ID. On the related linked article they mention it was for legacy reasons but still, seems odd to me that they had this setup.
This was a really nice read that helps really solidify some system design theoretical concepts seeing how it's applied in the real-world.
I also appreciated the diagram at the end too to help piece it all together.
Thanks for the article!
I always appreciate your input Jordan. Thanks so much!
Thank you for this convenient post!
About the S3 uploads, they cannot go direct to AWS servers. It has to go through the Instagram servers since the upload needs to be authenticated AND the S3 connection secrets can/should not be kept on the client end.
Hey Ravi! The diagram simplifies this a bit so that there aren’t too many arrows going everywhere. But yes, you are absolutely correct! The application server is involved in image uploads for the reasons you mentioned.
We don't want clients to upload images to the application servers, so the servers can then put them into your object store. This will just burn resources and slow things down. Instead, the application server will be used to create signed URL requests so that the client can upload directly to S3.
Thanks for the article!
Keep things simple is all your need! In my daily work, I also asked our team to do things as simple as we can.
If we get to the situation that all engineers have to fix bugs. No innvocation can happen.
Great article!
I am curious on why they needed to query Redis for the User ID instead of retrieving it in along with the Picture ID. On the related linked article they mention it was for legacy reasons but still, seems odd to me that they had this setup.
Maybe to identify the shard for the stored user data