Topic
32 articles
Amazon’s says: “We strive to offer our customers the lowest possible prices, the best available selection, and the utmost convenience.” When it comes to egress, their prices are far from the lowest.
Introducing Cloudflare’s S3-compatible Object Storage service, with zero egress bandwidth charges and automatic migration from S3-compatible services.
Use Sippy to incrementally migrate data from S3 to R2 as it’s requested and avoid migration-specific egress fees
A few months ago we released a new way for people to run serverless Javascript called Cloudflare Workers. We believe Workers is the fastest way to execute serverless functions, but lets prove it.
For a company like Slack that strives to be as data-driven as possible, understanding how our users use our product is essential. The Data Engineering team at Slack works to provide an ecosystem to help people in the company quickly and easily answer questions about usage, so they can make better and data informed decisions:…
Lyft is pleased to announce the initial open source release of our IPvlan-based CNI networking stack for running Kubernetes at scale in…
This post was co-authored with Andrew Johnson (@SecPrez).
By Curtis Castrapel, Patrick Sanders, and Hee Won Kim
I believe AWS Lambda is an exciting service, though maybe not for the reasons you’d expect. Rather than breathlessly talking about…
by David Berg, Ravi Kiran Chirravuri, Romain Cledat, Jason Ge, Savin Goyal, Ferras Hamad, Ville Tuulos
Excerpt By 2024, Slack’s data platform had accumulated 700+ SSH-based operators orchestrating critical data pipelines. We’re talking daily search indexing that processed terabytes of data, analytics jobs powering business intelligence, the whole shebang. Every single one of these jobs required direct SSH access to production AWS Elastic MapReduce (EMR) clusters. We had a massive security…
Over the past few years, we’ve been on a journey to modernise how we run Amazon Elastic Compute Cloud (EC2) instances at Slack. In our first post, Advancing Our Chef Infrastructure, we shared how we moved from a single Chef stack to a resilient, multi-stack setup with versioned cookbook deployments and safer promotion workflows. This…
In early 2023, Slack faced a foundational challenge: serving Large Language Models (LLMs) at enterprise scale with the security, reliability, and performance our customers expect. Over three years, we evolved from basic infrastructure to orchestrating a sophisticated multi-cloud architecture. We didn’t just want shiny new models; we needed a system resilient to regional outages and…
Authors: Brad Nicholson, Andrew Tanner, Marco Montagna Instacart runs one of the largest PostgreSQL fleets on top of AWS RDS, and part of managing a fleet like this is to keep it up to date....
The basics of Terraform and RDS. At Instacart we are approaching a hundred PostgreSQL instances running in AWS RDS. They are all managed in Terraform. Using Terraform with RDS has caused a lot of pain...
In today’s world of data breaches and privacy concerns, every measure you can take to protect your data is another brick in the wall defending you. At Instacart , no matter how confident we feel...
How we reduced error rates, and dropped latencies across merchants’ flows Reading Time: 6 Minutes Shopify merchants trust that when they build their stores on our platform, we’ve got their back. They can focus on their business, while we handle everything else. Any failures or degradations that happen put our promise of a sturdy, battle-tested platform at risk. To do so, we need to ensure that the platform stays up and stays reliable. Shopify since 2016 has grown from 375,000 merchants to over 600,000. As of today, an average of 450,000 S3 operations per second are being made through our platform. However, that rapid growth also came with an increased S3 error rate, and increased read and write latencies. While we use S3 at Shopify, if your application uses any flavor of cloud storage, and its use of cloud storage strongly correlates with the growth of your user base—whether it’s storing user or event data—I’m hoping this post provides some insight into how to optimize your cloud storage!
OpenTalk, a videoconferencing solution, needed a secure and scalable Identity and Access Management (IAM) solution to authenticate users across various services while maintaining strict security and…
At Instacart, we run our infrastructure on AWS, so our systems often deal with AWS ARNs . We often run into cluttered code, and needed to develop a solution for simpler, safer code. That’s why...
This blog shows how to find when something is wrong in production, avoid jumping between tabs/docs to find information, and resolving issues quickly in the troubleshooting process, using an AWS integration as a starting point.
Discover how to build reliable webhook handlers for Stripe events using AWS in this comprehensive guide. Learn about the challenges of processing webhook events at scale and how to address them with an enterprise-grade architecture.
For developers building on AWS, you have various choices for processing payments within your application. Most developers choose a payment processing service to handle this part of their application flow, which involves integrating with a third-party vendor outside of the AWS environment.