“Aaall aboooard the the Aaaaamazon Twiiiisterrr!” yelled the rollercoaster operator with the handlebar mustache. Our protagonist Andy stepped in line, along with the other bright-eyed and smiling to-be riders; the Amazon Twister is the talk of the town and tickets can be hard to come by. The inevitable chit chat helps pass the time as Andy steps closer and closer to hit seat. “Click!” goes the safety restraint as he’s strapped in. Anticipation builds. “Let’s see what the Amazon Twister has in store for me!” Andy ponders.
I wasn’t really sure what to think when I first got my spot in Amazon’s 2021 summer SDE internship. On one end, I was super excited about the opportunity to use my skills in an actual work environment, peer into the actual world of SDE work, and just try something new. On the other hand, I was nervous about working in such a large company like Amazon, especially since up until then I’d been the sole developer in all the games, apps and side projects I’d done. Collaboration is great (really, it is!), but the associated sacrifice in individual freedom can be a bit frightening sometimes.
The Amazon Twister might be the best ride in town, but its not necessarily a walk in the park! Before the ride begins, an onboarding video is shown on the monitors on the backs of the seats. “Huh, you don’t see that everyday,” Andy remarks.
First things first, I loved Amazon’s onboarding material, both from a technical and a non-technical perspective. From the outside, Amazon’s Leadership Principles seem like kind of a trope, and even some internal communications during my time there may make it seem so. However, the talks, anecdotes, and courses within Amazon’s onboarding material gave me some practical insight into how those principles are used and what effect they have on Amazon’s way of doing things.
To provide you with some context about my perspective, I should mention that I’m not usually someone who programs for the sake of it. Yes, programming is fun, but to me it’s ultimately a means to an end – a means to bring about creation and change.
So as I learnt about the early days of Amazon, their “It’s always Day 1” principle, how they manage and create small teams, what “customer obsession” actually looks like and works, the Empty Chair Principle, Customer Anecdotes, and the concepts of working backwards, Press Releases before products, and team tenets, I realized there’s actually a lot a startup founder like me can learn from it.
The ride jolts into motion, even before the video is over! “No time to waste!” the grinning operator declares. The cart starts escalating upwards. Tick…tick…tick…tick it goes as it rides up the chain to the top of the first (and largest) drop-off point. “The first rise is always the scariest,” Andy reminds himself.
The onboarding material was also helpful from a technical standpoint. It was relatively thorough and very easy to follow. That, however, doesn’t necessarily mean all of technical onboarding was easy to complete. Amazon is a company with impressive and mind-boggling scale – you truly don’t realize how large Amazon’s engineering is until you’ve worked inside of it. That size, however, translates to their stack as well. They have a lot of internal tools and layers, and to get something decent and quality going you have to use and understand a decent amount of those tools. There’s always more to their stack the more you dig.
Might be endless, to be frank. ¯\_(ツ)_/¯
It was rewarding to see me get used to their CI tools, build tools, dependency tools, container tools, permissions tools, website-generation tools, security tools, and the tools they use to manage internal AWS accounts. One of the things that made it rewarding, though, is that it was hard. I was being thorough because its important stuff to understand well from the get-go. I wanted to be sure that when I start my project, I’m starting it off right.
This is probably a good time to talk about what exactly I was tasked with building.
Over the summer, I was part of the Ingestion and Storage team for Amazon Timestream. Timestream is an AWS timeseries database service with some pretty large customers, including Disney Plus. A hallmark of timeseries data is that it is produced in large amounts, so in order to facilitate the large amounts of throughput needed for ingestion and storage, Timestream heavily utilizes partitioning.
You can think of timeseries data as a series of points on a 2D graph of an arbitrary measure (like temperature) against time. So, if you wanted to partition that very large and very dense series, you could split that graph into rectangular segments called tiles.
The way tiles are created and how they are split is an important thing to perfect, since it can affect data availability, ingestion speed, query performance, and everything in-between. Moreover, it’s important that the developers thoroughly understand that algorithm and have good tools to be able to debug issues that might arise from tile management.
What I was building was a web-based visualizer called Timestream Mondrian[1] to visualize the tile structure for development and production-scale Timestream tables.
Figure 1 A simplified diagram of the structure of Mondrian
By the time I finished, Timestream Mondrian helped the development team answer questions like:
- How many tiles constitute this table?
- How are the tiles arranged? (Mondrian can display tens of thousands of them at a time)
- How large are they temporally and spatially?
- How densely packed are they? (This is a surprisingly useful feature)
- How much data do they have? What queries are touching them? How often?
It was a bit of an ambitious project to finish in 12 weeks, so I was eager to get started! The first couple weeks was tough though, as most of my time was spent reading and tinkering to understand Amazon’s stack and tools. I should add, though, that my team was very supportive whenever I reached out to them for help. I made the choice to do most of this learning alone (which did provide me with some good depth), but in retrospect asking my team for pointers more often would have saved me some time. Remember that. There was also some time spent researching non-internal development tools and libraries as I made my design document for Mondrian and eventually defended it amongst the team. That was a great experience.
Admittedly, there was also a good chunk of time spent enjoying, exploring and biking the beautiful city of Seattle!
Finally, the ticking stopped and we reached the top of the first hill of the Twister. A strange silence filled the air; “The hardest part is over; now the fun begins!” One more tick erupts, and the cart starts zooming down.
Around the third or so week, I got the ball rolling. I started making the first React pages, AWS Lambdas and HTTP Gateways, and started programming the initial visualizer in D3. It was a blast! I was able to have demos with new features every week, I was able to engage with the team and get their feedback, and I was able to report news during the daily sync-ups. I came to the offices (they are really beautiful by the way) and met my team members and manager. This was my honeymoon period of the internship, and it was during this time I really got to appreciate my team. They continued to be super supportive whenever I had questions, and they were surprisingly fun. These people were exceptionally fluent in their field, veterans of 30+ years sometimes, but at meetings they would still make (sometimes contrived) jokes, bring snacks, and have a good time while also being good at their job. It was a joy to see this.
Over the next month my project was being fleshed out, but with the end of my internship looming in the next 6 or so weeks, I switched gears. I had to start moving Mondrian from the MVP stage to something that’s ready to be used and maintained when I’m gone.
Vroom! Swiiisssh! Rummm! Dummmm! “Woooohooooo!” Andy exclaimed. This ride was a blast; a great symphony of tuns, loops, tunnels and dives. But over the horizon another hill looms. This one isn’t as scary, and Andy might even have fun as the cart climbs this one.
These several weeks had a very different feel than before. As I was already getting Mondrian ready for hosting and handoff, I really started to feel that I was developing for a team and not myself. I started researching hosting tools from the lens of how it would be for others to maintain, I started making design decisions on the webapp and hosting based on security and what other team members are already familiar with. The team was super informative and helpful in this process. I also had to get Mondrian ready for much larger datasets, which required completely rewriting it from D3 to THREE.js. This also involved some fun and clever backend changes to deal with tables of arbitrary size (like over ten times the sizes I used for testing). I started documenting Mondrian more as well (especially during the last two weeks); I documented the tools used and why they were used, and also the features that should be added next. This was a pretty heads-down period for me, but it was where I was probably having the most growth.
The cornerstone of this period is when I was able to give a one-hour explainer presentation on how Mondrian works to the team. This happened at the beginning of the penultimate week. It was really fun for everyone; I was able to show the team how graphics and WebGL works, and how Mondrian makes use of WebGL to scale to large datasets. I was also able to go into the tricks I used to handle a few gotchas of Timestream data, and how the backend also minimizes reads from Timestream servers for visualization.
Around the time of the presentation, things were winding down. I had done most of the hard work, all that was left was polish, final security changes, putting Mondrian up on Timestream’s cellular hosts, and handoff.
We had reached the top of the hill, and everyone was expecting another fast descent. Rather, the Amazon Twister had a surprise for us. It completed the last stretch slowly, giving us time to savor the last few moments of the ride in serenity.
The last two weeks felt very rewarding; I could see the culmination of all my work in a project that the team loved and a project I was genuinely proud of. As I was having my last series of security meetings to get Mondrian hosted on Timestream’s cells, writing my last documentation pages and wiki page, and listening to my manager tell me about what people were already thinking of for the future of Mondrian, I thought to myself “Wow.” My work was going to leave a mark.
The ride came to a soft halt in the same spot it has started just moments before. Just like that, the Amazon Twister was over as quickly as it started.
Mondrian was a very fun project to build. Apart from making something that my team members appreciated, I also got better at React hooks, learned to use THREE.js and D3js (finally got to use my 3D graphics knowledge in a non-game project!), and also learnt about AWS services like Lambda, S3, DynamoDB, Timestream, CloudWatch, IAM, Kinesis, API Gateway, CloudFormation and CDK. Coming from a Google Cloud background, it was interesting to see how AWS handles things differently and how complicated industry backends can get.
I enjoyed being able to document, explain, design, and defend my project amongst other engineers. The process of making and updating my design doc, and going through code-reviews and feedback iterations was a mix of frustration and enjoyment – it was clear I was working in a team for a team, rather than alone.
My team was very fun, very supportive and very knowledgeable; they were a great compliment to the cool interns I met and the amazing city of Seattle.
The riders disembark, exhausted but satisfied. As they all leave in different directions, Andy looks back at the slumbering cart, wondering who will take his seat next time it awakens.
Cover Image Credit: Meg Boulden
