How We Reduced a $1,000/month Imgix Bill to $1 using Google Cloud
By Adam Fortuna
We store a LOT of images. Every single book has a cover image that we show and every reader has an avatar. With 800,000 books that’s already a massive amount of potential images – but it’s just the start.
For each book we have multiple covers. On Hardcover we use the term “Books” and “Editions”, where a Book has one or more Editions.
Each Edition represents a unique ISBN and release of the book. For a book that’s just released it might have a hardcover Edition, an audiobook Edition and a mass market paperback Edition. If it’s a worldwide release, it might have these in multiple languages.
For each of these Editions, we store multiple images of the cover. We save whatever we can find about this Edition from public sources – Google Books and OpenLibrary being our primary book sources so far.
Let’s think about The Hobbit. Released in 1937, we have 237 different editions of The Hobbit in our database. For each of those 237 we have zero or many cover images.
This means that for 800,000 books we might have 25,000,000 editions, and potentially 50,000,000 images.
We don’t have anywhere near that many yet. I ran a quick check to see our current image count (commas added for emphasis).
irb(main):001:0> Image.count
=> 1,206,455Code language: PHP (php)So, we have about 1.2 million images. Considering we have about 3,000 users, that’s a lot of covers. 😂
In our case we only store the original image file and not the resized versions.
Oh right, resized versions!
For each cover, we show it in different sizes across Hardcover. On the primary book page we show the largest version of the cover. When showing it on an Airlist, we allow readers to set their own size. Elsewhere we use the size of the parent component.
In our codebase we call these sizes xs, sm, md, and lg. Having these sizes hardcoded means we have a limited number of potential sizes.
But, unfortunately, that’s not all!
Covers aren’t uniform in their height and width. Audiobook covers, for example, are usually square. On our new Book pages, we decided to make the covers we show uniform there. We think this makes those pages look much better than if every cover was a different size.
However there are times when we want the cover to be displayed in it’s original proportions. Sometimes we still want a uniform height or width with those proportions. So solve this, we added an option to stretch covers by x or y. This raised the number of potential images from 4 (xs to lg), to 12 (xs to lg in both x and y).
Oh if it was only 12 sizes. It turns out that mobile devices have different pixel density. Due to this, you wouldn’t want to show a 100w by 150h image on a phone. You’d actually want to show a 200w by 300h image, but in a 100w x 150h space. The quality goes up but the space stays the same. In other words, we need a 2x version of every image.
So now we’re up to 24 versions of every image. 🤯
Resizing 1.2 million images 24 times wouldn’t be great. If we decided we needed to change the sizes later on we’d have to resize every single one of them just to get things working.
On Demand Image Resizing
When I started working on Hardcover I knew this would be an issue. Our backend is in Ruby on Rails, and it would be easy enough to use MiniMagick to resize each image into each proportion and store those on Google Cloud (where we store the original images).
Resizing isn’t the issue. It’s being confident that we have all image sizes accounted for (and we’re not even talking about the profile avatar image sizes).
After some research initially, the two biggest platforms for on-demand image resizing are Cloudinary and Imgix.
Both allow you to throw your original images behind a CDN that handles all resizing using URLs.
For example, if you wanted to get a version of an image that’s 700px wide, we could make a request to https://hardcover.imgix.net/images/logos/hardcover-logo-with-text.png?w=700.
Adding a fm=webp can further reduce the image size and make Google Page Speed a little happier.
Pricing for Cloudinary and Imgix is not straightforward.
Cloudinary
Cloudinary uses a credit system where you “spend” credits. On a $224/month plan that would give us 600k transformations or 600GB storage or 600GB bandwidth. With 1.2 million images resized 24 times, that’s 28.8m transformations, or about $10k/month – 47x their highest plan (if I understand it correctly).
Imgix is better. When we joined their pricing was entirely based on bandwidth. As a small startup our bandwidth was almost nothing! For two years we paid $50/month. It was great.
Imgix
Earlier this year Imgix changed their pricing to be based on the number of origin images you use in a given month. Their highest plan (before the dreaded “Call for pricing” tier) capped out at 50,000 origin images for $500/month.
I was so focused on Hardcover’s last big release that I missed this detail. Before I knew it we were drastically over their highest pricing tier.
This means that for August 116,735 different origin images were requested with a total of 277,641 different versions of those images. At Imgix’s current pricing that would be about $1,116/month.
Considering that Hardcover’s entire monthly spend is under $1,000, this would more than double our monthly spend. We’re not yet profitable, so this would be more money coming out of my pocket to keep things running.
That was enough incentive to find an alternative.
Imgix and Cloudinary Alternatives
When we started looking around for alternatives I originally hoped to find a drop in replacement. Something where we could change our URLs and they’d still be backed by our original images with transformations.
Imagekit looked promising, but after reading all the fine print we were still looking at $500 minumum.
With hosted solutions exhausted, we turned to open source. Full disclosure here: I’m not great at Devops. I don’t enjoy the details of hosting, which is why I rely on Vercel, Heroku and Hasura to keep Hardcover running. The thought of being on the hook for this image service gave me instant anxiety.
After searching a while I found Imagor, a Go-based image processing server. It seemed great! After digging more into it, I liked how it had an “in” bucket and an “out” bucket where it stored the resized images.
Unfortunately getting it setup with a plugin to support Google Cloud wasn’t straightforward. When looking for a Docker setup, I stumbled on this Hacker News Thread about it that mentioned Imaginary – another Go-based Image processing server. And the best part: Imaginary had a link to run it right on Google Cloud Run!
Hosting Imaginary on Google Cloud Run
I had no idea what Google Cloud Run was, but that seemed like a good thing.
It turns out there are three main ways to run a dockerized application on Google Cloud Platform: Compute Engine, Kubernetes Engine and Cloud Run.
Think of Cloud Run as a quick way to startup something which can be serverless. There’s even an option of Autoscale! We can scale Imaginary down to zero when it’s not needed or up to 5 (say, if we changed our image size and it needed to do a lot of resizing).
It took me about two days to get the settings right in Google Cloud for Imaginary, so I’m going to share the full setting here in the hope that it helps someone else. Just scroll on by if you don’t need it.
Some things to point out from this configuration:
- Aside from setting the port as 8080 you don’t need to do any other port proxying.
- Wrap container arguments with a space in them with single quotes. If you don’t nothing will work. ( Figuring this out required starting a docker container and experimenting until it worked 😭 ).
- That’s it. Those are the two main issues.
With this one change we had a new URL, https://imaginary-hcgk56u3uq-uc.a.run.app that we could use to resize images. We could switch off Imgix and use it!
But we quickly noticed a problem. Each image request took about 500ms. Imgix was responding in 40ms!
It turns out that Imaginary was fetching the image from our Google Bucket, resizing it and streaming the response for every single image. Nothing was cached, so every request required a full server request and internal network requests to complete.
Add a Content Delivery Network
Chances are you know what we need to solve this: a CDN. By adding a CDN in front of these requests, we can control the caching and serve images from a server that’s geographically close to our users.
The CDN I’ve used the most is Amazon’s CloudFront. In an effort to switch, I started looking into Google’s CDN options.
To a GCP practitioner I’m sure the tools make a lot of sense. To someone completely unfamiliar with the interface it took a lot of experimentation to figure out what I needed to do.
The process of setting up the CDN ended up taking me an entire day. If I knew then what I know now it’d take about an 15 minutes.
Here’s a walkthrough of the different pieces you’ll need to create to make this work:
Create a Load Balancer
Create a load balancer and set it to “Application Load Balancer (HTTP/S)”. Set “Internet facing or internal only” to “From Internet to my VMs or server less services” and “Global or Regional” to “Classic Application Load Balancer”.
Set the front-end to “HTTPS” and create a new IP address. For this you’ll need to create an SLL policy and set the URL that you’ll use. In our case, we added “cdn.hardcover.app” as the URL associated with that IP address, and Google generated a certificate for that URL.
The back-end part gave me the most trouble. I thought I’d just be able to add a URL as the backend like with CloudFront. I tried that approach but couldn’t get it to work.
When you create a new backend you can select a “Backend Type” of “Serverless network endpoint group” and then “Create Serverless network endpoint group” and set then select Google Run and your Imaginary service.
Here’s something that I struggled with: you can create a CDN and have it create a load balancer, or you can create a load balancer and have it create a CDN. When I created the CDN and had it create the load balancer it didn’t associate with a backend service. I had to create a load balancer and then select the checkbox for “Enable Cloud CDN”.
Imaginary doesn’t add any cache headers, so we needed the CDN to tack on some cache headers. We can set this here from the “Backend Configuration” under “Load Balancer”. I initially set this to 30 days, but Google Pagespeed Index recommended extend this out. We now have this as 6 months. These images never change so this works for us.
To recap, you’ll need the following parts setup:
- Google IP addresses: A dedicated IP address with a host certificate under your domain. For this, you’ll need to verify you own the domain and add some DNS settings to your DNS provider (CloudFlare in our case).
- Google Cloud Run: This is the service that will run Imaginary.
- Google Load Balancing: Two balancers with one redirecting http to https, and another sitting in front of Cloud Run with a CDN.
- Google Cloud CDN: with a backend set as the Cloud Run service.
Whew, that’s a lot to setup! On the bright side you don’t need to write a single line of code, or edit a single file. It’s 100% configuration you’re able to do from the Google Cloud Console.
The end result of this? Let’s go back to The Hobbit and check it out.
Every one of these images is using Google CDN backed by a load balancer and Google Run. The result? 30ms to 50ms per image! Each one of these is cached for 6 months, so every other request by other readers will be just as quick!
How Much Does this Cost to Run?
The cost of this for our small site is almost entirely on the networking side.
When I break this down by service so far, here’s how much we’ve spent so far in September since we launched this:
- Google Run (not shown): $0 → $0.38 😂 (
- Google Cloud Storage (red or dark orange): $15.54 → $25.82
- Networking (blue): $0 → $11.20
The networking cost was a surprise to me here. I knew the load balancer and the CDN wouldn’t be free, but I still expected Cloud Run to cost more. When I looked into more, the $11.20/month networking cost is broken down as “Cloud Load Balancing Forwarding Rule Minimum Global” ($6.12) and “Networking Cloud Armor *” ($5) and Google CDN ($0.17).
So it turns out resizing and serving the images costs roughly $1 a month! Adding a load balancer in front is what actually costs money.
I’m still trying to figure out how to setup the CDN without the load balancer. If you’re more experience in Google Cloud than I am (which isn’t a high bar), I’d love to get your thoughts on how we could improve this. Join the #development channel on the Hardcover Discord and let me know.