September 2020 Update

We’re republishing this article to help you prepare your app for Black Friday/Cyber Monday (BFCM). Though the information below is still valid, the best way of managing webhooks is now with EventBridge. EventBridge's server-less, event-driven architecture can help you reduce infrastructure costs while providing a highly scalable system.

Learn how to manage webhook events with Amazon EventBridge via the tutorial below.

View tutorial

Believe it or not, the holidays are just around the corner, and Shopify app developers need to be ready. The busiest shopping days of the year are no time to experience a failure or downtime with your app—you need to be prepared to handle the influx of orders and updates to customers, products, and more.

Many Shopify developers make use of webhooks: messages that Shopify sends when something happens inside of Shopify. Shopify webhooks are an efficient way to track changes in the ecommerce platform, and there’s no need to poll Shopify for changes—applications are notified via a push message when something happens in Shopify.

Until you start receiving webhooks at scale, you can't anticipate how many of them will be sent, and when. When we first launched our app Rewind, we were surprised to see webhooks were being sent every day at midnight for a large number of stores.

After talking to customers, we learned that these webhooks were sent when third-party systems synced Shopify data. For example, an inventory system that syncs daily can cause your app to be flooded by thousands or tens of thousands of webhooks, with absolutely no warning.

In this article, we’ll help you create a system that can handle webhooks offline, so your application can handle any load that Shopify sends your way. You can then process the webhooks as quickly as you want, on your terms. This system is built on top of a “serverless” architecture, so it scales to handle large loads automatically, and is inexpensive.

Creating a system to handle Shopify webhooks

Shopify requires that your application respond to webhooks in a timely manner, otherwise they'll be removed. If you process Shopify webhooks inline, as they happen, that can cause a major problem, as your application needs a powerful server to handle the webhooks that might be sent. This can be extremely expensive—you’ll need to run powerful servers 24/7, since you’ll never know when the webhooks will be sent.

Even with lots of large servers, you still run the risk of not responding quickly enough, and missing the data in the webhook, or having Shopify cancel all future webhooks.

A better idea is to handle them in a separate system, and have something that responds to Shopify quickly. If you acknowledge Shopify’s webhook quickly, and save the data somewhere else, your webhooks will never be cancelled, and you’ll have the data from the webhook so that you can process it in the future. You’ll never miss webhook data, and can ensure that your application never misses any data that’s sent to it.

"If you acknowledge Shopify’s webhook quickly, and save the data somewhere else, your webhooks will never be cancelled, and you’ll have the data from the webhook so that you can process it in the future."

This blog post describes the system we built for Rewind on top of Amazon Web Services, using API Gateway, Lambda, and SQS Queues. When Shopify sends a webhook to Rewind, we process it using API Gateway and Lambda, two “serverless” AWS services that autoscale to meet demand. The messages are stored in an SQS Queue, which saves the data from the webhook. Servers then process the webhook queues at a rate you control. Here are the steps to follow.

You might also like: 5 Key Strategies to Improve Your App Support.

1. Set up the SQS queue

The first step in this process is to set up the SQS queue. SQS has two types of queues: a first in/first out (FIFO) queue, or a regular queue. You may want to use a FIFO queue to ensure the order of the webhooks is maintained, or you can use a regular queue and check the updated_at time of the message. Either will work, and which one to use depends on your use case.

shopify webhooks: aws — Setting up the SQS Queue in AWS.

2. Set up the Lambda function

AWS Lambda allows you to run code on AWS servers, without having to manage the servers yourself. People often refer to this as "serverless" computing (although technically there are servers in the cloud.) Lambda servers are run by Amazon, and autoscale to handle increased loads. You only pay for the milliseconds that it takes for your code to execute, so it’s very inexpensive to run.

To get started with Lambda, create a new function. This code will take the incoming information from the API Gateway (which we'll set up in the next step), check the security of the webhook (to ensure it's coming from Shopify), and then add it to the SQS Queue that we created in step one.

Take a look at the code:

shopify webhooks: lambda function — Setting up the Lambda function.

You might also like: How to Get Reviews for Your Shopify Apps.

3. Set up the API Gateway

Once Lambda is set up, the next step is to set up the API Gateway. The API Gateway provides a way to define your API, which will run in AWS without any servers. Rather than hit your servers, webhooks will hit the API Gateway. This way you don't need to worry about running any servers, and scaling them as the Shopify webhooks come in.

shopify webhooks: api gateway — Setting up the API Gateway.

To set up the API gateway, first create a new model with the following schema:

For the integration request, we set up the following mapping template:

4. Set up Elastic Beanstalk workers

The last piece is to work off the SQS queue with an Elastic Beanstalk worker. Elastic Beanstalk workers automatically pull an item from the SQS queue, and POST it to localhost on the machine.

Our workers are written in Ruby and are a simple Ruby application. You can find the code for our workers in our Github repository. We’ve open-sourced our code for this serverless Shopify webhook system on Github, and welcome contributions from others to help improve it.

shopify webhooks: github — The code for handling webhooks is open source and hosted on GitHub.

The results we’ve seen

We've used this system for our own Shopify app with great success. We're able to process hundreds of thousands of webhooks per day, and are able to control how quickly and when we process the webhooks. This allows us the freedom to perform database maintenance tasks easily, without worrying about how our application will be affected.

The Elastic Beanstalk workers are very efficient and can process SQS messages quickly. Since you can add workers as the queue size increases, your limiting factor will likely be your database. It's also very inexpensive—at scale, this system costs approximately $0.01 every 10,000 webhooks.

You might also like: How to Get More App Downloads in the Shopify App Store.

Preparing to handle increased webhook volume is a must for BFCM

In order for your app to be effective for the merchants who use it, preparing for the seasonal spike of BFCM is a must. If your application is struggling to handle Shopify webhooks, this system can help you handle the load without costing you a lot of money.

How have you begun preparing your app for BFCM? Let us know in the comments below!

Mike Potter