Build a webhook system using AWS and Ruby on Rails framework
Have you ever wondered how you could use Amazon Web Services to start your own webhook triggering system? Many developers would go with the classic workflow by directly using HTTP requests without any fail handling or retry policy. Integrating with AWS can be easier than it looks.
This article tends to give more information about the webhook notification system for those who have a requirement from their client or if you just want to learn more about the implementation in the context of the AWS environment.
What are webhooks and why do we use them?
Sometimes, you will be working on an API product or SaaS which has a lot of features. You will have a lot of clients who want to get notified when their data has been changed, in order to keep track of their data. There are many ways of notifying these clients but the most known one is Webhooks. Lets imagine these webhooks as phone numbers and your core system as a customer service which calls these phone numbers each time their data is changed. In other words, these webhooks are predefined URLs which are triggered by the core system each time the database is changed ( it can be for other occasions not just the database change) and which allow two systems to communicate. Webhooks can be used for e.g telling your accounting app that a client has completed a payment or notifying or if you want to trigger a Slack notification when a new order is processed etc.
Below are some common terms found in webhooks.
Is a webhook just a HTTP post request?
Well, creating a webhook takes more time and resources than just creating a HTTP Post request to a specific endpoint. At Softup Technologies, we always strive to provide best in-class engineering solutions to our clients.
The implementation of Webhooks
We have a leading product in the real estate industry. We have clients who use our system API for their integrations. As such, they requested a kind of notification system if something changes in our data warehouse in order to be in sync with the application.
Before you start, you need to define what kind of events are necessary to be triggered and delivered to the clients. Is it worth it to trigger webhooks each time an Update or Create occurs or that should be more specific based on entities? Well, for sure it should be more specific to allow only some kind of entities to trigger events.
First, we have created an endpoint for our clients to create/update/delete their own rest URLs. They can also define for which events they want to be subscribed.
Using AWS for the implementation of the Webhooks
Building a serverless infrastructure on AWS for webhooks means less infrastructure management. This means you do not have to build all of the infrastructure on the core app, and as such you have servers managed by AWS. With this Function-as-a-service infrastructure, developers do not have to know in depth the server architecture.
For the real world applications, when multiple users start using it, you will realize that you cannot use external http requests into your code ( at least it is not a good practice). Why is that? Some clients may have a slow response from the apps and this will result with other complications to your core system. As such, you need to use a queueing system. If you are using Ruby on Rails the most known one is called Sidekiq (this depends on the architectural decisions you make). We have used Amazon SQS which stands for Simple Queue Service. It's a message queue service which has been available from 2006. The reason for this is that it's fast, scalable and reliable. Depending on the backend application you can fire events to SQS each time something happens.
There is also another reason to use SQS or other similar services behind the scenes. Let's consider your clients' endpoints fail from time to time. You will end up with failed events and a headache when the client wants to know what happened with the notification. In this case, you should get familiar with SQS retries. We used dead-letter queue (DLQ) configuration which is a good practice to be used to prevent any message loss.
Here comes the Lambda…
At this point, you might think what will the SQS trigger? There should be a logic that will decide from which environment the event is coming. As such, Lambda functions will be invoked. AWS Lambda can be used for a lot of use-cases. In our case it will be code that will run without having to take care of server management. In our case we have used NodeJs with the serverless framework which we will be discussing later.
Since AWS Lambda has revolutionized the serverless application, developers are getting more and more into the FaaS (function-as-a-service). With the traditional approach we had to initialize a new instance server for a single service. Lambda removed the need of provisioning and running the servers.
Lambda plays a critical role in Webhook implementations. The functionality of the Lambda is to take care of informing the clients for the events that occurred. In Lambda you can also add multi environment logic (development, staging or production). AWS Lambda has an inline code editor. You can also use Serverles Framework for the deployment.
When it comes to decide whether you want to go with the traditional approach or not, let's discuss some pros and cons of using serverless applications.
The first advantage is that you only need to pay for the server usage or computing costs. AWS Lambda are billed in the CPU time usage (milliseconds).
Another advantage is the application resiliency when the load is high. As such, you can reduce risk by not having to rely on a single machine to perform multiple tasks.
However, there might be a single drawback when considering to use the AWS Lambda. This can be the control over the environment. What does this mean? Since the AWS Lambda runs on the AMIs (amazon machine instances) there are no ways to install custom packages or softwares over it. To decide, AWS Lambda functions give you more flexibility to decouple your code from you main app architecture and create more lower-cost highly performing functions. If you think that this should be in your application, it's you that takes the final “move” - by also considering that you read the pros and cons to make an effective decision.
Yes, the security. Most of the API nowadays enforce users to include some kind of token on the request. For the webhooks there is a more challenging way to include this security. The strategy that we have taken for this case is to use the HMAC approach by providing the checksum at the payload. In this way, the clients can verify the data is indeed coming from our application. However, there can be other approaches when it comes to the security of the webhooks. Some of them might be: IP whitelist, HTTPS only, encrypt the entire data etc.
It's upon you to decide what kind of approach you want to take when it comes to the security and also the nature of the application you have.
Let's define the steps we have taken for the Webhooks implementation.
Setup the main application
On our main application, we have set up the callback functions on the entities that we want to have the Webhooks. For example: you can have Webhooks when a job post is created on your application. It can also be when the job post is updated or deleted. So, this is up to you to decide when you should trigger the Webhooks.
Create some callbacks to trigger the SQS
On our backend we use Ruby on Rails for development. Rails is very compliant when creating a webhook. It has some very handy callbacks that can help on triggering the background jobs. We have chosen AWS SQS as our queue service since it is very scalable.
Create the Lambda Function
To create a Lambda function on Amazon, you need to go to the console and search for the Lambda. You can choose “Create Function” and start your function from scratch. There is also a really good way to build a Lambda function by using the serverless framework. If you choose to have the serverless framework, you need to have Node.js and NPM installed on your computer. You can check more information here.
Create cloudwatch logs (optional)
It is highly recommended to keep an eye on the Webhook logs. CloudWatch is one of the best ways to achieve that. Lambda sends many metrics to CloudWatch, so in this way you can spot issues related with the Lambda function. Cloudwatch provides Invocation metrics, which measures how many times your Lambda is triggered. There is also an Error metric (for sure this is very important in Webhooks) which gives information each time your Lambda function has failed. There are also other metrics you can use in the CloudWatch.
A well built Webhook will make your product more professional and it will make your clients’ life easier. Depending on the team experience, the Webhook can be very easily implemented and will add a great value to your product. When it comes to choosing you and other similar products such as yours, Webhooks will make the difference. At the end, you're the one that chooses if it is worth building the Webhooks for your application or not.