I used the Async Queue several times. It was always easy peasy. Recently when I am using for one use case faced a problem. I searched to find the solution didn’t find anything useful so finally figured out the problem through a dirty hack and solved it. I thought let me share so that I might help someone who is struggling like this or I will get suggestions from the world how we can tackle this in a better way.
Before we discuss the problem and hack let me explain a bit about the Async Queue. Many times we need to run background tasks due to our application customers actions/requests, for reporting purpose and etc. Sometimes we get a lot of background tasks and sometimes we won’t have any. If we try to process everything as we are getting because of that we might put a lot of load on our systems like DB, other microservices and etc. So it’s better to process a certain number of tasks at a time. In these cases, the Async Queue comes as a handy tool.
If you want to use Async queue in your application then install this package using below command
npm install async --save
A small example for Async Queue
Basically, the Async Queue is kind of a resources pool. For example, You have 3 self-driving cabs and you will provide the cab as per customer requirement. If you get more than 3 customers at a time they need to wait, if not your cabs will be idle.
Here the most important thing is a callback function. It’s like once your customer is done with it they need to give it back or else you will feel they are using you will never know the cab is free or not.
If you miss calling callback function, then things get ugly. So we need to make sure we call callback function in any case. Here I got the problem basically my task is doing several things based on the requirements and somewhere I missed calling callback function so things got bad. I don’t where I missed.
Example of the problematic code.
const async = require('async');
//Code for processing the task
var processQueue = function (message, callback) {
setTimeout(function() {
console.log(`Task ${message} completed`);
if(message % 3 !== 0) {
callback(); } }, 500);
}
//Queue initialization. This queue process 3 tasks at a time var queue = async.queue(processQueue, 3);
//After all tasks completion queue process this function
queue.drain = function() {
console.log('Yuppie all tasks completed');
}
//To add tasks to queue we are using this function.
var processTasks = function () {
for (let index = 1; index <= 10; index++) {
queue.push(index);
}
}
processTasks();
You will get output like this
Task 1 completed Task 2 completed Task 3 completed Task 4 completed Task 5 completed Task 6 completed Task 7 completed Task 8 completed Task 9 completed
Above code will never able to run the 10th task and “All tasks completed” message also won’t come because queue didn’t drain.
We will never able to figure out the problem until we know which code didn’t call the callback function.
If you mess with callbacks, you mess with the whole application (Trust me you don’t want to).
Below hack helped me to figure out the problem
//To Figure out which tasks exactly troubling us
var lastTimeInQueue = new Set();
setInterval(function() {
let currentQueue = new Set();
queue.workersList().forEach(workerTask => {
currentQueue.add(workerTask.data);
if(lastTimeInQueue.has(workerTask.data)) {
console.error("Something wrong with this task: " + workerTask.data);
}
});
lastTimeInQueue = currentQueue;
}, 1000);
Basically what this code is doing is we know usually each task take around 500ms on average. So Every minute we are checking currently that tasks is running. We will compare in the last minute who are those and now who if we compare both we will get to know the culprits.
I get output like this
Task 1 completed Task 2 completed Task 3 completed Task 4 completed Task 5 completed Task 6 completed Task 7 completed Something wrong with this task: 3 Task 8 completed Task 9 completed Something wrong with this task: 3 Something wrong with this task: 6 Something wrong with this task: 3 Something wrong with this task: 6 Something wrong with this task: 9
Now we got the culprits. We just need to read 3,6,9 tasks code flow so that we can call the callback in a proper place.
You will find this code here.
Peace. Happy Coding.