Why you should avoid long running recursion in Node.

I don’t like recursion. I know its a controversial opinion, but I don’t like it. I’ve had too many issues with recursive functions, plus my brain never really got the concept when I first started programming. I avoid using recursion whenever I can, only using in the most obvious of cases (like the classic factorial example).

Not long ago, I was working on a project for work when I noticed that there were tons of errors in the logs, as the lambda that was running the code kept on running out of memory. The code was in production, and as a temporary fix the RAM for the lambda was cranked from 1GB to 3GB, which would also aid in finding where the bug was coming from. This script was written in NodeJS 14, made to run on a lambda, and acted as a download script. The data being downloaded was gotten from an API that could only return chunks of data, but we needed the whole dataset to run our algorithms on. Our solution was to get the data as a JSON array, then save it to AWS S3, using it as a sort of database for the JSON files. I noticed that to download 100MB of data, the RAM use was well over 1.5GB. While you’re almost never going to get a 1:1 data size to memory use ratio, it should not be as extreme as that.

High Memory Use

The shown example is quite extreme, as most of the time the data we’re downloading doesn’t go above 20MB, but there are edge cases were we could be downloading as much as 200MB. If the latter is the case, there’s no way going to run as intended.

I did some searching, and I found this StackOverflow post. It seems that Node’s garbage collector doesn’t clean up until after recursion is complete, and the recursion in this script did not end until after the main purpose of the script had finished. Here is the original recursive function code:


const allMessages = [];

const objectId = "someObjectId";

const callAPI = async (cursor = null) => {
    const headers = {'X-Api-Key': 'someApiKey'};
    const url = `https://api.url.here/${objectId}/${
        cursor ? `?cursor=${cursor}` : ''
    }`;
    const resp = await fetch(url, { headers });
    const { _next, comments } = await resp.json();
    allMessages.push(...comments);

    if (_next) {
        await callAPI(_next);
    }
};

await callAPI();

The basic idea is that this API returned us a cursor to to paginate the JSON data we were retrieving and storing for later in S3. If the cursor returned null from the API, we knew this was the last page of data and we could break recursion. The solution to this issue was really simple.

const allMessages = [];
const objectId = "someObjectId";

const callAPI = async (cursor = null) => {
    const headers = {'X-Api-Key': 'someApiKey'};
    const url = `https://api.url.here/${objectId}/${
        cursor ? `?cursor=${cursor}` : ''
    }`;
    const resp = await fetch(url, { headers });
    const { _next, comments } = await resp.json();
    allMessages.push(...comments);

    return _next;
};

var cursor = await callAPI();

while (cursor) {
    cursor = await callAPI(cursor);
}

This achieves the exact same functionality while fixing the garbage collector problem of before. Rather than recursively executing, the function is called once before starting a while loop, which conditionally runs provided that cursor is not null, appending the data like before into allMessages.

This is not the main reason I avoided recursive functions, but it has definitely been added to the list. I (as well as the man who wrote this code) are definitely more wary about using recursive functions on lots of data or long running processes, as you should be as well.

Read More

Python For Programmers Part 3

This is a series on Python and how to correctly use Python when coming from a background in another computer language. Because of this, this will not be a slow intro into programming and it will be assumed you have a preferred text editor and are smart enough to get Python running. You can download installers and packages from their official website found here. Basic knowledge of how to use Git and how to operate a computer is also preferred.

Read More

How to Install Node on Linux the Easy Way.

I recently started working on a few major NodeJS project, and found that installing the latest LTS release could be rather cumbersome on Linux. The application we’re developing is being hosted on Heroku, and will not be using a Docker container, and for a few reasons we would rather develop locally rather than in a container. Here is how I installed NodeJS and NPM on my Linux installations, and I found it quite easy. While this tutorial will be using Ubuntu’s apt and Arch Linux’s pacman, the process should be similar for most distributions. See here for more information about installing NodeJS and NPM via your package manager.

Read More

GistBin - The GitHub Gist command line client.

So I have always like the idea of being able to share simple snippets of code, but my first real foray into this was using PasteBin. The reason I was using PasteBin was because that is what the Minecraft mod, ComputerCraft, allowed you to download code off of the internet easily. I wrote a few programs that are still on my PasteBin. Since I found out that GitHub had a competitor in the form of GitHub’s Gists, I switched to that. I like the GitHub platform and its easy to switch between the gists and Git, and even clone the gists with git clone. I recently found HasteBin, an open source alternative to PasteBin that has a command-line tool. The command-line tool, called haste-client is extremely easy to use, only requiring you to pipe in the information you want to upload to HasteBin. You can even host your own HasteBin Server and point the haste-client to it. While I liked this, I also liked the ability to have other information about the code that I was sharing, along with being able to look at revisions like git, but haste-client is so easy to use, so I decided to create my own client for Gists.

Read More

Python For Programmers Part 2; Controls

This is a series on Python and how to correctly use Python when coming from a background in another computer language. Because of this, this will not be a slow intro into programming and it will be assumed you have a preferred text editor and are smart enough to get Python running. You can download installers and packages from their official website found here. Basic knowledge of how to use Git and how to operate a computer is also preferred.

Read More

Python For Programmers Part 1

This is a series on Python and how to correctly use Python when coming from a background in another computer language. Because of this, this will not be a slow intro into programming and it will be assumed you have a preferred text editor and are smart enough to get Python running. You can download installers and packages from their official website found here. Basic knowledge of how to use Git and how to operate a computer is also preferred.

Read More

Why Discord Bot Development is Flawed.

Before I complain about my experience with Discord bots, let me preamble with this: I enjoy developing the bots. I enjoy making bots that entertain people and that everyone uses for fun and memes. I like my Discord bots, I don’t regret developing them, and I will continue development of them. I do not think Discord’s current system for bot development should be replaced, it is too prevalent and there is too many bots currently using it.

Read More

3D Printing Is A Fun yet Rewarding Cruel Mistress.

Ever since I was 8 or 9, the idea of 3D printing seemed amazing. Objects appearing right before your very eyes, with nothing more than a computer file and a roll of plastic. Now the 3D printer that I saw didn’t really exists, as it was a piece of concept art from a book about technology I had bought at a Scholastic book fair, but when I saw it I thought it was science fiction. An idea. Nothing more than a picture in a book and an idea. I had no idea that no only was it going on at that time, but I wouldn’t have though that in the not-so-distant future I would be one of the many early adopters of the technology.

Read More

Make PowerPC Great Again.

Back when Apple didn’t try and screw over anyone with a broken iMac, back before Intel was the standard for Apple hardware, and back before Apple lost its only source of new innovation, that being Steve Jobs, there was the PowerMac, a line of computers running RISC architecture known as PowerPC, co-developed in a partnership between IBM, Apple, and Motorola. As the Power architecture is now fairly uncommon, as x86 and ARM dominate the processor market, it is now mainly used in server applications. The only “modern” OS you can run on them now is Linux, and I use that term lightly as not a lot of applications support the architecture.

Read More

How to Stream Any Game, on Any Console, to Anywhere.

Ever wanted to bring your library of home console games (PC, PS4, or Xbox One) on the go? Ever wanted to play a round or two of Call Of Duty while sitting in the school library studying for a class you never payed attention to? Well, with some basic knowledge of router settings, a half decent laptop or smartphone, and an internet connection that’s not dial-up, you can!

Read More