Running One-Offs in NestJS

[...] Recently, I had to do this at work to update some production database columns based on external IDs produced by an external service. Two of the bigger advantages in operating over data using one-offs (in my opinion) are that 1) you don't need to access production databases manually, and 2) you don't need to mix schema migrations with data migrations. [...]

Sometimes in business, we have the need to run specific operations in our backends or databases, once. Some people call these operations one-offs because they should run and complete only once. There is great power in running one-offs, as it allows to perform some actions using production data (e.g. send emails to specific groups of users) or update the production data based on external services. Recently, I had to do this at work to update some production database columns based on external IDs produced by an external service. Two of the bigger advantages in operating over data using one-offs (in my opinion) are that 1) you don't need to access production databases manually, and 2) you don't need to mix schema migrations with data migrations.

Currently, I'm working with NestJS, and I couldn't find any documentation on how to run one-offs with it. We decided to create tooling to run our one-offs, and this post will describe the steps needed to replicate this idea. For the sake of the post, let's imagine a scenario where we want to send emails to our admin users, using some kind of email service.

Create a trigger for the one-off

One of the things to consider here is that we need to run the one-off in production conditions most likely while our code is running. This means that we need to trigger the one-off function somehow in run-time. We considered 2 scenarios: a specific endpoint that we could call via Curl or a new entry point for NestJS. Creating a new endpoint would expose it to the public and some security measures would be needed to prevent unauthorized access, so we decided to go with the latter. When you start developing a NestJS application usually an entry point file is created for you, called main.ts. This file loads all the necessary models, controllers, and service dependencies for your server to run. We don't need all of this to run the one-off, so we will create a new entry point, with only the necessary dependencies. Let's call it one-off.ts.

Minimal dependency injection

The example above shows an OneOffModule that is yet to be defined. This NestJS module will be the one that will contain all the dependencies needed to run the one-off. In our case, we will need access to the database and to a service that will send the emails. For reference, your main.ts file loads a different module, usually called AppModule, that loads the entirety of your application's dependencies. The rationale behind splitting between two modules is that we don't want our one-offs to have an increased load time usually caused by the loading of all the dependencies that real production applications have.

As you can see we only initialized a connection to the database (I did it with TypeORM but you can use your own library), and only provided the only one-off that we want to run in this situation, the SendEmailToAdmins service.

Run on command

One of the requirements for this tool was that we could run it whenever we wanted and run specific one-offs. The first thing that we did to allow that was creating the one-off.ts entry point. Now - after compiling the application - we can simply run node node dist/one-off.js to load and run the entry point, but the way the code is written so far does not allow us to run specific one-offs. To do that, we need to tell the program which service to run. For that we are combining the use of Node's process.argv array to get the arguments passed in the terminal with the power of named exports to find, load, and run one specific service.

As you can see, the value passed to the terminal is then used against the Executables object which holds all the services that are exported by the one-off module. This gives more security on what you're allowed to run. Also, because we wanted to make it easier for future developers to create more one-offs, we defined an interface for the OneOff classes, where run is the only public method.

If you're using something like yarn, you just have to add this to your package.json scripts: "one-off": "node dist/src/one-off.js", and you'll be able to call it with yarn run one-off SendEmailToAdmins. With a PaaS like Heroku, you can run these commands from your local machines using Heroku CLI with the run command. So far, we've been successful and we no longer have to run data migrations manually, or use the schema migrations tool, in production.

Future work

So far the tool has been super useful, although I've been thinking about one small improvement. Right now, nothing forbids me from running the same one-off multiple times by using Heroku's run with the same arguments over and over again. One way to prevent this would be to add a new table to the database just like the schema migrations tool does, and keep track of the one-offs that have been run. This would allow us to run the one-offs only once, and also keep some kind of history of the one-offs that have been run. This would be useful for auditing purposes.

Conclusion

In this post, we've seen how to create a tool to run one-offs in NestJS. We've seen how to create a new entry point, how to load the dependencies needed to run the one-off, and how to run specific one-offs using the terminal. We've also seen how to use Heroku's CLI to run the one-offs in production. The obvious advantages of this approach are that we don't need to access production databases manually, and we don't need to mix schema migrations with data migrations. Developer experience is being valued more and more in the industry, and I think that this is a good example of how to improve it, not only in terms of having tools to do the heavy lifting but also in terms of keeping a record of development efforts and auditing.

JHFGloria Blog ♥️ 2023