Architecture behind tracksperanto-web
So it's been quite a while since tracksperanto-web went production, with more than 300 files converted so far with great results. I hope some sleepless nights for matchmovers worldwide have already been preserved and nothing could be better.
I would like to bring a little breakdown on how Tracksperanto on the web is put together (interesting case for background processing). The problem with this kind of applications (converters to be specific) is that the operation to convert something might take quite some time. To mitigate that, I decided to write the web UI for Tracksperanto in such a way that it would not be blocking for processing, so extensive use of multitasking is made.
Basically, the workers on the web app are fork-and-forget.
Very simple - the worker gets spun off directly from the main application, it writes it's status into Redis and at the end of the job to the database. Tracksperanto jobs use alot of processor time and alot of memory, and with Ruby nobody can guarantee that jobs won't leak memory.
fork() is tops here since after the job has been completed the forked worker will just die off, releasing any memory that has been consumed in the process.
This scheme has the following benefits:
- Status reporting does not load the database (not needed and the information is hardly crucial).
- Zero memory leakage
- No Ruby daemon processes
- Start/stop control is tied into the webserver.
Note: lots of stuff removed from this post since it's no longer relevant.comments powered by Disqus