Jun 032013
 

Recently a colleague of mine should run multiple parallel jobs on a server (around 20 in our specific case), every job does “an elaboration” and this can take a time longer or shorter in respect of the others, once a job finishes its task it must read from a text file the first line that indicate the ID of a new job, it removes that ID from the top of the text file and starts to elaborate it.

Problem: On a long run (a run of 50K+ jobs) it happens “frequently” that 2 jobs finish at the same time and so both take the same ID causing problems to the whole process, we tried to use a simple lock file “touched” just after the job was opening the file with the list of ID, but it seems that this operation it’s too slow and we had some case of failed concurrency again.

The solution was to use the bash function flock

Continue reading »

flattr this!

Oct 172012
 

In a former article I’ve talked about the commands cron and crontab  that are the standard way to schedule recurring things on a Gnu/Linux system.

But sometimes you need to do one thing at a specific date and time for just one time, and for these tasks the best option is to use at, another way to use at is to run a command later when the computer won’t be busy.

Or another reason could be that you want to run a command that requires a lot of time to end and you have to disconnect from that server, at could be used in this situation, but for these task i suggest to take a look at the articles on how to run commands on background and the utility screen

Continue reading »

flattr this!