Job System

Overview

Hope and v5 have a job system that can be used for running long operations from Hope. The code is essentially duplicated on both servers for now so if any changes are made to the job system they will have to be made in both the Hop and V5 repositories.

The job queue is implemented with a node.js package called kue which is a priority job queue backed by redis.

The general flow for implementing a long running job using the job system is:

• On the server that runs the job (v5/hope) register the job function that returns a promise.
• The endpoint that kicks off the job should just run the job and return the job id.
• The client can then poll for the status of the job using the id or the job name.

Registering a job

Jobs should be registered when the server starts up by calling the registerJob function. When registering a job you pass in the name of the job and a function that returns a promise which gets resolved when the job is complete.

registerJob(name, processor)

Registers a job and the function that processes it.

Params

• name string - The name of the job to register.
• processor function - A function that processes the job. This function should take a job as an argument and return a promise that is resolved when the job is complete.

Example

var jobs = require('./../../lib/services/jobs');

jobs.registerJob('test-job', function (job) {
  return new Promise(function (resolve, reject) {
    console.log('Starting job ' + job.id);
    setTimeout(function () {
      console.log('Finished job ' + job.id);
      resolve();
    }, 10000);
  });
});

Using job parameters

If the job requires parameters to be passed in they will be available in the data field of the job object.

Example

jobs.registerJob('test-job', function (job) {
  return new Promise(function (resolve, reject) {
    console.log(job.data);
    // ...
  });
});

Reporting progress

A job can report progress that will be returned to the client in the response of the status checks by calling the progress method on the job object.

When reporting progress you pass in the number of steps that are completed and the total number of steps.

Example

jobs.registerJob('test-job', function (job) {
  return new Promise(function (resolve, reject) {
    // ...
    job.progress(1, 3);
    // ...
    job.progress(2, 3);
    // ...
    job.progress(3, 3);
    // ...
  });
});

You can also pass an optional JSON object that will be returned to the client in the progress_data field. This can be used for anything you would like such as displaying a custom progress message.

Example

jobs.registerJob('test-job', function (job) {
  return new Promise(function (resolve, reject) {
    // ...
    job.progress(1, 3, { message: 'Step 1 complete.' });
    // ...
  });
});

Returning results

To return the results of a job to the client you simply pass a JSON object when resolving the job’s promise. When a client gets the status of a completed job the response will include the results if there were any.

Example

jobs.registerJob('test-job', function (job) {
  return new Promise(function (resolve, reject) {
    // ...
    resolve({
      results: 'This JSON object can contain any results of the job.'
    });
  });
});

Running a job

When an endpoint wants to run a job it should call the runJob function and return the job id.

runJob(name, data)

Adds a job of the given type to the queue.

Params

• name string - The name of the job to run.
• data object - Optional object containing data that will be available to the job processing function.

Example

router.post('/long-job', function (req, res) {
  jobs.runJob('test-job')
  .then(function (jobId) {
    res.json({
      success: true,
      jobId: jobId
    });
  })
  .catch(function (err) {
    res.status(500).send({ error: err });
  });
});

Passing data to a job

You can pass JSON data to a job processing function when you run the job.

Example

jobs.runJob('test-job', { param1: 'test data', param2: 5 });

Checking job status

The job system has two endpoints to check the status of a job, status and lastjob. These endpoints exist on both hope and v5. The hope client hits the hope endpoint telling it which server the job is running on and hope will forward the request to v5 if needed.

Checking job status by id

The status endpoint takes a job id and returns the status of that specific job. Only the client that starts a job will know it’s id so this endpoint is used when a client starts a job and wants to wait for it to complete.

[Hope] /api/jobs/status/:server/:id

Method GET

URL Params

• :server - required - [alphanumeric] - Valid valiues are hope or v5
• :id - required - [integer] - The id of the job.

Success Response See the Status response section below.

Error Responses

• 400 - { error: “Invalid request. Unknown server.” } - Sent is :server is anything other than hope or v5.
• 500 - { error: err.message } - Sent if there was an error getting the job. The message will have more information about the specific error.

[V5] /status/:id

Method GET

URL Params

• :id - required - [integer] - The id of the job.

Success Response See the Status response section below.

Error Responses

• 50001 - Sent if there was an error getting the job. The standard v5 error object returned will have more information about the specific error.

Checking job status by name

The lastjob endpoint can be used to get the status of the last job that was run for a given job name. This can be useful if there is a job that should only be running once at a time across all clients. The clients that didn’t start the job wouldn’t have the job id to check the status so they can use this endpoint to check if there is a job currently running.

[Hope] /api/jobs/lastjob/:server/:name

Method GET

URL Params

• :server - required - [alphanumeric] - Valid valiues are hope or v5
• :name - required - [alphanumeric] - The name of the job.

Success Response See the Status response section below.

Error Responses

• 400 - { error: “Invalid request. Unknown server.” } - Sent is :server is anything other than hope or v5.
• 404 - { error: ‘No job found for ‘ + name } - Sent if there are no jobs for the given job name.
• 500 - { error: err.message } - Sent if there was an error getting the job. The message will have more information about the specific error.

[V5] /lastjob/:name

Method GET

URL Params

• :name - required - [alphanumeric] - The name of the job.

Success Response See the Status response section below.

Error Responses

• 40401 - Sent if there are no jobs for the given job name.
• 50001 - Sent if there was an error getting the job. The standard v5 error object returned will have more information about the specific error.

Note: V5 endpoints are locked down to admin instances.

Status response

The response of the job status calls will include all the information about the current state of the job.

Response fields

Notes

• Jobs are stored in redis so the queue will be shared across all machines connecting to the same redis instance.
• Jobs will stay in redis once complete. There is a small server application in node_modules/kue/bin/kue-dashboard that can be run if we need to manually delete jobs. We could at some point implement some sort of job manageent dashboard in Hope.
• If there is an attempt to start a job that doesn’t have a process function registered (possibly a typo in the job name passed to registerJob or runJob) the runJob call will still succeed and a job will be created but it will stay inactive forever.
• You should ensure job names are unique, there is no error if you register a job that already exists, it will just overwrite the previous one.