queue_manager.py

Developer Documentation Only Below This Point:

Abstract queue management, the parent of each individual queue manager, LOCAL, PBS, LSF.

queue_manager.HOSTMAPPING(hostkey, which)

Return a queue manager mode based on hostkey

which is Cluster or Local

queue_manager.Q_LONG = 'long'

queue_manager.Q_MEDIUM = 'medium'

queue_manager.Q_PARALLEL = 'parallel'

queue_manager.Q_SHORT = 'short'

queue_manager.Q_VERYLONG = 'verylong'

queue_manager.check_for_cycle(paramdict, key, chain=None)

See if a key is referred to by any of its descendant references. Bails if it finds a single cycle.

queue_manager.determine_id(jobid)

pull a jobid from a string that contains one

queue_manager.determine_job(jobid)

pull a job shell name from a string that contains one

queue_manager.populate_command(command, obsoleteDict=None, _stderr=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>, _validateit=False, **kwargs)

Process command string.

First, check kwargs for reference cycles, and then replace defined parameters with the key, value pairs in kwargs.

raises an protocol error in the event of a reference cycle, a template error if the command string contains incorrect syntax.

If there are missing or obsolete values, raise a template error unless _validateit is True, in whic case return the sorted lists.

Otherwise, return the populated command, or empty lists of missing and obsolete values if _validateit is True.

class queue_manager.queue_manager(interface=None, logfunc=None, localconfig=None, testoverride=False)

This class manages the queue so it can monitor the jobs’ statuses.

check_status(jobid, batchroot, reset=False)

Check whether a job is still running. Return ERROR, RUNNING, or DONE

TODO: convert cluster queue managers to refer to self.processes instead of self.statuses. Would simplify some logic and potentially move that logic up here.

choose_bolus(jobestimate, processors, memory, pending, delta=0, qdesired=None, test_tuple=None)

Attempt to be a good citizen. Choose a batchsize that won’t overwhelm other users.

This is kind of twisted. Because PBS cluster sizes are governed by #processors, we are kind of tied to that here, until someone intelligent looks at it and improves the interface.

Also, by the way, don’t be too altruistic. The batch gotta make progress.

delta constrains the local queue manager strictly, and clustered ones loosely.

choose_processors(processors=1, gb_memory_needed=None)

Processor selection is interface-dependent

choose_queue(jobestimate, processors, gb_memory_needed=4, desiredqueue=None)

Attempt to be a good citizen of the hardware: Chooses an appropriate queue with the given information.

error_pid()

Return a unique value that won’t be confused with a pid. I considered negative, but that might get messy.

format_command(command, todir, label, region, rundir=None, iteration_label='', **kwargs)

Return a string with the format of subclass’s qsub_command() with the keyword arguments imbedded

ignoreable_warning(message)

Can this be ignored?

job_statistics_header()

Return a list of labels.

minimum_memory(gb_memory_needed=None)

The HMS cluster needs special attention

processor_usage()

Estimate current use of cpu resources by (user, everybody)

queue_status()

Report times for jobs in queue

select_interface_specific_parameters(kwargs, selectors)

Contract: Any term set by the subclass will be removed from selectors, so that parent class can set it to empty when this returns.

set_interface_specific_parameters(kwargs)

Allow special considerations of individual interfaces to be set.

A semi-hack for nautilus’s queue-naming policy

sleep_interval()

How long to sleep between checks

submit(qsubstream, queue=None)

Submit a job to the queueing system.

Return its job id and a warning, or None if no error.