queue_manager.py¶
Developer Documentation Only Below This Point:
Abstract queue management, the parent of each individual queue manager, LOCAL, PBS, LSF.
-
queue_manager.
HOSTMAPPING
(hostkey, which)¶ Return a queue manager mode based on hostkey
which is Cluster or Local
-
queue_manager.
Q_LONG
= 'long'¶
-
queue_manager.
Q_MEDIUM
= 'medium'¶
-
queue_manager.
Q_PARALLEL
= 'parallel'¶
-
queue_manager.
Q_SHORT
= 'short'¶
-
queue_manager.
Q_VERYLONG
= 'verylong'¶
-
queue_manager.
check_for_cycle
(paramdict, key, chain=None)¶ See if a key is referred to by any of its descendant references. Bails if it finds a single cycle.
-
queue_manager.
determine_id
(jobid)¶ pull a jobid from a string that contains one
-
queue_manager.
determine_job
(jobid)¶ pull a job shell name from a string that contains one
-
queue_manager.
populate_command
(command, obsoleteDict=None, _stderr=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>, _validateit=False, **kwargs)¶ Process command string.
First, check kwargs for reference cycles, and then replace defined parameters with the key, value pairs in kwargs.
raises an protocol error in the event of a reference cycle, a template error if the command string contains incorrect syntax.
If there are missing or obsolete values, raise a template error unless _validateit is True, in whic case return the sorted lists.
Otherwise, return the populated command, or empty lists of missing and obsolete values if _validateit is True.
-
class
queue_manager.
queue_manager
(interface=None, logfunc=None, localconfig=None, testoverride=False)¶ This class manages the queue so it can monitor the jobs’ statuses.
-
check_status
(jobid, batchroot, reset=False)¶ Check whether a job is still running. Return ERROR, RUNNING, or DONE
TODO: convert cluster queue managers to refer to self.processes instead of self.statuses. Would simplify some logic and potentially move that logic up here.
-
choose_bolus
(jobestimate, processors, memory, pending, delta=0, qdesired=None, test_tuple=None)¶ Attempt to be a good citizen. Choose a batchsize that won’t overwhelm other users.
This is kind of twisted. Because PBS cluster sizes are governed by #processors, we are kind of tied to that here, until someone intelligent looks at it and improves the interface.
Also, by the way, don’t be too altruistic. The batch gotta make progress.
delta constrains the local queue manager strictly, and clustered ones loosely.
-
choose_processors
(processors=1, gb_memory_needed=None)¶ Processor selection is interface-dependent
-
choose_queue
(jobestimate, processors, gb_memory_needed=4, desiredqueue=None)¶ Attempt to be a good citizen of the hardware: Chooses an appropriate queue with the given information.
-
error_pid
()¶ Return a unique value that won’t be confused with a pid. I considered negative, but that might get messy.
-
format_command
(command, todir, label, region, rundir=None, iteration_label='', **kwargs)¶ Return a string with the format of subclass’s qsub_command() with the keyword arguments imbedded
-
ignoreable_warning
(message)¶ Can this be ignored?
-
job_statistics_header
()¶ Return a list of labels.
-
minimum_memory
(gb_memory_needed=None)¶ The HMS cluster needs special attention
-
processor_usage
()¶ Estimate current use of cpu resources by (user, everybody)
-
queue_status
()¶ Report times for jobs in queue
-
select_interface_specific_parameters
(kwargs, selectors)¶ Contract: Any term set by the subclass will be removed from selectors, so that parent class can set it to empty when this returns.
-
set_interface_specific_parameters
(kwargs)¶ Allow special considerations of individual interfaces to be set.
A semi-hack for nautilus’s queue-naming policy
-
sleep_interval
()¶ How long to sleep between checks
-
submit
(qsubstream, queue=None)¶ Submit a job to the queueing system.
Return its job id and a warning, or None if no error.
-