jobs Package¶
jobs
Package¶
Support for running a tool in Galaxy via an internal job management system
-
class
galaxy.jobs.
ComputeEnvironment
[source]¶ Bases:
object
Definition of the job as it will be run on the (potentially) remote compute server.
-
class
galaxy.jobs.
JobConfiguration
(app)[source]¶ Bases:
object
A parser and interface to advanced job management features.
These features are configured in the job configuration, by default,
job_conf.xml
-
DEFAULT_NWORKERS
= 4¶
-
convert_legacy_destinations
(job_runners)[source]¶ Converts legacy (from a URL) destinations to contain the appropriate runner params defined in the URL.
Parameters: job_runners (list of job runner plugins) – All loaded job runner plugins.
-
default_job_tool_configuration
¶ The default JobToolConfiguration, used if a tool does not have an explicit defintion in the configuration. It consists of a reference to the default handler and default destination.
Returns: JobToolConfiguration – a representation of a <tool> element that uses the default handler and destination
-
get_destination
(id_or_tag)[source]¶ Given a destination ID or tag, return the JobDestination matching the provided ID or tag
Parameters: id_or_tag (str) – A destination ID or tag. Returns: JobDestination – A valid destination Destinations are deepcopied as they are expected to be passed in to job runners, which will modify them for persisting params set at runtime.
-
get_destinations
(id_or_tag)[source]¶ Given a destination ID or tag, return all JobDestinations matching the provided ID or tag
Parameters: id_or_tag (str) – A destination ID or tag. Returns: list or tuple of JobDestinations Destinations are not deepcopied, so they should not be passed to anything which might modify them.
-
get_handler
(id_or_tag)[source]¶ Given a handler ID or tag, return the provided ID or an ID matching the provided tag
Parameters: id_or_tag (str) – A handler ID or tag. Returns: str – A valid job handler ID.
-
get_job_runner_plugins
(handler_id)[source]¶ Load all configured job runner plugins
Returns: list of job runner plugins
-
get_job_tool_configurations
(ids)[source]¶ Get all configured JobToolConfigurations for a tool ID, or, if given a list of IDs, the JobToolConfigurations for the first id in
ids
matching a tool definition.Note
You should not mix tool shed tool IDs, versionless tool shed IDs, and tool config tool IDs that refer to the same tool.
Parameters: ids (list or str.) – Tool ID or IDs to fetch the JobToolConfiguration of. Returns: list – JobToolConfiguration Bunches representing <tool> elements matching the specified ID(s). Example tool ID strings include:
- Full tool shed id:
toolshed.example.org/repos/nate/filter_tool_repo/filter_tool/1.0.0
- Tool shed id less version:
toolshed.example.org/repos/nate/filter_tool_repo/filter_tool
- Tool config tool id:
filter_tool
- Full tool shed id:
-
get_tool_resource_parameters
(tool_id)[source]¶ Given a tool id, return XML elements describing parameters to insert into job resources.
Tool id: A tool ID (a string) Returns: List of parameter elements.
-
is_handler
(server_name)[source]¶ Given a server name, indicate whether the server is a job handler
Parameters: server_name (str) – The name to check Returns: bool
-
-
class
galaxy.jobs.
JobDestination
(**kwds)[source]¶ Bases:
galaxy.util.bunch.Bunch
Provides details about where a job runs
-
class
galaxy.jobs.
JobToolConfiguration
(**kwds)[source]¶ Bases:
galaxy.util.bunch.Bunch
Provides details on what handler and destination a tool should use
A JobToolConfiguration will have the required attribute ‘id’ and optional attributes ‘handler’, ‘destination’, and ‘params’
-
class
galaxy.jobs.
JobWrapper
(job, queue, use_persisted_destination=False)[source]¶ Bases:
object
Wraps a ‘model.Job’ with convenience methods for running processes and state management.
-
commands_in_new_shell
¶
-
fail
(message, exception=False, stdout='', stderr='', exit_code=None)[source]¶ Indicate job failure by setting state and message on all output datasets.
-
finish
(stdout, stderr, tool_exit_code=None, remote_working_directory=None)[source]¶ Called to indicate that the associated command has been run. Updates the output datasets based on stderr and stdout from the command, and the contents of the output files.
-
galaxy_lib_dir
¶
-
galaxy_system_pwent
¶
-
get_job_runner
()¶
-
get_output_destination
(output_path)[source]¶ Destination for outputs marked as from_work_dir. This is the normal case, just copy these files directly to the ulimate destination.
-
job_destination
¶ Return the JobDestination that this job will use to run. This will either be a configured destination, a randomly selected destination if the configured destination was a tag, or a dynamically generated destination from the dynamic runner.
Calling this method for the first time causes the dynamic runner to do its calculation, if any.
Returns: JobDestination
-
prepare
(compute_environment=None)[source]¶ Prepare the job to run by creating the working directory and the config files.
-
requires_setting_metadata
¶
-
set_job_destination
(job_destination, external_id=None)[source]¶ Persist job destination params in the database for recovery.
self.job_destination is not used because a runner may choose to rewrite parts of the destination (e.g. the params).
-
setup_external_metadata
(exec_dir=None, tmp_dir=None, dataset_files_path=None, config_root=None, config_file=None, datatypes_config=None, set_extension=True, **kwds)[source]¶
-
user
¶
-
user_system_pwent
¶
-
-
class
galaxy.jobs.
NoopQueue
[source]¶ Bases:
object
Implements the JobQueue / JobStopQueue interface but does nothing
-
class
galaxy.jobs.
ParallelismInfo
(tag)[source]¶ Bases:
object
Stores the information (if any) for running multiple instances of the tool in parallel on the same set of inputs.
Bases:
galaxy.jobs.SimpleComputeEnvironment
Default ComputeEnviornment for job and task wrapper to pass to ToolEvaluator - valid when Galaxy and compute share all the relevant file systems.
-
class
galaxy.jobs.
TaskWrapper
(task, queue)[source]¶ Bases:
galaxy.jobs.JobWrapper
Extension of JobWrapper intended for running tasks. Should be refactored into a generalized executable unit wrapper parent, then jobs and tasks.
-
finish
(stdout, stderr, tool_exit_code=None)[source]¶ Called to indicate that the associated command has been run. Updates the output datasets based on stderr and stdout from the command, and the contents of the output files.
-
get_output_destination
(output_path)[source]¶ Destination for outputs marked as from_work_dir. These must be copied with the same basenme as the path for the ultimate output destination. This is required in the task case so they can be merged.
-
handler
Module¶
Galaxy job handler, prepares, runs, tracks, and finishes Galaxy jobs
-
class
galaxy.jobs.handler.
JobHandler
(app)[source]¶ Bases:
object
Handle the preparation, running, tracking, and finishing of jobs
-
class
galaxy.jobs.handler.
JobHandlerQueue
(app, dispatcher)[source]¶ Bases:
object
Job Handler’s Internal Queue, this is what actually implements waiting for jobs to be runnable and dispatching to a JobRunner.
-
STOP_SIGNAL
= <object object>¶
-
manager
Module¶
Top-level Galaxy job manager, moves jobs to handler(s)
mapper
Module¶
-
exception
galaxy.jobs.mapper.
JobMappingException
(failure_message)[source]¶ Bases:
exceptions.Exception
-
exception
galaxy.jobs.mapper.
JobNotReadyException
(job_state=None, message=None)[source]¶ Bases:
exceptions.Exception
transfer_manager
Module¶
Manage transfers from arbitrary URLs to temporary files. Socket interface for IPC with multiple process configurations.
-
class
galaxy.jobs.transfer_manager.
Sleeper
[source]¶ Bases:
object
Provides a ‘sleep’ method that sleeps for a number of seconds unless the notify method is called (from a different thread).
-
class
galaxy.jobs.transfer_manager.
TransferManager
(app)[source]¶ Bases:
object
Manage simple data transfers from URLs to temporary locations.