mirror of
https://github.com/imjasonh/gcloud-help
synced 2026-07-11 23:49:35 +00:00
129 lines
4.9 KiB
Text
129 lines
4.9 KiB
Text
NAME
|
|
gcloud dataproc workflow-templates add-job spark-sql - add a SparkSql job
|
|
to the workflow template
|
|
|
|
SYNOPSIS
|
|
gcloud dataproc workflow-templates add-job spark-sql --step-id=STEP_ID
|
|
(--execute=QUERY, -e QUERY | --file=FILE, -f FILE)
|
|
(--workflow-template=WORKFLOW_TEMPLATE : --region=REGION)
|
|
[--driver-log-levels=[PACKAGE=LEVEL,...]] [--jars=[JAR,...]]
|
|
[--labels=[KEY=VALUE,...]] [--params=[PARAM=VALUE,...]]
|
|
[--properties=[PROPERTY=VALUE,...]] [--properties-file=PROPERTIES_FILE]
|
|
[--start-after=STEP_ID,[STEP_ID,...]] [GCLOUD_WIDE_FLAG ...]
|
|
|
|
DESCRIPTION
|
|
Add a SparkSql job to the workflow template.
|
|
|
|
EXAMPLES
|
|
To add a SparkSql job executing query 'QUERY' to a the workflow template
|
|
'my-workflow-template' in region 'us-central1' with step-id 'my-step-id' ,
|
|
run:
|
|
|
|
$ gcloud dataproc workflow-templates add-job spark-sql \
|
|
--step-id=my-step_id -e=QUERY \
|
|
--workflow-template=my-workflow-template --region=us-central1
|
|
|
|
REQUIRED FLAGS
|
|
--step-id=STEP_ID
|
|
The step ID of the job in the workflow template.
|
|
|
|
Exactly one of these must be specified:
|
|
|
|
--execute=QUERY, -e QUERY
|
|
A Spark SQL query to execute as part of the job.
|
|
|
|
--file=FILE, -f FILE
|
|
HCFS URI of file containing Spark SQL script to execute as the job.
|
|
|
|
Template resource - The name of the workflow template to add job to. The
|
|
arguments in this group can be used to specify the attributes of this
|
|
resource. (NOTE) Some attributes are not given arguments in this group but
|
|
can be set in other ways.
|
|
|
|
To set the project attribute:
|
|
◆ provide the argument --workflow-template on the command line with a
|
|
fully specified name;
|
|
◆ provide the argument --project on the command line;
|
|
◆ set the property core/project.
|
|
|
|
This must be specified.
|
|
|
|
--workflow-template=WORKFLOW_TEMPLATE
|
|
ID of the template or fully qualified identifier for the template.
|
|
|
|
To set the template attribute:
|
|
▸ provide the argument --workflow-template on the command line.
|
|
|
|
This flag argument must be specified if any of the other arguments in
|
|
this group are specified.
|
|
|
|
--region=REGION
|
|
Dataproc region for the template. Each Dataproc region constitutes an
|
|
independent resource namespace constrained to deploying instances
|
|
into Compute Engine zones inside the region. Overrides the default
|
|
dataproc/region property value for this command invocation.
|
|
|
|
To set the region attribute:
|
|
▸ provide the argument --workflow-template on the command line with
|
|
a fully specified name;
|
|
▸ provide the argument --region on the command line;
|
|
▸ set the property dataproc/region.
|
|
|
|
OPTIONAL FLAGS
|
|
--driver-log-levels=[PACKAGE=LEVEL,...]
|
|
A list of package to log4j log level pairs to configure driver logging.
|
|
For example: root=FATAL,com.example=INFO
|
|
|
|
--jars=[JAR,...]
|
|
Comma separated list of jar files to be provided to the executor and
|
|
driver classpaths. May contain UDFs.
|
|
|
|
--labels=[KEY=VALUE,...]
|
|
List of label KEY=VALUE pairs to add.
|
|
|
|
Keys must start with a lowercase character and contain only hyphens
|
|
(-), underscores (_), lowercase characters, and numbers. Values must
|
|
contain only hyphens (-), underscores (_), lowercase characters, and
|
|
numbers.
|
|
|
|
--params=[PARAM=VALUE,...]
|
|
A list of key value pairs to set variables in the Hive queries.
|
|
|
|
--properties=[PROPERTY=VALUE,...]
|
|
A list of key value pairs to configure Hive.
|
|
|
|
--properties-file=PROPERTIES_FILE
|
|
Path to a local file or a file in a Cloud Storage bucket containing
|
|
configuration properties for the job. The client machine running this
|
|
command must have read permission to the file.
|
|
|
|
Specify properties in the form of property=value in the text file. For
|
|
example:
|
|
|
|
# Properties to set for the job:
|
|
key1=value1
|
|
key2=value2
|
|
# Comment out properties not used.
|
|
# key3=value3
|
|
|
|
If a property is set in both --properties and --properties-file, the
|
|
value defined in --properties takes precedence.
|
|
|
|
--start-after=STEP_ID,[STEP_ID,...]
|
|
(Optional) List of step IDs to start this job after.
|
|
|
|
GCLOUD WIDE FLAGS
|
|
These flags are available to all commands: --access-token-file, --account,
|
|
--billing-project, --configuration, --flags-file, --flatten, --format,
|
|
--help, --impersonate-service-account, --log-http, --project, --quiet,
|
|
--trace-token, --user-output-enabled, --verbosity.
|
|
|
|
Run $ gcloud help for details.
|
|
|
|
NOTES
|
|
These variants are also available:
|
|
|
|
$ gcloud alpha dataproc workflow-templates add-job spark-sql
|
|
|
|
$ gcloud beta dataproc workflow-templates add-job spark-sql
|
|
|