NAME
    gcloud dataproc jobs submit pyspark - submit a PySpark job to a cluster

SYNOPSIS
    gcloud dataproc jobs submit pyspark PY_FILE
        (--cluster=CLUSTER | --cluster-labels=[KEY=VALUE,...])
        [--archives=[ARCHIVE,...]] [--async] [--bucket=BUCKET]
        [--driver-log-levels=[PACKAGE=LEVEL,...]]
        [--driver-required-memory-mb=DRIVER_REQUIRED_MEMORY_MB]
        [--driver-required-vcores=DRIVER_REQUIRED_VCORES] [--files=[FILE,...]]
        [--jars=[JAR,...]] [--labels=[KEY=VALUE,...]]
        [--max-failures-per-hour=MAX_FAILURES_PER_HOUR]
        [--max-failures-total=MAX_FAILURES_TOTAL]
        [--properties=[PROPERTY=VALUE,...]] [--properties-file=PROPERTIES_FILE]
        [--py-files=[PY_FILE,...]] [--region=REGION] [GCLOUD_WIDE_FLAG ...]
        [-- JOB_ARGS ...]

DESCRIPTION
    Submit a PySpark job to a cluster.

EXAMPLES
    To submit a PySpark job with a local script and custom flags, run:

        $ gcloud dataproc jobs submit pyspark --cluster=my-cluster \
            my_script.py -- --custom-flag

    To submit a Spark job that runs a script that is already on the cluster,
    run:

        $ gcloud dataproc jobs submit pyspark --cluster=my-cluster \
            file:///usr/lib/spark/examples/src/main/python/pi.py -- 100

POSITIONAL ARGUMENTS
     PY_FILE
        Main .py file to run as the driver.

     [-- JOB_ARGS ...]
        Arguments to pass to the driver.

        The '--' argument must be specified between gcloud specific args on the
        left and JOB_ARGS on the right.

REQUIRED FLAGS
     Exactly one of these must be specified:

       --cluster=CLUSTER
          The Dataproc cluster to submit the job to.

       --cluster-labels=[KEY=VALUE,...]
          List of label KEY=VALUE pairs to add.

          Keys must start with a lowercase character and contain only hyphens
          (-), underscores (_), lowercase characters, and numbers. Values must
          contain only hyphens (-), underscores (_), lowercase characters, and
          numbers.

          Labels of Dataproc cluster on which to place the job.

OPTIONAL FLAGS
     --archives=[ARCHIVE,...]
        Comma separated list of archives to be extracted into the working
        directory of each executor. Must be one of the following file formats:
        .zip, .tar, .tar.gz, or .tgz.

     --async
        Return immediately, without waiting for the operation in progress to
        complete.

     --bucket=BUCKET
        The Cloud Storage bucket to stage files in. Defaults to the cluster's
        configured bucket.

     --driver-log-levels=[PACKAGE=LEVEL,...]
        List of key value pairs to configure driver logging, where key is a
        package and value is the log4j log level. For example:
        root=FATAL,com.example=INFO

     --driver-required-memory-mb=DRIVER_REQUIRED_MEMORY_MB
        The memory allocation requested by the job driver in megabytes (MB) for
        execution on the driver node group (it is used only by clusters with a
        driver node group).

     --driver-required-vcores=DRIVER_REQUIRED_VCORES
        The vCPU allocation requested by the job driver for execution on the
        driver node group (it is used only by clusters with a driver node
        group).

     --files=[FILE,...]
        Comma separated list of files to be placed in the working directory of
        both the app driver and executors.

     --jars=[JAR,...]
        Comma separated list of jar files to be provided to the executor and
        driver classpaths.

     --labels=[KEY=VALUE,...]
        List of label KEY=VALUE pairs to add.

        Keys must start with a lowercase character and contain only hyphens
        (-), underscores (_), lowercase characters, and numbers. Values must
        contain only hyphens (-), underscores (_), lowercase characters, and
        numbers.

     --max-failures-per-hour=MAX_FAILURES_PER_HOUR
        Specifies the maximum number of times a job can be restarted per hour
        in event of failure. Default is 0 (no retries after job failure).

     --max-failures-total=MAX_FAILURES_TOTAL
        Specifies the maximum total number of times a job can be restarted
        after the job fails. Default is 0 (no retries after job failure).

     --properties=[PROPERTY=VALUE,...]
        List of key value pairs to configure PySpark. For a list of available
        properties, see:
        https://spark.apache.org/docs/latest/configuration.html#available-properties.

     --properties-file=PROPERTIES_FILE
        Path to a local file or a file in a Cloud Storage bucket containing
        configuration properties for the job. The client machine running this
        command must have read permission to the file.

        Specify properties in the form of property=value in the text file. For
        example:

              # Properties to set for the job:
              key1=value1
              key2=value2
              # Comment out properties not used.
              # key3=value3

        If a property is set in both --properties and --properties-file, the
        value defined in --properties takes precedence.

     --py-files=[PY_FILE,...]
        Comma separated list of Python files to be provided to the job. Must be
        one of the following file formats ".py, .zip, or .egg".

     --region=REGION
        Dataproc region to use. Each Dataproc region constitutes an independent
        resource namespace constrained to deploying instances into Compute
        Engine zones inside the region. Overrides the default dataproc/region
        property value for this command invocation.

GCLOUD WIDE FLAGS
    These flags are available to all commands: --access-token-file, --account,
    --billing-project, --configuration, --flags-file, --flatten, --format,
    --help, --impersonate-service-account, --log-http, --project, --quiet,
    --trace-token, --user-output-enabled, --verbosity.

    Run $ gcloud help for details.

NOTES
    These variants are also available:

        $ gcloud alpha dataproc jobs submit pyspark

        $ gcloud beta dataproc jobs submit pyspark

