1
0
Fork 0
mirror of https://github.com/imjasonh/gcloud-help synced 2026-07-12 16:08:31 +00:00
gcloud-help/gcloud/datastream/streams/update

579 lines
20 KiB
Text
Raw Normal View History

2022-03-01 04:29:52 +00:00
NAME
gcloud datastream streams update - updates a Datastream stream
SYNOPSIS
gcloud datastream streams update (STREAM : --location=LOCATION)
[--display-name=DISPLAY_NAME] [--state=STATE]
[--update-labels=[KEY=VALUE,...]] [--update-mask=UPDATE_MASK]
[--backfill-none
2025-05-07 10:56:18 +00:00
| --backfill-all --mongodb-excluded-objects=MONGODB_EXCLUDED_OBJECTS
| --mysql-excluded-objects=MYSQL_EXCLUDED_OBJECTS
2022-10-26 10:52:11 +00:00
| --oracle-excluded-objects=ORACLE_EXCLUDED_OBJECTS
2024-06-12 09:25:04 +00:00
| --postgresql-excluded-objects=POSTGRESQL_EXCLUDED_OBJECTS
2025-03-05 10:41:25 +00:00
| --salesforce-excluded-objects=SALESFORCE_EXCLUDED_OBJECTS
2024-06-12 09:25:04 +00:00
| --sqlserver-excluded-objects=SQLSERVER_EXCLUDED_OBJECTS]
2022-03-01 04:29:52 +00:00
[--clear-labels | --remove-labels=[KEY,...]]
[--destination=DESTINATION
2023-02-15 10:13:15 +00:00
--bigquery-destination-config=BIGQUERY_DESTINATION_CONFIG
2022-10-18 11:38:19 +00:00
| --gcs-destination-config=GCS_DESTINATION_CONFIG]
2022-03-01 04:29:52 +00:00
[--force | --validate-only]
2025-05-07 10:56:18 +00:00
[--source=SOURCE --mongodb-source-config=MONGODB_SOURCE_CONFIG
| --mysql-source-config=MYSQL_SOURCE_CONFIG
2022-10-26 10:52:11 +00:00
| --oracle-source-config=ORACLE_SOURCE_CONFIG
2024-06-12 09:25:04 +00:00
| --postgresql-source-config=POSTGRESQL_SOURCE_CONFIG
2025-03-05 10:41:25 +00:00
| --salesforce-source-config=SALESFORCE_SOURCE_CONFIG
2024-06-12 09:25:04 +00:00
| --sqlserver-source-config=SQLSERVER_SOURCE_CONFIG]
2022-10-26 10:52:11 +00:00
[GCLOUD_WIDE_FLAG ...]
2022-03-01 04:29:52 +00:00
DESCRIPTION
2023-04-26 10:24:37 +00:00
Update a Datastream stream. If successful, the response body contains a
newly created instance of Operation. To get the operation result, call:
describe OPERATION
2022-03-01 04:29:52 +00:00
EXAMPLES
To update a stream with a new source and new display name:
$ gcloud datastream streams update STREAM --location=us-central1 \
--display-name=my-stream --source=source \
--update-mask=display_name,source
To update a stream's state to RUNNING:
$ gcloud datastream streams update STREAM --location=us-central1 \
--state=RUNNING --update-mask=state
To update a stream's oracle source config:
$ gcloud datastream streams update STREAM --location=us-central1 \
--oracle-source-config=good_oracle_cp.json \
--update-mask=oracle_source_config.include_objects
POSITIONAL ARGUMENTS
Stream resource - The stream to update. The arguments in this group can be
used to specify the attributes of this resource. (NOTE) Some attributes
2023-05-04 10:43:54 +00:00
are not given arguments in this group but can be set in other ways.
To set the project attribute:
2022-03-01 04:29:52 +00:00
◆ provide the argument stream on the command line with a fully
specified name;
◆ provide the argument --project on the command line;
◆ set the property core/project.
This must be specified.
STREAM
2023-05-04 10:43:54 +00:00
ID of the stream or fully qualified identifier for the stream.
To set the stream attribute:
2022-03-01 04:29:52 +00:00
▸ provide the argument stream on the command line.
2022-08-10 08:48:58 +00:00
This positional argument must be specified if any of the other
arguments in this group are specified.
2022-03-01 04:29:52 +00:00
--location=LOCATION
2023-05-04 10:43:54 +00:00
The Cloud location for the stream.
To set the location attribute:
2022-03-01 04:29:52 +00:00
▸ provide the argument stream on the command line with a fully
specified name;
▸ provide the argument --location on the command line.
FLAGS
--display-name=DISPLAY_NAME
Friendly name for the stream.
--state=STATE
Stream state, can be set to: "RUNNING" or "PAUSED".
--update-labels=[KEY=VALUE,...]
List of label KEY=VALUE pairs to update. If a label exists, its value
is modified. Otherwise, a new label is created.
Keys must start with a lowercase character and contain only hyphens
(-), underscores (_), lowercase characters, and numbers. Values must
contain only hyphens (-), underscores (_), lowercase characters, and
numbers.
--update-mask=UPDATE_MASK
Used to specify the fields to be overwritten in the stream resource by
the update. If the update mask is used, then a field will be
overwritten only if it is in the mask. If the user does not provide a
mask then all fields will be overwritten. This is a comma-separated
list of fully qualified names of fields, written as snake_case or
camelCase. Example: "display_name, source_config.oracle_source_config".
At most one of these can be specified:
--backfill-none
2024-03-13 09:56:22 +00:00
Do not automatically backfill any objects. This flag is equivalent to
selecting the Manual backfill type in the Google Cloud console.
2022-03-01 04:29:52 +00:00
--backfill-all
Automatically backfill objects included in the stream source
2024-03-13 09:56:22 +00:00
configuration. Specific objects can be excluded. This flag is
equivalent to selecting the Automatic backfill type in the Google
Cloud console.
2022-03-01 04:29:52 +00:00
At most one of these can be specified:
2025-05-07 10:56:18 +00:00
--mongodb-excluded-objects=MONGODB_EXCLUDED_OBJECTS
Path to a YAML (or JSON) file containing the MongoDB data sources
to avoid backfilling.
The JSON file is formatted as follows, with camelCase field naming:
{
"databases": [
{
"database":"sample_database",
"collections": [
{
"collection": "sample_collection",
"fields": [
{
"field": "sample_field",
}
]
}
]
}
]
}
2022-03-01 04:29:52 +00:00
--mysql-excluded-objects=MYSQL_EXCLUDED_OBJECTS
2023-09-20 10:45:25 +00:00
Path to a YAML (or JSON) file containing the MySQL data sources to
avoid backfilling.
The JSON file is formatted as follows, with camelCase field naming:
{
2025-03-05 10:41:25 +00:00
"mysqlDatabases": [
{
"database":"sample_database",
"mysqlTables": [
2023-09-20 10:45:25 +00:00
{
2025-03-05 10:41:25 +00:00
"table": "sample_table",
"mysqlColumns": [
2023-09-20 10:45:25 +00:00
{
2025-03-05 10:41:25 +00:00
"column": "sample_column",
2023-09-20 10:45:25 +00:00
}
2025-03-05 10:41:25 +00:00
]
2023-09-20 10:45:25 +00:00
}
]
}
2025-03-05 10:41:25 +00:00
]
2023-09-20 10:45:25 +00:00
}
2022-03-01 04:29:52 +00:00
--oracle-excluded-objects=ORACLE_EXCLUDED_OBJECTS
2023-09-20 10:45:25 +00:00
Path to a YAML (or JSON) file containing the Oracle data sources to
avoid backfilling.
The JSON file is formatted as follows, with camelCase field naming:
{
2025-03-05 10:41:25 +00:00
"oracleSchemas": [
{
"schema": "SAMPLE",
"oracleTables": [
{
"table": "SAMPLE_TABLE",
"oracleColumns": [
{
"column": "COL",
}
]
}
]
}
]
2023-09-20 10:45:25 +00:00
}
2022-03-01 04:29:52 +00:00
2022-10-26 10:52:11 +00:00
--postgresql-excluded-objects=POSTGRESQL_EXCLUDED_OBJECTS
2023-09-20 10:45:25 +00:00
Path to a YAML (or JSON) file containing the PostgreSQL data
sources to avoid backfilling.
The JSON file is formatted as follows, with camelCase field naming:
{
2025-03-05 10:41:25 +00:00
"postgresqlSchemas": [
{
"schema": "SAMPLE",
"postgresqlTables": [
{
"table": "SAMPLE_TABLE",
"postgresqlColumns": [
{
"column": "COL",
}
]
}
]
}
]
}
--salesforce-excluded-objects=SALESFORCE_EXCLUDED_OBJECTS
Path to a YAML (or JSON) file containing the Salesforce data
sources to avoid backfilling.
The JSON file is formatted as follows, with camelCase field naming:
{
"objects": [
{
"objectName": "SAMPLE",
},
{
"objectName": "SAMPLE2",
}
]
2023-09-20 10:45:25 +00:00
}
2022-10-26 10:52:11 +00:00
2024-06-12 09:25:04 +00:00
--sqlserver-excluded-objects=SQLSERVER_EXCLUDED_OBJECTS
Path to a YAML (or JSON) file containing the SQL Server data
sources to avoid backfilling.
The JSON file is formatted as follows, with camelCase field naming:
{
2025-03-05 10:41:25 +00:00
"schemas": [
{
"schema": "SAMPLE",
"tables": [
{
"table": "SAMPLE_TABLE",
"columns": [
{
"column": "COL",
}
]
}
]
}
]
2024-06-12 09:25:04 +00:00
}
2022-03-01 04:29:52 +00:00
At most one of these can be specified:
--clear-labels
Remove all labels. If --update-labels is also specified then
--clear-labels is applied first.
For example, to remove all labels:
$ gcloud datastream streams update --clear-labels
To remove all existing labels and create two new labels, foo and baz:
$ gcloud datastream streams update --clear-labels \
--update-labels foo=bar,baz=qux
--remove-labels=[KEY,...]
List of label keys to remove. If a label does not exist it is
silently ignored. If --update-labels is also specified then
--update-labels is applied first.
Connection profile resource - Resource ID of the destination connection
profile. This represents a Cloud resource. (NOTE) Some attributes are not
2023-05-04 10:43:54 +00:00
given arguments in this group but can be set in other ways.
To set the project attribute:
2022-03-01 04:29:52 +00:00
◆ provide the argument --destination on the command line with a fully
specified name;
◆ provide the argument --project on the command line;
2023-05-04 10:43:54 +00:00
◆ set the property core/project.
To set the location attribute:
2022-03-01 04:29:52 +00:00
◆ provide the argument --destination on the command line with a fully
specified name;
◆ provide the argument --location on the command line.
--destination=DESTINATION
ID of the connection_profile or fully qualified identifier for the
2023-05-04 10:43:54 +00:00
connection_profile.
To set the connection_profile attribute:
2022-03-01 04:29:52 +00:00
▸ provide the argument --destination on the command line.
2022-10-18 11:38:19 +00:00
At most one of these can be specified:
2022-03-01 04:29:52 +00:00
2022-10-18 11:38:19 +00:00
--bigquery-destination-config=BIGQUERY_DESTINATION_CONFIG
Path to a YAML (or JSON) file containing the configuration for Google
BigQuery Destination Config.
2024-06-27 09:38:12 +00:00
The YAML (or JSON) file should be formatted as follows:
BigQuery configuration with source hierarchy datasets and merge mode
(merge mode is by default):
2022-10-18 11:38:19 +00:00
{
2023-02-15 10:13:15 +00:00
"sourceHierarchyDatasets": {
"datasetTemplate": {
2022-10-18 11:38:19 +00:00
"location": "us-central1",
2023-02-15 10:13:15 +00:00
"datasetIdPrefix": "my_prefix",
"kmsKeyName": "projects/{project}/locations/{location}/keyRings/{key_ring}/cryptoKeys/{cryptoKey}"
2022-10-18 11:38:19 +00:00
}
},
2024-06-27 09:38:12 +00:00
"merge": {}
"dataFreshness": "3600s"
}
BigQuery configuration with source hierarchy datasets and append only
mode: {
"sourceHierarchyDatasets": {
"datasetTemplate": {
"location": "us-central1",
"datasetIdPrefix": "my_prefix",
"kmsKeyName": "projects/{project}/locations/{location}/keyRings/{key_ring}/cryptoKeys/{cryptoKey}"
}
},
"appendOnly": {}
}
BigQuery configuration with single target dataset and merge mode:
{
"singleTargetDataset": {
"datasetId": "projectId:my_dataset"
},
"merge": {}
2023-05-04 10:43:54 +00:00
"dataFreshness": "3600s"
2022-10-18 11:38:19 +00:00
}
2025-04-02 10:41:45 +00:00
BigQuery configuration with Big Lake table configuration: {
"singleTargetDataset": {
"datasetId": "projectId:datasetId"
},
"appendOnly": {},
"blmtConfig": {
"bucket": "bucketName",
"tableFormat": "ICEBERG",
"fileFormat": "PARQUET",
"connectionName": "projectId.region.connectionName",
"rootPath": "/root"
}
}
2022-10-18 11:38:19 +00:00
--gcs-destination-config=GCS_DESTINATION_CONFIG
Path to a YAML (or JSON) file containing the configuration for Google
Cloud Storage Destination Config.
The JSON file is formatted as follows:
{
"path": "some/path",
2023-02-15 10:13:15 +00:00
"fileRotationMb":5,
"fileRotationInterval":"15s",
"avroFileFormat": {}
2022-10-18 11:38:19 +00:00
}
2022-03-01 04:29:52 +00:00
At most one of these can be specified:
--force
Update the stream without validating it.
--validate-only
Only validate the stream, but do not update any resources. The
default is false.
Connection profile resource - Resource ID of the source connection
profile. This represents a Cloud resource. (NOTE) Some attributes are not
2023-05-04 10:43:54 +00:00
given arguments in this group but can be set in other ways.
To set the project attribute:
2022-03-01 04:29:52 +00:00
◆ provide the argument --source on the command line with a fully
specified name;
◆ provide the argument --project on the command line;
2023-05-04 10:43:54 +00:00
◆ set the property core/project.
To set the location attribute:
2022-03-01 04:29:52 +00:00
◆ provide the argument --source on the command line with a fully
specified name;
◆ provide the argument --location on the command line.
--source=SOURCE
ID of the connection_profile or fully qualified identifier for the
2023-05-04 10:43:54 +00:00
connection_profile.
To set the connection_profile attribute:
2022-03-01 04:29:52 +00:00
▸ provide the argument --source on the command line.
At most one of these can be specified:
2025-05-07 10:56:18 +00:00
--mongodb-source-config=MONGODB_SOURCE_CONFIG
Path to a YAML (or JSON) file containing the configuration for
MongoDB Source Config.
The JSON file is formatted as follows, with snake_case field naming:
{
"includeObjects": {},
"excludeObjects": {
"databases": [
{
"database": "sampleDb",
"collections": [
{
"collection": "sampleCollection",
"fields": [
{
"field": "SAMPLE_FIELD",
}
]
}
]
}
]
}
}
2022-03-01 04:29:52 +00:00
--mysql-source-config=MYSQL_SOURCE_CONFIG
2022-10-26 10:52:11 +00:00
Path to a YAML (or JSON) file containing the configuration for MySQL
2022-03-01 04:29:52 +00:00
Source Config.
2023-02-15 10:13:15 +00:00
The JSON file is formatted as follows, with camelCase field naming:
2022-03-01 04:29:52 +00:00
{
2023-02-15 10:13:15 +00:00
"includeObjects": {},
"excludeObjects": {
"mysqlDatabases": [
2022-03-01 04:29:52 +00:00
{
"database":"sample_database",
2023-02-15 10:13:15 +00:00
"mysqlTables": [
2022-03-01 04:29:52 +00:00
{
"table": "sample_table",
2023-02-15 10:13:15 +00:00
"mysqlColumns": [
2022-03-01 04:29:52 +00:00
{
"column": "sample_column",
}
]
}
]
}
]
}
}
--oracle-source-config=ORACLE_SOURCE_CONFIG
Path to a YAML (or JSON) file containing the configuration for Oracle
Source Config.
2023-02-15 10:13:15 +00:00
The JSON file is formatted as follows, with camelCase field naming:
2022-03-01 04:29:52 +00:00
{
2023-02-15 10:13:15 +00:00
"includeObjects": {},
"excludeObjects": {
"oracleSchemas": [
2022-03-01 04:29:52 +00:00
{
"schema": "SAMPLE",
2023-02-15 10:13:15 +00:00
"oracleTables": [
2022-03-01 04:29:52 +00:00
{
"table": "SAMPLE_TABLE",
2023-02-15 10:13:15 +00:00
"oracleColumns": [
2022-03-01 04:29:52 +00:00
{
"column": "COL",
}
]
}
]
}
]
}
}
2022-10-26 10:52:11 +00:00
--postgresql-source-config=POSTGRESQL_SOURCE_CONFIG
Path to a YAML (or JSON) file containing the configuration for
PostgreSQL Source Config.
2023-02-15 10:13:15 +00:00
The JSON file is formatted as follows, with camelCase field naming:
2022-10-26 10:52:11 +00:00
{
2023-02-15 10:13:15 +00:00
"includeObjects": {},
"excludeObjects": {
"postgresqlSchemas": [
2022-10-26 10:52:11 +00:00
{
"schema": "SAMPLE",
2023-02-15 10:13:15 +00:00
"postgresqlTables": [
2022-10-26 10:52:11 +00:00
{
"table": "SAMPLE_TABLE",
2023-02-15 10:13:15 +00:00
"postgresqlColumns": [
2022-10-26 10:52:11 +00:00
{
"column": "COL",
}
]
}
]
}
]
},
2023-03-24 11:31:22 +00:00
"replicationSlot": "SAMPLE_REPLICATION_SLOT",
2022-10-26 10:52:11 +00:00
"publication": "SAMPLE_PUBLICATION"
}
2025-03-05 10:41:25 +00:00
--salesforce-source-config=SALESFORCE_SOURCE_CONFIG
2024-06-12 09:25:04 +00:00
Path to a YAML (or JSON) file containing the configuration for
2025-03-05 10:41:25 +00:00
Salesforce Source Config.
The JSON file is formatted as follows, with camelCase field naming:
{
"pollingInterval": "3000s",
"includeObjects": {},
"excludeObjects": {
"objects": [
{
"objectName": "SAMPLE",
"fields": [
{
"fieldName": "SAMPLE_FIELD",
}
]
}
]
}
}
--sqlserver-source-config=SQLSERVER_SOURCE_CONFIG
Path to a YAML (or JSON) file containing the configuration for SQL
Server Source Config.
2024-06-12 09:25:04 +00:00
The JSON file is formatted as follows, with camelCase field naming:
{
"includeObjects": {},
"excludeObjects": {
"schemas": [
{
"schema": "SAMPLE",
"tables": [
{
"table": "SAMPLE_TABLE",
"columns": [
{
"column": "COL",
}
]
}
]
}
]
},
"maxConcurrentCdcTasks": 2,
"maxConcurrentBackfillTasks": 10,
"transactionLogs": {} # Or changeTables
}
2022-03-01 04:29:52 +00:00
GCLOUD WIDE FLAGS
These flags are available to all commands: --access-token-file, --account,
--billing-project, --configuration, --flags-file, --flatten, --format,
--help, --impersonate-service-account, --log-http, --project, --quiet,
--trace-token, --user-output-enabled, --verbosity.
Run $ gcloud help for details.
NOTES
This variant is also available:
$ gcloud beta datastream streams update