mirror of
https://github.com/imjasonh/gcsbuf
synced 2026-07-16 20:33:38 +00:00
39 lines
2.3 KiB
Markdown
39 lines
2.3 KiB
Markdown
|
# `gcsbuf`
|
||
|
|
|
||
|
|
This Go library gives you an `io.Writer` to streamily write an object to [Google Cloud Storage](https://cloud.google.com/storage), and at the same time, streamily read that same object using an `io.Reader`.
|
||
|
|
|
||
|
|
### Writing
|
||
|
|
|
||
|
|
Writing works by buffering the input data and periodically flushing updates to GCS. Object updates are implemented using [composite objects](https://docs.cloud.google.com/storage/docs/composing-objects). When appending a new chunk of data, that data is first uploaded to a new temp object, then composed with the existing object contents over itself.
|
||
|
|
|
||
|
|
When the object reaches the max number of compositions, the count is reset by copying and then renaming the object back to its original location -- this is purely a metadata operation and does not require the client to read or write all the data again.
|
||
|
|
|
||
|
|
When the writer calls `Close()`, the object is finalized by updating object metadata, to tell consumers that the object is done being written.
|
||
|
|
|
||
|
|
Each compose/append and copy/reset operation is guarded with a precondition, so that the operations fail if the object changed due to outside operations, preventing silent corruption.
|
||
|
|
|
||
|
|
You can configure the flush timeout (default: 5s) and buffer size (default: 4MB), and the writer will flush when either of those conditions are met.
|
||
|
|
|
||
|
|
```go
|
||
|
|
w := gcsbuf.NewWriter(...)
|
||
|
|
fmt.Fprintln(w, "hello")
|
||
|
|
_ = w.Flush() // immediately flush to GCS
|
||
|
|
fmt.Fprintln(w, "world")
|
||
|
|
_ = w.Close() // flush and finalize
|
||
|
|
````
|
||
|
|
|
||
|
|
### Reading
|
||
|
|
|
||
|
|
Reading works by iteratively making [HTTP Range requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Range_requests) to the object and polling until either newly appended data is found, or until the object is marked as finalized.
|
||
|
|
|
||
|
|
```go
|
||
|
|
r := gcsbuf.NewReader(...)
|
||
|
|
_, _ = io.Copy(os.Stdout, r) // reads until the object is finalized
|
||
|
|
```
|
||
|
|
|
||
|
|
In the example above, if the writer is still writing data, `io.Copy` will block until the writer completes, and receive new content each time it flushes.
|
||
|
|
|
||
|
|
### Background
|
||
|
|
|
||
|
|
I originally used a ~hack~ scheme like this when creating [Google Cloud Build](https://cloud.google.com/build) to efficiently write logs to durable storage, and to allow clients like `gcloud` to simultaneously stream the logs objects to users until build completion. Logs were separately written to Google Cloud Logging, which didn't support a fast logs tailing API at the time.
|