gcsbuf

This Go library gives you an io.Writer to streamily write an object to Google Cloud Storage, and at the same time, streamily read that same object using an io.Reader.

Writing

Writing works by buffering the input data and periodically flushing updates to GCS. Object updates are implemented using composite objects. When appending a new chunk of data, that data is first uploaded to a new temp object, then composed with the existing object contents over itself.

When the object reaches the max number of compositions, the count is reset by copying and then renaming the object back to its original location -- this is purely a metadata operation and does not require the client to read or write all the data again.

When the writer calls Close(), the object is finalized by updating object metadata, to tell consumers that the object is done being written.

Each compose/append and copy/reset operation is guarded with a precondition, so that the operations fail if the object changed due to outside operations, preventing silent corruption.

You can configure the flush timeout (default: 5s) and buffer size (default: 4MB), and the writer will flush when either of those conditions are met.

w := gcsbuf.NewWriter(...)
fmt.Fprintln(w, "hello")
_ = w.Flush() // immediately flush to GCS
fmt.Fprintln(w, "world")
_ = w.Close() // flush and finalize

Reading

Reading works by iteratively making HTTP Range requests to the object and polling until either newly appended data is found, or until the object is marked as finalized.

r := gcsbuf.NewReader(...)
_, _ = io.Copy(os.Stdout, r) // reads until the object is finalized

In the example above, if the writer is still writing data, io.Copy will block until the writer completes, and receive new content each time it flushes.

Background

I originally used a hack scheme like this when creating Google Cloud Build to efficiently write logs to durable storage, and to allow clients like gcloud to simultaneously stream the logs objects to users until build completion. Logs were separately written to Google Cloud Logging, which didn't support a fast logs tailing API at the time.