From 6bd7460860eff383d332a12d85f757bd60e9f4c4 Mon Sep 17 00:00:00 2001 From: Jason Hall Date: Tue, 24 Mar 2026 13:55:27 -0400 Subject: [PATCH] readme Signed-off-by: Jason Hall --- README.md | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..e68fd3e --- /dev/null +++ b/README.md @@ -0,0 +1,38 @@ +# `gcsbuf` + +This Go library gives you an `io.Writer` to streamily write an object to [Google Cloud Storage](https://cloud.google.com/storage), and at the same time, streamily read that same object using an `io.Reader`. + +### Writing + +Writing works by buffering the input data and periodically flushing updates to GCS. Object updates are implemented using [composite objects](https://docs.cloud.google.com/storage/docs/composing-objects). When appending a new chunk of data, that data is first uploaded to a new temp object, then composed with the existing object contents over itself. + +When the object reaches the max number of compositions, the count is reset by copying and then renaming the object back to its original location -- this is purely a metadata operation and does not require the client to read or write all the data again. + +When the writer calls `Close()`, the object is finalized by updating object metadata, to tell consumers that the object is done being written. + +Each compose/append and copy/reset operation is guarded with a precondition, so that the operations fail if the object changed due to outside operations, preventing silent corruption. + +You can configure the flush timeout (default: 5s) and buffer size (default: 4MB), and the writer will flush when either of those conditions are met. + +```go +w := gcsbuf.NewWriter(...) +fmt.Fprintln(w, "hello") +_ = w.Flush() // immediately flush to GCS +fmt.Fprintln(w, "world") +_ = w.Close() // flush and finalize +```` + +### Reading + +Reading works by iteratively making [HTTP Range requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Range_requests) to the object and polling until either newly appended data is found, or until the object is marked as finalized. + +```go +r := gcsbuf.NewReader(...) +_, _ = io.Copy(os.Stdout, r) // reads until the object is finalized +``` + +In the example above, if the writer is still writing data, `io.Copy` will block until the writer completes, and receive new content each time it flushes. + +### Background + +I originally used a ~hack~ scheme like this when creating [Google Cloud Build](https://cloud.google.com/build) to efficiently write logs to durable storage, and to allow clients like `gcloud` to simultaneously stream the logs objects to users until build completion. Logs were separately written to Google Cloud Logging, which didn't support a fast logs tailing API at the time.