Use streaming RPCs
Move file data as chunks instead of requiring the whole file in memory.
← Back to projects
Case Study · Deep Backend Engineering
Service File Service
A gRPC file service designed around reliable streaming, storage integrity, and production observability rather than simple upload and download endpoints.
My role
Backend development & reliability engineering
Stack
NestJS · gRPC · S3 / MinIO · Node.js Streams · SHA-256 · Redis · SigNoz · Sentry · Docker
The problem
The service had to move potentially large files between clients and storage while supporting both object storage and direct filesystem modes. A simple buffer-based upload was not enough: the system needed ordering checks, integrity verification, safe overwrite behavior, and useful production telemetry.
What I built
I implemented the file workflow around gRPC streaming. Reads use server streaming, while uploads use client streaming so file data arrives in chunks. The service routes storage through S3/MinIO or the local filesystem and applies validation, checksums, atomic writes, cleanup, and observability around the workflow.
My contributions
- Implemented ReadFileStream and UploadFileStream using gRPC streaming RPCs.
- Built DIRECT storage writes with temporary files, fsync, atomic swap, rollback handling, and stale-file cleanup.
- Built S3 multipart upload with a rolling buffer that respects the minimum 5 MB part size.
- Added SHA-256 streaming checksums and post-write integrity verification.
- Added magic-byte and extension validation plus path-traversal and invalid-path guards.
- Implemented structured telemetry, success/error metrics, request latency, and production-oriented error handling.
Architecture
The same gRPC contract supports two storage paths. DIRECT mode streams bytes into local storage; S3 mode uses multipart upload for writes and presigned URLs for reads so large binary payloads do not need to pass through the service on download.
Key technical decisions
Atomic DIRECT writes
Write to .tmp, fsync the completed file, then swap it into place so incomplete uploads are not exposed as the final file.
Verify after writing
Re-hash the file on disk and compare it with the upload checksum; S3 uploads also verify the final object size.
Keep S3 reads lightweight
Return a presigned URL for S3 reads instead of streaming the object bytes through the backend.
Challenges & solutions
Large-file streaming
Chunked gRPC streaming keeps the workflow bounded instead of loading an entire file into memory.
Partial or corrupted writes
DIRECT uploads use temporary files and atomic replacement, followed by a read-back SHA-256 verification.
S3 multipart constraints
A rolling buffer accumulates chunks into valid multipart sizes and sends the final remainder as the last part.
Production visibility
Telemetry covers request lifecycle, success/error paths, latency, chunk information, and upload/read attributes for investigation in SigNoz.
Engineering evidence
- gRPC streaming RPC implementation for file reads and uploads.
- Atomic overwrite flow with .tmp and .deleting states plus rollback protection.
- fsync before rename and post-write SHA-256 verification on DIRECT storage.
- S3 multipart upload with post-upload HeadObject size verification.
- Structured SigNoz telemetry and centralized gRPC error handling across error paths.
Result
The service provides a reusable file-transfer layer with explicit streaming semantics, two storage backends, integrity checks, safer overwrite behavior, and production telemetry. The engineering focus is on making file operations predictable and recoverable when things go wrong.