Skip to main content

Design: Uniform File-Upload Support for smithy-hono Generated Services

0. Scope & summary​

Today a service generated by this codegen can only accept a file as base64 embedded in a JSON body, fully buffered in memory, hashed whole for HMAC signing, and with nowhere to land the bytes (there is no object-storage port in any adapter). This document proposes a uniform upload abstraction — one Smithy trait + model convention, one generated route shape, and one ObjectStore storage-port — that lets a modeled operation stream an upload into R2 / S3 / local-or-S3 across the three deploy targets, while staying inside the existing security pipeline's body-size, content-type, signing, and authorization guarantees. No implementation is proposed here — design only.

1. Current limitations (with file:line evidence)​

1.1 The codegen speaks JSON bodies only​

  • Body binding collapses to JSON. InputBindings maps every body-bound member to either PAYLOAD (a member carrying @httpPayload) or IMPLICIT_BODY, and isBodyMethod only recognizes POST/PUT/PATCH (InputBindings.java:43-57). There is no binding class for a raw/binary/multipart request body.
  • The validator chain is always zValidator('json', …). In RouteEmitter.buildValidatorChain, both the @httpPayload branch and the implicit-body branch emit a JSON validator (RouteEmitter.java:256-267); buildPayloadValidator hard-codes zValidator('json', …) (RouteEmitter.java:318-320). Input assembly likewise reads c.req.valid('json') (RouteEmitter.java:353-358). There is no c.req.parseBody() / formData() / raw-stream path anywhere in the emitter.
  • A blob is validated as a base64 JSON string. ZodEmitter.emitBlob (ZodEmitter.java:310-323) emits z.string().regex(/^[A-Za-z0-9+/]*={0,2}$/) for a non-streaming blob — i.e. the file must be base64-encoded and wrapped in JSON (≈33% size inflation, and it is fully materialized twice: once as base64 text, once decoded). A @streaming blob degrades to an opaque z.string() passthrough (ZodEmitter.java:311-315) — it is typed as a stream but nothing in RouteEmitter ever reads the request body as a stream, so streaming input is not actually representable at the route level.
  • Input-side @streaming is not wired. ModelIndex.isStreaming only returns true for the SSE response trait (ModelIndex.java:188-190, SseStreamTrait); it has no notion of a streaming request body. The metadata registry emits streaming: true solely from that SSE trait (MetadataRegistryEmitter.java:123-124).
  • The generated client can only send JSON. ClientEmitter.bodyExpr always JSON.stringify(...) (ClientEmitter.java:275-285) — it cannot post multipart/form-data, application/octet-stream, or a stream.

Net: multipart and binary uploads are not representable. The only upload story is "base64 blob inside a JSON object," fully buffered, size-capped by the JSON body limit.

1.2 The security pipeline actively rejects real uploads​

  • Content-type allowlist → 415 for multipart/binary. headerGuards/bodyGuards require the request media type to equal the single modeled protocol content-type (restJson1 → application/json), else 415 (bodyGuards.ts:257, bodyGuards.ts:268-274). A multipart/form-data or application/octet-stream upload is rejected before the handler runs.
  • The whole body is buffered in memory, then replaced. readBoundedBody accumulates every chunk into a Uint8Array up to maxBodyBytes (bodyGuards.ts:155-187) and rebuildBoundedRequest rebuilds the Request around those bytes (bodyGuards.ts:196-209). There is no streaming-to-storage path; peak memory ≈ file size, and any file over maxBodyBytes (per-op or global default) is 413'd (bodyGuards.ts:234, bodyGuards.ts:321-330). This is correct and desirable for JSON APIs, but it is exactly wrong for large uploads.
  • HMAC signing hashes the entire body. The canonical string's field 5 is the SHA-256 of the exact received body bytes (canonical.ts:49-52, canonical.ts:226-241), and the verifier obtains those bytes via readRawBody(c) → c.req.arrayBuffer(), which buffers the whole body (rawBody.ts:113-118, verifySignature.ts:331,:359). For a signed upload the entire file must be read into memory and digested before auth completes — no streaming, and the buffering bound is only safe because bodyGuards capped it first (rawBody.ts:54-68).

1.3 There is no object-storage port anywhere​

  • The only data port is DataStore<T> — row/entity semantics (get/create/put/update/patch/delete/list), not blobs (data-core/src/index.ts:97-149).
  • adapter-cf implements KV / Durable Objects / D1 (adapter-cf/src/dataStore.ts, ports.ts), no R2. A repo-wide search for R2 / S3 / multipart / PutObject / ObjectStore returns no production hits.
  • adapter-aws implements DynamoDB (dynamoPort.ts, stores/), no S3.
  • adapter-node implements Redis/in-memory (stores.ts, dataStore.ts), no filesystem/S3.
  • The CF deploy tool renders only kv_namespaces, durable_objects, d1_databases bindings (deploy-cf/src/wrangler.ts:64-108); BindingsSpec has no r2 field (deploy-cf/src/config.ts:40-44). So even if bytes arrived, there is no provisioned bucket to hold them.

2. Proposed uniform upload abstraction​

Three coordinated pieces: (A) a model convention, (B) a generated route shape, (C) an ObjectStore storage-port. The design goal is that the security pipeline is unchanged in spirit — the same size cap, an extended content-type allowlist, and a streaming-friendly body-hash mode — and that a handler author never touches raw bytes unless they opt in.

2.A Model convention: @upload trait + ObjectRef shape​

Add a service-owned trait (mirroring the existing traits/ package, e.g. RequiresAuthTrait, SseStreamTrait):

// model/traits.smithy (new)
@trait(selector: "operation")
structure upload {
    /// wire framing the route accepts for the payload member
    mode: UploadMode          // "binary" (octet-stream) | "multipart" (form-data)
    /// hard byte ceiling for the streamed payload (drives per-op maxBodyBytes)
    maxBytes: Long
    /// content-type allowlist for the stored object (validated, not trusted)
    accept: ContentTypeList
    /// logical bucket/namespace the ObjectStore writes into
    store: String
}

The upload payload member is modeled as a @streaming blob under @httpPayload for the binary mode, or as a dedicated multipart part in multipart mode. The operation's input keeps all its other members as ordinary path/query/header bindings (unchanged), and its output returns an ObjectRef:

structure ObjectRef {
    @required key: String        // opaque storage key the service assigned
    @required size: Long
    @required contentType: String
    checksumSha256: String       // hex; ties into signing (§4)
    etag: String
}

This keeps uploads first-class in the model (discoverable, versioned, documented) rather than an out-of-band side channel. It composes with @requiresAuth, @readonly, pagination, etc., because it only changes how the payload member is transported.

2.B Generated route shape​

When RouteEmitter.emitRoute sees @upload, it emits a different body branch instead of the JSON validator (new branch alongside RouteEmitter.java:256-267):

  • No JSON body validator for the payload member. Path/query/header validators are emitted exactly as today (RouteEmitter.java:244-255) — those still validate.
  • A generated upload preamble that:
    1. reads the payload as a bounded stream (binary: c.req.raw.body; multipart: c.req.parseBody() / formData() for the file part),
    2. validates the declared content-type against @upload.accept (defense-in-depth, since the pipeline allowlist is widened for this route — see §4),
    3. calls the injected ObjectStore.put(...) port, obtaining an ObjectRef,
    4. assembles input with the payload member replaced by the ObjectRef (or the store handle), so the handler receives a reference, not bytes.
  • The MetadataRegistryEmitter gains an upload descriptor and a per-op constraints.maxBodyBytes = @upload.maxBytes (extending the existing constraints emission at MetadataRegistryEmitter.java:93,:134), so the pipeline's per-op body cap and the content-type widening are driven by the model — no hand-editing.

The handler interface (RouteEmitter.emitOperationsInterface, RouteEmitter.java:67-78) is unchanged in shape: the operation still takes a typed input and returns a typed output; only the payload member's TS type becomes ObjectRef/an upload handle rather than a base64 string.

2.C Storage-port interface (data-core)​

Add an ObjectStore port next to DataStore (data-core/src/index.ts), following the established narrow-structural-port convention (ARCH-01) so no adapter imports a cloud SDK into the type surface:

export interface ObjectStore {
  /** Stream bytes to storage under `scope`; returns the assigned ref. */
  put(
    key: string,
    body: ReadableStream<Uint8Array> | Uint8Array,
    meta: { contentType: string; contentLength?: number; checksumSha256?: string },
    scope: DataScope,
  ): Promise<ObjectRef>

  /** Open a stored object for download (streamed back out). */
  get(key: string, scope: DataScope): Promise<ObjectBody | null>

  delete(key: string, scope: DataScope): Promise<boolean>

  /** Optional: presigned direct-to-storage upload/download URL (capability-graded). */
  presign?(
    key: string,
    op: 'put' | 'get',
    opts: { expiresInSeconds: number; contentType?: string },
    scope: DataScope,
  ): Promise<string>
}

Reusing DataScope (data-core/src/index.ts:35) means uploads inherit the same tenant-isolation key-prefixing that DataStore already enforces (tenant A cannot address tenant B's objects). presign? is optional and capability-graded exactly like DataStore.count? (data-core/src/index.ts:145-149) — some backends (S3, R2) support it, others (local FS) do not.

3. Per-target implementation sketch​

Each adapter implements ObjectStore behind its own narrow *Like port, mirroring how dataStore.ts sits behind D1DatabaseLike/KvNamespaceLike (adapter-cf/src/ports.ts).

3.1 Cloudflare — R2 (adapter-cf)​

  • New R2BucketLike structural port ({ put, get, delete, createMultipartUpload? }) that a consumer's real R2Bucket binding satisfies without importing @cloudflare/workers-types (same discipline as KvNamespaceLike, ports.ts:30-38).
  • createR2ObjectStore(bucket: R2BucketLike) implements ObjectStore.put by passing the bounded ReadableStream straight to bucket.put(key, stream, { httpMetadata, sha256 }). R2 accepts a stream, so once the pipeline widens to allow streamed bodies (§4), bytes flow edge→R2 without full buffering. presign → R2 presigned URLs.
  • deploy-cf gains an r2?: R2BindingSpec[] field on BindingsSpec (deploy-cf/src/config.ts:40-44) and a [[r2_buckets]] render block in wrangler.ts alongside the existing kv/DO/d1 blocks (wrangler.ts:64-108), plus provisioning in bin/deploy.ts.

3.2 AWS — S3 via Lambda (adapter-aws)​

  • New S3ClientLike structural port (putObject / getObject / deleteObject / createPresignedPost), satisfied by the AWS SDK v3 client the consumer already injects (same pattern as dynamoPort.ts).
  • createS3ObjectStore(client, bucket) implements put. Caveat: API Gateway + Lambda buffers and base64-encodes the request body (documented in rawBody.ts:76), so true streaming-to-S3 is not available on that path for the payload itself. The recommended AWS mode is therefore presigned PUT (ObjectStore.presign('put', …)): the modeled operation returns a presigned URL and the client uploads directly to S3, bypassing the Lambda body limit entirely. Small uploads may still go through the buffered Lambda path into putObject. This tradeoff should be surfaced in @upload docs.

3.3 Node / k8s — local FS or S3 (adapter-node)​

  • Two implementations behind the same port: createFsObjectStore(rootDir) (streams to a temp file, fsync, rename — atomic; no presign) for single-node/dev, and createS3ObjectStore(...) reused from a shared S3 helper for production k8s (MinIO/S3-compatible). Node's @hono/node-server exposes a real Web ReadableStream body (rawBody.ts runtime table), so streaming-to-disk/S3 works without the Lambda buffering constraint.

All three land bytes via the same ObjectStore.put call the generated route emits — the adapter is the only thing that varies.

4. Composition with body-size, content-type, signing, and auth​

This is the crux: uploads must not punch a hole in the pipeline.

  • Body-size limits. @upload.maxBytes becomes the operation's constraints.maxBodyBytes, which the pipeline already reads per-op (bodyGuards.ts:234, resolveLimit). So an upload route gets a higher, explicit, modeled cap while every other route keeps the global default. The during-read cap in readBoundedBody (bodyGuards.ts:155-187) still applies — an upload over its own maxBytes is 413'd. The change needed: allow the bounded stream to be forwarded to ObjectStore rather than only buffered-then-parsed. Cleanest approach: for upload routes, bodyGuards enforces the byte cap during read but skips the JSON parse + structural walk (bodyGuards.ts:339-356), which are meaningless for binary, and hands the bounded stream/bytes to the route.
  • Content-type. The single-value allowlist (bodyGuards.ts:257,:271) must become per-op: for an @upload route the accepted media types are @upload.accept (binary → application/octet-stream; multipart → multipart/form-data), resolved from the metadata registry the same way maxBodyBytes already is. Non-upload routes are unchanged (still restJson1 application/json). The route-level re-validation in §2.B is defense-in-depth.
  • HMAC body-hash signing. The canonical contract hashes exact body bytes (canonical.ts:49-52). Two paths:
    1. Small/buffered uploads (Lambda path, or files under a "sign-inline" threshold): unchanged — readRawBody buffers and the existing verifier works as-is (verifySignature.ts:331).
    2. Large/streamed uploads: buffering the whole file just to sign it defeats streaming. Proposal: for @upload routes, sign over the client-computed X-SH-Body-Sha256 verified incrementally — the pipeline streams the body through a running SHA-256 while forwarding chunks to ObjectStore, and rejects (401) if the final digest ≠ the signed bodySha256Hex. This preserves the "never trust the client-declared hash" rule (canonical.ts:50-52) without a second full-body buffer. The ObjectRef.checksumSha256 returned to the handler is that same verified digest. Presigned-URL uploads (AWS/CF) move the bytes out of the signed request entirely: the modeled request (which returns the presigned URL) is a normal JSON request signed the ordinary way; the actual bytes go direct-to-storage under the presign's own auth. This is the recommended posture for anything large.
  • Auth / authorization. No change to ordering. @requiresAuth still drives authenticate + the op-tier authorize(OPERATIONS.x) hook, which RouteEmitter already emits after validators and before the handler (RouteEmitter.java:208-216). For streamed uploads the ordering must be: size-cap + content-type gate → authenticate/verifySignature → then stream to ObjectStore, so an unauthenticated caller never causes a write. Because authorize runs before the handler and the handler is what receives the ObjectRef, authZ is enforced before the bytes are durably committed (or, for presign, before a URL is minted).

5. Backwards-compatibility notes​

  • Purely additive at the model layer. Operations without @upload are byte-for-byte unchanged: InputBindings still routes @httpPayload/implicit members to the JSON branch (InputBindings.java:43-53), ZodEmitter.emitBlob still emits base64/streaming string schemas (ZodEmitter.java:310-323), and the pipeline still enforces application/json-only. Existing generated snapshots do not churn.
  • Pipeline config is additive. ValidationConfig (bodyGuards.ts:45-55) gains no required field; the per-op content-type/maxBytes come from the metadata registry, and absent an @upload op the resolved allowlist is exactly today's single protocolContentType. SecurityConfig is untouched for non-upload deployments.
  • ObjectStore is an opt-in injected port, exactly like DataStore/AuditSink/SecretProvider (ARCH-05). A service with no upload operation never constructs one; adapters ship it as a new named export, not a breaking change to existing exports (adapter-cf/src/index.ts).
  • Signing contract is preserved, not forked. The default (buffered) path uses the unchanged canonical string. The streamed path produces the same bodySha256Hex field — it only changes how that digest is obtained (incrementally), so old clients/signers still interoperate on non-upload routes and on small uploads. The canonicalization spec (plan/security/07a-canonicalization-spec.md, referenced at canonical.ts:9) would get an addendum, not a version bump, for the streamed-hash mode.
  • Generated client gains an upload overload (multipart/binary/presigned-follow) only for @upload ops; the JSON bodyExpr path (ClientEmitter.java:275-285) is untouched for everything else.

6. Phased rollout​

  • Phase U0 — Port + convention (no wire change). Land ObjectStore + ObjectRef in data-core, the conformance suite (mirroring data-core/src/conformance.ts), and an in-memory fake. No codegen change. Ships value immediately: handlers can hand-wire uploads against a stable port.
  • Phase U1 — Node/local first. Implement createFsObjectStore + createS3ObjectStore in adapter-node and run them through conformance. Node has the friendliest streaming model (rawBody.ts runtime table), so it de-risks the streaming semantics before touching edge/serverless.
  • Phase U2 — @upload trait + codegen (binary mode, buffered). Add the trait in traits/, the UPLOAD binding branch in InputBindings/RouteEmitter, and per-op maxBodyBytes + content-type widening in MetadataRegistryEmitter. Start with buffered binary uploads so the existing signing path (readRawBody) works unchanged. Golden-file/snapshot tests for the new route shape.
  • Phase U3 — Pipeline streaming + incremental hash. Extend bodyGuards to forward the bounded stream and skip JSON parse for upload routes, and add the streamed SHA-256 verification path to verifySignature. This is the highest-risk change (touches the signing contract) and is gated behind U2's tests.
  • Phase U4 — Cloudflare R2. R2BucketLike + createR2ObjectStore in adapter-cf, r2 binding in deploy-cf config/wrangler.ts/provisioning. Validate on miniflare (there is already a live.miniflare test convention in adapter-cf).
  • Phase U5 — AWS S3 + presigned. S3ClientLike + createS3ObjectStore in adapter-aws, and the presigned-PUT mode end-to-end (the recommended large-upload posture given the API Gateway/Lambda buffering limit noted at rawBody.ts:76). Add presign to the CF and Node stores for parity.
  • Phase U6 — Client + docs. Generated-client upload overloads (multipart/binary/presign-follow) and a canonicalization-spec addendum for the streamed-hash mode.

Key files this design touches (for implementers)​

  • Codegen: src/main/java/com/smithyhono/writers/{InputBindings,HttpBinding,RouteEmitter,ZodEmitter,MetadataRegistryEmitter,ClientEmitter}.java, src/main/java/com/smithyhono/ModelIndex.java, new trait under src/main/java/com/smithyhono/traits/.
  • Pipeline: packages/security-core/src/pipeline/bodyGuards.ts, packages/security-core/src/signing/{rawBody,canonical,verifySignature}.ts, packages/security-core/src/config.ts.
  • Port: packages/data-core/src/index.ts (+ conformance.ts).
  • Adapters: packages/adapter-cf/src/{ports,index}.ts, packages/adapter-aws/src/{port,index}.ts, packages/adapter-node/src/{ports,index}.ts.
  • Deploy: packages/deploy-cf/src/{config,wrangler,bin/deploy}.ts.