[Bug]: Downloading only working if Garage configured as an external site

[arcoast]

What happened?

If I configure Garage as an external site with a reverse proxy downloading works properly

S3_ENDPOINT=http://typetype-garage:3900
S3_PUBLIC_ENDPOINT=https://typetype-garage.${DOMAIN}

Reading the docs here it appears this shouldn't be necessary

"The supported stack serves artifacts through /api/downloader/.... Server follows the internal Garage redirect and streams the result, so a normal deployment does not need a second public hostname for Garage.

Only expose Garage separately when a deliberate custom Downloader configuration uses a public S3 endpoint. In that topology, protect the endpoint according to the object store's documentation and keep the access credentials private."

However, if I run the stack with the following

S3_ENDPOINT=http://typetype-garage:3900
S3_PUBLIC_ENDPOINT=http://typetype-garage:3900

and attempt to download a video the browser attempts to access the following URL

http://typetype-garage:3900/typetype-downloads/artifacts/b2422146efe47523bbfa4538/Linux_Kernel_Devs_Keep_Removing_Ancient_Drivers.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GK3301521df0d4dd1f6debd319%2F20260812%2Fgarage%2Fs3%2Faws4_request&X-Amz-Date=20260812T091219Z&X-Amz-Expires=7200&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3D%22Linux_Kernel_Devs_Keep_Removing_Ancient_Drivers.mp4%22&X-Amz-Signature=26d926215c2e032e90d423bef5d8170647e4b80364297b76eb6efc11d711dddc

No errors in the browser console.

It appears that the routing to /api/downloader/.... isn't working properly for me.

I can however get a response from

# curl -fsS https://typetype.DOMAIN.COM/api/version/downloader
{"buildTime":"2026-08-11T10:46:01Z","revision":"68624aad523dca3b886efdceb98c934432fbdca7","service":"downloader","shortRevision":"68624aad523d","version":"1.5.1"}

although I recognise this is a slightly different endpoint, but thought it might be relevant.

There are no errors in typetype-downloader logs

# dtail typetype-downloader
2026-08-12T08:53:07.936800294Z 2026/08/12 08:53:07 INFO server listening addr=:18093

The typetype-server does however show some errors which I think might be relevant

2026-08-12T09:18:36.082107779Z 09:18:36.081 [eventLoopGroupProxy-7-4] INFO  Application - requestId=4643eda2-e261-43be-b3b1-bef1ce96c0be method=GET path=/sabr/playback/85PQ4Mm2QN9QsNazJXXKf_QHOdcpN5_ww-D_hjBzHuY/137/segment/6 status=200 durationMs=0
2026-08-12T09:18:36.314657780Z 09:18:36.314 [DefaultDispatcher-worker-21] INFO  d.t.s.services.SabrDownloadStreamer - sabr_download videoId=0zn1pGRgEoo part=1/4 tracks=2 durationMs=872 pumps=9 upstreamBytes=34468756 peakResponseBytes=5614916
2026-08-12T09:18:36.317835455Z 09:18:36.317 [eventLoopGroupProxy-7-10] INFO  Application - requestId=edbc843a-e541-4e76-b872-ca00e44b2472 method=GET path=/sabr/download/0zn1pGRgEoo status=200 durationMs=1084
2026-08-12T09:18:36.468621108Z YT_PERF videoId=0zn1pGRgEoo stage=ejs.batch.decode durationMs=35
2026-08-12T09:18:36.471630510Z 09:18:36.470 [DefaultDispatcher-worker-6] INFO  d.t.server.services.SabrInfoFetcher - sabr_info_fetch videoId=0zn1pGRgEoo startTimeMs=0 source=isolated_network elapsedMs=149
2026-08-12T09:18:36.529554967Z 09:18:36.528 [eventLoopGroupProxy-7-5] INFO  Application - requestId=05f027ad-936e-4a31-a98c-f7d52a428248 method=POST path=/sessions/playback/start status=204 durationMs=3
2026-08-12T09:18:36.549219538Z 09:18:36.549 [eventLoopGroupProxy-7-6] INFO  Application - requestId=43a6088c-08de-4029-a90e-72a930f7369c method=GET path=/proxy status=200 durationMs=15
2026-08-12T09:18:36.716913531Z 09:18:36.716 [DefaultDispatcher-worker-9] INFO  d.t.s.services.SabrDownloadStreamer - sabr_download videoId=0zn1pGRgEoo part=2/4 tracks=2 durationMs=1324 pumps=9 upstreamBytes=34222319 peakResponseBytes=5296099
2026-08-12T09:18:36.717078178Z 09:18:36.717 [eventLoopGroupProxy-7-11] INFO  Application - requestId=e44beda9-6e0c-43b6-9f2f-f70dfb7cfd48 method=GET path=/sabr/download/0zn1pGRgEoo status=200 durationMs=1483
2026-08-12T09:18:37.282665010Z 09:18:37.282 [DefaultDispatcher-worker-26] INFO  d.t.s.services.SabrDownloadStreamer - sabr_download videoId=0zn1pGRgEoo part=3/4 tracks=2 durationMs=1855 pumps=9 upstreamBytes=31723156 peakResponseBytes=5341987
2026-08-12T09:18:37.284124562Z 09:18:37.283 [eventLoopGroupProxy-7-12] INFO  Application - requestId=7e5a0ae3-f2af-43c0-9244-d0be94a13203 method=GET path=/sabr/download/0zn1pGRgEoo status=200 durationMs=2049
2026-08-12T09:18:37.377391473Z 09:18:37.377 [eventLoopGroupProxy-7-7] INFO  Application - requestId=5ebf4cde-b428-4460-b72a-f4234df57331 method=GET path=/downloader/jobs/{id} status=200 durationMs=2
2026-08-12T09:18:37.649214751Z 09:18:37.649 [DefaultDispatcher-worker-6] INFO  d.t.s.services.SabrDownloadStreamer - sabr_download videoId=0zn1pGRgEoo part=4/4 tracks=2 durationMs=1176 pumps=10 upstreamBytes=35350477 peakResponseBytes=5315054
2026-08-12T09:18:37.649604043Z 09:18:37.649 [eventLoopGroupProxy-7-10] INFO  Application - requestId=62b0513c-28cf-4e9e-bf0d-e78bcdd674d3 method=GET path=/sabr/download/0zn1pGRgEoo status=200 durationMs=1331
2026-08-12T09:18:39.129653143Z 09:18:39.129 [eventLoopGroupProxy-7-8] INFO  Application - requestId=d92f9947-04a1-4617-b7b0-1a29ab187324 method=GET path=/downloader/jobs/{id} status=200 durationMs=2
2026-08-12T09:18:39.370358007Z 09:18:39.370 [eventLoopGroupProxy-7-9] INFO  Application - requestId=fcf8427e-dea8-4079-9141-356e0510329c method=GET path=/downloader/jobs/{id}/artifact status=302 durationMs=1
2026-08-12T09:18:40.345600267Z 09:18:40.345 [eventLoopGroupProxy-7-10] INFO  Application - requestId=c769284e-ad8c-4454-b2d8-3ac99cfce9b7 method=PUT path=/settings status=200 durationMs=12
2026-08-12T09:18:45.720189751Z 09:18:45.719 [eventLoopGroupProxy-7-1] ERROR RequestLogger - Unhandled exception requestId=18a5ad26-8cd1-473c-8fc1-49242b17be0d path=/downloader/jobs/39705bff72571330b1cf7013/events
2026-08-12T09:18:45.720268313Z io.ktor.util.cio.ChannelWriteException: Cannot write to channel
2026-08-12T09:18:45.720280345Z  at io.ktor.server.engine.BaseApplicationResponse.respondWriteChannelContent$suspendImpl(BaseApplicationResponse.kt:191)
2026-08-12T09:18:45.720288331Z  at io.ktor.server.engine.BaseApplicationResponse$respondWriteChannelContent$1.invokeSuspend(BaseApplicationResponse.kt)
2026-08-12T09:18:45.720296050Z  at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:34)
2026-08-12T09:18:45.720302366Z  at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:98)
2026-08-12T09:18:45.720309875Z  at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:148)
2026-08-12T09:18:45.720317453Z  at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:141)
2026-08-12T09:18:45.720324560Z  at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:535)
2026-08-12T09:18:45.720331317Z  at io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:201)
2026-08-12T09:18:45.720337813Z  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1204)
2026-08-12T09:18:45.720344483Z  at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2026-08-12T09:18:45.720351129Z  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2026-08-12T09:18:45.720357688Z  at java.base/java.lang.Thread.run(Unknown Source)
2026-08-12T09:18:45.720363845Z Caused by: io.ktor.utils.io.ClosedWriteChannelException: Cannot write to channel
2026-08-12T09:18:45.720370959Z  at io.ktor.utils.io.ByteChannel.getWriteBuffer(ByteChannel.kt:76)
2026-08-12T09:18:45.720377842Z  at io.ktor.utils.io.ByteWriteChannelOperationsKt.writeFully(ByteWriteChannelOperations.kt:85)
2026-08-12T09:18:45.720385003Z  at io.ktor.utils.io.jvm.javaio.BlockingKt$toOutputStream$1$write$2.invokeSuspend(Blocking.kt:63)
2026-08-12T09:18:45.720391963Z  at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:34)
2026-08-12T09:18:45.720398759Z  at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
2026-08-12T09:18:45.720405362Z  at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:256)
2026-08-12T09:18:45.720412656Z  at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:54)
2026-08-12T09:18:45.720420957Z  at kotlinx.coroutines.BuildersKt__BuildersKt.runBlockingImpl(Builders.kt:30)
2026-08-12T09:18:45.720428338Z  at kotlinx.coroutines.BuildersKt.runBlockingImpl(Unknown Source)
2026-08-12T09:18:45.720435998Z  at kotlinx.coroutines.BuildersKt__Builders_concurrentKt.runBlockingK(Builders.concurrent.kt:172)
2026-08-12T09:18:45.720442538Z  at kotlinx.coroutines.BuildersKt.runBlockingK(Unknown Source)
2026-08-12T09:18:45.720449818Z  at kotlinx.coroutines.BuildersKt__Builders_concurrentKt.runBlockingK$default(Builders.concurrent.kt:157)
2026-08-12T09:18:45.720464917Z  at kotlinx.coroutines.BuildersKt.runBlockingK$default(Unknown Source)
2026-08-12T09:18:45.720472977Z  at io.ktor.utils.io.jvm.javaio.BlockingKt$toOutputStream$1.write(Blocking.kt:63)
2026-08-12T09:18:45.720480769Z  at dev.typetype.server.routes.DownloaderGatewaySseProxyKt$forwardDownloaderSseRequest$2$2$1.invokeSuspend(DownloaderGatewaySseProxy.kt:46)
2026-08-12T09:18:45.720487840Z  at dev.typetype.server.routes.DownloaderGatewaySseProxyKt$forwardDownloaderSseRequest$2$2$1.invoke(DownloaderGatewaySseProxy.kt)
2026-08-12T09:18:45.720495409Z  at dev.typetype.server.routes.DownloaderGatewaySseProxyKt$forwardDownloaderSseRequest$2$2$1.invoke(DownloaderGatewaySseProxy.kt)
2026-08-12T09:18:45.720504875Z  at io.ktor.http.content.OutputStreamContent$writeTo$$inlined$withBlockingOutputStream$default$1.invokeSuspend(BlockingBridge.kt:47)
2026-08-12T09:18:45.720513105Z  at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:34)
2026-08-12T09:18:45.720520469Z  at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
2026-08-12T09:18:45.720527447Z  at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:124)
2026-08-12T09:18:45.720534588Z  at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)
2026-08-12T09:18:45.720541029Z  at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
2026-08-12T09:18:45.720548632Z  at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:798)
2026-08-12T09:18:45.720556484Z  at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
2026-08-12T09:18:45.720564054Z  at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)
2026-08-12T09:18:45.720571552Z Caused by: io.ktor.util.cio.ChannelWriteException: Cannot write to channel
2026-08-12T09:18:45.720578297Z  at io.ktor.server.netty.cio.NettyHttpResponsePipeline.respondWithFailure(NettyHttpResponsePipeline.kt:123)
2026-08-12T09:18:45.720603133Z  at io.ktor.server.netty.cio.NettyHttpResponsePipeline.respondWithBodyAndTrailerMessage(NettyHttpResponsePipeline.kt:288)
2026-08-12T09:18:45.720612964Z  at io.ktor.server.netty.cio.NettyHttpResponsePipeline.access$respondWithBodyAndTrailerMessage(NettyHttpResponsePipeline.kt:44)
2026-08-12T09:18:45.720620817Z  at io.ktor.server.netty.cio.NettyHttpResponsePipeline$respondWithBodyAndTrailerMessage$1.invokeSuspend(NettyHttpResponsePipeline.kt)
2026-08-12T09:18:45.720627655Z  at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:34)
2026-08-12T09:18:45.720633769Z  at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
2026-08-12T09:18:45.720640427Z  at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:148)
2026-08-12T09:18:45.720647851Z  at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:141)
2026-08-12T09:18:45.720665709Z  at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:535)
2026-08-12T09:18:45.720673224Z  at io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:201)
2026-08-12T09:18:45.720680347Z  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1204)
2026-08-12T09:18:45.720687754Z  at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2026-08-12T09:18:45.720694802Z  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2026-08-12T09:18:45.720707067Z  at java.base/java.lang.Thread.run(Unknown Source)
2026-08-12T09:18:45.720714626Z Caused by: io.netty.channel.StacklessClosedChannelException
2026-08-12T09:18:45.720722799Z  at io.netty.channel.AbstractChannel$AbstractUnsafe.write(Object, ChannelPromise)(Unknown Source)
2026-08-12T09:18:45.720730208Z 09:18:45.720 [eventLoopGroupProxy-7-1] INFO  Application - requestId=18a5ad26-8cd1-473c-8fc1-49242b17be0d method=GET path=/downloader/jobs/{id}/events status=500 durationMs=10104

The typetype container logs I don't think show anything relevant

2026-08-12T09:21:35.985286253Z 10.57.133.253 - - [12/Aug/2026:09:21:35 +0000] "GET /api/sabr/playback/BqndFLFhH1Ft2mUHv2nZu72jSi3HAcCHc01ZOToGk8Y/137/segment/5?generation=0 HTTP/1.1" 200 758212 "https://typetype.DOMAIN.COM/watch?v=W7LAIODVYA0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36" "10.57.0.254"
2026-08-12T09:21:36.405102324Z 10.57.133.253 - - [12/Aug/2026:09:21:36 +0000] "GET /api/sabr/playback/BqndFLFhH1Ft2mUHv2nZu72jSi3HAcCHc01ZOToGk8Y/137/segment/6?generation=0 HTTP/1.1" 200 1099384 "https://typetype.DOMAIN.COM/watch?v=W7LAIODVYA0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36" "10.57.0.254"
2026-08-12T09:21:36.730318232Z 127.0.0.1 - - [12/Aug/2026:09:21:36 +0000] "GET /api/version/web HTTP/1.1" 200 156 "-" "Wget" "-"
2026-08-12T09:21:36.916727961Z 10.57.133.253 - - [12/Aug/2026:09:21:36 +0000] "GET /api/sabr/playback/BqndFLFhH1Ft2mUHv2nZu72jSi3HAcCHc01ZOToGk8Y/137/segment/7?generation=0 HTTP/1.1" 200 526475 "https://typetype.DOMAIN.COM/watch?v=W7LAIODVYA0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36" "10.57.0.254"
2026-08-12T09:21:37.193406391Z 10.57.133.253 - - [12/Aug/2026:09:21:37 +0000] "POST /api/sessions/playback/start HTTP/1.1" 204 0 "https://typetype.DOMAIN.COM/watch?v=W7LAIODVYA0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36" "10.57.0.254"
2026-08-12T09:21:37.216631389Z 10.57.133.253 - - [12/Aug/2026:09:21:37 +0000] "GET /api/proxy?url=https%3A%2F%2Fi.ytimg.com%2Fvi%2FW7LAIODVYA0%2Fhq720.jpg%3Fsqp%3D-oaymwEhCK4FEIIDSFryq4qpAxMIARUAAAAAGAElAADIQj0AgKJD%26rs%3DAOn4CLBAUnwbjPvJFvjSBj2KNC3bqQd0Ew HTTP/1.1" 200 91945 "https://typetype.DOMAIN.COM/watch?v=W7LAIODVYA0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36" "10.57.0.254"
2026-08-12T09:21:37.504699858Z 10.57.133.253 - - [12/Aug/2026:09:21:37 +0000] "GET /api/downloader/jobs/ad5c0d9dfbb808a8e7dff43f HTTP/1.1" 200 455 "https://typetype.DOMAIN.COM/watch?v=W7LAIODVYA0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36" "10.57.0.254"
2026-08-12T09:21:39.290407845Z 10.57.133.253 - - [12/Aug/2026:09:21:39 +0000] "GET /api/downloader/jobs/ad5c0d9dfbb808a8e7dff43f HTTP/1.1" 200 433 "https://typetype.DOMAIN.COM/watch?v=W7LAIODVYA0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36" "10.57.0.254"
2026-08-12T09:21:39.886796509Z 10.57.133.253 - - [12/Aug/2026:09:21:39 +0000] "GET /api/downloader/jobs/ad5c0d9dfbb808a8e7dff43f/events HTTP/1.1" 200 8116 "https://typetype.DOMAIN.COM/watch?v=W7LAIODVYA0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36" "10.57.0.254"
2026-08-12T09:21:40.132261898Z 10.57.133.253 - - [12/Aug/2026:09:21:40 +0000] "GET /api/downloader/jobs/ad5c0d9dfbb808a8e7dff43f/artifact HTTP/1.1" 302 570 "https://typetype.DOMAIN.COM/watch?v=W7LAIODVYA0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36" "10.57.0.254"
2026-08-12T09:21:41.084641120Z 10.57.133.253 - - [12/Aug/2026:09:21:41 +0000] "PUT /api/settings HTTP/1.1" 200 622 "https://typetype.DOMAIN.COM/watch?v=W7LAIODVYA0" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36" "10.57.0.254"

Steps to reproduce

As above

Area

Downloader

Channel

main (stable)

Anything else?

It might be that I'm completely misreading and misunderstanding the docs and for my use case I should be using a garage reverse proxy.