Batch #BATCH-94281Đang xử lý (84.2%)Khởi chạy 14:20
Cụm 8x NVIDIA H100 SXM5 80GB•Hiệu suất: 54.2 items / phút
Tổng Số Tác Vụ500Multi-story images
Thành Công40981.8% hoàn thành
Đang Sinh (8 Nodes)7915.8% trong hàng đợi
Lỗi Cần Xử Lý1210 VRAM OOM • 2 Safety
Tiến độ thực thi tổng quan84.2% Hoàn Tất
#410Thành cônggpu-node-h100-04 • 1.2s
#398GPU_OUT_OF_MEMORYgpu-node-h100-02 • VRAM spike 98%
CUDA error: out of memory. Tried to allocate 4.20 GiB (GPU 0; 79.15 GiB total capacity)
#411Đang sinh Denoise [12/25]gpu-node-h100-07
52%#408Thành cônggpu-node-h100-01 • 1.1s
Console Logs (Live Stream)
SSE CONNECTED
[14:20:01.000][INFO] Batch #BATCH-94281 initialized with 500 items.
[14:20:02.150][INFO] Worker cluster active: 8x NVIDIA H100 SXM5 80GB allocated.
[14:25:30.812][INFO] Worker #03 completed item #380 (hash: a3f1c9, 1024x1024, 20 steps, seed: 441029).
[14:27:10.400][WARN] Worker #02 reported VRAM pressure on item #398 (98.2% allocation).
[14:27:11.020][ERROR] Worker #02 CUDA OOM on item #398. Automatic retry #1 queued.
[14:28:02.104][INFO] Worker #04 completed item #409 (hash: 7a8f9c, 1024x1024, 20 steps, seed: 981249).
[14:28:06.180][INFO] Worker #07 started latent denoise pass for #411 [Step 12/25].
Cluster: 8x H100 SXM5 80GBPort: 8000 (gRPC / SSE)