cellbender:remove-background: Command: cellbender remove-background --input=/net/vast-storage/scratch/vast/kellislab/benjames/raw/pbmc_granulocyte_sorted_10k.h5ad --output cellbender.h5 --low-count-threshold 5 --cuda cellbender:remove-background: CellBender 0.3.0 cellbender:remove-background: (Workflow hash f0e598238f) cellbender:remove-background: 2023-12-05 11:21:29 cellbender:remove-background: Running remove-background cellbender:remove-background: Loading data from /net/vast-storage/scratch/vast/kellislab/benjames/raw/pbmc_granulocyte_sorted_10k.h5ad cellbender:remove-background: AnnData format cellbender:remove-background: Using `.X` for count data. cellbender:remove-background: Features in dataset: 36601 Gene Expression cellbender:remove-background: Trimming features for inference. cellbender:remove-background: 29972 features have nonzero counts. cellbender:remove-background: Prior on counts for cells is 3756 cellbender:remove-background: Prior on counts for empty droplets is 98 cellbender:remove-background: Excluding 12767 features that are estimated to have <= 0.1 background counts in cells. cellbender:remove-background: Including 17205 features in the analysis. cellbender:remove-background: Trimming barcodes for inference. cellbender:remove-background: Excluding barcodes with counts below 49 cellbender:remove-background: Using 6069 probable cell barcodes, plus an additional 12241 barcodes, and 57681 empty droplets. cellbender:remove-background: Largest surely-empty droplet has 126 UMI counts. cellbender:remove-background: Attempting to unpack tarball "ckpt.tar.gz" to /tmp/tmp6nu4nbmx cellbender:remove-background: No saved checkpoint. cellbender:remove-background: No checkpoint loaded. cellbender:remove-background: Running inference... cellbender:remove-background: [epoch 001] average training loss: 6245.2039 cellbender:remove-background: [epoch 002] average training loss: 5335.9412 (4.7 seconds per epoch) cellbender:remove-background: Will checkpoint every 89 epochs cellbender:remove-background: [epoch 003] average training loss: 4726.3676 cellbender:remove-background: [epoch 004] average training loss: 4304.6500 cellbender:remove-background: [epoch 005] average training loss: 4020.3913 cellbender:remove-background: [epoch 005] average test loss: 4074.4488 cellbender:remove-background: [epoch 006] average training loss: 3942.8185 cellbender:remove-background: [epoch 007] average training loss: 3882.1089 cellbender:remove-background: [epoch 008] average training loss: 3881.6317 cellbender:remove-background: [epoch 009] average training loss: 3853.9924 cellbender:remove-background: [epoch 010] average training loss: 3838.4773 cellbender:remove-background: [epoch 010] average test loss: 3844.9618 cellbender:remove-background: [epoch 011] average training loss: 3838.5748 cellbender:remove-background: [epoch 012] average training loss: 3826.8834 cellbender:remove-background: [epoch 013] average training loss: 3823.6867 cellbender:remove-background: [epoch 014] average training loss: 3791.8949 cellbender:remove-background: [epoch 015] average training loss: 3699.8284 cellbender:remove-background: [epoch 015] average test loss: 3680.6972 cellbender:remove-background: [epoch 016] average training loss: 3652.2267 cellbender:remove-background: [epoch 017] average training loss: 3647.4564 cellbender:remove-background: [epoch 018] average training loss: 3642.5386 cellbender:remove-background: [epoch 019] average training loss: 3635.3883 cellbender:remove-background: [epoch 020] average training loss: 3626.8211 cellbender:remove-background: [epoch 020] average test loss: 3631.4953 cellbender:remove-background: [epoch 021] average training loss: 3621.3936 cellbender:remove-background: [epoch 022] average training loss: 3610.7437 cellbender:remove-background: [epoch 023] average training loss: 3606.7036 cellbender:remove-background: [epoch 024] average training loss: 3604.0070 cellbender:remove-background: [epoch 025] average training loss: 3600.9665 cellbender:remove-background: [epoch 025] average test loss: 3613.0462 cellbender:remove-background: [epoch 026] average training loss: 3594.6501 cellbender:remove-background: [epoch 027] average training loss: 3591.7902 cellbender:remove-background: [epoch 028] average training loss: 3585.6073 cellbender:remove-background: [epoch 029] average training loss: 3582.1962 cellbender:remove-background: [epoch 030] average training loss: 3577.6049 cellbender:remove-background: [epoch 030] average test loss: 3589.3787 cellbender:remove-background: [epoch 031] average training loss: 3572.6396 cellbender:remove-background: [epoch 032] average training loss: 3566.4552 cellbender:remove-background: [epoch 033] average training loss: 3556.9453 cellbender:remove-background: [epoch 034] average training loss: 3550.2673 cellbender:remove-background: [epoch 035] average training loss: 3545.5523 cellbender:remove-background: [epoch 035] average test loss: 3557.0495 cellbender:remove-background: [epoch 036] average training loss: 3542.9559 cellbender:remove-background: [epoch 037] average training loss: 3540.1738 cellbender:remove-background: [epoch 038] average training loss: 3536.3145 cellbender:remove-background: [epoch 039] average training loss: 3534.8236 cellbender:remove-background: [epoch 040] average training loss: 3532.2206 cellbender:remove-background: [epoch 040] average test loss: 3544.8734 cellbender:remove-background: [epoch 041] average training loss: 3529.6217 cellbender:remove-background: [epoch 042] average training loss: 3528.4603 cellbender:remove-background: [epoch 043] average training loss: 3526.4319 cellbender:remove-background: [epoch 044] average training loss: 3523.2009 cellbender:remove-background: [epoch 045] average training loss: 3520.2710 cellbender:remove-background: [epoch 045] average test loss: 3531.2639 cellbender:remove-background: [epoch 046] average training loss: 3518.3897 cellbender:remove-background: [epoch 047] average training loss: 3515.9635 cellbender:remove-background: [epoch 048] average training loss: 3513.5228 cellbender:remove-background: [epoch 049] average training loss: 3511.0858 cellbender:remove-background: [epoch 050] average training loss: 3509.5055 cellbender:remove-background: [epoch 050] average test loss: 3522.2648 cellbender:remove-background: [epoch 051] average training loss: 3507.7334 cellbender:remove-background: [epoch 052] average training loss: 3505.7836 cellbender:remove-background: [epoch 053] average training loss: 3503.6195 cellbender:remove-background: [epoch 054] average training loss: 3501.7005 cellbender:remove-background: [epoch 055] average training loss: 3500.1393 cellbender:remove-background: [epoch 055] average test loss: 3517.0951 cellbender:remove-background: [epoch 056] average training loss: 3498.0538 cellbender:remove-background: [epoch 057] average training loss: 3498.0112 cellbender:remove-background: [epoch 058] average training loss: 3494.6830 cellbender:remove-background: [epoch 059] average training loss: 3493.7408 cellbender:remove-background: [epoch 060] average training loss: 3492.0124 cellbender:remove-background: [epoch 060] average test loss: 3508.7523 cellbender:remove-background: [epoch 061] average training loss: 3491.1321 cellbender:remove-background: [epoch 062] average training loss: 3492.4780 cellbender:remove-background: [epoch 063] average training loss: 3490.2347 cellbender:remove-background: [epoch 064] average training loss: 3490.5769 cellbender:remove-background: [epoch 065] average training loss: 3490.8367 cellbender:remove-background: [epoch 065] average test loss: 3500.9343 cellbender:remove-background: [epoch 066] average training loss: 3488.9361 cellbender:remove-background: [epoch 067] average training loss: 3488.1708 cellbender:remove-background: [epoch 068] average training loss: 3489.7072 cellbender:remove-background: [epoch 069] average training loss: 3486.3491 cellbender:remove-background: [epoch 070] average training loss: 3484.1980 cellbender:remove-background: [epoch 070] average test loss: 3502.5484 cellbender:remove-background: [epoch 071] average training loss: 3488.9028 cellbender:remove-background: [epoch 072] average training loss: 3484.3890 cellbender:remove-background: [epoch 073] average training loss: 3486.8867 cellbender:remove-background: [epoch 074] average training loss: 3483.7478 cellbender:remove-background: [epoch 075] average training loss: 3482.7552 cellbender:remove-background: [epoch 075] average test loss: 3496.2787 cellbender:remove-background: [epoch 076] average training loss: 3478.5837 cellbender:remove-background: [epoch 077] average training loss: 3479.1508 cellbender:remove-background: [epoch 078] average training loss: 3478.4264 cellbender:remove-background: [epoch 079] average training loss: 3480.4851 cellbender:remove-background: [epoch 080] average training loss: 3478.3454 cellbender:remove-background: [epoch 080] average test loss: 3499.6263 cellbender:remove-background: [epoch 081] average training loss: 3478.4874 cellbender:remove-background: [epoch 082] average training loss: 3475.5848 cellbender:remove-background: [epoch 083] average training loss: 3478.1212 cellbender:remove-background: [epoch 084] average training loss: 3478.2719 cellbender:remove-background: [epoch 085] average training loss: 3473.6307 cellbender:remove-background: [epoch 085] average test loss: 3496.3796 cellbender:remove-background: [epoch 086] average training loss: 3473.9059 cellbender:remove-background: [epoch 087] average training loss: 3478.2153 cellbender:remove-background: [epoch 088] average training loss: 3471.7605 cellbender:remove-background: [epoch 089] average training loss: 3471.7460 cellbender:remove-background: Saving a checkpoint... cellbender:remove-background: Saved checkpoint as /net/vast-storage/scratch/vast/kellislab/benjames/cellbender/pbmc_granulocyte_sorted_10k/ckpt.tar.gz cellbender:remove-background: [epoch 090] average training loss: 3470.5014 cellbender:remove-background: [epoch 090] average test loss: 3489.8384 cellbender:remove-background: [epoch 091] average training loss: 3470.9893 cellbender:remove-background: [epoch 092] average training loss: 3470.4666 cellbender:remove-background: [epoch 093] average training loss: 3472.8550 cellbender:remove-background: [epoch 094] average training loss: 3468.8221 cellbender:remove-background: [epoch 095] average training loss: 3465.9232 cellbender:remove-background: [epoch 095] average test loss: 3491.0120 cellbender:remove-background: [epoch 096] average training loss: 3466.7866 cellbender:remove-background: [epoch 097] average training loss: 3469.1800 cellbender:remove-background: [epoch 098] average training loss: 3466.5437 cellbender:remove-background: [epoch 099] average training loss: 3463.6385 cellbender:remove-background: [epoch 100] average training loss: 3465.6179 cellbender:remove-background: [epoch 100] average test loss: 3484.9758 cellbender:remove-background: [epoch 101] average training loss: 3461.3223 cellbender:remove-background: [epoch 102] average training loss: 3462.6019 cellbender:remove-background: [epoch 103] average training loss: 3462.0358 cellbender:remove-background: [epoch 104] average training loss: 3460.1336 cellbender:remove-background: [epoch 105] average training loss: 3460.1466 cellbender:remove-background: [epoch 105] average test loss: 3481.5633 cellbender:remove-background: [epoch 106] average training loss: 3460.6314 cellbender:remove-background: [epoch 107] average training loss: 3462.6026 cellbender:remove-background: [epoch 108] average training loss: 3457.6737 cellbender:remove-background: [epoch 109] average training loss: 3458.7869 cellbender:remove-background: [epoch 110] average training loss: 3459.3092 cellbender:remove-background: [epoch 110] average test loss: 3494.3444 cellbender:remove-background: [epoch 111] average training loss: 3459.7881 cellbender:remove-background: [epoch 112] average training loss: 3460.5990 cellbender:remove-background: [epoch 113] average training loss: 3459.7100 cellbender:remove-background: [epoch 114] average training loss: 3456.2765 cellbender:remove-background: [epoch 115] average training loss: 3457.0486 cellbender:remove-background: [epoch 115] average test loss: 3481.9750 cellbender:remove-background: [epoch 116] average training loss: 3457.3899 cellbender:remove-background: [epoch 117] average training loss: 3459.0252 cellbender:remove-background: [epoch 118] average training loss: 3456.3971 cellbender:remove-background: [epoch 119] average training loss: 3457.0670 cellbender:remove-background: [epoch 120] average training loss: 3454.4341 cellbender:remove-background: [epoch 120] average test loss: 3478.4955 cellbender:remove-background: [epoch 121] average training loss: 3454.9422 cellbender:remove-background: [epoch 122] average training loss: 3456.7048 cellbender:remove-background: [epoch 123] average training loss: 3456.9594 cellbender:remove-background: [epoch 124] average training loss: 3455.8315 cellbender:remove-background: [epoch 125] average training loss: 3453.6457 cellbender:remove-background: [epoch 125] average test loss: 3487.8577 cellbender:remove-background: [epoch 126] average training loss: 3456.0411 cellbender:remove-background: [epoch 127] average training loss: 3452.8060 cellbender:remove-background: [epoch 128] average training loss: 3454.3284 cellbender:remove-background: [epoch 129] average training loss: 3453.2383 cellbender:remove-background: [epoch 130] average training loss: 3452.4292 cellbender:remove-background: [epoch 130] average test loss: 3477.4156 cellbender:remove-background: [epoch 131] average training loss: 3451.7754 cellbender:remove-background: [epoch 132] average training loss: 3451.2123 cellbender:remove-background: [epoch 133] average training loss: 3452.8150 cellbender:remove-background: [epoch 134] average training loss: 3450.4944 cellbender:remove-background: [epoch 135] average training loss: 3452.7905 cellbender:remove-background: [epoch 135] average test loss: 3479.3774 cellbender:remove-background: [epoch 136] average training loss: 3452.0012 cellbender:remove-background: [epoch 137] average training loss: 3451.6656 cellbender:remove-background: [epoch 138] average training loss: 3452.1855 cellbender:remove-background: [epoch 139] average training loss: 3452.8406 cellbender:remove-background: [epoch 140] average training loss: 3452.3515 cellbender:remove-background: [epoch 140] average test loss: 3482.1101 cellbender:remove-background: [epoch 141] average training loss: 3451.3181 cellbender:remove-background: [epoch 142] average training loss: 3450.6421 cellbender:remove-background: [epoch 143] average training loss: 3452.1037 cellbender:remove-background: [epoch 144] average training loss: 3451.9165 cellbender:remove-background: [epoch 145] average training loss: 3450.5817 cellbender:remove-background: [epoch 145] average test loss: 3479.3682 cellbender:remove-background: [epoch 146] average training loss: 3451.6374 cellbender:remove-background: [epoch 147] average training loss: 3452.1351 cellbender:remove-background: [epoch 148] average training loss: 3452.7384 cellbender:remove-background: [epoch 149] average training loss: 3451.0595 cellbender:remove-background: [epoch 150] average training loss: 3452.2634 cellbender:remove-background: [epoch 150] average test loss: 3479.4066 cellbender:remove-background: Saving a checkpoint... cellbender:remove-background: Saved checkpoint as /net/vast-storage/scratch/vast/kellislab/benjames/cellbender/pbmc_granulocyte_sorted_10k/ckpt.tar.gz cellbender:remove-background: 2023-12-05 11:35:28 cellbender:remove-background: Inference procedure complete. cellbender:remove-background: Attempting to unpack tarball "ckpt.tar.gz" to /tmp/tmpcaz7jotc cellbender:remove-background: Successfully unpacked tarball to /tmp/tmpcaz7jotc /tmp/tmpcaz7jotc/f0e598238f_random.pyro /tmp/tmpcaz7jotc/f0e598238f_random.cuda /tmp/tmpcaz7jotc/f0e598238f_model.torch /tmp/tmpcaz7jotc/f0e598238f_optim.torch /tmp/tmpcaz7jotc/f0e598238f_optim.pyro /tmp/tmpcaz7jotc/f0e598238f_params.pyro /tmp/tmpcaz7jotc/f0e598238f_train.loaderstate /tmp/tmpcaz7jotc/f0e598238f_test.loaderstate /tmp/tmpcaz7jotc/f0e598238f_args.npy cellbender:remove-background: Posterior not currently included in checkpoint. cellbender:remove-background: Computing posterior noise count probabilities in mini-batches. cellbender:remove-background: Working on chunk (1/96) cellbender:remove-background: [0.04 mins per chunk] cellbender:remove-background: Working on chunk (2/96) cellbender:remove-background: Working on chunk (3/96) cellbender:remove-background: Working on chunk (4/96) cellbender:remove-background: Working on chunk (5/96) cellbender:remove-background: Working on chunk (6/96) cellbender:remove-background: Working on chunk (7/96) cellbender:remove-background: Working on chunk (8/96) cellbender:remove-background: Working on chunk (9/96) cellbender:remove-background: Working on chunk (10/96) cellbender:remove-background: Working on chunk (11/96) cellbender:remove-background: Working on chunk (12/96) cellbender:remove-background: Working on chunk (13/96) cellbender:remove-background: Working on chunk (14/96) cellbender:remove-background: Working on chunk (15/96) cellbender:remove-background: Working on chunk (16/96) cellbender:remove-background: Working on chunk (17/96) cellbender:remove-background: Working on chunk (18/96) cellbender:remove-background: Working on chunk (19/96) cellbender:remove-background: Working on chunk (20/96) cellbender:remove-background: Working on chunk (21/96) cellbender:remove-background: Working on chunk (22/96) cellbender:remove-background: Working on chunk (23/96) cellbender:remove-background: Working on chunk (24/96) cellbender:remove-background: Working on chunk (25/96) cellbender:remove-background: Working on chunk (26/96) cellbender:remove-background: Working on chunk (27/96) cellbender:remove-background: Working on chunk (28/96) cellbender:remove-background: Working on chunk (29/96) cellbender:remove-background: Working on chunk (30/96) cellbender:remove-background: Working on chunk (31/96) cellbender:remove-background: Working on chunk (32/96) cellbender:remove-background: Working on chunk (33/96) cellbender:remove-background: Working on chunk (34/96) cellbender:remove-background: Working on chunk (35/96) cellbender:remove-background: Working on chunk (36/96) cellbender:remove-background: Working on chunk (37/96) cellbender:remove-background: Working on chunk (38/96) cellbender:remove-background: Working on chunk (39/96) cellbender:remove-background: Working on chunk (40/96) cellbender:remove-background: Working on chunk (41/96) cellbender:remove-background: Working on chunk (42/96) cellbender:remove-background: Working on chunk (43/96) cellbender:remove-background: Working on chunk (44/96) cellbender:remove-background: Working on chunk (45/96) cellbender:remove-background: Working on chunk (46/96) cellbender:remove-background: Working on chunk (47/96) cellbender:remove-background: Working on chunk (48/96) cellbender:remove-background: Working on chunk (49/96) cellbender:remove-background: Working on chunk (50/96) cellbender:remove-background: Working on chunk (51/96) cellbender:remove-background: Working on chunk (52/96) cellbender:remove-background: Working on chunk (53/96) cellbender:remove-background: Working on chunk (54/96) cellbender:remove-background: Working on chunk (55/96) cellbender:remove-background: Working on chunk (56/96) cellbender:remove-background: Working on chunk (57/96) cellbender:remove-background: Working on chunk (58/96) cellbender:remove-background: Working on chunk (59/96) cellbender:remove-background: Working on chunk (60/96) cellbender:remove-background: Working on chunk (61/96) cellbender:remove-background: Working on chunk (62/96) cellbender:remove-background: Working on chunk (63/96) cellbender:remove-background: Working on chunk (64/96) cellbender:remove-background: Working on chunk (65/96) cellbender:remove-background: Working on chunk (66/96) cellbender:remove-background: Working on chunk (67/96) cellbender:remove-background: Working on chunk (68/96) cellbender:remove-background: Working on chunk (69/96) cellbender:remove-background: Working on chunk (70/96) cellbender:remove-background: Working on chunk (71/96) cellbender:remove-background: Working on chunk (72/96) cellbender:remove-background: Working on chunk (73/96) cellbender:remove-background: Working on chunk (74/96) cellbender:remove-background: Working on chunk (75/96) cellbender:remove-background: Working on chunk (76/96) cellbender:remove-background: Working on chunk (77/96) cellbender:remove-background: Working on chunk (78/96) cellbender:remove-background: Working on chunk (79/96) cellbender:remove-background: Working on chunk (80/96) cellbender:remove-background: Working on chunk (81/96) cellbender:remove-background: Working on chunk (82/96) cellbender:remove-background: Working on chunk (83/96) cellbender:remove-background: Working on chunk (84/96) cellbender:remove-background: Working on chunk (85/96) cellbender:remove-background: Working on chunk (86/96) cellbender:remove-background: Working on chunk (87/96) cellbender:remove-background: Working on chunk (88/96) cellbender:remove-background: Working on chunk (89/96) cellbender:remove-background: Working on chunk (90/96) cellbender:remove-background: Working on chunk (91/96) cellbender:remove-background: Working on chunk (92/96) cellbender:remove-background: Working on chunk (93/96) cellbender:remove-background: Working on chunk (94/96) cellbender:remove-background: Working on chunk (95/96) cellbender:remove-background: Working on chunk (96/96) cellbender:remove-background: Writing full posterior to cellbender_posterior.h5 cellbender:remove-background: Succeeded in writing posterior to file cellbender_posterior.h5 cellbender:remove-background: Added posterior object to checkpoint file. cellbender:remove-background: 2023-12-05 11:41:55 cellbender:remove-background: Saved summary plots as cellbender.pdf cellbender:remove-background: Saved cell barcodes in cellbender_cell_barcodes.csv cellbender:remove-background: Computing target noise counts per gene for MCKP estimator cellbender:remove-background: Using MCKP noise targets computed for FPR 0.01 cellbender:remove-background: Computing denoised counts using mckp estimator cellbender:remove-background: Working on chunk (1/7) cellbender:remove-background: [2.05 mins per chunk] cellbender:remove-background: Working on chunk (2/7) cellbender:remove-background: Working on chunk (3/7) cellbender:remove-background: Working on chunk (4/7) cellbender:remove-background: Working on chunk (5/7) cellbender:remove-background: Working on chunk (6/7) cellbender:remove-background: Working on chunk (7/7) cellbender:remove-background: 2023-12-05 11:57:08 Total MCKP estimation time = 872.02 sec cellbender:remove-background: Succeeded in writing CellRanger format output to file cellbender.h5 cellbender:remove-background: Succeeded in writing CellRanger format output to file cellbender_filtered.h5 cellbender:remove-background: Saved output metrics as cellbender_metrics.csv cellbender:remove-background: Succeeded in writing report to cellbender_report.html cellbender:remove-background: Completed remove-background. cellbender:remove-background: 2023-12-05 11:57:55