cellbender:remove-background: Command: cellbender remove-background --input /om2/user/benjames/raw/human_brain_3k.h5ad --output cellbender.h5 --low-count-threshold 5 --cuda cellbender:remove-background: CellBender 0.3.0 cellbender:remove-background: (Workflow hash d4ef6573dc) cellbender:remove-background: 2023-12-05 20:54:21 cellbender:remove-background: Running remove-background cellbender:remove-background: Loading data from /om2/user/benjames/raw/human_brain_3k.h5ad cellbender:remove-background: AnnData format cellbender:remove-background: Using `.X` for count data. cellbender:remove-background: Features in dataset: 36601 Gene Expression cellbender:remove-background: Trimming features for inference. cellbender:remove-background: 31527 features have nonzero counts. cellbender:remove-background: Prior on counts for cells is 7048 cellbender:remove-background: Prior on counts for empty droplets is 14 cellbender:remove-background: Excluding 15025 features that are estimated to have <= 0.1 background counts in cells. cellbender:remove-background: Including 16502 features in the analysis. cellbender:remove-background: Trimming barcodes for inference. cellbender:remove-background: Excluding barcodes with counts below 7 cellbender:remove-background: Using 1719 probable cell barcodes, plus an additional 6440 barcodes, and 43071 empty droplets. cellbender:remove-background: Largest surely-empty droplet has 18 UMI counts. cellbender:remove-background: Attempting to unpack tarball "ckpt.tar.gz" to /tmp/tmpsoavcg10 cellbender:remove-background: No saved checkpoint. cellbender:remove-background: No checkpoint loaded. cellbender:remove-background: Running inference... cellbender:remove-background: [epoch 001] average training loss: 9421.8953 cellbender:remove-background: [epoch 002] average training loss: 8365.7674 (2.4 seconds per epoch) cellbender:remove-background: Will not checkpoint due to projected run completion in under 7.0 min cellbender:remove-background: [epoch 003] average training loss: 7770.0120 cellbender:remove-background: [epoch 004] average training loss: 6674.9113 cellbender:remove-background: [epoch 005] average training loss: 6179.7925 cellbender:remove-background: [epoch 005] average test loss: 6995.0687 cellbender:remove-background: [epoch 006] average training loss: 5630.3922 cellbender:remove-background: [epoch 007] average training loss: 5456.5508 cellbender:remove-background: [epoch 008] average training loss: 4987.2090 cellbender:remove-background: [epoch 009] average training loss: 4780.5033 cellbender:remove-background: [epoch 010] average training loss: 4460.2830 cellbender:remove-background: [epoch 010] average test loss: 4578.8601 cellbender:remove-background: [epoch 011] average training loss: 4393.0149 cellbender:remove-background: [epoch 012] average training loss: 4359.9656 cellbender:remove-background: [epoch 013] average training loss: 4272.2469 cellbender:remove-background: [epoch 014] average training loss: 4144.7282 cellbender:remove-background: [epoch 015] average training loss: 4087.8661 cellbender:remove-background: [epoch 015] average test loss: 4270.2753 cellbender:remove-background: [epoch 016] average training loss: 4088.0304 cellbender:remove-background: [epoch 017] average training loss: 4087.8990 cellbender:remove-background: [epoch 018] average training loss: 4046.0746 cellbender:remove-background: [epoch 019] average training loss: 3757.8872 cellbender:remove-background: [epoch 020] average training loss: 3727.3619 cellbender:remove-background: [epoch 020] average test loss: 4018.3126 cellbender:remove-background: [epoch 021] average training loss: 3697.0241 cellbender:remove-background: [epoch 022] average training loss: 3667.4991 cellbender:remove-background: [epoch 023] average training loss: 3639.7000 cellbender:remove-background: [epoch 024] average training loss: 3631.2596 cellbender:remove-background: [epoch 025] average training loss: 3604.5991 cellbender:remove-background: [epoch 025] average test loss: 3830.9796 cellbender:remove-background: [epoch 026] average training loss: 3618.9747 cellbender:remove-background: [epoch 027] average training loss: 3576.6604 cellbender:remove-background: [epoch 028] average training loss: 3565.6209 cellbender:remove-background: [epoch 029] average training loss: 3556.7193 cellbender:remove-background: [epoch 030] average training loss: 3574.3290 cellbender:remove-background: [epoch 030] average test loss: 3624.3006 cellbender:remove-background: [epoch 031] average training loss: 3549.2925 cellbender:remove-background: [epoch 032] average training loss: 3537.7752 cellbender:remove-background: [epoch 033] average training loss: 3511.9332 cellbender:remove-background: [epoch 034] average training loss: 3523.1212 cellbender:remove-background: [epoch 035] average training loss: 3484.9293 cellbender:remove-background: [epoch 035] average test loss: 3593.1136 cellbender:remove-background: [epoch 036] average training loss: 3475.4907 cellbender:remove-background: [epoch 037] average training loss: 3474.5101 cellbender:remove-background: [epoch 038] average training loss: 3468.3219 cellbender:remove-background: [epoch 039] average training loss: 3448.3782 cellbender:remove-background: [epoch 040] average training loss: 3450.1370 cellbender:remove-background: [epoch 040] average test loss: 3577.6782 cellbender:remove-background: [epoch 041] average training loss: 3453.5002 cellbender:remove-background: [epoch 042] average training loss: 3435.9435 cellbender:remove-background: [epoch 043] average training loss: 3458.4780 cellbender:remove-background: [epoch 044] average training loss: 3441.2540 cellbender:remove-background: [epoch 045] average training loss: 3418.5099 cellbender:remove-background: [epoch 045] average test loss: 3527.2257 cellbender:remove-background: [epoch 046] average training loss: 3446.6400 cellbender:remove-background: [epoch 047] average training loss: 3485.3224 cellbender:remove-background: [epoch 048] average training loss: 3410.5484 cellbender:remove-background: [epoch 049] average training loss: 3422.7325 cellbender:remove-background: [epoch 050] average training loss: 3436.5323 cellbender:remove-background: [epoch 050] average test loss: 3473.7390 cellbender:remove-background: [epoch 051] average training loss: 3444.3200 cellbender:remove-background: [epoch 052] average training loss: 3416.9715 cellbender:remove-background: [epoch 053] average training loss: 3399.0875 cellbender:remove-background: [epoch 054] average training loss: 3410.9901 cellbender:remove-background: [epoch 055] average training loss: 3405.4037 cellbender:remove-background: [epoch 055] average test loss: 3476.9709 cellbender:remove-background: [epoch 056] average training loss: 3418.2370 cellbender:remove-background: [epoch 057] average training loss: 3434.9277 cellbender:remove-background: [epoch 058] average training loss: 3399.6817 cellbender:remove-background: [epoch 059] average training loss: 3411.4972 cellbender:remove-background: [epoch 060] average training loss: 3447.4528 cellbender:remove-background: [epoch 060] average test loss: 3495.3979 cellbender:remove-background: [epoch 061] average training loss: 3408.2628 cellbender:remove-background: [epoch 062] average training loss: 3392.1347 cellbender:remove-background: [epoch 063] average training loss: 3429.8604 cellbender:remove-background: [epoch 064] average training loss: 3463.5573 cellbender:remove-background: [epoch 065] average training loss: 3391.5057 cellbender:remove-background: [epoch 065] average test loss: 3485.3428 cellbender:remove-background: [epoch 066] average training loss: 3427.8305 cellbender:remove-background: [epoch 067] average training loss: 3443.3930 cellbender:remove-background: [epoch 068] average training loss: 3527.2606 cellbender:remove-background: [epoch 069] average training loss: 3393.6105 cellbender:remove-background: [epoch 070] average training loss: 3430.7477 cellbender:remove-background: [epoch 070] average test loss: 3682.4548 cellbender:remove-background: [epoch 071] average training loss: 3382.8507 cellbender:remove-background: [epoch 072] average training loss: 3423.2718 cellbender:remove-background: [epoch 073] average training loss: 3449.6687 cellbender:remove-background: [epoch 074] average training loss: 3432.6429 cellbender:remove-background: [epoch 075] average training loss: 3451.0101 cellbender:remove-background: [epoch 075] average test loss: 3472.7948 cellbender:remove-background: [epoch 076] average training loss: 3395.3973 cellbender:remove-background: [epoch 077] average training loss: 3364.5842 cellbender:remove-background: [epoch 078] average training loss: 3430.8637 cellbender:remove-background: [epoch 079] average training loss: 3538.8890 cellbender:remove-background: [epoch 080] average training loss: 3420.3357 cellbender:remove-background: [epoch 080] average test loss: 3432.9108 cellbender:remove-background: [epoch 081] average training loss: 3473.3313 cellbender:remove-background: [epoch 082] average training loss: 3358.6112 cellbender:remove-background: [epoch 083] average training loss: 3394.1608 cellbender:remove-background: [epoch 084] average training loss: 3351.4519 cellbender:remove-background: [epoch 085] average training loss: 3365.1057 cellbender:remove-background: [epoch 085] average test loss: 3429.6408 cellbender:remove-background: [epoch 086] average training loss: 3357.2040 cellbender:remove-background: [epoch 087] average training loss: 3483.6334 cellbender:remove-background: [epoch 088] average training loss: 3371.5743 cellbender:remove-background: [epoch 089] average training loss: 3362.0617 cellbender:remove-background: [epoch 090] average training loss: 3363.3602 cellbender:remove-background: [epoch 090] average test loss: 3456.8331 cellbender:remove-background: [epoch 091] average training loss: 3347.8683 cellbender:remove-background: [epoch 092] average training loss: 3444.2721 cellbender:remove-background: [epoch 093] average training loss: 3394.1066 cellbender:remove-background: [epoch 094] average training loss: 3349.7894 cellbender:remove-background: [epoch 095] average training loss: 3448.7298 cellbender:remove-background: [epoch 095] average test loss: 3474.9186 cellbender:remove-background: [epoch 096] average training loss: 3353.5865 cellbender:remove-background: [epoch 097] average training loss: 3361.6570 cellbender:remove-background: [epoch 098] average training loss: 3420.3975 cellbender:remove-background: [epoch 099] average training loss: 3356.0131 cellbender:remove-background: [epoch 100] average training loss: 3347.2577 cellbender:remove-background: [epoch 100] average test loss: 3513.3181 cellbender:remove-background: [epoch 101] average training loss: 3387.5554 cellbender:remove-background: [epoch 102] average training loss: 3384.2154 cellbender:remove-background: [epoch 103] average training loss: 3382.8191 cellbender:remove-background: [epoch 104] average training loss: 3341.3567 cellbender:remove-background: [epoch 105] average training loss: 3332.9728 cellbender:remove-background: [epoch 105] average test loss: 3427.8598 cellbender:remove-background: [epoch 106] average training loss: 3375.2365 cellbender:remove-background: [epoch 107] average training loss: 3386.0028 cellbender:remove-background: [epoch 108] average training loss: 3484.4521 cellbender:remove-background: [epoch 109] average training loss: 3360.0883 cellbender:remove-background: [epoch 110] average training loss: 3339.8403 cellbender:remove-background: [epoch 110] average test loss: 3409.1647 cellbender:remove-background: [epoch 111] average training loss: 3335.9104 cellbender:remove-background: [epoch 112] average training loss: 3386.3336 cellbender:remove-background: [epoch 113] average training loss: 3387.1701 cellbender:remove-background: [epoch 114] average training loss: 3371.8190 cellbender:remove-background: [epoch 115] average training loss: 3372.6329 cellbender:remove-background: [epoch 115] average test loss: 3416.4539 cellbender:remove-background: [epoch 116] average training loss: 3370.0098 cellbender:remove-background: [epoch 117] average training loss: 3347.8948 cellbender:remove-background: [epoch 118] average training loss: 3380.4447 cellbender:remove-background: [epoch 119] average training loss: 3368.0514 cellbender:remove-background: [epoch 120] average training loss: 3402.0689 cellbender:remove-background: [epoch 120] average test loss: 3436.2230 cellbender:remove-background: [epoch 121] average training loss: 3338.1733 cellbender:remove-background: [epoch 122] average training loss: 3360.0686 cellbender:remove-background: [epoch 123] average training loss: 3328.2134 cellbender:remove-background: [epoch 124] average training loss: 3383.8421 cellbender:remove-background: [epoch 125] average training loss: 3340.4931 cellbender:remove-background: [epoch 125] average test loss: 3585.7217 cellbender:remove-background: [epoch 126] average training loss: 3414.3335 cellbender:remove-background: [epoch 127] average training loss: 3379.9300 cellbender:remove-background: [epoch 128] average training loss: 3363.2496 cellbender:remove-background: [epoch 129] average training loss: 3421.5268 cellbender:remove-background: [epoch 130] average training loss: 3346.3972 cellbender:remove-background: [epoch 130] average test loss: 3430.1746 cellbender:remove-background: [epoch 131] average training loss: 3414.0120 cellbender:remove-background: [epoch 132] average training loss: 3344.9683 cellbender:remove-background: [epoch 133] average training loss: 3347.1061 cellbender:remove-background: [epoch 134] average training loss: 3328.7894 cellbender:remove-background: [epoch 135] average training loss: 3377.9700 cellbender:remove-background: [epoch 135] average test loss: 3532.2181 cellbender:remove-background: [epoch 136] average training loss: 3335.9759 cellbender:remove-background: [epoch 137] average training loss: 3329.8462 cellbender:remove-background: [epoch 138] average training loss: 3386.2915 cellbender:remove-background: [epoch 139] average training loss: 3346.7995 cellbender:remove-background: [epoch 140] average training loss: 3374.6088 cellbender:remove-background: [epoch 140] average test loss: 3421.4607 cellbender:remove-background: [epoch 141] average training loss: 3323.0598 cellbender:remove-background: [epoch 142] average training loss: 3362.4035 cellbender:remove-background: [epoch 143] average training loss: 3326.2700 cellbender:remove-background: [epoch 144] average training loss: 3347.4532 cellbender:remove-background: [epoch 145] average training loss: 3410.2793 cellbender:remove-background: [epoch 145] average test loss: 3408.1082 cellbender:remove-background: [epoch 146] average training loss: 3407.9777 cellbender:remove-background: [epoch 147] average training loss: 3423.8554 cellbender:remove-background: [epoch 148] average training loss: 3338.5923 cellbender:remove-background: [epoch 149] average training loss: 3411.2795 cellbender:remove-background: [epoch 150] average training loss: 3327.2752 cellbender:remove-background: [epoch 150] average test loss: 3603.2829 cellbender:remove-background: Saving a checkpoint... cellbender:remove-background: Saved checkpoint as /net/vast-storage/scratch/vast/kellislab/benjames/cellbender/human_brain_3k/ckpt.tar.gz cellbender:remove-background: 2023-12-05 21:01:10 cellbender:remove-background: Inference procedure complete. cellbender:remove-background: Attempting to unpack tarball "ckpt.tar.gz" to /tmp/tmp76umqw31 cellbender:remove-background: Successfully unpacked tarball to /tmp/tmp76umqw31 /tmp/tmp76umqw31/d4ef6573dc_random.pyro /tmp/tmp76umqw31/d4ef6573dc_random.cuda /tmp/tmp76umqw31/d4ef6573dc_model.torch /tmp/tmp76umqw31/d4ef6573dc_optim.torch /tmp/tmp76umqw31/d4ef6573dc_optim.pyro /tmp/tmp76umqw31/d4ef6573dc_params.pyro /tmp/tmp76umqw31/d4ef6573dc_train.loaderstate /tmp/tmp76umqw31/d4ef6573dc_test.loaderstate /tmp/tmp76umqw31/d4ef6573dc_args.npy cellbender:remove-background: Posterior not currently included in checkpoint. cellbender:remove-background: Computing posterior noise count probabilities in mini-batches. cellbender:remove-background: Working on chunk (1/28) cellbender:remove-background: [0.05 mins per chunk] cellbender:remove-background: Working on chunk (2/28) cellbender:remove-background: Working on chunk (3/28) cellbender:remove-background: Working on chunk (4/28) cellbender:remove-background: Working on chunk (5/28) cellbender:remove-background: Working on chunk (6/28) cellbender:remove-background: Working on chunk (7/28) cellbender:remove-background: Working on chunk (8/28) cellbender:remove-background: Working on chunk (9/28) cellbender:remove-background: Working on chunk (10/28) cellbender:remove-background: Working on chunk (11/28) cellbender:remove-background: Working on chunk (12/28) cellbender:remove-background: Working on chunk (13/28) cellbender:remove-background: Working on chunk (14/28) cellbender:remove-background: Working on chunk (15/28) cellbender:remove-background: Working on chunk (16/28) cellbender:remove-background: Working on chunk (17/28) cellbender:remove-background: Working on chunk (18/28) cellbender:remove-background: Working on chunk (19/28) cellbender:remove-background: Working on chunk (20/28) cellbender:remove-background: Working on chunk (21/28) cellbender:remove-background: Working on chunk (22/28) cellbender:remove-background: Working on chunk (23/28) cellbender:remove-background: Working on chunk (24/28) cellbender:remove-background: Working on chunk (25/28) cellbender:remove-background: Working on chunk (26/28) cellbender:remove-background: Working on chunk (27/28) cellbender:remove-background: Working on chunk (28/28) cellbender:remove-background: Writing full posterior to cellbender_posterior.h5 cellbender:remove-background: Succeeded in writing posterior to file cellbender_posterior.h5 cellbender:remove-background: Added posterior object to checkpoint file. cellbender:remove-background: 2023-12-05 21:03:56 cellbender:remove-background: Saved summary plots as cellbender.pdf cellbender:remove-background: Saved cell barcodes in cellbender_cell_barcodes.csv cellbender:remove-background: Computing target noise counts per gene for MCKP estimator cellbender:remove-background: Using MCKP noise targets computed for FPR 0.01 cellbender:remove-background: Computing denoised counts using mckp estimator cellbender:remove-background: Working on chunk (1/7) cellbender:remove-background: [0.82 mins per chunk] cellbender:remove-background: Working on chunk (2/7) cellbender:remove-background: Working on chunk (3/7) cellbender:remove-background: Working on chunk (4/7) cellbender:remove-background: Working on chunk (5/7) cellbender:remove-background: Working on chunk (6/7) cellbender:remove-background: Working on chunk (7/7) cellbender:remove-background: 2023-12-05 21:09:40 Total MCKP estimation time = 327.87 sec cellbender:remove-background: Succeeded in writing CellRanger format output to file cellbender.h5 cellbender:remove-background: Succeeded in writing CellRanger format output to file cellbender_filtered.h5 cellbender:remove-background: Saved output metrics as cellbender_metrics.csv cellbender:remove-background: Succeeded in writing report to cellbender_report.html cellbender:remove-background: Completed remove-background. cellbender:remove-background: 2023-12-05 21:10:16