I have a small number of naive questions. I already have a fine-tuned tardigrade detection model that gives me tardigrade bounding boxes (data comes from labelled images on my microscope). I want to do masks as well.
Right now my home server w/ an RTX 2080 is able to do mask prediction in about ~4 seconds (I'm running the sample script in "directory" mode) per image (640x480).
I'd love to be able to get the first mask back in 0.1ms, so I can do 10FPS on the scope. Is there a practical way to speed things up (my guess would be buying the absolute fastest GPU I can afford)? I can run the object detection to get a reasonable seed location, is that what the paper means by prompting?
A 4s run time for object segmentation at 640x480 sounds like it's not using the GPU at all. Something like that should run on a VGA image in at most a few hundred ms.
For the second part of the question, a 2080 should get you close to 10FPS operation. For a ballpark estimate, using an off-the-shelf repo like Ultralytics's YOLOv5 lets you run object detection (not masking) at something like 100FPS. Masking should not add that much overhead.
w.r.t. GPUs yes, these days more money equals more speed for GPU NN inference, though there are diminishing returns. A 3090 might get you the best bang for your buck these days while still having enough VRAM to run fancier models which may need more than the 12 GiB many other GPUs have.
Finally, I haven't read the paper too carefully but I believe that by prompting they mean that you have the option of describing in human language what you want the model to select, rather than the model being "hardwired" to do this. In other words, you could prompt the model to "segment the red car only" and it would do it, rather than just having the model blindly segment every object in the image, and then relying on custom scripting to potentially post-process these segments.
I'm using the first model on the SAM website ( ViT-H SAM).
It's definitely using the GPU- I'm running nvidia-smi and I see near 100% utilization on the GPU while the CPU is using 1 core. If I run the script with --device=cpu then I see my server using 4 cpu cores and no GPU and it takes tens o seconds per image.
I'm trying to check with people who have experience with this specific model.
I've checked on the repo and other folks report the same numbers as me- in fact my 2080 is just as fast as the 3090 (5 seconds per mask generation/image).
I have bounding box already, so I could prompt the model with that, but all of this runs counter to the published performance numbers.
If you want to try it on one reach out to me (email in profile). We rent those out in the cloud. Would allow you to confirm performance before buying one for local use.
the RTX 2080ti 11GB model is a little more than 2 times slower than the flagship RTX (12.5it/s vs 28it/s) for torch/diffusion. extrapolate that from what you will.
however, CLIP/BLIP and boxes should be much faster than 4 seconds, even on a 2080. I had a python CLIP CSV tag script running in a directory with thousands of images and it was taking <=2 seconds per image on a Geforce GTX 1070 TI, with 8GB of memory - an old card without any tensors. CLIP is much slower than some other mechanisms, for instance a deepbooru classifier is about 4x faster than CLIP/BLIP on my RTX 3060 12GB. CLIP of a random image around your dimensions takes ~4-5 seconds, and deepbooru takes about 1.5 seconds. edit: the additional time is the overhead of the webUI, i am guessing
What will probably have to happen is some sort of auto-crop that only forces the model to view a very tiny section of the image. You mentioned you already had a model, was it trained from scratch, or using an existing model?
What model are you using? 4s/image seems extremely slow. I've been experimenting with Detectron2 and most of their models give me less than 1s on the CPU for instance segmentation on images 4x the size
Right now my home server w/ an RTX 2080 is able to do mask prediction in about ~4 seconds (I'm running the sample script in "directory" mode) per image (640x480).
I'd love to be able to get the first mask back in 0.1ms, so I can do 10FPS on the scope. Is there a practical way to speed things up (my guess would be buying the absolute fastest GPU I can afford)? I can run the object detection to get a reasonable seed location, is that what the paper means by prompting?