Pardon me if I'm telling you what you already know, but . . . .
Most cacti need lots of sunlight and heat, very little water, and not too much cold. Many of them can go for several months without being watered. Really. They do better with neglect than with overattention.
You said you brought the plants inside. I take that to mean they're in pots. Either way, whether in pots or in the ground, cacti absolutely need to have good drainage, and not too much organic material in the soil. They don't need "sand", necessarily, but if you put the cacti in peat moss or clay, or if you water them too much, they will rot.
Then there's the usual varmints (bugs, etc) that will nibble on anything you have, anyway.
And about growth: some cacti do grow slowly. My mom had a saguaro that grew to a height of 2 or 3 inches in ten years.
If you can tell us where you have put your cacti, that might help to clarify the problem. Do you live in Yuma (great cactus climate), or in Seattle (a great place to kill cacti)? Do your cacti have sun, shade, rain, fungus, etc?
Ted Shoemaker certified grey thumb