Hi Raymond,
I don't know if this is what you have, but a friend of mine had dirt dobbers and wasps start building nests in his walls due to some openings he didn't know about under a window seal. At first he heard some scratching every so often then he saw similar things to what you're describing. It wasn't until he started seeing 2-3 wasps and dobbers a day that he started putting the pieces together.
Putting your ear to his wall you could hear the buzzing and sounds of the critters, and just pushing the wall with a finger it went through exposing a massive nest in several places inside the wall. He ended up removing the drywall and after an exterminator found the spot they were coming in, he fixed the hole and all was good.
Just a thought... walk around your house and see if you see massive accumulations of flying bugs (dirt dobbers, wasps, yellow jackets, anything) flying to a point where you see no nest. Chances are they might be going inside the brick or walls and nesting. In my house I saw hornets starting to do this.. coming in and out a spot in my brick, and I used some mortar and a little caulk to seal the hole.
HTH,
Sam