I assume that you just pull the wallpaper off without wetting it. If so, the brown paper left on the wall is the backing of the wallpaper. Spray it with water and let the water soak in. With luck, the brown backing will then be easly removed with a scraper (a putty knife works well). After removal, make sure all the glue from the wallpaper has been removed. Use DIF (a solution made to soften the glue) if necessary. The wallpaper backing must be removed.