The original poster didn't ask if WE LIKE GRASS !! He asked if it was
ok to plant it in an area that has some pea gravel on it.
It is none of our darned business if he wants to plant grass, so we should't
lecture him on reasons why HE should not plant grass on HIS property.
To the OP: I would try to spread a minimum of 2-3 inches of topsoil on
the area, if that is feasible for you. That would give the grass a better
chance to grow and spread. As long as the existing gravel is "mixed"
with some soil, the grass will grow, but adding a few inches of topsoil
would give you a quicker and more pleasing result.
After the grass has taken hold and doing fine, you might add a few more
inches of topsoil the next year, and before you know it, you will not be
able to tell the gravel was there.
Good Luck !!