The concern itself is valid but the application makes any transfer of potentially bad chemicals unlikely. Unless you store the water in a PVC container, there's not going to be enough time for any chemical transfer from the pipe to the water while it's flowing through and out of the pipes. Also, the PVC itself is extremely stable (the main reason they make pipes from it) and the rigid pipe does not contain as many phthalate plasticizers as a flexible hose does. The phthalate plasticizers are the main point of contention here. They are suspected of causing certain types of liver cancer in people with prolonged exposure. But the thing with phtalates is that they are not as stable (their instability is one of the reasons they are toxic) and cannot stay long in the environment before breaking down. So, whatever minute amounts the water may pick up while in the pipe, they will break down in the soil soon after.
I would not drink that water because we are very sensitive to tiny amounts of additives in water and if you drink it straight away, you may taste something foreign in the water. But for irrigation electrical conduit will be just as fine as plumbing PVC. Note that without special additives PVC is not particularly resistant against UV. There are UV-resistant PVC electrical conduits (the gray electric grade conduit you can find at HD or Lowes is one) but there are no UV-resistant plumbing PVC pipes I know of, since they are generally installed sub-terrain and never see any Sun. So, if your pipe is exposed, you may in fact, do better with an electrical PVC conduit for irrigation.