In order not to limit the power available to the rating of one panel it may be better to locate an enclosed breaker for one panel at the other panels location or to group two disconnects at one location on the exterior of the building with each disconnect feeding one of the panels. There is only supposed to be a single set of service entry or feeder conductors supplying the building except for an emergency feeder from an alternate power source or a supply with different phase or voltage characteristics.
Just a reminder that the code specifically allows a separate disconnect for a water pump located at the yard pole so that you can kill power to the building without shutting off the source of water used for first aid fire suppression. The conductors from the water pump disconnect are another exception to the one set of supply conductors rule.
-- Tom Horne