No, it is true. It is not the quantity of oil infrastructure, but the grades and types they are. The US crude is mostly light sweet crude after the shift to oil shale. The refinery infrastructure was originally built for heavy crude with high sulfur content. Thus the US imports the type of oil our refineries were built to handle, and exports the portion of the oil that is domestically produced, but the wrong type.
The lack of investment in the types of oil refineries to refine US oil domestically isn’t as much for optics purposes. But that relative to the amount of investment required to build new refineries to compete with the current foreign ones isn’t a good return on investment relative to the up front cost and the existing profits of the current arrangement.