welcome to the site! lots of good info here. as far as I know, there’s no reason that you can;t run new trains on the old track. I have one MTH steam engine I run on my old tube track powered by my Prewar V transformers.
On the subject of DC vs. AC O gauge trains, unless it was a cheap set made in the 1980’s, all Lionel trains are made to run on AC including the items you mentioned above. They should work fine with your transformers, though if you get the boxcab diesel you may want to install an inline fuse to protect the circuit boards in the locomotive in case of a short. Postwar transformers have circuit breakers, but they aren’t fast enough with modern locomotives and can cause some expensive problems for your locomotives.
The circuit board fails first. Then the circuit breaker or fuse shuts off the current after the board is failed. No circuit breaker or fuse is fast enough to shut down the transformer before the damage is done. The best protection for modern locomotive electronics is a transient-voltage suppressor (TVS), like the 1.5KE36CA, which you can get from Mouser, for example. They’re very cheap.
The TVS would be put across terminal posts like; A & U, rather then in line like a fuse. Won’t work when put in line like a fuse, so it must be wired across the terminals to cut or clip spike voltages.
If adding an extra circuit breaker I would use a 7 amp breaker in line like a fuse, as it is not too strong and not that week, and much better than LIonel’s internal breakers on the post war transformers.
Even the brand new transformers could use a TVS unit to help eliminate spikes in A.C. voltage.
The Williams diesel should be able to navigate any 027 or O gauge switch in tubular track. I have not heard of any problem with Williams engines and Lionel switches except for one stock number in Lionel and it is the 6-23010 & 23011 series switch. Williams service claims the design of the 23010 series switch is very much flawed!
I use most of my Williams with Gargraves 042 switches and older Lionel 027 switches and have no problems with them.