Sobering chatter about a near head-on incident at Sax on the Missabe
Sub last night. Apparently the new computer dispatching system being
implemented on CN allowed a Minntac limestone to head north through
Kelsey while at the same time allowing an overlapping authority to a
southbound CN manifest at MP 51. Fortunately that area of the
mainline has good visibility and it allowed both trains to see the
opposing movement’s headlights and stop before colliding. The
limestone ended up backing into Kelsey to allow the core train to
continue south. It appears there are some serious safety issues with the new system that
could lead to tragedy on the railroad.
Now does the computer system get 30 or 60 days off while being investigated?
Glad nobody was hurt physically and the crews were aware the potential disaster that was staring at them.Highly doubt it that the CN will give an “atta-boy”. (jokenly) The crew will probably get a week off cause they were looking too far down the line.
I am not quite sure why the CN is painted as the bad guy here. A dispatching computer is only as good as the dispatcher sitting behind the screen. There is a lot of things that a CAD system will let you do that under normal operations you shouldn’t, because in special situations you will need to be able to do it. In the end it is up to the dispatcher know what is going on when giving out authoritys.
I suspect the people writing the program may not have had much experience with railroads. They take the specifications for the desired program and write it to fit. If someone missed a parameter, the program won’t know how to properly deal with a given situation when it encounters it.
Too, railroads are chock full of “unique” situations, each requiring its own solution. This one may have been missed.
That said, and as pointed out already - the dispatcher should still have situational awareness and be able to prevent such incidents.
Fortunately, this one falls in the “lessons learned” column, with no loss of life or equipment.
I’ve been in software development since 1985, and a lot of it in real-time control systems. They’ll no doubt identify the particular component(s) in the software system where the root cause, in terms of implementation, exists, if it was indeed caused by a “bug” in the implementation (i.e. coding) of the software. However, it’s entirely possible the software was coded precisely according to its design specification, where the root cause may lie instead. It’s also possible the design was created precisely according to software requirements, which need to be finalized and approved before the software design begins, and would quite likely have to be approved by CN staffers who would be quite familiar with railroad operations.
In any event, whether the defect exists in the requirements definition, design or implementation (or some combinati
Hollerith cards. At least with rectangular holes by the time I got there.
Mea culpa?
One axiom concerning development, SW or otherwise, is that as the complexity of the system increases, so do the number of corner cases. There comes a point where the code system becomes so complex that it is temporally impossible to test every combination of branches. That being said, it is not uncommon for some corner case failure to slip through extensive requirements reviews, design reviews, coding reviews, unit testing and system testing. It’s the nature of the beast.
Commercial development projects usually do not have the deep structural processes in place that accompany DOD projects, for instance, so not all mistakes will get caught before the product gets into the hands of the end user (just as even in DOD projects, some things slip through). Anybody who thinks that a complex SW product is going to be absolutely flawless on delivery is uninformed about history, the development process or the impact of complexity factors on design, implementation and testing.
There are a limited number of companies that offer CAD’s software in the rail industry. All of those companies have had much experience within the industry. Not knowing which vendor’s CAD’s system CN chose to install I can’t make any comments concerning the vendors reputation.
The Primary purpose of a CAD’s system, in addition to absolute protection, is to make the various signaling systems the the Class I railroads have installed over the year, each with their quirks, appear as a seamless, transparent operating system to the Dispatcher as the ultimate user of the system. The CAD’s System must also be able to provide absolute protection on dark territories where DTC or Track Warrant operating rules are in effect.
I’m a retired software engineer, and I’d like to read public, technical papers on these CAD systems. Do the companies have websites? “CAD” means Computer Automated Dispatching, right?
One of the other posts spoke of “new format”. I wonder - did the railroad just install some new or revised software? Putting information technology to work is a hard job. In addition to the development processes that the other guys wrote about, they need to manage the configuration management processes that puts the software to work in the field. I bet that is pretty tough in a railroad!
CN just got rid of Digicon, and went to Wabtec for the system. I’m pretty sure all the RTCs are in Homewood now. I think a good number didn’t want to move down there, so the RTC running the desk may not be as familiar with the territory as s/he should be. Track warrants and track & time are gone, now a form called “Track Authority” is used. I haven’t been out on the CN for a while; I have about 20 bulletins to go through before I go out there.
My understanding is that “CAD” in this context* stands for either “Computer-Aided Dispatching” or “Computer-Assisted Dispatching”.
(* - It’s Computer-Aided or-Assisted Drafting in the engineering and architectural worlds.)
Suggest a Google Advanced Search. For example, I input “Digicon” as an “All these words”, and “Computer Aided Dispatching” as “This exact wording or phrase”, and found a couple of trade publication articles (from about 5 and 12 years ago), and an extensive discussion on another forum. Other combinations - using “Wabtec” and / or “Computer Assisted Dispatching” might be equally fruitful.
One link I did find that was unexpected was to a March 2003 FRA “Final Report” titled “Selection of Railroad Dispatcher Candidates”, Report No. DOT/FRA-ORD-03/09, 118 pages. Notably, it was for the purpose of assisting in “assessing candidates with no prior railroad experience”. [emphasis added] The preparers visited and interviewed dispatchers at BNSF, ConRail, WC, and UP, as well several commuter railroads - LIRR, Metra, and MetroNorth. I’d be interested in seeing what our members with real-life experieinces as dispatchers think of this report. Here’s the link to it:
Mea culpa, myacopa, whatever. I’d HOPE the software development processes employed by CN or whoever provides their software systems would’ve had some type of code coverage analysis as part of their validation process. Code coverage (depending on what type it is - simple line execution counts, branch coverage, MC/DC, etc.) may not catch every possible glitch, but it can at least force developers to take a second look at their testing to understand why a certain line(s) weren’t hit during unit and/or integration testing, and that can sometimes lead to questioning how various scenarios are handled. I work in the
Truth be that no matter how extensive the testing, there is always one more idiotic occurrence that can occur that the testers were unable to think of. If the testers utilize a million scenarios, and the system handles them all…there will always be that million+1 scenario that will catch the system out. CAD’s systems (the A is for aided or assisted, NOT automated) are very complex pieces of rule compliant and process control integrated software. As a result, Dispatchers using CAD’s systems ALWAYS need to remain vigilant that the system provides the protection that the Dispatcher intends.
While the CAD’s systems do have flaws, when incidents are fully investigated - the failures are 90% or higher human caused, as a Dispatcher tries to ‘short cut’ the system or is flat out mistaken on the part of the operation that needs protection or authority (the hundred mile mistake).
The simple dictum - ‘Protect, then Authorize’ is the hallmark of dispatching. All to frequently, dispatchers overlook the first word of that dictum.
That “nice, warm, fuzzy feeling” is actually the residuals of the adrenaline that shot through your system when you realized that it could have been you out there “testing” the new system.
I would imagine most crews on the CN share those feelings…
I am sure that is one of many questions being asked. Without details, it is hard to speculate, which is why sometimes a detailed root-cause-analysis is called for, if the answer is not obvious.
I am wondering if there may have been a misuse of a user override in this case, as it would seem that one thing that should have been thoroughly tested were the consistency checks and exception handling associated with putting two trains into the same authority block (for want of proper nomenclature, as I certainly do not claim domain expertise in this area).
It is certainly a puzzlement, at any rate, and I can surmise with some accuracy what is being done post mortem to correct the deficiency. I can also surmise that details will likely not filter out to us unwashed masses.