building in a test path
As far as testing systems is concerned, it really helps to have an automatic or semi-automatic test path built into the product, starting at the high-level design stage. There should be a lot of points in the development process where scripts and utilities can be used to check people's work objectively.
It makes a difference to build on an architecture that has enforces good structure, and where validation utilities can be used to create detailed build logs. This may mean thinking about software design in a non-traditional way.
Applications and systems that do not have a test path incorporated in them are a nightmare to diagnose when they are put into service.
Usually, they are required to interoperate with other systems that are similarly lacking this feature. When these interdependent systems are put into operation, and a problem occurs, it is hard to figure out which is the offending part of the process, becuase none of the applications leave an audit trail.
Ultimately, poorly-designed systems never end up getting redesigned, due to expense and organizational inertia. Instead, humans are expected to make up for the application's shortcomings, by performing the task of a computer, just to keep up the illusion that things are running smoothly.
People's time should be freed-up to spend on things that computers can't do - things that don't require mindless repetition.