Some of the most dangerous PLC behaviour appears after the original fault has already occurred.

A motor trips. Communication disappears. Control power fails. A drive reports a fault, or the safety system removes permission to operate.

The equipment stops.

At that moment, the control system must decide what happens next.

Without structured recovery logic, old commands may remain stored, outputs may return when permissives recover, and an interrupted sequence may continue from a state that no longer matches the physical machine.

The fault itself may be handled correctly.

The recovery may not be.

Why Fault Recovery Is So Difficult

Normal operation usually follows an expected sequence:

Ready
↓
Start request
↓
Startup validation
↓
Equipment starts
↓
Normal operation

Fault recovery begins from a less predictable condition.

After a trip or interruption:

  • Actuators may no longer be in their expected positions.
  • Pneumatic pressure may have decayed.
  • Drives may have lost their internal state.
  • Remote I/O may still be reconnecting.
  • Analog values may be stale or invalid.
  • Product may remain inside the machine.
  • Internal sequence latches may still be active.
  • Operators may have moved equipment manually.

The PLC cannot safely assume that the condition before the fault still exists afterward.

A recovered machine should therefore behave like a controlled startup—not an uncontrolled continuation.

A Safe Recovery Structure

A professional recovery sequence can follow this structure:

Fault detected
↓
Enter faulted state
↓
Remove or control commands
↓
Confirm fault cause is gone
↓
Validate reset request
↓
Recheck communication and permissives
↓
Enter recovery state
↓
Verify physical machine condition
↓
Return to ready state
↓
Require normal startup

Each stage has a specific purpose.

Skipping one stage may allow the machine to move before the controller has confirmed that recovery is complete.

The Faulted State

When a fault is detected, the sequence should enter a clearly defined fault state.

For example:

Machine_State := FAULTED;

The faulted state should define:

  • Which output requests are removed
  • Which outputs may remain active
  • Whether a controlled stop must finish
  • Which fault information is retained
  • Whether manual recovery is permitted
  • What conditions are required before reset
  • What state follows a successful reset

It should not simply clear random latches throughout the program.

A structured fault state provides one authoritative description of the machine’s condition.

Not Every Fault Requires the Same Response

Faults can require different stopping and recovery behaviour.

Possible categories include:

Immediate stop faults

These require rapid removal of ordinary control commands.

Examples may include:

  • Motor overload
  • Drive trip
  • Critical pressure loss
  • Mechanical jam

Controlled shutdown faults

Some processes need an orderly stop.

The machine may need to:

  • Stop material feeding
  • Clear a conveyor
  • Reduce speed
  • Close a valve
  • Complete cooling or purge operation

Communication faults

The response depends on which data was lost and how important it is.

A communication failure may require:

  • Immediate stop
  • Controlled shutdown
  • Blocking new starts
  • Holding a limited operating state temporarily

Safety-system stops

Emergency-stop and guard-related functions must be handled through the required safety-rated architecture.

The standard PLC may receive status from the safety system and coordinate normal recovery, but ordinary PLC logic must not replace the safety function.

Fault Detection Should Be Latched Clearly

Some faults disappear as soon as the output turns off.

For example:

  1. Motor runs.
  2. Pressure falls.
  3. Low-pressure fault stops the motor.
  4. Pressure immediately recovers.
  5. The original fault condition disappears.

Without fault memory, maintenance personnel may arrive and find no active cause.

A fault latch preserves the event:

IF Low_Pressure_Condition THEN
    Low_Pressure_Fault := TRUE;
END_IF;

The latch should clear only when the required reset conditions are satisfied.

IF Low_Pressure_Condition THEN
    Low_Pressure_Fault := TRUE;

ELSIF Reset_Request
      AND Pressure_Recovery_Valid THEN
    Low_Pressure_Fault := FALSE;
END_IF;

The active fault has priority over reset.

Reset Is Not Fault Recovery

Pressing RESET should not automatically make the machine ready to run.

A reset request usually means:

The operator acknowledges the fault and requests that the control system evaluate recovery.

It does not prove:

  • The physical cause has disappeared
  • Communication data is fresh
  • Actuators are correctly positioned
  • Process pressure has recovered
  • Old commands have been cleared
  • Automatic operation is safe

A strong sequence separates:

Fault acknowledgement
Fault clearing
Recovery validation
Ready state
Start command

These are different actions.

Reset Must Not Mean Start

Combining reset and restart creates dangerous operator expectations.

The operator may press RESET only to clear an alarm, yet a retained run request can immediately become active.

A safer sequence is:

Fault condition removed
↓
Operator presses RESET
↓
Reset conditions checked
↓
Machine enters RECOVERY
↓
Recovery completed
↓
Machine enters READY
↓
Fresh START request required

The RESET button restores eligibility for operation.

The START button requests operation.

Validate the Reset Request

A reset should be accepted only when the original fault and required recovery conditions are healthy.

For example:

Reset_Valid :=
    Reset_Request
    AND NOT Fault_Condition
    AND Safety_System_Healthy
    AND Communication_Healthy
    AND Device_Data_Valid;

Where appropriate, the conditions should remain stable for a defined time.

Recovery_Conditions continuously healthy
for validation period
→ Reset permitted

This prevents a fault from clearing during a one-scan signal flicker.

Never Allow Reset to Override an Active Fault

Weak logic may use separate set and reset instructions:

Fault condition → SET Fault_Latched
Reset button    → RESET Fault_Latched

If both are active, the final result may depend on execution order.

A more explicit structure gives the fault priority:

IF Fault_Condition THEN
    Fault_Latched := TRUE;

ELSIF Reset_Request
      AND Recovery_Conditions_Valid THEN
    Fault_Latched := FALSE;
END_IF;

The system cannot be reset while the fault remains present.

Retained Commands and Unexpected Motion

Retentive memory can preserve values through power loss or CPU restart.

This is useful for:

  • Production totals
  • Recipes
  • Setpoints
  • Calibration data

It is dangerous when applied carelessly to:

  • Motor run commands
  • Valve movement requests
  • Automatic-cycle latches
  • Temporary bypasses
  • Sequence-complete bits

Consider this sequence:

Motor running
↓
Power failure
↓
Output turns off
↓
Retained run latch remains true
↓
Power returns
↓
Drive becomes ready
↓
Motor starts unexpectedly

The PLC may be executing exactly as programmed.

The unsafe behaviour comes from an undefined recovery strategy.

Movement requests should normally return to a defined safe condition after startup unless automatic restart has been specifically assessed and designed.

Clear Requests Without Destroying Diagnostics

At fault entry, the program may need to clear:

  • Pending start requests
  • Automatic run requests
  • Temporary manual commands
  • Sequence transition requests
  • Maintenance overrides

However, diagnostic information should remain available.

Preserve:

  • First fault
  • Previous machine state
  • Active sequence step
  • Output commands
  • Field feedback
  • Process measurements
  • Communication status
  • Timestamp

Clearing every internal bit immediately may remove the evidence needed to understand the event.

Do Not Resume From an Unknown Sequence State

A retained sequence state may no longer match the physical equipment after a fault.

For example, the PLC may remember:

State = FILLING_TANK

During the interruption:

  • The inlet valve closes through its fail-safe action.
  • Product continues draining.
  • The operator manually moves another valve.
  • The level signal becomes invalid.
  • Pneumatic pressure decays.

Resuming directly from FILLING_TANK would assume that all required physical conditions remain unchanged.

A stronger recovery sequence compares stored state with actual feedback.

When the condition cannot be validated, the machine should enter:

RECOVERY_REQUIRED

rather than continuing automatically.

A Dedicated Recovery State

Recovery should be represented as an explicit operating state.

For example:

FAULTED
↓
WAITING_FOR_RESET
↓
RECOVERY_REQUIRED
↓
VALIDATING_POSITIONS
↓
READY

During recovery, the program may:

  • Keep automatic outputs blocked
  • Re-establish communication
  • Confirm fresh analog data
  • Validate actuator positions
  • Restore utilities
  • Remove incomplete product
  • Permit controlled manual movement
  • Rehome motion systems
  • Require operator confirmation

The exact actions depend on the machine.

The important point is that recovery is a controlled phase—not a shortcut back into production.

Manual Recovery Must Be Restricted

Some faults require maintenance personnel to move equipment manually.

Manual recovery may allow:

  • Jogging a conveyor
  • Opening or closing a valve
  • Returning a cylinder home
  • Clearing material
  • Repositioning an axis

These actions should be restricted by:

  • Selected maintenance mode
  • Appropriate authorization
  • Hold-to-run controls where required
  • Valid interlocks
  • Clear HMI indication
  • Reduced speed or limited movement where applicable

Manual movement should not silently advance the automatic sequence.

After recovery movement is complete, the PLC should verify the resulting machine state before returning to READY.

Revalidate Every Important Permissive

Recovery logic should not reuse the permissive result that existed before the fault.

Each important condition should be checked again.

Examples include:

  • Safety system healthy
  • Drive communication established
  • Drive ready for operation
  • Remote I/O data valid
  • Air pressure stable
  • Valve positions confirmed
  • Cooling flow available
  • Analog values current and plausible
  • Downstream equipment ready

For networked values, validate both the value and its quality:

Remote_Pressure_Permissive :=
    Remote_Communication_Healthy
    AND Remote_Data_Valid
    AND Remote_Pressure > Minimum_Pressure;

A retained healthy value is not proof that the current process condition is healthy.

Communication Recovery Is Not Immediate Readiness

A device may become connected before its process data is usable.

A typical sequence can be:

  1. Network connection returns.
  2. Device appears online.
  3. Old or default data becomes visible.
  4. Device completes initialization.
  5. Fresh cyclic data begins updating.
  6. Ready status becomes valid.

Recovery logic should distinguish between:

Device_Connected
Device_Communicating
Device_Data_Valid
Device_Ready

Treating the first connection bit as complete readiness can create false permissives and unsafe startup transitions.

Analog Recovery Requires Fresh Data

Analog values also require validation after recovery.

A filter may contain:

  • An old retained value
  • Zero after initialization
  • A default substitute value
  • A slowly recovering measurement

Before using the value, confirm:

  • Channel healthy
  • Signal inside valid electrical range
  • Fresh input received
  • Scaling configuration valid
  • Filter initialized
  • Process value plausible
  • Required stability period completed

For example:

Pressure_Recovery_Valid :=
    Pressure_Channel_Healthy
    AND Pressure_Data_Fresh
    AND Pressure_Signal_Valid
    AND Pressure_Filter_Initialized
    AND Pressure_Stable;

Recovery Timeouts

Every recovery action should have a defined timeout.

Examples include:

  • Drive communication must return within 10 seconds.
  • Valve must reach its recovery position within 5 seconds.
  • Pressure must recover within 30 seconds.
  • Axis homing must complete within 60 seconds.

Without timeouts, the machine may remain in recovery indefinitely.

A useful structure is:

IF Recovery_Condition_Complete THEN
    Next_State := READY;

ELSIF Recovery_Timeout THEN
    Recovery_Fault := TRUE;
    Next_State := FAULTED;
END_IF;

The timeout alarm should identify the exact failed action.

Avoid generic messages such as RECOVERY FAILED when the PLC knows that a specific valve or drive did not respond.

Prevent Recovery Loops

Poorly designed logic can repeatedly switch between fault and recovery states.

For example:

  1. Fault clears.
  2. RESET sends the machine to recovery.
  3. One unstable permissive drops.
  4. Machine returns to fault.
  5. Permissive returns.
  6. RESET remains active.
  7. Machine enters recovery again.

This creates repeated transitions and confusing alarms.

Prevent this by:

  • Requiring a new reset edge
  • Validating conditions before leaving the faulted state
  • Using hysteresis or stabilization where appropriate
  • Clearing pending reset requests
  • Recording the reason recovery failed

A held RESET button should not repeatedly attempt recovery without deliberate design.

Fault Priority During Recovery

A new fault may occur while recovery is already in progress.

The logic should define priority clearly:

IF Critical_Stop THEN
    Next_State := SAFE_STOPPED;

ELSIF New_Fault_Active THEN
    Next_State := FAULTED;

ELSIF Recovery_Timeout THEN
    Next_State := FAULTED;

ELSIF Recovery_Complete THEN
    Next_State := READY;
END_IF;

Normal recovery completion must not override a newly detected fault in the same scan.

Separate Fault Cause From Machine State

The machine state and fault code should be separate.

For example:

Machine_State = FAULTED
Fault_Code = DRIVE_2_COMMUNICATION_LOST

The state describes what the machine is doing.

The fault code explains why it entered that state.

Useful records include:

Current_State
Previous_State
First_Fault_Code
Current_Faults
Fault_Time
Faulted_Sequence_Step
Recovery_Failure_Code

This structure allows the machine to remain in one faulted state while preserving detailed diagnostics.

Record the First Fault

One failure often creates several secondary alarms.

For example:

  1. Drive communication fails.
  2. Motor stops.
  3. Flow disappears.
  4. Pressure falls.
  5. Downstream equipment stops.
  6. Sequence timeout activates.

By the time maintenance arrives, five alarms are active.

The first fault was the communication loss.

A first-out recorder should capture:

  • Original fault
  • Time
  • Machine state
  • Output request
  • Process values
  • Communication status
  • Operator mode

This prevents troubleshooting from beginning with secondary consequences.

Controlled Recovery Example

A simplified state-machine structure could look like this:

CASE Machine_State OF

    RUNNING:

        IF Critical_Fault THEN
            Previous_State := RUNNING;
            Machine_State := FAULTED;

        ELSIF Stop_Request THEN
            Machine_State := STOPPING;
        END_IF;

    FAULTED:

        Motor_Run_Request := FALSE;
        Automatic_Mode_Allowed := FALSE;

        IF Fault_Cause_Removed
           AND Reset_Rising_Edge
           AND Reset_Conditions_Valid THEN

            Machine_State := RECOVERY_REQUIRED;
        END_IF;

    RECOVERY_REQUIRED:

        Motor_Run_Request := FALSE;
        Automatic_Mode_Allowed := FALSE;

        IF New_Fault THEN
            Machine_State := FAULTED;

        ELSIF Recovery_Timeout THEN
            Recovery_Fault := TRUE;
            Machine_State := FAULTED;

        ELSIF Communication_Validated
              AND Positions_Validated
              AND Process_Data_Validated THEN

            Machine_State := READY;
        END_IF;

    READY:

        IF Fresh_Start_Request
           AND Start_Permissives_Validated THEN

            Machine_State := STARTING;
        END_IF;

END_CASE;

The recovered machine returns to READY rather than directly to RUNNING.

Output Ownership During Faults

Fault logic should not write physical outputs from several unrelated locations.

Instead, the sequence generates requests, and one device-control block owns the final command.

For example:

Motor_Run_Request :=
    Machine_State = RUNNING;

Then:

Motor_Command :=
    Motor_Run_Request
    AND Motor_Run_Permissive
    AND NOT Motor_Fault
    AND NOT Stop_Required;

Finally:

Physical_Motor_Output := Motor_Command;

This prevents the recovery routine, manual routine and automatic sequence from competing for the same output.

Emergency-Stop Recovery

Resetting an emergency-stop device or safety controller must not automatically restart ordinary machine movement.

A typical control sequence may require:

Emergency-stop released
↓
Safety system reset
↓
Safety status healthy
↓
Standard PLC confirms machine recovery
↓
Operator issues normal START command

The exact behaviour depends on the safety design and machine risk assessment.

The standard PLC should treat restored safety permission as one recovery condition—not as a start command.

Power-Recovery Behaviour

Power restoration should enter a defined startup or recovery state.

The program should decide:

  • Which values remain retained
  • Which commands are cleared
  • Whether the previous state is trusted
  • Whether field positions must be checked
  • Whether automatic operation is blocked
  • Whether operator reset is required

A common safe approach is:

On PLC startup:
    Clear movement requests
    Clear temporary bypasses
    Block automatic mode
    Enter INITIALIZING
    Validate I/O and communication
    Validate process data
    Check machine positions
    Enter READY_FOR_RESET

Automatic restart may be appropriate in some continuous processes, but it must be deliberately engineered and supported by the relevant risk assessment.

Bypasses Must Not Short-Circuit Recovery

A maintenance bypass may allow normal permissive logic to appear healthy.

For example:

Valve_Permitted :=
    Valve_Open_Feedback
    OR Valve_Bypass;

During recovery, this can hide the fact that the valve position is unknown.

Recovery should preserve the actual condition:

Valve_Feedback_Healthy := Valve_Open_Feedback;
Valve_Bypass_Active := Valve_Bypass;

The machine may block automatic recovery while a critical bypass remains active.

A bypass should never allow the controller to pretend that physical validation has succeeded.

HMI Recovery Information

A useful recovery screen should show:

  • Current machine state
  • Original fault
  • Active faults
  • Conditions blocking reset
  • Conditions blocking recovery
  • Required manual actions
  • Active bypasses
  • Communication status
  • Field-position status
  • Recovery timer
  • Whether a fresh START will be required

Avoid a generic message such as:

Machine not ready

A better message is:

Recovery blocked: discharge valve not confirmed closed.

Clear information reduces repeated reset attempts and unnecessary bypassing.

Recovery Tests That Should Be Performed

Commissioning should deliberately test:

  • Motor trip while running
  • Drive fault during startup
  • Communication loss during a transition
  • Remote I/O reconnecting late
  • Power failure while outputs are active
  • PLC restart with retained sequence data
  • Emergency-stop activation during automatic operation
  • Safety reset while process permissives remain invalid
  • RESET held while a fault clears
  • Recovery timeout
  • Manual actuator movement during fault
  • Analog data returning slowly
  • Bypass active during recovery
  • New fault appearing during recovery
  • Output feedback not returning after reset

For every test, confirm:

  • The machine enters the correct fault state
  • Output requests are controlled correctly
  • The first fault is preserved
  • Reset is rejected when conditions are invalid
  • Recovery does not skip startup validation
  • Old run requests do not restart equipment
  • A fresh START is required where intended
  • HMI diagnostics explain the blocked condition

Final Thoughts

Fault recovery should never be treated as the removal of an alarm bit.

A fault can leave the physical machine, internal sequence and process data in different states. Simply clearing the fault may allow old commands to become active before the system has confirmed that recovery is complete.

Professional recovery logic should define:

  • Fault states
  • Output behaviour
  • Reset requirements
  • Permissive revalidation
  • Communication recovery
  • Physical-position checks
  • Timeouts
  • Manual recovery actions
  • Safe return to startup

The recovered machine should not continue blindly from where it stopped.

It should return through a controlled path, confirm that the process is trustworthy and require the appropriate deliberate command before normal operation resumes.

A safe recovery behaves like a structured startup—not an uncontrolled continuation.

Leave a Reply

Your email address will not be published. Required fields are marked *