When a network fails during visual inspection, treat the outcome as unknown until the system can confirm what happened. Capturing an image, completing an upload, accepting a submission, processing an inspection, and saving a result are different milestones. Retries can help with temporary faults, but they need limits and duplicate protection. A connection failure must never be treated as a passing inspection.
A line-side camera or application may lose contact with a server at any step. A timeout does not tell you whether the server received or processed a request. A recovery plan should make uncertainty visible to operators and preserve the link between the part, its image, and its result. Microsoft’s guidance recommends retrying only when a failure may be transient and setting a retry limit. HTTP semantics also warn against automatically repeating non-idempotent requests unless there is a way to establish whether the first request took effect. (Microsoft Learn: Retry pattern; IETF RFC 9110)
Before you start
- Map the inspection lifecycle: Define what “captured,” “uploaded,” “accepted,” “processed,” and “saved” mean in your system, and how each state is confirmed.
- Define the production response: Use the approved quality and control procedure to decide whether a part is held, diverted, reinspected, or allowed to proceed when its inspection status is unresolved.
- Identify the record key: Decide which stable identifier ties an inspection request to the part, image, station, product or work order, and configuration used at inspection time.
- Agree on recovery ownership: Specify who or what checks uncertain requests after connectivity returns, and who resolves cases the system cannot reconcile automatically.
- Review existing retry behavior: Check every layer—device, client library, gateway, service, and queue—to avoid overlapping retry policies that multiply attempts unexpectedly.
Practical steps for outage recovery
Step 1: Treat each lifecycle milestone separately
A camera can capture an image locally without sending it to a server. An upload can reach storage without the inspection request being accepted. A server can accept a request while processing remains pending. Processing can finish while saving or delivering the result is still unconfirmed.
Use state names that describe confirmed evidence, such as captured locally, upload unconfirmed, submission accepted, processing, result saved, and outcome unknown. Avoid a single “done” indicator unless the system has confirmed the specific condition the next step depends on.
For a broader view of how inspection records support traceability, see AI and real-time data for manufacturing traceability.
Step 2: Give each inspection a durable identity
Create an identifier for the inspection event before sending it. Keep the identifier stable across retries and associate it with the part, the relevant image, and the inspection context. If the interface or API supports an idempotency key, reuse the same key for attempts representing the same inspection—not a new key for every retry.
Idempotency means repeated requests have the same intended effect as one request. It does not mean every system guarantees exactly-once execution or that every step—uploading, processing, saving, and notifying—happens only once. The IETF’s HTTP specification says clients generally should not automatically retry non-idempotent requests unless they can establish that repeating the request is safe or determine that the original request was not applied. (RFC 9110)
For an upload, a stable inspection ID alone may not prove that the complete image arrived. Where supported, use the upload protocol’s completion status and integrity checks. For example, Google Cloud Storage documents resumable uploads with session identifiers and recommends checking the final uploaded object’s integrity. This is a service-specific example, not a guarantee about other systems. (Google Cloud Storage: Resumable uploads)
Step 3: Separate retryable faults from faults that need correction
Retry only errors your system classifies as temporary and safe to repeat. A connection reset, timeout, or temporary service-unavailable response may be retryable, but the right policy depends on the operation and service contract. Invalid data, missing permissions, unsupported configuration, or a rejected image format generally call for correction or escalation—not repeated identical attempts.
Set a finite retry budget and a total time limit that fit the station’s process and decision window. Use exponential backoff with jitter—progressively longer, randomized waits—instead of letting many clients retry at the same moment. Coordinate policies across layers: if a client and a downstream service both retry independently, attempts can multiply and increase pressure on a struggling dependency. Microsoft recommends bounded retries and cautions against duplicated retry layers; AWS guidance also describes backoff with jitter and retry limits as ways to control retry load. (Microsoft Learn: Transient fault handling; AWS SDK retry behavior)
Record the error category, attempt count, elapsed time, and final state. Repeated retries may indicate an underlying service or connectivity problem; they are not a reason to increase the retry count indefinitely.
Step 4: Show uncertainty instead of guessing
A timeout after a request is sent means the caller did not receive a timely response. It does not prove that the server failed to receive, accept, or complete the request. Display a state such as outcome unknown or submission unconfirmed until the system can check.
After connectivity returns, reconcile using the original inspection ID: query the receiving system or its durable records, if that capability exists, to determine whether the submission was accepted, is processing, was completed, or is absent. If the system cannot determine the outcome, send the same logical request again only when duplicate protection is in place. Otherwise, use the defined manual recovery procedure. Never create a second inspection record just to make the status appear resolved.
Step 5: Recover records in a controlled order
If an implementation includes a persistent local queue or buffer, specify how it handles device restarts, storage limits, expired records, duplicate messages, and prolonged outages. These are design choices, not capabilities to assume. If no persistent buffer exists, state clearly what can and cannot be recovered after connectivity is lost.
When queued records are sent later, retain the original capture time and any ordering needed to understand the inspection sequence. Also retain the inspection configuration or version used at capture, where applicable. If configurations can change during an outage, define whether pending inspections are processed under the original configuration, reassessed under the current one, or escalated for review. Do not silently mix the two.
For more on image quality as a separate part of the inspection chain, see why high-speed defect detection starts with a better image.
Worked example: Connection drops after submission
This example is fictional and is included to explain the recovery path.
A station captures an image and submits inspection event E-1042. The server accepts the request and starts processing, but the response is lost when the network drops. The station must not assume the request failed and send a new event ID. It must not display a pass.
Instead, it marks E-1042 outcome unknown and follows the plant’s approved rule for the part while the status is unresolved. After reconnecting, the software checks for E-1042. If the system confirms a saved result, the station displays it. If the system confirms the request was never accepted, it retries the same event under the agreed duplicate-protection policy. If the status remains indeterminate, the inspection goes to the designated manual recovery path.
Network-outage recovery checklist
Answer yes or no for each question:
- Can an operator distinguish image capture from upload completion?
- Does the workflow distinguish server acceptance, completed processing, and saved results?
- Does each inspection keep the same durable identifier across retries?
- Can the system prevent or detect duplicate submissions for the same inspection?
- Are retryable errors defined separately from errors that need correction?
- Are retries bounded, delayed with backoff and jitter, and coordinated across system layers?
- Is there a documented reconciliation or manual recovery path for an unknown outcome?
- Does the approved line procedure define what happens to the part while its inspection status is unresolved?
A “no” indicates a design or procedure gap to resolve before relying on automatic recovery.
Limitations
Retries address some temporary communication failures. They do not correct bad images, invalid requests, incorrect configuration, or authorization failures. Delayed retries may help with temporary overload, but retries alone cannot resolve sustained overload. Persistent queues and resumable uploads can support recovery in some implementations, but their behavior depends on how they are built and configured. Do not assume they exist or guarantee that no data will be lost.
Even with idempotency, a multi-step workflow can leave work partially completed—for example, with a stored image but no processed result. Recovery therefore needs observable states and a reconciliation rule, not just a retry loop. No general pattern can promise exactly-once execution across every component or eliminate every duplicate, delay, or lost record.
Most importantly, network recovery is not a quality disposition. A successful upload, a completed software task, or the absence of an error message does not mean the part passed. Decisions to hold, divert, reinspect, or release a part must follow the plant’s approved quality and control procedure. See when people should review AI inspection results for more on handling inspections that need human review.
Next step
This week, map one inspection station’s lifecycle from image capture to operator feedback. Mark where the system has affirmative confirmation, where it only has a timeout, and what the approved procedure requires for a part at each unresolved point.
Planning recovery behavior for a visual-inspection workflow? Review the Loopr platform overview to discuss inspection requirements for your environment.
Frequently asked questions
Does a timeout mean the inspection failed?
No. It means the sender did not receive a response in time; the server may still have received or processed the request. Keep the outcome marked unknown until the system can confirm it or the defined recovery process resolves it.
Should inspection software retry every failed request?
No. Some failures may be temporary, while others require correcting data, permissions, or configuration. Retry only operations and error types your system is designed to repeat safely, with a finite retry limit.
What is the difference between idempotency and exactly-once processing?
Idempotency means repeating the same logical request has the same intended effect as making it once. Exactly-once execution is a stronger end-to-end guarantee across components. Retries and idempotency alone do not establish that guarantee.
Is a captured image the same as a saved inspection result?
No. The image may be captured locally or uploaded without the inspection being accepted, processed, or saved. Define and communicate each milestone separately so operators know what has actually been confirmed.
What should the line do while inspection status is unknown?
Follow the approved plant procedure for that inspection and part. The software should not infer a pass from a network outage, a timeout, or the completion of a retry process.
Sources
- Retry pattern, Microsoft Learn. Supports retry choices, limits, idempotency considerations, and risks from aggressive or layered retries.
- Transient fault handling, Microsoft Learn. Supports logging and monitoring retry behavior, planning for continual failures, and considering idempotency in multi-step operations.
- RFC 9110: HTTP Semantics, Internet Engineering Task Force, June 2022. Defines idempotent request semantics and advises against automatically retrying non-idempotent requests without evidence that the operation is safe or was not applied.
- Resumable uploads, Google Cloud documentation. Gives a service-specific example of upload sessions, recovery after an interrupted transfer, and final-object integrity checking.
- Retry behavior, AWS SDKs and Tools Reference Guide. Describes bounded retry configuration and exponential backoff with jitter in AWS SDK retry modes.





