A factory demonstration detects defects on a brightly lit bench. On the production line, a replacement lamp changes glare on the metal surface, the camera vibrates, and the operator turns parts at a slightly different angle. The most important question is how the system behaves after that change. An evaluation built around a curated image collection will miss much of what determines whether the tool works.
Computer vision platforms can cover image collection, annotation, training, deployment, and workflow logic. Some tools specialize in one stage. Evaluate the chain around the camera and the business action, then choose which stages should be managed together. A platform designed for reviewing images may solve the largest problem even when model training happens somewhere else.
Specify the output before collecting images
Image classification assigns labels to an image or crop. Object detection predicts locations, often represented by rectangles. Segmentation marks regions or pixels, and tracking links objects across frames. Pick the output that matches the action. Counting parcels might need detection and tracking, while measuring the size of a surface defect may need a calibrated segmentation workflow. An application that rejects an entire item may only need a reliable result for a known crop.
Write down the cost of different errors. A false rejection may slow a line and send good items for inspection. A missed defect may be much more consequential. Choose an operating threshold that reflects the costs and the capacity for manual review. A model metric measured over all thresholds is useful for development, but the deployed system acts at particular thresholds and under particular conditions.
Define the unit that triggers action. Adjacent video frames are correlated, and a per-frame false alarm rate does not directly tell you how many items are sent for review. Test complete clips or production units and count the actual events created by your workflow. Include the time available to make the decision before an item leaves the inspection station.
Collect the variation that changes the answer
Create a matrix of sites, camera positions, lighting, backgrounds, and product variants. Sample normal variation as well as defects. For rare events, keep track of which images were staged and which came from regular operations. A staged example can help the model see a useful pattern while differing in ways a model can exploit. Reserve a real sample for evaluation when possible.
Avoid splitting neighboring frames or nearly identical products across training and validation. Group samples by clip, batch, site, or inspection session according to the future deployment. Include a held-out camera or shift if new cameras or shifts are expected. Report performance by the conditions that matter. A strong average can hide a failure on one finish, lens, or background.
Labeling rules should explain ambiguous cases. Give reviewers examples of partially visible objects, borderline defects, and regions that should be ignored. For segmentation, define whether an uncertain border should be labeled tightly or flagged for review. Have two reviewers label a small common set and investigate disagreements. Better annotation policy can improve the dataset before any model change.
Use tools where they remove a real bottleneck
CVAT is a specialist option when annotation is the main workload. Its documentation covers image and video labeling and related workflows. Test the geometry you need, the review process, and the export format. Import an exported sample into your training pipeline and check whether class labels, masks, and coordinates survive the trip. CVAT documentation
Roboflow is relevant when a team wants a broader computer vision workflow, from dataset work to model and inference workflows. Run the exact project type and deployment style you intend to use. Assess whether dataset versions can be connected to a production model, how edge or hosted inference fits your network, and how much of the pipeline can be exported. Confirm feature availability for your account and target hardware. Roboflow documentation
OpenVINO belongs in the inference evaluation when your team needs to optimize and run a model on supported hardware. It complements labeling and training tools. Test conversion, supported operations, and output parity on your chosen device. Compare the complete application, including decoding, resizing, and postprocessing, because network inference is only one part of the latency. OpenVINO documentation
A managed workflow can reduce the number of integrations a small team owns. A modular setup can give teams more control over annotations, hardware, and runtime. Both approaches need a way to reproduce the dataset used for a release and to trace a bad prediction back to the camera and model version. During evaluation, have a second person retrieve that evidence.
Put the runtime next to the camera for a real test
An edge deployment may reduce transfer requirements and continue operating during a network interruption. It also creates a fleet management problem. Models, device images, and configuration need a controlled update and a recovery path. Central inference can simplify deployment in some environments, while network capacity, upload latency, and privacy constraints may limit the images it can receive.
Measure end-to-end timing. Include capture, decoding, preprocessing, inference, tracking or business logic, and delivery of the result to the actuator or interface. Run the system with the resolution, frame rate, and simultaneous streams expected in production. Observe how it behaves when the input rate exceeds capacity. Dropping frames, delaying decisions, and queueing all have different effects on an inspection workflow.
Decide which data is retained. A useful error review may need selected frames, but storing every frame can create unnecessary cost and exposure. Record enough context to investigate a decision under the applicable policies. For installations that capture people or sensitive locations, confirm where images are processed and how reviewers obtain access. Test deletion across local devices, upload buffers, and central storage.
Consider what a device should do when its own inputs become unreliable. A blocked lens, dropped feed, or exposure that leaves the frame nearly black should produce a clear operational state. Configure the review interface to distinguish an unobservable item from an item the model assessed as normal. Test the recovery behavior when the camera becomes available again. Operators need to know whether the system skipped items during the interruption and whether those items can be inspected elsewhere.
Test changeovers between product types as well as ordinary steady operation. If the same line handles several components, the application must choose the correct model or thresholds at the right time. Inspect what happens to items already in flight. A wrong configuration can produce a burst of errors even when each model works under the conditions for which it was validated.
Design the improvement loop before going live
Run a shadow trial on representative cameras and compare predictions with the existing inspection or review process. Collect samples of false alarms, missed events, and uncertain cases. When a reviewer corrects a prediction, preserve the input, original output, and reason for correction. Use those cases for analysis and define which ones are eligible for training. Keep a separate holdout so improvements can be evaluated consistently.
Track changes in the physical setup as operational events. Replacing a camera, altering exposure, or moving a light can change accuracy before any code changes. Maintain a small calibration or reference set for each important station. Define a check that operators can run after maintenance and a fallback if the check fails.
Select the platform after watching it handle the full path from camera to action and then back to review. The deciding evidence is how it behaves on awkward images, under realistic timing, and after an ordinary change on the line. A platform that makes those cases easy to reproduce gives the team a practical way to improve the system after installation.