Automating GUI Clicks with Image Recognition

When you record and replay macros, many people hit the phenomenon "it worked yesterday but today the clicks are off." The cause is usually that the recorded macro acts on fixed screen coordinates. A window opened elsewhere, the resolution changed, you replayed on a different monitor — such changes make clicks land somewhere other than intended. Image-recognition automation solves this fundamental problem.

Image recognition finds on-screen elements by appearance rather than coordinates. You register a small image of the button you want to click, and at replay time the tool scans the whole screen for a region that matches that image. Wherever the button has moved, as long as it looks the same it can be found and clicked accurately — that's the strength of image recognition.

How image recognition finds elements

The mechanism is simple. First, capture and save a small image (a template) of the element you want as a landmark — a "Submit" button or a specific icon, say. When the macro runs, the tool grabs a snapshot of the screen at that moment and searches for a region matching the template. When it finds one, it returns the centre coordinates, and the macro moves the mouse there and clicks.

A key setting here is the match threshold (matching accuracy). You can tune it so a region counts as a match even without being 100% identical — for instance, at 90% similarity or above. A high threshold reduces false matches but may fail on tiny visual differences; a low one is more flexible but risks grabbing a similar but wrong element. The trick is to try it on the real screen and find the sweet spot.

When to use FIND_IMAGE vs WAIT_IMAGE

In makuroku, two representative commands let you use image recognition from a script. FIND_IMAGE searches for "where on the screen that image is right now." It can store the found coordinates in a variable, so it suits cases like "click a button that could appear anywhere."

WAIT_IMAGE, on the other hand, "waits until that image appears." A web page loading or an app launching takes time, and doing the next action before it finishes causes failures. With WAIT_IMAGE you can "wait until the icon signalling load-complete appears, then proceed" — reliable automation that adapts to the screen's state. Combining the two makes automation resilient to the timing variability common in real work, so it breaks far less often.

Where image recognition is especially useful

Image recognition is especially powerful in situations like these:

• • Apps where buttons or windows appear in different positions each time
• • When you want to wait for a load or process to finish before proceeding
• • Dynamic layouts that fixed coordinates can't handle
• • When you want to trigger a step when a specific icon or state appears

Image recognition has limits too. Designs whose appearance changes often, or translucent and animated elements, tend to make matching unstable. And because it scans the whole screen, it takes slightly longer than coordinate targeting. Rather than replacing everything with image recognition, the practical, fast approach is to mix them: use coordinates where they suffice, and image recognition only where positions move.

How to prepare good template images

Image-recognition accuracy depends heavily on the quality of the template you register. A few tips. First, crop the landmark region "small and distinctive." Cropping just the icon or text unique to a button, rather than the whole button, makes it less likely to be confused with other elements. Cropping too large a region, conversely, can fail to match on tiny surrounding changes (background colour or a neighbouring element).

Second, avoid parts whose colour or state changes easily. If you use a button that changes colour on hover, or a spot highlighted when selected, matching fails depending on the situation. Choose a stable part that always keeps the same look. Also, since the image size changes when the display scaling changes, it's safest to capture the template at the resolution and scaling you actually run at. Such small considerations, stacked up, greatly raise the success rate of image recognition.

Combining image recognition with other methods

Image recognition is powerful but not a cure-all. To build robust automation in practice, combining several methods, each where it fits, is the right approach. A button that never moves is fine with quick coordinate targeting. Leave to image recognition only the buttons that move or the elements that take time to appear. Further, use OCR when you want to read on-screen text, and HTTP requests when you want to integrate with an external service — choosing the method by purpose.

makuroku lets you freely combine these methods within a single script. "Wait for the screen with WAIT_IMAGE, find and click a button with FIND_IMAGE, read the resulting number with OCR, and branch with a condition" — you can write flexible automation close to human work as one flow. Rather than sticking to a single method, combine them to play to each one's strengths. This mindset is the shortcut to automation that truly works in the field.

What to do when image recognition fails

Once you start using image recognition, you'll sometimes hit "the image I registered isn't found." Several causes are possible. The most common is that the display scaling or resolution differs between when you captured the template and when you actually search. Then the image size changes and it won't match. Recapturing the template at the same settings as your operating environment is the reliable fix.

The next most common is too strict a match threshold. Demanding a 100% exact match, it fails on the slightest colour difference or anti-aliasing variation. Lowering the threshold a little, to count roughly 90% similarity as a match, is often more stable. But lowering too far increases false matches, so the trick is to probe for the sweet spot by testing on the real screen.

If that still doesn't work, review the region you chose as the template. Using a landmark whose background changes easily, or one highlighted when selected, makes the look change by situation and fail to match. Narrowing to a distinctive, stable part that always keeps the same look — a unique icon or logo — raises the success rate. "Choosing an easy-to-find landmark" is the biggest trick to stabilising image recognition.

Image recognition takes a little tuning at first, but once set up well it delivers a flexibility coordinates can't. It reacts accurately as long as the "look" is the same, even when the window moves or the layout changes — this robustness brings great peace of mind for using automation over the long term in real work.

Practical scenarios using image recognition

Let's see image recognition's ability in a concrete scenario. Suppose you automate periodically downloading data from a web app. This app's layout shifts slightly at each login, and the download button's position isn't fixed. A coordinate-based macro can't handle this change and breaks quickly.

This is where image recognition comes in. First, register the download button's icon as a template. The macro waits for the page to finish loading with WAIT_IMAGE, then finds the download button on screen with FIND_IMAGE and clicks its position. Wherever the button appears, as long as it looks the same it can be found and clicked accurately, yielding stable automation unaffected by layout changes.

Further, waiting with WAIT_IMAGE until the download-complete message appears before proceeding means no misses even on slow-network days. In this way, image recognition brings into automation the judgement a person does as a matter of course — "look at the screen's state and operate the right place at the right time." That's exactly why you can build automation that rarely breaks even in the "change-filled place" that is a real work environment.

Such scenarios apply not only to web apps but to desktop apps and business systems — every situation. Work you'd given up on automating because "the position moves" or "the timing is unpredictable" comes within reach with image recognition. First picture a task whose position kept shifting on you, and start by stabilising it with image recognition.

Building automation that rarely breaks

Whether automation is genuinely usable in the field comes down to how well it withstands the unexpected. Automation that relies only on recorded coordinates may work the first time but tends to stall the moment the environment changes a little. Adding image recognition lets you build in the judgements a person makes unconsciously — "click by looking," "wait until the screen settles."

makuroku includes image-recognition commands like FIND_IMAGE and WAIT_IMAGE as standard, alongside GUI recording. Record actions to build the skeleton, then replace only the moving positions and timing waits with image recognition — this combination yields automation that endures real-world messiness instead of breaking on the first surprise. Every feature is free to try, so start by stabilising, with image recognition, the task whose position kept shifting on you.