How can I automate repeat mouse clicks, with delay, from fixed coordinates within a browser window?

Question

Looking for a quick and dirty way to automate this mundane activity. I presume this could be done with an automation to press the mouse button from two separate pixel coordinates, with delay, so long as the browser window doesn't move -- unless there's a better way.

I tried building a Watch Me Do workflow, in Automator, but can't seem to get it to work. I get the error "application Chrome is not running when it is expected to." It also never lists mouse clicks for some reason.

I do have accessibility set up, in system preferences > security & privacy > privacy > accessibility. Automator, Chrome and System Events are all checked within there (among other stuff of course).

I tried the markashworth workaround as detailed here -- which also doesn't seem to work. Using his method, I get this error:

UPDATE: I just downloaded "cliclick" and figured out how do both clicks as a terminal command, with the proper delays. BUT -- I don't know code and need a way for this terminal command to repeat "x" amount of times. Anyone know how I could do that in Script Editor? The command line is:

cliclick c:132,286 w:100 c:132,286 w:5000 c:1100,238

I would need a few seconds of delay before the loop repeats, which I presume I could do with a final "w:" in that line. Thanks for the help!!

Answer 1

The following steps are a solution for automating the repeat clicking of a mouse, back and forth, at two different fixed positions within a web page -- i.e. clicking two buttons or links within in a web-page, with a user-defined delay between each click.

All this is, is a slight modification of what was described over in this thread -- this one being for two website "buttons" rather than one.

IMPORTANT -- this automation does NOT work in the background. In other words, you can't be doing something else on your computer at the same time this automation is running. In addition, your browser can't move its physical position on your screen after the automation begins, because you're literally telling the computer to press two buttons at fixed X,Y pixel coordinates. The "buttons" on the web-page, therefore need to stay at the same spots on your screen for this to work.

Steps as follow:

1. Click the Apple icon (upper left), select "System Preferences." Type "security" in the upper-right search window. This should highlight Security & Privacy. Open that.
2. Click lock icon, enter your password, click Unlock.
3. Click Accessibility from the scrollable list (blue icon). Now click the plus sign below the box on the right. Then navigate to Applications and scroll down to Utilities. From within there, put check marks on Terminal AND Script Editor. Then finally select whichever web browser you use under the Applications list. You can then close out of this window.
4. Launch Terminal. (I do this by typing "Terminal" as a Spotlight search (the magnifying glass icon, upper right of screen)).
5. Remember (generally speaking) what that bottom-most line appears like when Terminal opens up. You'll want Terminal to revert back to whatever that is after each of two short downloads occur within terminal -- both are idiot-proof. `
6. First Download -- Home Brew. As strange as this sounds, all you do to download this "program" is copy the single line of text you see below "Install Home Brew" on its website. Paste that text into Terminal, hit Enter and wait for the process to complete. When the bottom line in Terminal appears as it was when you first launched it, you're done.
7. Second Download -- CliClick (second download). Simply type these three words in Terminal, then hit by Enter:
   brew install cliclick
   (note: CliClick took about 90 seconds to install as I recall)
8. When the download is complete, you can minimize Terminal -- as you're done with it now. Close it entirely if you're certain cliclick has been downloaded.
9. Next, simply determine the X,Y pixel coordinates for both website "buttons" you want the robot to click on. Before doing this, I recommend placing the browser window on your screen in such a way that its upper-left corner abuts the upper-left corner of your display. To determine the coordinates, simply press COMMAND+SHIFT+4 and jot down both sets of numbers atop each "button" you over over. The top digits are X coordinates and the bottom are therefore Y. You'll end up with two pairs of numbers for each button, obviously. Write these down somewhere, then press Escape to turn off the crosshairs.
10. Launch Script Editor. Like Terminal, I did this by typing "script editor" as a Spotlight search
11. Now, copy and paste ALL of the following text into Script Editor. It must show up there exactly ias it does below, including indents etc:

tell application "Google Chrome"
    activate
    repeat 208 times
        do shell script "/usr/local/bin/cliclick c:" & "132,286"
        delay 3.5
        do shell script "/usr/local/bin/cliclick c:" & "1100,230"
        delay 4.5
    end repeat
end tell

Now you're pretty much home free! All you do is replace the first set of comma-separated digits with the coordinates you of your first website button in X,Y format. Leave the quote marks as they are. Repeat this for your second button and change the delays as you wish, which are in seconds.

Finally, set repeat value to the amount of loops you want to run and save the script. Voila! Pressing "play" in Script Editor will roll the press. You can stop it by simply stealing your pointer back and clicking on the "stop" button, which I did multiple times to change the delays based on internet speed and server speed.

Answer 2

There is a much easier way to do this on the Mac: Keyboard Maestro.

Despite the name, it automates the mouse as well as the keyboard.

It let you use GUI tools to setup the clicks (all you have to go is tell it to watch where you click, then click, and it will do the calculation for you).

It will let you set a specific delay between the clicks.

It also has built-in tools for clicking links in Safari and Chrome, which may also help.

It will also let you use AppleScript or JavaScript (or anything else) as part of the macro.

All of the actions are drag-and-drop.

There is a very active forum where lots of helpful users hang out (and also the developer of the program).

There are about 1,000 other things you can do with Keyboard Maestro too. Once you start using it, you'll realize all the ways it can help.

(To be clear, I have absolutely no connection to the app except being an extremely grateful user for a number of years. I pay for my own upgrades as soon as a new version comes out because it's the most useful utility on the Mac.)

Answer 3

So glad you figured out your script!

Just wanted to include that you also may be able to use Selenium to do this for you. https://www.seleniumhq.org/ Selenium is used by Quality Assurance professionals to automate the testing of websites, and is made to do everything a user would do and more. Here is an article that describes automating mouse clicks with Selenium... https://www.guru99.com/keyboard-mouse-events-files-webdriver.html And here is a description of including a wait or timeout in a Selenium script... https://stackoverflow.com/questions/20009211/getting-selenium-to-pause-for-x-seconds

Hope this helps.