0

With the amount of spam the Stack Exchange network has been seeing recently, I decided to reduce the number of clicks I need to flag these posts, by using a 'Watch Me Do' action in Automator.
Click, click, click, makes a recording, which I then wanted to just playback each time I wanted to raise a flag.
It doesn't work.

I've never used 'Watch me do' before so it took a little research to figure out how I could examine what it had recorded. I now have it as an Applescript.
Excerpt, highly abbreviated to just the pertinent call…
¬ newlines added to make it easier to read in here, whilst still being able to copy/paste to Script Editor.

-- Click the “Flag” button.

    set uiScript to "click UI Element \"Flag\" of group 7 of group 8 of ¬ 
        UI Element 1 of scroll area 1 of group 1 of group 1 of tab group 1 of ¬ 
        splitter group 1 of window \"Long long spammy page name\" ¬ 
        of application process \"Safari\""

So, it seems it has recorded the exact page title, which is completely useless. I can't even test it works on the same page, because I can't repeat the flagging action.

What I need to figure out is how to genericise that uiScript call to just be "my current window".
I had the naïve idea I could maybe change the reference to window 1 but that didn't work & my limited talent ran out at that point.

Is there a simple 'this window' substitute?

13
  • As an aside, I have an email address that I use for signups to various services. I created a rule that moves everything that comes in on that address to the Junk folder except for what’s in an exception list. It took a month or two of just checking that folder to add legit items to the exception list but since doing this, my mailbox has been virtually spam free.
    – Allan
    Jul 12, 2023 at 17:07
  • 1
    @Allan - I bought SpamSieve about 20 years ago, to similar end ;))I also own my own domain & associated mail servers, so I can make black holes for anything I never want to hear from again. I meant the SE network, btw, not email. I'll add clarification.
    – Tetsujin
    Jul 12, 2023 at 17:10
  • Gotcha…AS is definitely not in my wheelhouse, but I don’t think you can do a “Watch me do” action on a webpage. You might be able to select elements like <div> or <radiobutton> with some Apple/JavaScript. Maybe there’s a plugin for your browser of choice that automates keystrokes/clicks?
    – Allan
    Jul 12, 2023 at 17:18
  • The action itself records just fine. Playback stalls because the page changes after initial flagging… then it won't work on any other page because it considers the title a part of the action. I've learned how to convert it to plain Applescript to make it editable - I just need to get rid of the page title & have it work on 'this page'. My Applescript skills are… woeful:\
    – Tetsujin
    Jul 12, 2023 at 17:23
  • 1
    At least you have AppleScript skills! I’ve got AppleScript stress headaches
    – Allan
    Jul 12, 2023 at 17:29

3 Answers 3

1
+100

Looking at the sample AppleScript code you supplied...

-- Click the “Flag” button.

set uiScript to "click UI Element \"Flag\" of group 7 of group 8 of ¬ 
    UI Element 1 of scroll area 1 of group 1 of group 1 of tab group 1 of ¬ 
    splitter group 1 of window \"Long long spammy page name\" ¬ 
    of application process \"Safari\""

I realized there was not enough here to identify what the problem was. I, too, have also used Automator's "Watch me do" then copied those recorded steps into Script Editor.app.

I used Automator's "Watch me do" , like you did, to generate an AppleScript which will click the “Flag” button. Then I pasted the "recorded action" (the following code) into Script Editor.app.

-- Click the “Flag” button.
delay 4.881676
set timeoutSeconds to 2.0
set uiScript to "click UI Element \"Flag\" of group 12 of group 9 of UI Element 1 of scroll area 1 of group 1 of group 1 of tab group 1 of splitter group 1 of window 1 of application process \"Safari\""
my doWithTimeout(uiScript, timeoutSeconds)

on doWithTimeout(uiScript, timeoutSeconds)
    set endDate to (current date) + timeoutSeconds
    repeat
        try
            run script "tell application \"System Events\"\n" & uiScript & "\nend tell"
            exit repeat
        on error errorMessage
            if ((current date) > endDate) then
                error "Can not " & uiScript
            end if
        end try
    end repeat
end doWithTimeout

The problem is that the above code is long-winded, and not easy to understand. But if you look at it closely, you can see that it can be "backward engineered" and trimmed down, in a sense.

The following code is my version of re-writing the code created by Automator's "Watch me do".


tell application "System Events"
    tell application process "Safari"
        click UI element "Flag" of group 12 of group 9 of ¬
            UI element 1 of scroll area 1 of group 1 of group 1 ¬
            of tab group 1 of splitter group 1 of window 1
    end tell
end tell

So, it seems it has recorded the exact page title, which is completely useless. I can't even test it works on the same page, because I can't repeat the flagging action.

What I need to figure out is how to genericise that uiScript call to just be "my current window". I had the naïve idea I could maybe change the reference to window 1 but that didn't work & my limited talent ran out at that point.

Is there a simple 'this window' substitute?

After repeated testing of my version of how you were trying to click the "Flag" button, I soon realized the problem was not with "window 1" but that 1 or more of the element numbers get changed (group 7) may now be (group 8)... for example.

For now though, you won't need to use the above code to click the “Flag” button because I already provided you with an alternative option that should always work for that... With the following code.


In Safari, select "Allow JavaScript from Apple Events" in the Developer menu item.

This following AppleScript should work for you.

clickFlag(0) -- 0 = 1st Flag button, 1 = 2nd Flag button, etc.

on clickFlag(theNumber)
    tell application "Safari"
        do JavaScript "document.getElementsByClassName" & ¬
            "('js-flag-post-link s-btn s-btn__link js-gps-track')[" & ¬
            theNumber & "].click();" in document 1
    end tell
end clickFlag

The beauty of the method above is that Safari doesn't even need to be the front most application and the code will work on the currently visible tab.


Taking it one step further, this following AppleScript code will click the first "Flag" button on the page then attempts to click the "Spam" option

NOTE: This following AppleScript code is only meant to target the 1st post of any thread.

clickFlag(0) -- Clicks the "Flag" button

tell application "System Events"
    tell application process "Safari"
        set frontmost to true
        repeat until frontmost
            delay 0.1
        end repeat
        repeat until UI element 1 of scroll area 1 of group 1 of group 1 of ¬
            tab group 1 of splitter group 1 of window 1 exists
            delay 0.1
        end repeat
        delay 0.1 -- can probably be removed
        try
            click radio button 1 of group 1 of group 1 of group 2 of group 13 ¬
                of group 9 of UI element 1 of scroll area 1 of group 1 of ¬
                group 1 of tab group 1 of splitter group 1 of window 1
        on error from offendingObject
            log offendingObject
            repeat with i from 1 to 20
                try
                    click radio button 1 of group 1 of group 1 of ¬
                        group 2 of group i of group 9 of UI element 1 ¬
                        of scroll area 1 of group 1 of group 1 of ¬
                        tab group 1 of splitter group 1 of window 1
                    exit repeat
                on error from offendingObject
                    try
                        click radio button 1 of group 1 of group 1 of group 2 ¬
                            of group i of group 8 of UI element 1 of ¬
                            scroll area 1 of group 1 of group 1 of ¬
                            tab group 1 of splitter group 1 of window 1
                        exit repeat
                    end try
                end try
            end repeat
        end try
    end tell
end tell

on clickFlag(theNumber)
    tell application "Safari"
        do JavaScript "document.getElementsByClassName" & ¬
            "('js-flag-post-link s-btn s-btn__link js-gps-track')[" & ¬
            theNumber & "].click();" in document 1
    end tell
end clickFlag

If you look at my on error from offendingObject command above, I use the seldom used on error from which will log the actual item that triggers the error. For example, the result of the error log may be something like (*group 2 of group 13 of group 9 of UI element 1 of scroll area 1 of group 1 of group 1 of tab group 1 of splitter group 1 of window 1 of application process Safari*). What this is telling us is "group 2" triggered the error. While this is true, the actual problem is with the second item "group 13". There is no group 2 because there is no group 13. Long story short, it's pretty much always the second item of the error log that is the problem. This is why I use the repeat commands... It tries group 1, then group 2, etc. ... All the way up to group 20 until it finds the correct value.


This following .GIF demonstrates using the code directly above, on 4 different tabs in 1 window of Safari. I tested the code successfully on 2 different macs with 2 different macOS versions and 2 different versions of Safari. What it does is clicks the "Flag" button, then selects "Spam". I left out the commands to automatically press the "Flag question" button so the code can be used for testing purposes on the first post in any thread. I can add those commands at a later time. Doing that would automate the entire process.

enter image description here

8
  • I can only get it to open the flag dialog [even if it's out of sight below the viewport]. But, it does nothing else. [Hard to test as I need to wait for new spam to come in each time. At one point we were getting about 20 an hour, which is why I first thought of trying to automate the task.]
    – Tetsujin
    Jul 13, 2023 at 10:14
  • @Tetsujin I made major edits to my post. I'm pretty sure all of my code should work for you. You can test the code on any thread since I did not include the code that will actually "submit it" by pushing the "Flag question" button. Let me know how it works out please
    – wch1zpink
    Jul 16, 2023 at 7:25
  • I tried it… still nothing. If I could even figure out why it's not seeing the same elements in the same order, I could probably finagle it from here - but I don't know how to get the UI Inspector to even have a poke at it. It seems to be part of Xcode, which I really don't want to have to download & install just to pick one bit out of it. You've done more than enough work on this now, I can't ask you to do more, but if you know where I could get hold of this Inspector without all of Xcode, I could at least have a poke at it myself.
    – Tetsujin
    Jul 16, 2023 at 7:43
  • reams & reams of --> error number -1719 from group 2 of group 20 of group 8 of UI element 1 of scroll area 1 of group 1 of group 1 of tab group 1 of splitter group 1 of window 1 of application process "Safari" as it goes through each iteration.
    – Tetsujin
    Jul 16, 2023 at 7:44
  • 1
    AH! Don't know what I did, but it worked. I copy/pasted from the Automator code, but still no joy. I then took out all the repeats & then the offendingObject references as I couldn't tell what they were doing - and Tadaahhh. I'll paste my resulting code into another answer, but this one's yours for figuring it out. Thank you.
    – Tetsujin
    Jul 16, 2023 at 8:25
1

This is a preliminary script. Using the click command will make it easy to specify the ui element in question. Use command-shift-4 to get the crosshair with pixel coordinates while hovering over the desired element. In my case, it is x=375, y=385.

tell application "Safari"
    activate
    delay 0.4
    tell application "System Events" to tell application process "Safari"
        
        set bf to click at {375, 385} -- or wherever the `flag` 'button' is on your screen
        --> button "Flag" of group 12 of group 8 of UI element 1 of scroll area 1 of group 1 of group 1 of tab group 1 of splitter group 1 of window "applescript - How do I convert a specific browser page name to just be \"this page\"? - Ask Different" of application process "Safari" of application "System Events"
        
    end tell
end tell

Now that you know the specifier, you can streamline the script.

tell application "Safari"
    activate
    delay 0.4
    tell application "System Events" to tell application process "Safari"
                
        -- since you are inside a tell block, strip out extraneous
        set bf to button "Flag" of group 12 of group 8 of UI element 1 of scroll area 1 of group 1 of group 1 of tab group 1 of splitter group 1 of window 1
        click bf
    end tell
end tell

On these pages, groups are various bits and pieces of the page.

If you do something like this, and walk through some of the groups, then you will see the pattern:

        set g8 to group 8 of UI element 1 of scroll area 1 of group 1 of group 1 of tab group 1 of splitter group 1 of front window
        UI elements of group 5 of g8
        --> {static text "So, it seems it has recorded the exact page title, which is completely useless. I can't even test it works on the same page, because I can't repeat the flagging action." of group 5 of group 8 of UI element 1 of scroll area 1 of group 1 of group 1 of tab group 1 of splitter group 1 of window "applescript - How do I convert a specific browser page name to just be \"this page\"? - Ask Different" of application process "Safari" of application "System Events"}

As you can see, group 5 specifies the paragraph beginning with "So, it seems…" *Group 6 refers to the subsequent paragraph. For me, group 12 is the flag button.

set g8 to group 8 of UI element 1 of scroll area 1 of group 1 of group 1 of tab group 1 of splitter group 1 of front window
set bflag to UI elements of group 12 of g8 whose title is "Flag"
--> {button "Flag" of group 12 of group 8 of UI element 1 of scroll area 1 of group 1 of group 1 of tab group 1 of splitter group 1 of window "applescript - How do I convert a specific browser page name to just be \"this page\"? - Ask Different" of application process "Safari" of application "System Events"}
properties of item 1 of bflag

Here are the properties. I moved them outside of the code block to make formatting easier. Note that I used item 1 of above because the result of the prior line was a list.

--> {minimum value:missing value, orientation:missing value, position:{365, 458}, class:button, role description:"button", accessibility description:"", focused:false, title:"Flag", size:{27, 18}, value:missing value, help:"Flag this post for serious problems or moderator attention", enabled:true, maximum value:missing value, role:"AXButton", entire contents:{}, subrole:missing value, selected:false, name:"Flag", description:""}

1
  • 1
    +1 for showing me an alternate way to get an item's UI Element's hierarchy by using "command-shift-4 to get the crosshair with pixel coordinates while hovering over the desired element." then using the click at command to return its value. I was using Automator to get the UI Elements. However your technique does not always return "nice, pretty, and easy to use" results. Just try your posted solutions with Google Chrome instead of Safari. You'll see exactly what I mean. Either way, +1 for you.
    – wch1zpink
    Jul 18, 2023 at 13:41
0

Just to close the loop on this, following wch1zpink's hard work, this is what eventually worked. I'm not sure how, but it does & that's good enough for me ;)

clickFlag(0) -- Clicks the "Flag" button

tell application "System Events"
    tell application process "Safari"
        set frontmost to true
        repeat until frontmost
            delay 0.1
        end repeat
        repeat until UI element 1 of scroll area 1 of group 1 of group 1 of ¬
            tab group 1 of splitter group 1 of window 1 exists
            delay 0.1
        end repeat
        delay 0.1 -- can probably be removed
        try         
            --on error from offendingObject
            --log offendingObject
            --repeat with i from 1 to 20
            try
                click static text 1 of group 1 of group 1 of group 1 of group 2 of group 8 of group 8 of UI element 1 of scroll area 1 of group 1 of group 1 of tab group 1 of splitter group 1 of window 1
                
                --exit repeat
                --on error from offendingObject
                try
                    click UI element "Flag question" of group 8 of group 8 of UI element 1 of scroll area 1 of group 1 of group 1 of tab group 1 of splitter group 1 of window 1
                    --exit repeat
                end try
            end try
            --end repeat
        end try
    end tell
end tell

on clickFlag(theNumber)
    tell application "Safari"
        do JavaScript "document.getElementsByClassName" & ¬
            "('js-flag-post-link s-btn s-btn__link js-gps-track')[" & ¬
            theNumber & "].click();" in document 1
    end tell
end clickFlag

There was one additional hoop to jump. Saving this as a Service [Quick Action] meant that Gatekeeper would sandbox it & not allow it to run.

The workaround was to save the script as an app, then launch the app itself from automator, as a Service. A bit lumpy, but it works.

Discovered from Stack Overflow - com.automator.runner.xpc is not allowed assistive access - cannot click when running a quick action

3
  • I usually try avoiding Automator to run code or launch applications, whenever possible. An alternate option, which I would use instead, would be to save your code as a .scpt file and run the script file directly from the Script menu in the Menu Bar. Because I had nothing better to do I decided to create a script to automate the process of enabling the Script menu in the Menu Bar and adding your .scpt file to it. The full description and the code is here pastebin.com/t7s72FzQ and a .GIF of running that code is here i.imgur.com/g1xgG76.gif
    – wch1zpink
    Jul 17, 2023 at 3:34
  • Yeah, appreciate it. My intent, though, was to get this on a key command. Flagging spam is way too many clicks for my liking, especially when you're doing it 30 times a day, so I've just put it on F19, out of the way of casual or accidental interaction. I'm going to bounty your work, btw - I just need to wait for the timer to be up. Thanks again for going the extra mile :))
    – Tetsujin
    Jul 17, 2023 at 8:14
  • 1
    Thank you very much for the +100 Bounty. Yeah I did put a good amount of effort into this one. Honestly though, a significant part of my willingness to go the extra mile sometimes, is for selfish reasons. The more I help others… The more I continue to learn and the better I get. lol.
    – wch1zpink
    Jul 18, 2023 at 13:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .