I agree with Korey's solution, but I have updated it a bit by using Automator!

Setup:
Open Automator. Create a new Quick Action.
Set "Workflow receives current" to "folders" in "Finder".
Drag in a "Run Shell Script".
IMPORTANT: Change "Pass input" to "as arguments"
Paste in the following code:
cd $1
find . -type f -exec brctl evict {} \;
Save (CMD+S). I saved mine as "Remove Downloads (Recursively)"
Using:
This will automatically be available under the Quick Actions menu. Simply select the folder, right click (CTRL+click), and the Quick Actions menu is near the bottom. You can also press CMD+SHIFT+P to toggle the preview window. You should see the action in the bottom right of your screen.

However, I found it most useful to add a keyboard shortcut. This is a little different in macOS 13 (Ventura) and I personally hate it, but you can search the system settings for "keyboard shortcut". A new box will appear. Navigate to the "Services" tab and then the "Files and Folders" drop-down. Double-click where it says "none" and add your shortcut. I first tested a few shortcuts in Finder until I found one that prompted the "error sound".
NOTE: After you save your shortcut, close settings and "Relaunch" finder (simply OPTION+right_click or OPTION+CTRL+click finder and click "Relaunch". Otherwise, it doesn't work!

Now, I simply select the folder and press CMD+OPTION+R and it works. This is awesome!

Explanation:
The "New Terminal at Folder" essentially is a "cd /path/to/folder" as the first line in the new terminal window. We've replaced this with passing in the action of selecting a file/folder. Thus, as long as we pass in the path to the folder, it should work just fine.