There is no way I can think of to use AppleScript to filter messages directly, but I guess you could go the ugly route of redirecting "All Messages" to an AppleScript and then handling them from there back to Mail.app.
==== Update:
To give you a better idea of what I´m talking about, you´ll have to create a new rule with condition "All Messages" and execute an Applescript. For example to check for the subject, you´ll want something similar to this:
using terms from application "Mail"
on perform mail action with messages theMessages for rule theRule
tell application "Mail"
repeat with eachMessage in theMessages
set theSubject to subject of eachMessage
if theSubject contains "test" then
-- do something here, like move message to a folder
end if
end repeat
end tell
end perform mail action with messages
end using terms from
Note that of course you could easily use a do shell script statement here to use whatever command line tool you want to compare the subject with a regexp, like if (do shell script "*your regexp that returns 'true' if matched*" ) is true … and so on.