3

I have an AppleScript .scpt file, triggered by a key combination in FastScripts.app, that functions as a thesaurus. The script looks up the selected word in a pre-formatted list, and, if the word is found in this list, it displays the synonyms of this word to the user1.

This list is contained in a plain text (.txt) file. The list is already formatted in the AppleScript list format. I would like my .scpt file to be able to accept this text as a true list2.

It is important to note that the .txt file contains 2.5 million words3.

This is why I am not simply copying the contents of the .txt file into the .scpt file itself, despite the fact that the text file is 100% static and will never be altered. Inserting the text directly into my script would bring with it considerable lag and sluggishness as I edit and compile my .scpt file in Script Editor.app.

Script Editor.app froze every time that I tried to read the .txt file. The problem is that Script Editor reads into the memory a given text file in its entirety, instead of streaming the contents in a more efficient manner. So, I broke this text file up into 10 smaller text files4, each new .txt file containing about 250,000 words.

At 250,000 words, of course, the text files are still extremely large (by any standard).

Here is a (severely condensed) example of what the contents of each text file looks like:

{{"exaltation","accolade","adulation","advance","advancement"},{"exalted","winnowing","winsome"},{"exam","audition","blue book","examen","examination","final","examination","test","trial","tripos","viva","written","written examination"},{"examination","Pap test","Socratic method","airing","analysis","anatomic diagnosis","appraisal","work-up","written","written examination"},{"examine","air","analyze","appraise","archetype","asleep","assess","canvass","case"},{"examiner","analyst","analyzer","asker"},{"examining","analytic","examinational","exploratory"},{"example","admonishment","admonition","alarm","archetype"},{"exasperate","bedevil","vex","work up","worry"},{"exasperated","aggravated","amplified","angry","annoyed"},{"exasperating","annoying","bothering","bothersome"}}

As you can see, the contents of the text file is a nested list5 that is organized in the same way that AppleScript formats a list. Each text file contains no line breaks or paragraphs.

I am looking for a method to get this list into my AppleScript, with as little latency as possible6. This is why I pre-formatted it. So, speed is key.


Footnotes:

1. My thesaurus script is similar to the built-in thesaurus feature that exists in Microsoft Word. One notable difference is that my script works system-wide.

2. By true list, I mean that I can call, for example, item 12 of this list later on in my AppleScript.

3. My source for the thesaurus data is Grady Ward's "Moby" Thesaurus. I found this database from this answer: Looking for Thesaurus Data - Stack Overflow

4. I had to use Hex Fiend.app to cut from the text file and paste into a new text file. I could not edit the file in TextEdit.app, without TextEdit freezing on me.

5. The outer list contains each thesaurus entry. The inner lists contain all of the synonyms for that entry. The first item of each inner list is the entry title. Both the outer list and each inner list are ordered alphabetically (with the exception of the first word of each inner list, because, again, this word is the entry title).

6. I understand that even the fastest method will still have several seconds of latency, since the text file is so large.


  • Why don't you use an SQLite db? – klanomath May 11 '17 at 12:03
  • I had never heard of SQLite, until you mentioned it. I am wholly unfamiliar with it. However, if you know how to accomplish what I want with it, I'm happy to use anything. – rubik's sphere May 11 '17 at 12:16
  • What's the purpose of the 2.5 MWords? The idea behind my proposal is to use a database instead of a basically flat text file. – klanomath May 11 '17 at 12:20
  • Please see my most recent edit. I've provided more specifics about my script. – rubik's sphere May 11 '17 at 20:22
2

Obviously, I do not know the total scope of what you're doing or how you have other things coded, as you have not supplied all the details and code, however, I would take a different approach.

I downloaded the Moby Thesaurus from the linked page in your question and preformed the following actions on it.

  1. Extracted the contents of the mthes.tar.Z file.
  2. Opened the ./mthes/mobythes.aur file in TextWrangler and noticed two things to change.
    • Change the line endings from Classic Mac (CR) to Unix (LF).
    • Removed unwanted trailing commas from 6 lines.

Note that while I could make these changes in TextWrangler, nonetheless I prefer to use Terminal, and did so using the following command:

tr "\r" "\n" < mobythes.aur | sed -E 's/[,]{1,}$//' > mobythes.txt

Which took but literally a second to do (as I actually prefaced the above command with time, out of curiosity). With the mobythes.aur file having now been processed, saved to mobythes.txt and copied to my Documents folder, I will use this new plain CSV file as is, to query the search string for a match to the first field of each record and return the record, sans the first field, as a list to choose from in AppleScript. I found this method to be extremely fast, while searching for "zoom" the last record in the CSV file, it took but a second to return and create the list for that record on the fly.

In AppleScript Editor I use the following code to test against the plain CSV file as a single file containing the 30,260 lines with 2.5 million synonyms and related words.

set AppleScript's text item delimiters to ""
set theMobyThesaurus to POSIX path of (path to documents folder) & "mobythes.txt"

set theSearchString to the text returned of (display dialog "Find synonyms for:" default answer "" buttons {"Cancel", "Search"} default button 2 with title "Search Moby Thesaurus")

if theSearchString is not equal to "" then

    try
        set theSearchResults to (do shell script "grep -i -m 1 '^" & theSearchString & ",' " & theMobyThesaurus)
    on error
        display dialog "No match for \"" & theSearchString & "\" available." buttons {"OK"} default button 1
        return
    end try

    if theSearchResults is not equal to "" then
        set AppleScript's text item delimiters to ","
        set theSynonymsList to items 2 thru -1 of text items of theSearchResults as list
        set AppleScript's text item delimiters to ""

        choose from list theSynonymsList with prompt "Choose a synonym for: " & linefeed & theSearchString
        if the result is not false then
            set theChosenWord to (item 1 of the result)
        end if
    end if

end if

In this example, assuming a search match was made and nothing canceled, then the theChosenWord variable now contains what was chosen from the displayed list and can be processed further as needed/wanted.

Note that this is of course strictly example code for testing purposes and will need to be adapted to your use case scenario while incorporating appropriate error handling as needed.

I believe this is going to be the fastest way while leaving the Moby Thesaurus as a single CSV file, and is probably faster then whatever methods you tried thus far.

  • I'm stuck on the Terminal line. When I enter it in Terminal, nothing occurs; the new .txt file is never created. Terminal just hangs and displays a > on the next line. I'm not familiar with Terminal, so I don't know what the issue is. – rubik's sphere May 12 '17 at 6:29
  • @rubik's sphere, Sorry, had a typo, left an ' off of the sed command. I fixed the typo. – user3439894 May 12 '17 at 6:46
  • Masterfully done, user3439894. Very thoughtful answer (e.g., removing trailing commas from the source file). I too found your method to be amazingly fast. And, you read my mind about placing the synonyms within a choose from list dialog. (Just in case you are curious, the script will then keystroke the synonym that the user chooses, overwriting the original word that was selected.) I cannot thank you enough. – rubik's sphere May 13 '17 at 5:56
  • I've realized that the grep code is case-sensitive. For example, if theSearchString is Zoom, no matches will appear. Similarly, a search for socratic produces no results. Do you know how to fix this, so that the case of the characters in theSearchString would not matter? – rubik's sphere May 14 '17 at 10:33
  • @rubik's sphere, In Terminal, type man grep and press enter, or just type grep and right-click on grep and select Open man page. I've updated the answer. BTW Like with many commands the options can be concatenated, i.e. -i -m 1 can be written -im 1 but not -m 1i. – user3439894 May 14 '17 at 12:34
0

I had devised a solution before user3439894 had posted their answer.

Despite the fact that user3439894's solution is superior to my solution in every way, I figure that I may as well post my code, if for no other reason than to highlight the fast response time of user3439894's solution.


File adjustments:

Here are the two modifications that I made to the source file, mobythes.aur, for my solution:

  1. I converted the .aur file to a .txt file, simply by renaming the file extension in Finder.

  2. I inserted (1) carriage return before the first character of the .txt file (just in case the user ever searches for the first thesaurus entry, i.e., a cappella).

I realized that I was barking up the wrong tree in my original post—there is no need (or benefit, really) of pre-formatting the .txt file contents in AppleScript's list format, within the file itself. Hence, I did not modify the file's original delimiter structure whatsoever (in the way that I had done so in my original post).


My code:

display dialog "Find synonyms of:" default answer ""
set theSearchQuery to text returned of the result

-- Referencing the default delimiters of the "mobythes.txt" file:
set theOuterListDelimiter_oneCarriageReturn to (ASCII character 13)
set theInnerListDelimiter_oneComma to ","

set theSearchQueryAsAThesaurusEntry to (theOuterListDelimiter_oneCarriageReturn & theSearchQuery & theInnerListDelimiter_oneComma)

set theThesaurusAsString to (read POSIX file "/Users/Me/Desktop/mobythes.txt")

if theThesaurusAsString contains theSearchQueryAsAThesaurusEntry then

    set theSynonymsAsText to extractBetween(theThesaurusAsString, theSearchQueryAsAThesaurusEntry, theOuterListDelimiter_oneCarriageReturn)
    set theSynonymsInList to splitStringIntoList(theSynonymsAsText, theInnerListDelimiter_oneComma)

    choose from list theSynonymsInList
else
    display dialog "No thesaurus entry exists for \"" & theSearchQuery & "\"!"
end if


-- Subroutines:

to extractBetween(SearchText, startText, endText)
    --  Source: http://macscripter.net/viewtopic.php?id=24725
    set tid to AppleScript's text item delimiters -- save them for later.  
    set AppleScript's text item delimiters to startText -- find the first one.  
    set endItems to text of text item -1 of SearchText -- everything after the first.  
    set AppleScript's text item delimiters to endText -- find the end one.  
    set beginningToEnd to text of text item 1 of endItems -- get the first part.  
    set AppleScript's text item delimiters to tid -- back to original values.  
    return beginningToEnd -- pass back the piece.  
end extractBetween

on splitStringIntoList(theString, theDelimiter)
    -- Source: http://erikslab.com/2007/08/31/applescript-how-to-split-a-string/
    -- save delimiters to restore old settings:
    set oldDelimiters to AppleScript's text item delimiters
    -- set delimiters to delimiter to be used:
    set AppleScript's text item delimiters to theDelimiter
    -- create the array:
    set theArray to every text item of theString
    -- restore the old setting:
    set AppleScript's text item delimiters to oldDelimiters
    -- return the result:
    return theArray
end splitStringIntoList

Runtime performance comparison:

Out of curiosity, I conducted a "shoot-out" of execution times between user3439894's approach and my approach.

I commented out every dialog in both of our solutions. I set the test search term as the fixed string, "planet".

Entering time osascript /Users/Me/Desktop/MyOriginalSolution.scpt in Terminal.app returned:

real    0m1.257s
user    0m0.728s
sys     0m0.409s

Entering time osascript /Users/Me/Desktop/user3439894Solution.scpt returned:

real    0m0.250s
user    0m0.193s
sys     0m0.030s

Based on this test, user3439894's solution is over 5 times faster than mine, with a difference of 1.007 seconds.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.