49

We recently had a meltdown of a Tomcat server, that produced an 82.7 GB "catalina.out" log file, which I saved for forensic analysis.

What macOS editors can open monster text files without consuming 80 GB of RAM or causing 15 minute freezes?

  • 17
    Do you need to read the file to skim it for interesting details or faults or do you need to search the file? Does the file have a consistent timestamp? The answers below are all suitable but at 80GB+ you should consider some log analysis and search techniques to find the data you need for your analysis. One example, but off topic, question is serverfault.com/questions/63297/good-free-tomcat-log-analyser – Freiheit Feb 17 at 14:31
  • 2
  • Would it be reasonable to write a parser for the file that extracts records and adds them as rows in a database? Databases are designed to efficiently sort and search millions of records; text editors are not. – Eric Lippert Feb 19 at 21:50

13 Answers 13

5

Try Glogg. There is a MacOs build on the download page:

https://glogg.bonnefon.org/download.html

I don't know about 80 GB files, but I regularly used it (on Windos) to open log files up to 5 GB, and it works great on those (memory footprint after indexing is about 100-150MB, and searching is very fast).

One note though - it's read-only analyzer, not an editor.

|improve this answer|||||
  • 1
    It took nearly an hour to open the file (showing a progress bar, to let me know it hadn't simply locked up), but it opened it, allowed me to scroll and search through it, and it led me right to the problem (apparently a tightloop). – hbquikcomjamesl Mar 2 at 17:56
  • 1
    Oh, and a read-only analyzer was exactly what I had in mind, especially given that it allowed me to copy the relevant lines into a document of more manageable size. – hbquikcomjamesl Mar 2 at 17:57
  • 1
    One other thing: if examining enormous files, requiring many minutes to open, it is probably a good idea to go into Preferences first, and disable "Load Last Session." – hbquikcomjamesl Mar 2 at 21:58
  • @hbquikcomjamesl Thanks for the info! It's good to know that Glogg can handle such behemoths. – user2384366 Mar 4 at 10:26
91

less filename

From the command line, it lets you view files straightaway without loading the full file into memory.

|improve this answer|||||
  • 10
    GNU less only uses a default of 64k of buffer space when viewing an arbitrarily large file. I assume that the less in macos does the same, so this is a great answer. less also has regex search, will let you paginate through the file, and much more. – Wayne Conrad Feb 17 at 15:58
  • 5
    This is exactly what first more and then less was made for. A lot of navigational shortcut keys as well. The Unix toolset is very useful, and worth learning. – Thorbjørn Ravn Andersen Feb 17 at 16:17
  • 7
    @WayneConrad less isn't a standard program with multiple implementations; less is the GNU pager based on more, and is what ships with macOS. – chepner Feb 17 at 21:46
  • 1
    +1 here for simplicity, it’s not an editor which is what was asked, but it’s super pragmatic and a little buried in my answer. – bmike Feb 18 at 1:47
  • 3
    I doubt whether the average macOS user thinks too carefully about what editing means when using words like editor anyway. OP may very well have just meant viewer or pager. +1 – kojiro Feb 18 at 13:11
37

I would not try to open it... I'd rather do:

  1. grep - look for some text
  2. split - chop the file into say 10Mb chunks.

Something like:

grep "crash" My80GbFile.txt | more 

If the big file is not "Line delimited"

split -b 10M My80GbFile.txt

But if the big file is just a load of lines, then (as was posted), split by line (100,000 per sub-file) in this case.

 split -l 100000 My80GbFile.txt
|improve this answer|||||
  • 11
    You might want to use grep -C5 crash just to have a few lines of context above and below each match. – terdon Feb 17 at 13:10
  • 7
    This. Do not open an 85 GB file in an editor. Get rid of all the fluff first (without compromising the original file, of course). If the file is big because of long logging time, inspect the time close to the incident. If it is big because it is a snapshot of a huge system state, e.g. dumps a database or such, try to focus on the relevant data. – Peter - Reinstate Monica Feb 17 at 14:22
  • 3
    If the file is made up of lines, instead of split -b it would be better to do split -l. Otherwise, you'd split lines in half. – JoL Feb 17 at 17:26
  • 2
    I would suggest grep "crash" My80GbFile.txt | less instead of grep "crash" My80GbFile.txt | more , just for the sake of easier navigation and use of search with / key. – LucasBr Feb 18 at 8:27
26

In terms of your immediate needs, the best free visual editor for macOS is BBEdit (linked to the Mac App Store download) and it does so much - a true powerhouse. Once you have it, you can also pay up for the pro / automation / out of gratitude features, but it's free forever if you want and like that price.

I also use vi to edit things, but that opens a can of worms for needing the shell, terminal app or another app and some studying to learn how to exit the editor (tldr; try ZZ or ZQ), customize it and teach your brain to think about operating on text in the abstract as opposed to using the mouse to select items. Also, a pager like less or more or bat is also very friendly to get started and navigate around massive files. (And bat gives you wings awesome colors and syntax awareness).

brew install bat

In your case, the console app that comes with macOS might also be worth looking at if you can use the search functionality there. Launch the app from spotlight and drag your monster file on the window to have a peek.

|improve this answer|||||
  • 10
    +1 for BBEdit -- the BareBones team have specifically optimised this app to deal with massive text files over the years. – calum_b Feb 16 at 20:44
  • 10
    Please add whether this editor can actually open a 82.7G "catalina.out" log file. And whether it requires 85G of RAM. – reinierpost Feb 17 at 16:02
  • 10
    @T.J.L. It doesn't need to be confirmed. It is stated right there in the question. An answer is supposed to answer the question asked. – reinierpost Feb 17 at 16:46
  • 2
    Vim on linux can be used to edit very large files, but you need to know how to do that before trying to open them, and may need to disable plugins etc; stackoverflow.com/questions/908575/… I assume mac os is a similar story. Would not really recommend despite vim being my go-to text editor. – Clumsy cat Feb 17 at 19:20
  • 2
    @T.J.L. Confirmation bias The more upvotes they get, the more future upvotes they receive. Also, they're a "181K reputation moderator" of the apple SE. That's a big fish in a small pond. I haven't personally tried BBEdit, but I have tried using less and more with large files, and they're not a good idea, unless you like waiting while programs seek through files. grep is good. ag (The Silver Surfer) is great: I dunno about an 82.7G (!) text file, but it can find a string in all the files on my 128GB (!!) SSD in less than 60 seconds. – Aaron F Feb 17 at 19:41
12

Just don't (open it as ONE file)

Is there any specific reason that you can not simply break it into about 1GB chunks with a script?

Yes, searching and similar functionality will suffer but that will already be the case with a 80GB file.

If you have specific break points in the script (days in the timestamp, startup / shutdown messages) you could also split it up for that. That way you would probably even get additional meaning into the file.

Also: once split up any decent IDE (like IntelliJ IDEA or any other) will give you search functionality over the text back.

[Beware: This comes from a programmer so might not be your approach or overkill, I can only say that it WOULD work in the end, you'll have to know if its worth it]

|improve this answer|||||
10
  1. Use less in a terminal window. It will show you one page at a time of the file, will only load about that much in memory, so you can navigate multi-TB files with it if you want.

    You probably should add the -n option to prevent less from trying to compute line numbers. So:

    less -n /path/to/file
    

    Remember you can type less -n (don't forget the final space) and drag-and-drop the file from the Finder to the Terminal window to add the path to that file.

  2. Once you are viewing the file in less, you can:

    • navigate using up/down arrows, space (one page down), b (one page back)...
    • search using /. You can also search for lines not containing a pattern with /!. Reverse search uses ?. But all searches will scan the whole file. Better have it on an SSD if you do that a lot.
    • navigate to a specific line in the file using <number> followed by G (capital G)
    • navigate to a specific part of the file using <number> followed by %. So 50% will get you to the middle of the file, 90% to the last 10%, etc.

If your log file has timestamps and you know when you want to look, the quickest approach is to:

  1. open the file
  2. Use a "binary search" to find the rough part of the file you are interested in:

    • Type 50%, which will show you the middle of the file
    • If the part you want is after, go to 75%, otherwise 25%
    • Repeat until you have narrowed down to the relevant part
  3. Use a regular search (using / to go forward or ? to go backwards) to find the exact line you're looking for (based on either the exact timestamp, or a specific word you know shows the issue).

This should allow you to navigate quickly quickly to the relevant part of the file.


If you think you'll have a lot of searching within a subset of the file, you could alternatively use grep with a specific date or date-time combination (in the right format) to first extract that subset to another smaller file. For instance, if you know the crash occurred today a bit after noon while your log covers months, you could

grep '2020-02-17 12:' /path/to/file > extracted-log.txt

This would give you all lines which contain a timestamp between 12:00:00 and 12:59:59 inclusive. Of course, the exact format will depend on the actual format used for timestamps.

grep will scan the whole file once to find all the relevant lines, which will take a little while on a very large file, but you'll then have a much more manageable file.


An alternative may be to use dd to "extract" a part of the original file, using offsets and lengths found in less (Ctrl-G to get the current offset). dd is a very powerful tool but can be very dangerous to use, so use with caution (and most definitely not as root or with sudo if you are not 100% sure of what you're doing):

dd if=/path/to/original/file of=destination_file.txt bs=1 skip=<start offset> count=<length>

Note that this is not very efficient, it's better to use a larger block size (bs), ideally a power of 2 such as 1024, and divide skip and count by that block size.

I'm pretty sure there must be other tools that do the same, though I'm drawing a blank. I think some versions of cat can do it, but not the one on macOS apparently.

|improve this answer|||||
9

With disk based text editors, the file is not loaded entirely into memory - what you see on the UI is a peek into the contents the editor has loaded into memory. I have used UltraEdit successfully in the past to do large log file analysis. Its regex based search tools and location bookmarks are especially useful. It loads the file snappily, and you can do regular expression based searches. The url takes you to a download page where you can download a 30 day trial version. There are other disk based text editors as well.

Since it has been a few years, I installed UltraEdit and opened the largest file I had. It was a 64 GB binary file and it opened instantly. Ran a search for a term and that took about 90 seconds. I have highlighted the file size with a red rectangle in the bottom right. The mac is an MBP 2018 with 8 GB RAM running Mojave.

Screenshot of UltraEdit with a 64GB file open and the search window open

|improve this answer|||||
  • yes, UltraEdit will do the trick. But not "instantly". It WILL be churning for 5-10 minutes on a file that size :) – jwenting Feb 18 at 5:00
  • 1
    @jwenting You might be surprised - UE is VERY good at dealing with large files. – Mike Brockington Feb 18 at 11:07
  • 2
    @MikeBrockington I know, I use UE. Took it about 5 minutes to open a 25GB SQL dump though (which helped a lot, nothing else would open it) that had to be changed to load on a different machine a few weeks ago. – jwenting Feb 18 at 11:44
  • @jwenting - you are right. It might have been the combination of available RAM (the system was fresh off a reboot, with minimal apps running) + SSD (and the file on the same disk) + OSX Version (Mojave) + UE version (the latest). If the system disk is metal (one of my mac's has a 5400 RPM metal disk), the file may be better analyzed by copying it to a UHSII 128GB SD card. – Vinil Feb 19 at 6:42
3

You wouldn't

Even a Tolkien fan doesnt want 82.7GB of anything. You only want certain bits out of that; you'll know it when you see it.

And even contemplating a tool that analyzes the whole file is a waste of time, literally; it's going to spend 15 minutes just reading through the file assuming 100MB/sec. A lot slower if it's doing analysis of any complexity.

Terminal is your friend

The lifesaver here is that OS X is built on top of Unix. That was a big part of Apple buying NeXT and Steve Jobs coming back. That means you can use the entire suite of Unix tools, which are extremely well-honed, and very well supported here.

There are dozens of ways to do it without perl, but since perl is built into MacOS and is infinitely extensible, I prefer to start there (rather than do it in a simpler tool, want to improve the query somewhat, hit the limits of that tool, and have to re-craft it in a different tool). So something like this in a file called, say "xx":

 $len = -s "filename.log";            # variable becomes length of file
 open ($IN,  "<", "filename.log"); 
 seek ($IN, $len - 10_000_000, 0);    # perl allows _ in numbers for readability

 while (<$IN>) {         # <> reads a line.  Default variable is metavariable $_
   print;                # with no argument, default is metavariable $_
 }

That won't read the whole file, just seek to the specified location (10MB from the end), then read and print everything to the end. It will just print it to the screen, so to send it to the file, do this when you call it:

 perl xx > tailfile.txt

Now you have a 10MB tailfile.txt that you can open with something else.

There are simpler ways to do just that, but suppose you realize "Wait, I want to do more. I only want errors and warnings.” So you change the print command to

 print if /error/i or /warning/i;    # // matches text, defaults to $_ 

That too can be accomplished in simpler tools if you spend enough time rooting through docs. But then, you decide you need to see the three lines after the error. Just like that... you've outgrown the simpler tools, but this is trivial in Perl. You can just keep shimming Perl pretty much forever. There's a full programming language in there. Object oriented and everything.

|improve this answer|||||
  • 2
    I'm a big perl fan, but if you just want the end of a file, tail -c <offset> is probably a lot easier :-) – jcaron Feb 18 at 9:52
  • @jcaron Sure, if your needs stop there. As I discussed. But when do your needs ever stop there? – Harper - Reinstate Monica Feb 18 at 10:16
3

A file that large is probably 99.999999% redundant (literally), so the key is to remove lines that occur a zillion times, to some degree of similarity, and examine what's left over.

On Linux there's a utility called petit, designed for analyzing huge log files, that does this. An example usage is petit --hash /var/log/kern.log. The utility can probably be found or built for Mac.

It processes each line of the file to remove things that make the line unique; for example, strip the date from each line, and substitute all strings of digits with a single # character. Each generic line is then hashed to become a fingerprint for detection of similar lines.

The result is that it outputs each line only once with a count of occurrences, vastly reducing the size of the data. Anything out of the ordinary is likely to show up clearly, and then one can search for that specifically, using utilities from some of the other answers here.

I don't know if this particular utility is performant enough for something that size. I would bet yes, because it has options for plotting graphs on the order of months or years of input, and wouldn't need to store much besides a small number of fingerprints. In the worst case you could write your own: for each input line, genericize it to a fingerprint, hash it, and add it to a database of hash+fingerprint+count, indexed by hash.

EDIT: petit seems to use more CPU and memory than desired, so I wrote my own simple implementation: https://github.com/curtmcd/hashlog. It makes one pass through the log file; it processes at about 6.5 sec/GB on my home Ubuntu server.

|improve this answer|||||
  • petit won't be a success story. I've just tried it with a ~1.1 GB log file. It consumed all available memory and took ~5 minutes before I interrupted it. A tool that creates a hash of each line in a file in order to detect duplicates is bound to fail at this task. – Schmuddi Feb 18 at 16:46
  • The expectation is that it would only need to store a hash table of unique-ified signature strings, not every line, scanning the file a single time. The number of entries should be not much more than the number of unique printf's in program(s) that write to the log, typically on the order of hundreds. petit might not be such a great implementation, and I admit I only tried it with a 30MB log file. I wrote my own and will update the answer. – Curt Feb 25 at 9:48
1

"joe", aka Joe's Own Editor, was designed to only load parts of the file as needed. I've never used it on a file that large but I never came across a text.file too large for it to open.

|improve this answer|||||
1

Definitely Hex Fiend. It opens files WITHOUT using RAM. It simply reads from the disk. Performance is absolutely incredible. I've examined 500gb password dumps with it before.

https://ridiculousfish.com/hexfiend/

|improve this answer|||||
-1

Open terminal and use vim to open it

vim filename.txt

P/s:

Type vim and drag the file to your terminal. Then hit enter.

To quit vim (without editing):

:q!
|improve this answer|||||
  • 2
    How does this work with a file of the size described in the question? – nohillside Feb 18 at 14:40
  • Better use vim -r to avoid the creation of huge swap files. – lhf Feb 19 at 12:21
  • I'm not sure people would know how to understand the :q!. It's not all obvious that you just type it directly. – Alexander - Reinstate Monica Feb 19 at 14:49
-1

I would recommend using Sublime Text. Although it requires a license it can be downloaded and evaluated for free without time or functionality limitations. That means that you or your company can have the chance to try it out as much and however you want. I personally use it for investigating logs of maybe even 3-4GB in most cases, or SQL dumps of even 12GB. On initial openning it does go through the entire file in order to perform 1st level indexing etc, but it comes with a progress bar indicating the whole process's progress.

|improve this answer|||||
  • Do you have personal experience of using Sublime Text to open an 83GB file ? Positive personal experience ? Your answer makes mention only of files almost an order of magnitude smaller. – High Performance Mark Feb 19 at 9:59
  • No, that's why I recommend trying it out to assess its suitability. The fact that from my personal experience I had no issues processing files up to 12GB and the fact that the application's limitations do not mention anything about max. file size imply that there shouldn't be a problem reading a file of any size. The OP is interested in 3 things: reading the file, keeping the memory usage low, not getting signs of app freeze. Sublime renders a progress bar while indexing and is very good for especially just reading and searching through enormous files – Panos Kordis Feb 19 at 13:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .