7

I have text file with 250,000 lines - and I need to append the same single character at the start of each line. I have tried to use various multiline /column edit plugins in Atom and Sublime but they just hang - I guess due to size of file.

Is this something I could do with with a bash/zsh script - or AppleScript/Automator maybe ?

10

Prepend each line of a file with a capital A and write a new file-

awk '{print "A"$0}' < FILE > NEWFILE
9

To prepend X to the start of every line of file, writing to newfile, in Terminal:

sed 's/^/X/' file > newfile

Here I'm using sed, the Unix stream editor, to use a very simple regular expression to substitute the beginning of every line (the ^ symbol) with an X.

  • 1
    As an extension to this solution, using sed -i'.bak' 's/^/X/' file will create a backup of the original file (with specified extension file.bak) and then the substitution is applied in-place. – ziddarth Nov 24 '19 at 4:53
  • I find it hard to believe that someone would down vote an answer that is shown as an example in the venerable sed one liners. Any mods out there????? – fd0 Nov 24 '19 at 19:02
  • Did someone downvote it, @fd0? I didn't see that. I suppose if one had a Classic-style text file with CRs as newlines this (and all the other shell examples) would misbehave. – scruss Nov 24 '19 at 19:37
2

The stream editor sed is likely the fastest and sharpest tool built for exactly this task.

Use the insert command (the newline after \ is part of the syntax):

sed 'i\
X' file > newfile

$ time sed 'i\                     
X' line250000 >/dev/null

real    0m0.118s
user    0m0.102s
sys     0m0.012s

The delay or overhead for this operation is extremely low making it very efficient for huge files.

  • 2
    Writing to disk file, and using sudo purge in between, your version verses sed 's/^/X/' file > newfile was only 0.021s faster in my testing, and hardly worth considering it as an option to the single line regexp sed command for me as I find it easier to write the single line regexp sed command. That said, either of the two sed answers was no less then 1.224s faster than the awk command to achieve the same goal. Go sed! 🙂 – user3439894 Nov 23 '19 at 19:04
  • @user3439894 Disk write time basically is the sys part of the measured time, processing time the user part. I see about 70% more processing time for the regexp version here, but would agree that it doesn't really matter unless one needs to process a huge number of files with a huge number of lines. – nohillside Nov 23 '19 at 19:17
1

This bash snippet will prepend each line of a file with a hash (#) and save it to a new file:

IFS=$'\r\n'; printf '#%s\n' $(</path/to/file.txt) > /path/to/newfile.txt

Explanation

  • IFS=$'\r\n': This sets the field separator so that only the characters \r (carriage return) and \n (newline) are used to delimit a string. The default setting can be restored using: IFS=$' \t\n'

  • printf: This is a builtin command in bash that is used to format text according to the pattern supplied as the first argument. In this case, #%s\n tells it to format every subsequent argument as a regular string (%s), before which it will print a hash character and after which it will print a newline.

  • $(</path/to/file.txt): This is a command substitution that uses an input redirection to supply it the contents of a file path. It's equivalent to $(cat /path/to/file.txt), without having to call out to cat.

  • >: This redirects the output of the previous command, i.e. printf, so that it is written to the file at the specified file path. If your file paths contain spaces, either escape each space using a preceding backslash (\), or surround the file path with double quotes, making sure anything you want the shell to expand sits outside of the quotes, e.g. ~/Documents/My\ Massive\ File.txt or ~/"My Massive File.txt"

  • Does this work with files > 250‘000 lines? $(< might produce a lot of data here – nohillside Nov 23 '19 at 13:05
  • 2
    @nohillside It does. I created a file with 412,740 lines, each of 80 characters length. This method performed the task in bash 4.4 in under 8 seconds (processor: Intel Core i5 2.40Ghz). During its run, it used approximately 0.6% of the available memory at a constant rate (total RAM: 8.00GB). I compared it to a buffered input/output solution (using while and read). Whilst this solution achieved the same desired outcome, and obviously used approximately 0.0% memory per unit time, those units of time were too many to sit through: after 2 minutes, it had 96816 lines left to process. – CJK Nov 23 '19 at 18:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .