Ask Different is a question and answer site for power users of Apple hardware and software. It only takes a minute to sign up.
Sign up to join this community
I have text file with 250,000 lines - and I need to append the same single character at the start of each line. I have tried to use various multiline /column edit plugins in Atom and Sublime but they just hang - I guess due to size of file.
Is this something I could do with with a bash/zsh script - or AppleScript/Automator maybe ?
Prepend each line of a file with a capital A and write a new file-
awk '{print "A"$0}' < FILE > NEWFILE
To prepend X to the start of every line of file, writing to newfile, in Terminal:
sed 's/^/X/' file > newfile
Here I'm using sed, the Unix stream editor, to use a very simple regular expression to substitute the beginning of every line (the ^ symbol) with an X.
sed -i'.bak' 's/^/X/' file will create a backup of the original file (with specified extension file.bak) and then the substitution is applied in-place.
The stream editor sed is likely the fastest and sharpest tool built for exactly this task.
Use the insert command (the newline after \ is part of the syntax):
sed 'i\
X' file > newfile
$ time sed 'i\
X' line250000 >/dev/null
real 0m0.118s
user 0m0.102s
sys 0m0.012s
The delay or overhead for this operation is extremely low making it very efficient for huge files.
sudo purge in between, your version verses sed 's/^/X/' file > newfile was only 0.021s faster in my testing, and hardly worth considering it as an option to the single line regexp sed command for me as I find it easier to write the single line regexp sed command. That said, either of the two sed answers was no less then 1.224s faster than the awk command to achieve the same goal. Go sed! 🙂
Nov 23, 2019 at 19:04
sys part of the measured time, processing time the user part. I see about 70% more processing time for the regexp version here, but would agree that it doesn't really matter unless one needs to process a huge number of files with a huge number of lines.
This bash snippet will prepend each line of a file with a hash (#) and save it to a new file:
IFS=$'\r\n'; printf '#%s\n' $(</path/to/file.txt) > /path/to/newfile.txt
Explanation
IFS=$'\r\n': This sets the field separator so that only the characters \r (carriage return) and \n (newline) are used to delimit a string. The default setting can be restored using: IFS=$' \t\n'
printf: This is a builtin command in bash that is used to format text according to the pattern supplied as the first argument. In this case, #%s\n tells it to format every subsequent argument as a regular string (%s), before which it will print a hash character and after which it will print a newline.
$(</path/to/file.txt): This is a command substitution that uses an input redirection to supply it the contents of a file path. It's equivalent to $(cat /path/to/file.txt), without having to call out to cat.
>: This redirects the output of the previous command, i.e. printf, so that it is written to the file at the specified file path. If your file paths contain spaces, either escape each space using a preceding backslash (\), or surround the file path with double quotes, making sure anything you want the shell to expand sits outside of the quotes, e.g. ~/Documents/My\ Massive\ File.txt or ~/"My Massive File.txt"
$(< might produce a lot of data here
bash 4.4 in under 8 seconds (processor: Intel Core i5 2.40Ghz). During its run, it used approximately 0.6% of the available memory at a constant rate (total RAM: 8.00GB). I compared it to a buffered input/output solution (using while and read). Whilst this solution achieved the same desired outcome, and obviously used approximately 0.0% memory per unit time, those units of time were too many to sit through: after 2 minutes, it had 96816 lines left to process.