Take the 2-minute tour ×
Ask Different is a question and answer site for power users of Apple hardware and software. It's 100% free, no registration required.

I need to merge multiple .CSV files (using the cat command) but without copying the header for each file.

What's the best way to accomplish this task?

share|improve this question
add comment

3 Answers

You'll need more than the cat command, as described here:

Say you have 3 CSV-files: file1.csv, file2.csv, and file3.csv and want to join them to bigfile.csv and your header is always (only) the first line, then use

either (keep header from first file):

cat < file1.csv <(tail +2 file2.csv) <(tail +2 file3.csv) > bigfile.csv

or (remove header from all files who's names begin with "file"):

awk 'FNR > 1' file*.csv > bigfile.csv
share|improve this answer
    
Thanks so much! :-D –  Dranian Feb 2 '13 at 22:55
    
I found this looking for a generic linux answer, but in my case this didn't exactly work. It would silently ignore file1.csv. I needed to cat that file. cat <(cat file1.csv) <(tail +2 file2.csv) <(tail +2 file3.csv) > bigfile.csv –  Lelon Jun 18 '13 at 20:41
    
I am getting tail+2: command not found when i used cat < file1.csv <(tail +2 file2.csv) <(tail +2 file3.csv) > bigfile.csv method –  user64636 Dec 11 '13 at 20:01
    
@user64636 there should be a space character between tail and +2 –  patrix Dec 11 '13 at 22:20
    
actually I had to use tail -n+2, tail +2 wouldn't work –  Matthieu Napoli Jan 9 at 16:51
add comment

You could also use a group command ({ ; }) instead of process substitution (<()):

{ head -n1 file1.csv; for f in file*.csv; do tail -n+2 "$f"; done; } > new.csv

It also works with CRLF line endings as long as the files end with an empty line (\r\n).

The number-only versions of head and tail were made obsolete by POSIX 1003.1-2001, and they result in warnings in some environments.

share|improve this answer
add comment

Using the command sequence above resulted in a file looking like this:

header,of,csv1
contents,of,csv1
==> csv2.csv

contents,of,csv2

To make it a proper CSV, with one header line and all the relevant values, I employed the following sed incantation... sed -ie "/^$/d;/^==>/d" bigfile.csv

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.