I'm working on a script that reads the header of some files (medical images), filters the data and then creates a file directory based on that data. But there are millions of images and I need to make it faster in a task. After reading the image, the script creates a block of information that I need to filter, swap the letters accented by normal letters, swap any space for an underline, and then delete any character other than "AZ, AZ, 0-9,_)( ][-". At the moment I use the SED and Tr commands as shown below, Is there a faster way to do this ? TIA

BLOCK=$( command to read the images ) 
FILTER=$( echo $BLOCK | tr 'ÁÉÍÓÚáéíóúçÇ ^\' 'AEIOUaeioucC_:' | sed -e 's/[^A-_a-z0-9() -]//g' )
  • 2
    Did you check the execution time of command to read the images? I'm nearly sure that this takes much longer than the echo .. | tr ... | sed ... pipeline, so optimizing that won't help (even though you could go with only tr here for a tiny bit of performance) – nohillside Jun 11 '18 at 17:31
  • yes, it consumes about 15% of the script time. But I can not change it, so I need to improve performance at another point. In script I do not use 'echo' but FILTER = $ (command | tr '' '' | sed -e ''). I separated in two parts here just to be more understandable. – Vilefort Jun 11 '18 at 17:50
  • What consumes the other 85%? – nohillside Jun 11 '18 at 17:54
  • filter, separate the data in name, date, id, type, create new directories with this data, copy the files to these directories and put the operation in a log – Vilefort Jun 11 '18 at 18:02
  • 1
    Or use a Perl/Python script to drive the process, not a bunch of utilities combined with a shell script – nohillside Jun 11 '18 at 19:18

This does not address the "faster way" part of the question: more of a code review

  1. make sure to quote your variables: echo "$BLOCK"
  2. "delete any character other than "AZ, AZ, 0-9,_)( ][-" ->

    tr -dc '][:alnum:]_)( [-'

    where -d means "delete" and -c mean "the complement of the given set". [:alnum:] is a character class matching letters and numbers.

  • Thanks, it got faster, but I had to change it a bit. tr -dc '][:alnum:]_)(\n [-' ) – Vilefort Jun 11 '18 at 18:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .