3

I need to create 64GB worth of files with a size between 30 and 50 MB to evaluate copying a large number of files from an SDXC card to an external HDD. So I need a few files to try with.

I figure I could create an arbitrary number of files (can be random bytes) with a file size between x MB and y MB who in total do not cap a size of z GB on OSX?

Is this easy to do in terminal for a non-programmer?

6

64 GB of files, 30 to 50 MB each

First, there are a couple of ways to create large files. This command creates a 30 MB file of zeros very quickly.

dd if=/dev/zero of=file.dat count=30 bs=1048576

This command creates a 50 MB file of random bytes. It's not as fast as a file of zeros.

dd if=/dev/urandom of=file.dat count=50 bs=1048576

Here are some other useful commands for this project.

jot -r 1 30 50             # makes a random number between 30 and 50
du -sm . | awk {'print $1'} # finds the size in MB for the current folder

Shell script

Choosing files of zeros (it's faster), and putting everything together in a shell script:

#!/bin/sh
min=1          # minimum file size
max=1          # maximum file size
limit=1        # limit of total files size
filetype="dat" # filetype
folder="."     # folder to put files in
zeros=1        # fill with zeros or random bytes?

while getopts ":h:n:m:l:f:t:z:" opt; do
  case $opt in
    h) echo "-n {min. file size in MB} -m {max. file size in MB} -l {limit of total size of all files in GB} -t {string filetype without dot ex.: 'jpg'}-z {1 = fill with zeros | 0 = fill with random bytes (slower) }"
    ;;
    n) min="$OPTARG"
    ;;
    m) max="$OPTARG"
    ;;
    l) limit="$OPTARG"
    ;;
    f) folder="$OPTARG"
    ;;
    t) filetype="$OPTARG"
    ;;
    z) zeros="$OPTARG"
    ;;
    \?) echo "Invalid option -$OPTARG" >&2
    ;;
  esac
done

if [ $zeros -eq 1 ]
then
  source=/dev/zero
else
  source=/dev/urandom
fi

n=1                                      # count files
lm=$((($limit*1000)-($max-1)))           # real total file size limit
sz=`du -sm "$folder" | awk {'print $1'}` # size of folder in MB

while [ $sz -lt $lm ];
do
   cnt=`jot -rn 1 $min $max`;
   dd if=$source of=$folder/file$n.$filetype count=$cnt bs=1048576 2> /dev/null;
   status=$?;
   if [ $status -eq 0 ]; then
      echo file$n.$filetype $cnt MB;
   else
      echo write file$n.$filetype failed;
      exit $status;
   fi
   let n=n+1;
   sz=`du -sm "$folder" | awk {'print $1'}`;
done
exit;
  • n counts files so that each file has a different name
  • cnt is a random number from $min to $max that the dd command uses to make a file between $min MB and $max MB

Copy the script and paste into a new TextEdit window. Select Format->Make Plain Text from the menu bar. Save the file on the Desktop and name it random_data_files.sh.

Creating the files

Open a Terminal window and run these commands:

cd ~/Desktop
chmod a+x random_data_files.sh

Make a folder or mount an SDXC card. cd to the folder or card. Run the the shell script.

~/Desktop/random_data_files.sh -n 30 -m 50 -l 64 -t jpg -f {your_folder}

The script will create approximately 1,600 files (sometimes more, sometimes fewer) for 64 GB of 30-to-50 MB files. Open another Terminal window in the same folder or the card, and use:

du -h

to watch the progress of the script. Please comment if something's not clear or if you have a problem.

  • Or you could just let it run until the SD card is full (error message). – WGroleau Aug 28 '16 at 3:23
  • Thanks your answer is really helpful and extensive. I have adapted it and expanded it a little. Now you can call it with some options. Works really great. But has at least 2 bugs still, if anyone cares to check: 1) the cap size limit works only till 1GB above it will create a few more files running over the cap - 2) if you create files directly on an sd-card (in my case on the sd-reader in macbook pro) it will start generating zero byte files if the cap size is reached; it works fine in a folder though – steros Aug 28 '16 at 9:29
  • I'm glad the script works for you. As for the bugs... the lm variable is a heuristic, a guess, for when to stop creating files. It's hard to know how much overhead each file will need on a particular device. Also, a lot of small files have more overhead than fewer large ones, so adjust lm downward if the script creates files above the cap. The number that works for a folder on a hard disk may be different than an SD card. – creidhne Aug 28 '16 at 13:45
  • The original script lacks error-checking... always a good thing. I added a check for the status from the dd command. It should stop the zero-length files when a device is full. – creidhne Aug 28 '16 at 13:55
3

Alternate:

mkfile -v $(echo "$((30+$RANDOM*20/32767))")m "/tmp/$(date)"
sleep 1

On my SSD, the sleep ensures that 'date' doesn't repeat the file name, as the mkfile takes 100-200 milliseconds. The files are all zeroes. The -v makes it tell you the file size.

Repeat this until you get a device full error message.

'dd' takes only 15-20 milliseconds for all zeros and three to five seconds for the random variation.

  • Note that I offer the mkfile alternative only to show it exists. 'dd' IS faster. – WGroleau Aug 28 '16 at 3:36
  • Hi, thanks for the answer. It is a nice, quick one liner for filling up an sdcard. I still checked the other answer as it was more extensive. – steros Aug 28 '16 at 9:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .