1

Trying to get the bytes offset of a text word in a binary file (with some text here and there). I need to know the offset of this word from the beginning of the file. I'm wondering if on osx the command options are slightly different from other unix.

strings Agent_of_Change.mobi | grep -o -b --binary EXTH

This gives me : 31:EXTH

Not sure what 31 is, the bytes from the beginning of the line? The correct value I need would be 1617, from the beginning of the file.

1
  • 1
    My initial thought is that you could save the output of strings to a separate file. My guess is the offset is relating to the output of strings and not to Agents_of_Change.mobi. Also the grep I have, only has --binary-files=XXX and no -binary command - perhaps you have another grep in your path than the OS X provided one?
    – bmike
    Commented Apr 3, 2014 at 13:39

2 Answers 2

3

Grep is likely counting the offset from the output of the strings command since it's not actually parsing the .mobi file directly.

3
  • I see. The reasons why I'm using strings is that, I don't get the offset when I use this line: "grep -o -b -U EXTH Agent_of_Change.mobi"
    – aneuryzm
    Commented Apr 3, 2014 at 13:49
  • I just get Binary file Agent_of_Change.mobi matches, but I don't get the offset
    – aneuryzm
    Commented Apr 3, 2014 at 13:50
  • Yup - grep is protecting you from seeing binary characters and that also is removing the chance for it to tell you the offset of what you already know is a text safe string. You'll likely need a different grep or a different tool to find that offset.
    – bmike
    Commented Apr 3, 2014 at 14:05
1

This answer assumes the version of grep is either 2.5.1-FreeBSD or 2.6.0-FreeBSD. The 2.5.1-FreeBSD version of grep is included with Apple's operating system releases for Mac from macOS 11 (Big Sur) down to at least Mac OS X 10.6 (Snow Leopard). The 2.6.0-FreeBSD version of grep is included with Apple's operating system releases for Mac from macOS 12 (Monterey) through the current release of macOS 14 (Sonoma). When the either the -b or --byte-offset option is included and both the -o or --only-matching options are omitted, then grep will produce the offset of the start of the line where a pattern appears.

If you are using a version of macOS newer than Big Sur, then see the section below titled “Simpler Alternative Available When Using grep Version 2.6.0-FreeBSD”.

The output from the following command can be used to determine the offset of the first occurrence of the string EXTH in the binary file Agent_of_Change.mobi. Note that the tr command replaces each byte that is not in the EXTH string (which is $'\x45\x58\x54\x48') with a linefeed character (which is $'\x0A'). This converts the binary stream piped into tr into a text stream piped into grep.

cat Agent_of_Change.mobi | LC_ALL=C tr -c 'EXTH' '\n' | grep -Fbam1 'EXTH'

To better explain how to interpret the output from the above commands, the following example is presented. Instead of searching for a text string, the hexadecimal sequence of D4 0A 42 D4 bytes will be used. Note that the third byte is a linefeed. The following command was used to generate the binary file named binary.dat. This is the file that will be searched.

xxd -r -p - binary.dat <<<"E4 34 0A 17 7D D4 42 D4 0A 42 D4 D4 0A 42 D4 C3 DD AB D4 0A 42 D4 0A 27 40 D4 0A 42 D4 92"

The output below is from the command hexdump -Cv binary.dat. Note that the sequence D4 0A 42 D4 appears 4 times.

00000000  e4 34 0a 17 7d d4 42 d4  0a 42 d4 d4 0a 42 d4 c3  |.4..}.B..B...B..|
00000010  dd ab d4 0a 42 d4 0a 27  40 d4 0a 42 d4 92        |....B..'@..B..|
0000001e

Before grep can perform the search, the following needs to happen.

  • The non‑printable ASCII and non‑ASCII values in the sequence D4 0A 42 D4 need to be replaced with printable ASCII values that do not already occur in the sequence. The table below shows the chosen replacements. This replaces the sequence D4 0A 42 D4 with the text string ACBA.

    Hexadecimal
    Value
    Character
    Type
    Printable ASCII
    Character
    Chosen Printable ASCII
    Replacement Character
    D4 non‑ASCII A
    0A non‑printable ASCII C
    42 printable ASCII B
  • The following substitutions shown in the table below need to take place after the bytes are read from the binary.dat file. This table was generated from the previous table.

    Existing Byte
    Size Character
    Replacement Byte
    Size Character
    $'\xD4' 'A'
    $'\x0A' 'C'
    'A' $'\xD4'
    'C' $'\x0A'
  • After making the substitutions given in the previous table, all characters that are not in the text string ACBA need to be replaced with the ASCII linefeed character ($'\n').

Note: The strings passed to the tr command should not contain the ASCII null character ($'\x00'). If an ASCII null character is needed, then use the octal representation ($'\\000').

The commands below include the changes listed above, then use the grep command to find the first occurrence of the text string ACBA. Note that a subshell was used so the results of the export LC_ALL=C command would not persist.

(export LC_ALL=C; cat binary.dat | tr $'\xD4\nAC' $'AC\xD4\n' | tr -c ACBA '\n' | grep -Fbam1 ACBA)

The output is shown below.

5:ABACBAACBA

This output shows the line containing the string ABACBAACBA occurred at offset 5 in the stream that was originally read from the binary.dat file. The first occurrence of the string ACBA in the string ABACBAACBA is at offset 2. The offset of the first occurrence of the sequence D4 0A 42 D4 in the binary.dat file is at the sum of these two values, which is 5+2=7.

Simpler Alternative Available When Using grep Version 2.6.0-FreeBSD

With the 2.6.0-FreeBSD version of grep, the -b and --byte-offset options, when combined with the -o or --only-matching options, produce the offset of where a matching pattern appears. This simplifies the search since most non‑printable ASCII and all non‑ASCII characters no longer need to be replaced with printable ASCII characters.

Before grep can perform the search, the following needs to happen.

  • If the sequence D4 0A 42 D4 contains an ASCII null (00) or linefeed (0A), then these characters need to be replaced with other ASCII values that do not already occur in the sequence. The table below shows the chosen replacement. This replaces the sequence D4 0A 42 D4 with D4 41 42 D4.

    Hexadecimal
    Value
    Character
    Type
    Printable ASCII
    Character
    Chosen Printable ASCII
    Replacement Character
    D4 non‑ASCII
    0A non‑printable ASCII A
    42 printable ASCII B
  • The following substitutions shown in the table below need to take place after the bytes are read from the binary.dat file. This table was generated from the previous table.

    Existing Byte
    Size Character
    Replacement Byte
    Size Character
    $'\x0A' 'A'
    'A' $'\x0A'
  • After making the substitutions given in the previous table, all values not in the sequence D4 41 42 D4 need to be replaced with the ASCII linefeed character ($'\n').

Note: The strings passed to the tr command should not contain the ASCII null character ($'\x00'). If an ASCII null character is needed, then use the octal representation ($'\\000').

The commands below include the changes listed above, then use the grep command to find the first occurrence of the sequence D4 41 42 D4. Note that a subshell was used so the results of the export LC_ALL=C command would not persist. The sed command shown is only needed if the pattern passed to grep contains at least one non‑printable ASCII or non‑ASCII character.

(export LC_ALL=C; cat binary.dat | tr $'\x0AA' $'A\x0A' | tr -c $'\xD4A\x42\xD4' '\n' | grep -Fboam1 $'\xD4A\x42\xD4' | sed 's/:.*//')

The below is the same a above, except the strings have been simplified.

(export LC_ALL=C; cat binary.dat | tr '\nA' 'A\n' | tr -c $'\xD4AB' '\n' | grep -Fboam1 $'\xD4AB\xD4' | sed 's/:.*//')

The output is shown below.

7
11

This output shows the sequence D4 41 42 D4 occurred at offset 7 in the stream that was originally read from the binary.dat file. Therefore, the offset of the first occurrence of the sequence D4 0A 42 D4 in the binary.dat file is also 7. Since the sequence D4 41 42 D4 occurred again in the same line in the stream, a second output line was produced showing an offset of 11. This second line can be ignored.

Under the 2.6.0-FreeBSD version of grep, the output from the following command can be used to determine the offset of the first occurrence of the string EXTH in the binary file Agent_of_Change.mobi.

cat Agent_of_Change.mobi | LC_ALL=C tr -c 'EXTH' '\n' | grep -Fboam1 'EXTH'

The only difference between the above and the original, is the addition of the -o option to the grep command. Note that since the search pattern EXTH only contains printable ASCII characters, the sed 's/:.*//' command was omitted.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .