5

I'm trying to paste text into Terminal (v2.5 build 343, I'm running Yosemite), and it seems to consistently omit characters.

For example if I copy the following text to the pasteboard:

1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890

Then in Finder Edit -> Show Clipboard I can verify that the text is all there.

Next I open Terminal and type:

cat > test

Then paste the above text and press Ctrl+d.

On the screen I get this:

1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
1234567890,12345678,1234567890,1234567890,1234567890
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890

but the contents of the file "test" match the original text which I copied to the pasteboard.

Is there some reason why Terminal would drop characters when echoing to the screen during a paste operation?

Improve this question
5
  • 1
    What an awesome find.
    – Asmus
    Feb 27, 2015 at 18:37
  • From first testing it seems that when pasting, the missing characters are before the 1024th character; so in your example 14 characters before number 1024 are missing, if you add two columns of "1234567890," per line, 10 characters before the 1024th are missing. Not sure what to make of it, though.
    – Asmus
    Feb 27, 2015 at 19:34
  • 2
    This is fairly common, you're just bumping up against the input buffer for the terminal. I'd suggest just doing pbpaste > test instead.
    – miken32
    Feb 27, 2015 at 21:46
  • @miken32 that's interesting, I would think that processes would block on the input buffer being full, or at the very least drop the end of the input. How could the input buffer filling lead to dropped characters in the middle of the pasted string?
    – nonagon
    Feb 27, 2015 at 22:07
  • I was going under the assumption that the characters were being dropped just after the 1024 byte mark. Looking more closely, I now see it's slightly before. So not sure what to make of that. With lines of 256 characters, it only misses one at 1022 consistently.
    – miken32
    Feb 27, 2015 at 22:29

1 Answer 1

Reset to default
3

Stdout in this case is line-buffered and is 1024 bytes which is causing the output truncation on the display. When you paste from the clipboard the non-printable New-Line (EOL) characters at the end of each line are simply counted as another character on the line as far as the buffer count is concerned. The reason the math doesn’t add up for where the truncation happens that @miken32 was trying to calculate, is because you are not counting the New-Line characters.

If you cat the test file using a –e option you can see the non printable EOL characters as $, including them in the math should add up.

$ cat -e test
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890$
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890$
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890$
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890$
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890$
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890$
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890$
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890$
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890$
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890$
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890$
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890$
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890$
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890$
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890$
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890$
1234567890,1234567890,1234567890,1234567890,1234567890,1234567890$

You can get around filling the line buffer with pasted bytes if you specify your own end-of-file (EOF) marker to the cat command. This way when the cat command encounters the EOL characters it will reset the line buffer at the beginning of each new line instead of just counting them as another byte in a continuous stream because the cat command is parsing each new line looking for the EOF string on a line by itself to know when to exit.

You can do something like this :

cat << EOF > test

The stdin redirect (<< EOF) tells cat to keep printing until it encounters input matching the specified end-of-file characters on a new line by themselves. This way stdout will then print every pasted printable character line by line, the drawback is that you get a “>" character in the output at the start of each new line.

$ cat << EOF > test
> 1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
> 1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
> 1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
> 1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
> 1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
> 1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
> 1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
> 1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
> 1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
> 1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
> 1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
> 1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
> 1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
> 1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
> 1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
> 1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
> 1234567890,1234567890,1234567890,1234567890,1234567890,1234567890
> EOF

The end-of-file marker is typed literally as the characters E, O, F, after your paste and can be anything you like. Also the EOF characters, and the > characters are not included in the redirected output sent to the file. Typing the letters as EOF is just symbolic and can be anything you like, XXX for example, you just have to be certain that what ever EOF marker you specify does not appear in the paste buffer.

For the record, you can still use ^D as the EOF marker when using stdin, even though you specify something else. The use of the EOF string is an old school scripting convention for delimiting a block of text to redirect from within the script.

Hope this helps.

Improve this answer
3
  • Sorry if this is a silly question but can you explain how a line buffer of 1024 bytes leads to the observed missing chars? If a buffer is filling (and then waiting for a real newline, not a pasted EOL character) it seems to me that all of the missing characters should be at the end of what I pasted.
    – nonagon
    Apr 1, 2015 at 19:04
  • I don't know exactly how the stty terminal is working, but I believe that once it stops at 1024 characters, it starts printing again on the same line because it has not encountered a new line. If you do the cat command without the redirect to the test file you can get a better idea of what the terminal is seeing. Simply type cat with no other arguments, then past the test contents, you will see what was pasted echo'ed to the terminal, then you will also see the output of the cat command.
    – Michael Gagnon
    Apr 6, 2015 at 17:34
  • I tried running cat with no arguments and pasting long strings both with and without newlines. It's still pretty difficult to make sense of what it's doing. The chars it echoes back are right, but the chars it echoes on paste are missing some from the line containing the 1024th char. I still don't see how this could be the correct behavior for Terminal and/or the cat command.
    – nonagon
    Apr 6, 2015 at 22:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .