Why Windows line endings kill your Unix scripts

Friends don’t let friends use Windows line endings.
Somedude

1 Summary

This page applies to the effect of Windows line endings on scripts running on a Unix server. The examples in this document happen to be CGI scripts that use Python3, but the principles apply just as well to ordinary scripts that use Perl, Ruby, Shell, or any other language.

2 The Problem

Sometimes your script will run great on your Windows box, but when youupload it to a Linux server, you get mysterious error messages like this:

Server error!

The server encountered an internal error and was unable to complete your request. Error message: Premature end of script headers: lab1.2.py

When you test your script on the server by executing it on the command line, like this:

python3 lab2.3.py

it runs perfectly — but it still doesn’t work as when invoked as a CGI web script.

Next, you run it on the command line to debug it, like this:

python3 lab1.2.py

and it runs perfectly! At this point you begin to suspect that you may have found a bug in Linux! But, when you try to run your absolutely perfect script as a standalone, like this,

./lab1.2.py

it goes down in flames with another cryptic error message:

unable to exec joesmith88/public_html/lab1.2.py: No such file or directory

If you’re lucky, you’ll see this more informative error message:

-bash: ./lab1.2.py: /usr/local/bin/python3^M: bad interpreter: No such file or directory

3 What’s going on?

You may wonder, “How can it be no such file or directory?” That doesn’t make sense. You can see the program on the file system. It’s there. What’s going on?

What’s going on is that you really do have a fatal error in your code, and it’s an error that you put into the code yourself. The thing is, you can’t see the error with a simple visual inspection of the code because the error is caused by an invisible character that your editor inserted into the code.

As we all know that there’s nothing intrinsically wrong with Windows. But we also know that Windows ain’t Linux — and one of the differences between the two is that Windows and Linux use different line endings. Where Linux uses a single character for line endings, the \n newline, Windows uses two characters, a return and a newline: \r\n. While you’re
working on Windows, languages such as Perl, Python, and Ruby are OS-aware and will understand that line endings have two characters. But, when you upload the Windows files to a Linux server, the line endings can cause mysterious errors.

4 Common Line Endings

Windows OS X, Linux, Unix MacOS
\r\n \n \r

5 Why do Windows line endings break CGI scripts?

Assume that you create a script on your Windows machine and upload the code below to your Linux server. I’ll make the line endings conspicuous by using “\r\n” to indicate them for this example:

#!/usr/local/bin/python3\r\n
print("Content-type:text/html\n")\r\n
print("Hello, world")\r\n

The problem occurs when Linux reads the shebang line and sees this:

#!/usr/local/bin/python3\r\n

Linux doesn’t know about Windows line endings; the \r character is just a literal character, no different than an “a”, “b”, or “c”. Linux sees the shebang and looks for a program file named /usr/local/bin/python3\r. Of course, that file doesn’t exist, so you get a mysterious fatal error, the error caused by the invisible r character.

unable to exec joesmith88/public_html/lab1.2.py: No such file or directory

6 od (octal dump)

Your can see all of the characters in your scripts by using the od program to dump all of the characters in your script. The od program represent the invisible \r and \n characters with cr and nl, respectively.

Here’s an example that shows how to use od. This a dump of an actual script that a student uploaded from a Windows machine. The “cr” and “nl” characters in the output represent the “” Windows line endings, and they appear at the end of every line of code.

As we’ve already discussed, the “cr” will cause fatal errors when they are at the end of the shebang line. In a Unix CGI script, there should not be ANY “cr” characters unless you put them there intentionally.

% od -a joesmith88/public_html/lab1.2.py
0000000   #   !   /   u   s   r   /   l   o   c   a   l   /   b   i   n
0000020   /   p   y   t   h   o   n   3  cr  nl   #  sp   N   a   m   e
0000040   :  sp   J   o   s   e   p   h  sp  sp   S   m   i   t   h  cr
0000060  nl   #  sp   F   i   l   e  sp   n   a   m   e   :  sp   l   a
0000100   b   1   .   2   .   p   y  cr  nl   #  sp   D   a   t   e   :
0000120  sp   J   u   n   e  sp   2   1   ,  sp   2   0   1   1  cr  nl
0000140  cr  nl   #  sp   *   *   *   *   *   *   *   *   *   *  sp   L
0000160   a   b  sp   1  sp   E   x   e   r   c   i   s   e  sp   1   .
0000200   2  sp   *   *   *   *   *   *   *   *   *   *  cr  nl  cr  nl
0000220   p   r   i   n   t   (   ’   C   o   n   t   e   n   t   -   t
0000240   y   p   e   :  sp   t   e   x   t   /   h   t   m   l      n
0000260   ’   )  cr  nl   p   r   i   n   t   (   ’      n   ’   )  cr
0000300  nl   p   r   i   n   t   (  sp   ’   *   ’  sp   *  sp   1   0
0000320  sp   +  sp   ’  sp   L   a   b  sp   E   x   e   r   c   i   s
0000340   e  sp   1   .   2  sp   ’  sp   +  sp   ’   *   ’  sp   *  sp
0000360   1   0  sp   )  cr  nl   p   r   i   n   t   (   ’   H   e   l
0000400   l   o   ,  sp   w   o   r   l   d   !   ’   )  cr  nl  cr  nl

You can see the “cr” and “nl” characters in the shebang line. This script will not run on a Linux server.

7 Solution

The solution, fortunately, is really simple: tell your Windows or Mac editors (yes, even Mac editors misbehave), to stop using Windows-style line endings. Every coding editor has the a way of setting the default line endings. Drill down into your editor’s configuration panel and tell your editor to always use Unix-style line endings.

8 FTP causes problems, Too

Some SFTP clients add \r\n line ending by default. If you upload scripts that you know have Unix-style line endings, and they still fail, check them with the od command to see whether your SFTP client has added Windows line endings. If so, configure your client to behave properly.

Leave a Reply

Your email address will not be published. Required fields are marked *