Amazon EMR job failed: Shut down as step failed

Question

I'm writing a simple streaming map reduce job using Python to be run on Amazon EMR. It's basically an aggregator of user records which groups the entries for each user id together. Mapper ``` #!/usr/bin/env python import sys def main(argv): line = sys.stdin.readline() try: while line: line = line.rstrip() elements = line.split() print '%s	%s' % (elements[0] , (elements[1],elements[2]) ) line = sys.stdin.readline() except "end of file": return None if __name__ == '__main__': main(sys.argv) ``` Reducer: ``` #!/usr/bin/env python import sys def main(argv): users=dict() for line in sys.stdin: elements=line.split('	',1) if elements[0] in users: users[elements[0]].append(elements[1]) else: users[elements[0]]=elements[1] for user in users: print '%s	%s'% ( user, users[user] ) if __name__ == '__main__': main(sys.argv) ``` This job should run on a directory with five text files. The parameters of the EMR job are: Input: [Bucket name]/[Input folder name] Output: [Bucket name]/output Mapper:…

Accepted Answer

Solved. Make sure you don't have any comments at the beginning of your Python scripts and that the very first line in the script is the `#!/usr/bin/env python` line

Amazon EMR job failed: Shut down as step failed

Solution

Problem