python: read csv, execute command and write results to new vertical column

csv, python

Solution

You're not using subprocess.Popen correctly, which is leading to the immediate problem (`OSError: [Errno 2] No such file or directory`).

In general, the first argument to Popen should be a sequence, not a string unless you also pass the `shell=True` keyword parameter. If the first argument is a string and `shell=False` (the default), Popen will attempt to execute the file named for the value of the string. There is no file named `"python '/root/my_script.py'"` (the whole string), therefore you get an `OSError`.

So,

p = subprocess.Popen(
    "python '/root/my_script.py'", 
    stdout=subprocess.PIPE, stderr=subprocess.PIPE
)

should probably become something like ...

p = subprocess.Popen(
    ["python", "'/root/my_script.py'"], 
    stdout=subprocess.PIPE, stderr=subprocess.PIPE
)

or (essentially equivalent)

p = subprocess.Popen(
    "python '/root/my_script.py'".split(), 
     stdout=subprocess.PIPE, stderr=subprocess.PIPE
)

or (see warning)

p = subprocess.Popen(
    "python '/root/my_script.py'", shell=True,
    stdout=subprocess.PIPE, stderr=subprocess.PIPE
)

Update: The answer to your question is Yes. Python can help you accomplish all that you'd like to do. Here's a breakdown of you list.

SPOILER ALERT! Do not read beyond this line if you want to figure things out for yourself.

- `read a csv FILE`

What you've done is fine. Another way ...

with open('my_list.csv', 'rb') as fp:
    my_data_list = [row for row in csv.reader(fp)]

... which introduces some potentially new concepts, the with statement, and list comprehensions. But you don't really need an intermediate list to act upon, you can read and write in the same loop (see below)

- `executes COMMAND on fourth (vertical) column of FILE`

- `the COMMAND prints`

- `loop until first empty row of FILE`

I've assumed you want to print the output, or result, of running the command.

for row in my_data_list:
    command = row[3] #<- 4th column is index 3, 1st is 0
    p = Popen(command.split(), stdout=PIPE, stderr=STDOUT) #<- stderr to stdout
    stdout, empty = p.communicate()
    print stdout

- `read each line for HEALTHY (from COMMAND)`

- `if multiple lines of HEALTHY and all do not equal Yes, HEALTHY equals "No"`

- `if HEALTHY is not found on any lines, HEALTHY equals "Error Scanning"`

`write HEALTHY on new fifth column to NEW_FILE with all five columns`

if 'HEALTHY:No' in stdout:
    writer.writerow(row + ['No'])
elif 'HEATHLY:Yes' in stdout:
    writer.writerow(row + ['Yes'])
else: 
    writer.writerow(row + ['Error Scanning'])

And putting it all together (untested) ...

import csv
from subprocess import Popen, PIPE, STDOUT

with open('my_list.csv', 'rb') as incsv:
    with open('new_data.csv', 'wb') as outcsv:
        reader = csv.reader(incsv)
        writer = csv.writer(outcsv)

        for row in reader:
            p = Popen(row[3].split(), stdout=PIPE, stderr=STDOUT)
            stdout, empty = p.communicate()

            print 'Command: %s\nOutput: %s\n' % (row[3], stdout)

            if 'HEALTHY:No' in stdout:
                writer.writerow(row + ['No'])
            elif 'HEATHLY:Yes' in stdout:
                writer.writerow(row + ['Yes'])
            else: 
                writer.writerow(row + ['Error Scanning'])

Update: fixed poor naming choice of csv reader and writer file objects

Update: Python 2.5 introduced the `from __future__ import with_statement` directive. For versions of python older than 2.5, the with statement is unavailable. In this case, the common approach is to wrap file operations in a try finally. As in,

import csv
from subprocess import Popen, PIPE, STDOUT

incsv = open('my_list.csv', 'rb')
try:
    reader = csv.reader(incsv)
    outcsv = open('new_data.csv', 'wb')
    try:    
        writer = csv.writer(outcsv)

        for row in reader:
            p = Popen(row[3].split(), stdout=PIPE, stderr=STDOUT)
            stdout, empty = p.communicate()

            print 'Command: %s\nOutput: %s\n' % (row[3], stdout)

            if 'HEALTHY:No' in stdout:
                writer.writerow(row + ['No'])
            elif 'HEATHLY:Yes' in stdout:
                writer.writerow(row + ['Yes'])
            else: 
                writer.writerow(row + ['Error Scanning'])
    finally:
        outcsv.close()
finally:
    incsv.close()

HTH!

Problem

Original source

Related problems

• How to safely open/close files in python 2.4