uWSGI and Django segmentation fault
django, segmentation-fault, uwsgi, zip
Solution
The problem happens because your process fails to allocate a chunk of memory.
When I first look at your code, the first thing that drew my attention was that you are loading entire content of file into memory and read them all at once at each iteration.
zip.writestr("{0} - {1}.{2}".format(num, name, ext), f.read())
`f.read()` here will read the entire content of the file all at once and put that into memory, then will try to write it into another section of the memory, the `StringIO` object, which in turn ending up storing same data twice in different parts of the RAM.
I would recommend you to read from file one line at a time or at least reasonable amount of data and don't forget to close the file when you are done with it:
for i, f in enumerate(track_files):
f.seek(0)
...
while True:
data = f.read(2**16)
if not data:
break
else:
zip.write(data)
...
f.close()
You might want to take a look at this SO answer about writing to a `zipfile.ZipFile` instance chunk by chunk.
Problem
I've been dealing with a strange error that is intermittent and hard to trace. One of the pages on my django-powered site allows users to download music they have purchased. The problem: Some users are reporting a 502 error upon downloading. A further investigation of the problem shows that there is a segmentation fault in uWSGI. I don't really know how to debug this, and am looking for anything that might help find a resolution. Attempts to solve the problem: I thought it may be that I was running too many uWSGI processes, so I've lowered that, but since the problem only happens sometimes (some users have no problem, some try again with success), I can't be sure the error is resolved. Thoughts on the issue: I am thinking it might be due to the way I'm handling the files here. The files that I'm zipping are also being sent to other users in other views. Could it be an issue when the uWSGI processes are both trying to read a file? No processes write to the files. Next steps: I'm looking for some guidance on this, or any ideas on how I can get a little more information on what's going on here. uWSGI config: ``` [uwsgi] plugin=/etc/uwsgi/python_plugin.so wsgi-file = /path/to/my/project/projectname/wsgi.py home = /path/to/my/project/ master = true socket = /tmp/uwsgi.sock chmod-socket = 666 vacuum = true processes = 3 workers = 15 min-worker-lifetime = 45 max-requests = 100 reload-mercy = 5 harakiri = 20 buffer-size = 16384 ``` Relevant view: ``` def zipForDownload(album): bonus = BonusContent.objects.filter(album=album) bonus_files = [open(f.bonus_file.path, 'rb') for f in bonus] tracks = trackSort(list(Track.objects.filter(album=album))) track_files = [open(f.audio_file.path, 'rb') for f in tracks] zipped_file = StringIO.StringIO() with zipfile.ZipFile(zipped_file, 'w') as zip: for i, f in enumerate(track_files): f.seek(0) num = tracks[i].track_number name = tracks[i].name ext = os.path.basename(f.name).split('.')[-1] zip.writestr("{0} - {1}.{2}".format(num, name, ext), f.read()) for i, f in enumerate(bonus_files): name = bonus[i].name ext = os.path.basename(f.name).split('.')[-1] zip.writestr("{0}.{1}".format(name, ext), f.read()) zipped_file.seek(0) response = HttpResponse(zipped_file, content_type='application/octet-stream') response['Content-Disposition'] = 'attachment; filename=%s.zip' % (album.name) return response def downloadPostPin(request, purchase): if request.POST.get('PIN').encode('utf-8') == purchase.download_pin.encode('utf-8'): # Get zip response with all tracks and bonus content resp = zipForDownload(purchase.album) return resp else: # Otherwise, return the same page again with an error error = "Invalid PIN. Please try again!" return downloadPrePin(request, purchase, error) ``` uWSGI error log: ``` !!! uWSGI process 8094 got Segmentation Fault !!! *** backtrace of 8094 *** /home/web/.envs/music/bin/uwsgi(uwsgi_backtrace+0x2e) [0x46a1be] /home/web/.envs/music/bin/uwsgi(uwsgi_segfault+0x21) [0x46a581] /lib/x86_64-linux-gnu/libc.so.6(+0x36c30) [0x7fdc1ffd6c30] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Malloc+0x248) [0x7fdc206ef508] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyString_FromStringAndSize+0xa2) [0x7fdc206de232] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3be7) [0x7fdc206fe0e7] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) [0x7fdc2070117d] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x48d8) [0x7fdc206fedd8] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) [0x7fdc2070117d] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x162310) [0x7fdc20701310] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fdc206c8e23] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x7d30d) [0x7fdc2061c30d] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fdc206c8e23] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7fdc20687837] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xff706) [0x7fdc2069e706] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2920) [0x7fdc206fce20] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x161883) [0x7fdc20700883] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1a7d02) [0x7fdc20746d02] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PySequence_List+0x2c) [0x7fdc207471bc] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PySequence_Fast+0x3d) [0x7fdc2074807d] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1aa995) [0x7fdc20749995] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4abc) [0x7fdc206fefbc] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) [0x7fdc2070117d] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x162310) [0x7fdc20701310] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fdc206c8e23] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_CallFunction+0xbb) [0x7fdc20706efb] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x16a15d) [0x7fdc2070915d] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(_PyObject_GenericSetAttrWithDict+0x107) [0x7fdc205f00b7] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_SetAttr+0x8f) [0x7fdc206aeadf] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x1cda) [0x7fdc206fc1da] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) [0x7fdc2070117d] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1623e5) [0x7fdc207013e5] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fdc206c8e23] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x7d30d) [0x7fdc2061c30d] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fdc206c8e23] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x12e48f) [0x7fdc206cd48f] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x12c4df) [0x7fdc206cb4df] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fdc206c8e23] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2316) [0x7fdc206fc816] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4b59) [0x7fdc206ff059] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4b59) [0x7fdc206ff059] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) [0x7fdc2070117d] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x1623e5) [0x7fdc207013e5] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fdc206c8e23] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0xeb1) [0x7fdc206fb3b1] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) [0x7fdc2070117d] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x48d8) [0x7fdc206fedd8] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x80d) [0x7fdc2070117d] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x162310) [0x7fdc20701310] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fdc206c8e23] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x7d30d) [0x7fdc2061c30d] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fdc206c8e23] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0x12e5f5) [0x7fdc206cd5f5] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fdc206c8e23] /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7fdc20687837] /home/web/.envs/music/bin/uwsgi(python_call+0x11) [0x4808a1] /home/web/.envs/music/bin/uwsgi(uwsgi_request_wsgi+0x116) [0x482a96] /home/web/.envs/music/bin/uwsgi(wsgi_req_recv+0xa2) [0x41f1f2] /home/web/.envs/music/bin/uwsgi(simple_loop_run+0xc4) [0x466664] /home/web/.envs/music/bin/uwsgi(uwsgi_ignition+0x194) [0x46a7d4] /home/web/.envs/music/bin/uwsgi(uwsgi_worker_run+0x2dd) [0x46f02d] /home/web/.envs/music/bin/uwsgi(uwsgi_run+0x3b4) [0x46f554] /home/web/.envs/music/bin/uwsgi(_start+0) [0x41e8ae] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fdc1ffc1ec5] *** end of backtrace *** DAMN ! worker 9 (pid: 8094) died :( trying respawn ... Respawned uWSGI worker 9 (new pid: 8371) ```