I'm trying to make a long-polling chat app using Twisted and jQuery (with Django). How do I pass queries back to the JS?

Question

This is the first long-polling application that I have ever built, and second project with Twisted, so I would appreciate any feedback that anyone has about anything in my code at all, as I may be going about this entirely the wrong way. I have been cobbling together various examples as I've gone along and it's almost working, but I can't seem to find a way to get the data back to the Javascript. I have a Django site running on Twisted, and it seems to work fine so I'm not going to include the Django bits unless someone thinks it's important, and the only thing the Django site does is host a chat. I had originally set it up using regular polling, but I've been asked to change it to long-polling and I'm almost there (I hope). Here's the HTML/JS (long.html): ```

Accepted Answer

Sometimes, a function in a twisted application may conditionally return data, and at other times return a `Deferred`. In those cases, you can't check to see if you got data; you probably won't, and in the cases that you do get a deferred, no amount of rechecking will change that; you must always turn such functions into real Deferreds, with `maybeDeferred`, and then attach a callback to the result.

That said, `t.e.adbapi.ConnectionPool.runQuery()` is not such a function. it always returns a deferred. The only way to work with that data is to attach a callback. In general, you won't ever see the result of an asynchronous call in twisted applications in the same function that makes the initial call.

This means that, since you want to run a query for every long polling request, and since those are unconditionally asynchronous (you have to return from your `render()` function before they can even start), your `render()` always returns `NOT_DONE_YET`:

```
def render(self, request):
    """Handle a new request"""
    request.setHeader('Content-Type', 'applicaton/json')
    self.getData(request)
    return server.NOT_DONE_YET
```

and now everything needs to happen correctly in getData. As it turns out, the handling of the deferred from `runQuery` is fine; but the sql itself has a pretty big issue. To understand why, imagine a clever hacker tried to access

```
http://yoursite?last_update=5+and+"secret"+in+(select+password+from+users)
```

The fix is easy, though, don't do string interpolation, use bind parameters. switch the `%s` for a `?` in the query, and the `%` for a `,` in the function call itself. While we're at it, lets move the `ConnectionPool` out of this method and into `__init__`, you don't want or need a whole pool for each retry for each request.

```
def getData(self, request):
    last_update = request.args['last_update']
    print "LAST UPDATE: ", last_update
    new_messages = self.dbpool.runQuery("SELECT *"
                                        " FROM chat_message"
                                        " WHERE time > ?", request.last_update)
    #                                                  ^ ^
    return new_messages.addCallback(self.gotRows, request)
```

The callback attached to the deferred returned by `runQuery` is returning the formatted result; but there's nobody to return it to; it needs to do all the work itself. Fortunately, we're already giving it the `request` to work with, so that's not too difficult. We also need to handle the case of when there was no data to return, since there's nobody on the other end to add it to the list of delayed requests.

```
def gotRows(self, rows, request):
    if rows:
        # we have data to send back to the client! actually finish the
        # request here.
        data = {"messages": [{'author': row[1], 'message': row[2], 'timestamp': row[3]} for
                             row in rows]}
        request.write(self.__format_response(request, 1, data))
        request.finish()

else:
        self.delayed_requests.append(self)
```

Lastly, we need to make a similar change to `processedDelayedRequests()` as we made in `render()`. it can only fire off the query, it can't update its state based on the results because it doesn't have them. To simplify things, we'll just eat items off the list.

```
def processDelayedRequests(self):
    delayed_requests = self.delayed_requests
    self.delayed_requests = []
    while self.delayed_requests:
        # grab a request out of the "queue"
        request = self.delayed_requests.pop()

# we can cause another attempt at getting data, but we'll never get
        # to see what hapened with it in this function.
        self.getData(request)
```

I'm trying to make a long-polling chat app using Twisted and jQuery (with Django). How do I pass queries back to the JS?

Solution

Problem