WebFaction
Community site: login faq

I'm leaning toward using nginx push module, tornado or twisted to add real time features to a flask app served on the web. Ajax polling seems not to scale well. The market doesn't seem ready for web sockets. I have looked at what I could find on this forum and on stackoverflow about these solutions. I understand about creating a custom app. I prefer python.
Can I do this on webfaction?
What features on webfaction do I need to specify to accomplish this?
Do I need a static ip number? Extra memory?
Can I do this behind apache or do I need to be on a nginx server?
Is there anything else I need?
Is there another reasonably-well-documented, efficient solution you would recommend instead?

asked 24 Oct '11, 14:28

bbergmann
313
accept rate: 0%


The setup I personally prefer is:

Essentially, the goal is to keep the majority of work in Python and Django. So you would set up a Django application as normal. Then, you would install a new Custom Application (listening on port) and install Node.js under that. Then, map the main http://domain.com/ to the Django App, and map something like http://comet.domain.com/ to the Node.js app.

Trigger Faye from your visitors' browser via javascript, which then signals to Node.js, which then adds the event to the ZeroMQ queue, which Django picks up and acts upon.


Installation happens in a few parts. First, let's install node.js 0.4.12 with NPM. In this example, I am installing it into $HOME/node rather than directly into $HOME:

# install tar
mkdir -p $HOME/src
cd $HOME/src
wget http://ftp.gnu.org/gnu/tar/tar-1.26.tar.gz
tar -xvzf tar-1.26.tar.gz
cd tar-1.26/
./configure --prefix=$HOME
make
make install
export PATH="$HOME/bin:$PATH"
which tar # should give ~/bin/tar

# add new PATH to ~/.bashrc so it's always active
echo 'export PATH="$HOME/bin:$PATH"' >> $HOME/.bashrc

###############

# install node 0.4.12
mkdir -p $HOME/node
mkdir -p $HOME/src
cd $HOME/src
wget http://nodejs.org/dist/node-v0.4.12.tar.gz
tar -xzf node-v0.4.12.tar.gz
cd node-v0.4.12
alias python=python2.6
./configure --jobs=8 --prefix=$HOME/node
make && make install

export PATH="$HOME/node/bin:$PATH"
echo 'export PATH="$HOME/node/bin:$PATH"' >> $HOME/.bashrc

# install npm
curl http://npmjs.org/install.sh | sh

Next, install ZeroMQ 2.1. This needs to be done into $HOME/node as well:

mkdir -p $HOME/src
cd $HOME/src
wget http://download.zeromq.org/zeromq-2.1.10.tar.gz
tar -xzf zeromq-2.1.10.tar.gz
cd zeromq-2.1.10
./configure --prefix=$HOME/node
make && make install

export LD_LIBRARY_PATH="$HOME/node/lib:$LD_LIBRARY_PATH"
export NODE_PATH="$HOME/node/lib/node_modules:$NODE_PATH"
export CPPFLAGS="-I$HOME/node/include $CPPFLAGS"
export LDFLAGS="-L$HOME/node/lib $LDFLAGS"
echo 'export LD_LIBRARY_PATH="$HOME/node/lib:$LD_LIBRARY_PATH"' >> $HOME/.bashrc
echo 'export NODE_PATH="$HOME/node/lib/node_modules:$NODE_PATH"' >> $HOME/.bashrc
echo 'export CPPFLAGS="-I$HOME/node/include $CPPFLAGS"' >> $HOME/.bashrc
echo 'export LDFLAGS="-L$HOME/node/lib $LDFLAGS"' >> $HOME/.bashrc

Now, install ZeroMQ bindings for Python and Node.js:

# python bindings
mkdir -p $HOME/src
cd $HOME/src
wget https://github.com/zeromq/pyzmq/downloads/pyzmq-2.1.10.tar.gz
tar -xzf pyzmq-2.1.10.tar.gz
cd pyzmq-2.1.10
python2.6 setup.py install --zmq=$HOME/node

# node.js bindings
PKG_CONFIG_PATH="$HOME/node/lib/pkgconfig" npm install -g zeromq

Next, install Faye for Node.js:

npm install -g faye

And, install a Django instance from the control panel via one-click installer. Connect it to a website from the WebFaction Control Panel. Create a Custom Application (listening on por) for Node.js. Connect it to a website as well under another domain (like http://comet.domain.com).

The basic frameworks are now installed.


Now, you're going to need the Faye server. Here is a faye_server.js which can be run using node faye_server.js &. It should use the port assigned by the Control Panel for your Faye Custom Application (replace 55555 in the code below):

// To Run:
// node faye_server.js &

var http = require('http');
var httpServer = http.createServer(function(req, res) {
        res.writeHead(200, {'Content-Type':'text/html'});
        res.end('<h1>Hello There</h1>');
    });
httpServer.listen(55555, '127.0.0.1');

var faye = require('faye');
var bayeux = new faye.NodeAdapter({
    mount: '/faye',
    timeout: 60
});
bayeux.attach(httpServer);

faye.Logging.logLevel = 'info';

var zeromq = require('zeromq'),
zmqSocket = zeromq.createSocket('subscriber');
zmqSocket.subscribe('');  // Subscribe to everything.
zmqSocket.connect('ipc:///home/USERNAME/webapps/NODE/ipc');
zmqSocket.on('message', function(dataObj) {
    data = JSON.parse(dataObj.toString('utf8'));
    bayeux.getClient().publish(data['channel'], data);
});

console.log('Server started');

In the above code, USERNAME is your username and NODE is the Node.js custom application. The "ipc" is a unix-domain socket. You don't need to create it; it will be created when you import and use the Django zmq library.

In this sense, Node.js is the "ZeroMQ Client" (which is why it uses zmqSocket.connect()). Django will be the "ZeroMQ Server" part, and that process creates the ipc socket with zmq_socket.bind(). Something like this:

zmq_context = zmq.Context()
zmq_socket = zmq_context.socket(zmq.PUB)
zmq_socket.bind('ipc:///home/USERNAME/webapps/NODE/ipc')

Lastly, the actual Javascript is something like this:

var bayeux = new Faye.Client('http://comet.domain.com/faye', { timeout: 120 });
var global_subscription = bayeux.subscribe('/_GLOBAL', messageHandler);
var room_subscription = bayeux.subscribe('/room/'+my_room, messageHandler);  # use this to separate channels
bayeux.publish('/room/'+my_room, {joined: my_username});

bayeux.publish('/room/'+my_room, {command: 'someTextualCommand',
                                  timestamp: new Date().getTime(),
                                  username: my_username,
                                  a: a,
                                  b: b
                                 });

You will definitely want to be using JSON (and the json python library) to send messages back and forth.

Hope that helps!

permanent link

answered 25 Oct '11, 19:17

ryans ♦♦
5.0k103960
accept rate: 43%

edited 26 Oct '11, 21:29

Thanks. More light is coming on. Here are a few more questions. The process chain goes something like below? For http and ajax processing: client=>django=>client For long-polling/asynchronous processing: request: client=>faye => node.js => zeromq => django <=>database reply: django=>zeromq=>node.js=>faye=>client

node does it asynchronous magic because it sends it requests on and doesn't block waiting for processing in its loop. Heavily loaded, django might slow down, but node will allow a heavier load of traffic.

From watching this talk presented by Alex Gaynor http://python.mirocommunity.org/video/1495/pycon-2010-building-leafy-chat I had two take-aways. Keep processes from bringing each other down, and don't try to reinvent orbited. It seems in the stack you recommend that the processes are independent. They won't bring each other down. Is faye a successful orbited reinvention? Further enlightenment?

(26 Oct '11, 16:17) bbergmann

This is essentially correct. Your pipeline (Client -> Faye -> Node.js -> ZeroMQ -> Django -> Database) and (Django -> ZeroMQ -> Node.js -> Faye -> Client) is correct.

Node.js is not used because it can handle a larger load than Django. It's used because Faye only supports Node.js and Rails. Since I'm only using this as a Faye server, Node.js is the obvious choice over Rails.

Faye can be considered a successful "orbited reinvention". Orbited development activity unfortunately seems to have dropped off. Orbited2 seems to use WebSockets and require a TCP server - that means you need to purchase a dedicated IP address and have an open port. Faye speaks HTTP, works in every browser (even IE6), handles reconnection well, and has an elegant light-weight interface.

So, no, I don't care for Node.js really. But until Faye has a Mongrel2 adapter, this is just the best way to do it right now.

(26 Oct '11, 20:45) ryans ♦♦

Hey, excellent. Thank you. I will dig into this. I hope the explanation and code is of good use to others besides me. If there is anything useful I can add, I will.

(26 Oct '11, 21:37) bbergmann

Can I do this on webfaction?

You should be able to, we do not restrict the building of software from source like custom HTTP servers. We provide SSH access so you can do this.

What features on webfaction do I need to specify to accomplish this?

I do not think any special features.

Do I need a static ip number? Extra memory?

You only need a dedicated IP for non-HTTP traffic, that module appears to be HTTP so should work without one, http://pushmodule.slact.net/. You may need more RAM, but you will only know once its built. We have no way of predicting how much RAM your custom stack will use.

Can I do this behind apache or do I need to be on a nginx server?

Everything is behind an nginx proxy server and is assigned a port to listen to by the control panel. This picture explains.

Is there anything else I need?

Not sure, I have not seen those 3 technologies in use together on our platform.

Is there another reasonably-well-documented, efficient solution you would recommend instead?

Personally I like AJAX since it is supported by almost every javascript framework and HTML5. I am not sure how much info you will find for the exact setup you are asking for as its highly customized.

permanent link

answered 24 Oct '11, 17:25

johns
5.4k412
accept rate: 23%

edited 24 Oct '11, 17:37

seanf
12.2k42136

Thanks for the quick reply. It really helped me. Here are some other things I sorted out that others may be wanting to know. In many cases short polling using ajax is efficient enough at lower scales. Here is a a good starting point: http://stackoverflow.com/questions/5313641/scaling-a-chat-app-short-polling-vs-long-polling-ajax-php

I had a misunderstanding that I had to connect to a web server supplied by webfaction. Generally, users can install the web server and stack they need to do what they want. I can install nginx or apache behind the nginx proxy. For example, I can set up tornado behind an nginx server behind the webfaction nginx server. Once I have my own web server, there seem to be many choices in python to solve what I am doing: A couple useful articles for me were
http://nichol.as/asynchronous-servers-in-python http://nichol.as/benchmark-of-python-web-servers.

If I am clearly understanding, if I choose to use long polling through an http connection I can listen and reply through the port assigned by webfaction. My server will listen, process the request, and then send the reply, asynchronous or not, through the same port. Am I on the right page?

(25 Oct '11, 14:05) bbergmann
Your answer
toggle preview

Follow this question

By Email:

Once you sign in you will be able to subscribe for any updates here

By RSS:

Answers

Answers and Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "title")
  • image?![alt text](/path/img.jpg "title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Question tags:

×98
×4
×3
×3
×2

question asked: 24 Oct '11, 14:28

question was seen: 8,361 times

last updated: 26 Oct '11, 21:37

                              
WEBFACTION
REACH US
SUPPORT
LEGAL
© COPYRIGHT 2003-2021 SWARMA LIMITED - WEBFACTION IS A SERVICE OF SWARMA LIMITED
REGISTERED IN ENGLAND AND WALES 5729350 - VAT REGISTRATION NUMBER 877397162
5TH FLOOR, THE OLD VINYL FACTORY, HAYES, UB3 1HA, UNITED KINGDOM