Uwsgi

Changelog [20130622]

Bugfixes
^^^^^^^^

- Fixed a corner case bug when response offloading is enabled, but no request plugin is loaded
- Fixed harakiri routing when multiple rules are in place (return NEXT instead of CONTINUE)
- Fixed curl crashing master on slow dns responses (Łukasz Mierzwa)
- Removed PTRACE check in uwsgi.h (it is no more needed since uWSGI 1.0)
- Fixed --print-sym
- Added a newline in --cflags
- Improved python3 detection and compilation
- Fixed Coro::AnyEvent loop engine (John Berthels)
- Rack api functions are now static
- Better fastcgi handling of big uploads
- Improved GCC usage on Darwin for Python non-apple builds
- Fixed XCLIENT usage in rawrouter
- Use the clang preprocessor instead of hardcoded 'cpp' when CC=clang is used
- Set 16bit options to 65535 when higher values are requested
- Fixed virtualhosting (it is now compatible with 1.4 configurations)

New features
^^^^^^^^^^^^

PyPy performance and features improvents
****************************************

The PyPy plugin has been improved a lot. The amount of C code has been reduced by 70%, so, now, the vast majority of the plugin is
written in python. The c helpers have been removed allowing the python part to directly call native uWSGI functions via cffi.

Support for PyPy continulets (and their greenlet abstraction) has been added (while waiting for a solid gevent port for pypy) and a chat example is already available
(using the uwsgi async api):

https://github.com/unbit/uwsgi/tree/master/t/pypy

https://github.com/unbit/uwsgi/blob/master/contrib/pypy/uwsgi_pypy_greenlets.py

The pypy uwsgi api has been improved and now you can use the uwsgidecorators module (even if the spooler subsystem is still missing)


Chunked input api
*****************

In the last days there have been a bunch of discussions on how to correctly manage chunked input. As basically none
of the available standards support it in a "definitive" way, we have defined a low-level api (so we can easily adapt it
in the feature).

The api exposes two functions:

uwsgi.chunked_read()

and

uwsgi.chunked_read_nb()

A non blocking chat example:

.. code-block:: py

import uwsgi
def application(e, sr):
while True:
uwsgi.wait_fd_read(uwsgi.connection_fd())
uwsgi.suspend()
msg = uwsgi.chunked_read_nb()
if msg: print "core %d" % e['uwsgi.core'], msg


Toward better third-party plugins management: the --dot-h option
****************************************************************

As the --cflags option shows the CFLAGS used to build the server, the --dot-h option shows the content of uwsgi.h

This means the content of uwsgi.h is now embedded in the binary (compressed where available).

It could look a bizarre choice but the objective is to allow easy compilation of plugins out of the uwsgi sources
(somethign that will be available in 2.0 for sure)

setmethod, seturi and setpathinfo routing action
************************************************

we continue extending the routing api.

Three new actions have been added to dinamically modify the request

UWSGI_INCLUDES
**************

You can now ovverride the include search path (while building uWSGI) with this environment variable.

Improved set_user_harakiri api function
***************************************

Now the uwsgi.set_user_harakiri automatically recognize mules and spoolers. It has been added to the ruby/rack, pypy and perl/psgi plugins

--add-cache-item [cache ]KEY=VALUE
**********************************

this is a commodity option (mainly useful for testing) allowing you to store an item in a uWSGI cache during startup

the router_xmldir plugin
************************

This is a proof of concept plugin aimed at stressing the transformation api.

It basically generates an xml representation of a directory. This could be useful to
implement apache-style directoryindex:

Check this example using xslt:

https://github.com/unbit/uwsgi/issues/271#issuecomment-19820204

Implement __call__ for @spool* decorators
*****************************************

Thanks to 'anaconda', you can now directly call functions mapped to the spooler, so instead of

.. code-block:: py

myfunc.spool(args)

you can directly do:

.. code-block:: py

myfunc(args)

the old way is obviously supported

the uwsgi[lq] routing var
*************************

this routing var exports the current size of the listen_queue:

.. code-block:: ini

[uwsgi]
...
route-if = higher:${uwsgi[lq]};70 break:503 Server Overload
...

--use-abort
***********

On some system the SEGV signal handler cannot be correctly restored after the uWSGI backtrace.

If you want to generate a core file, you may want to trigger a SIGABRT soon after the backtrace.

Availability
^^^^^^^^^^^^

uWSGI 1.9.13 will be available 22th June 2013 at this url:

https://projects.unbit.it/downloads/uwsgi-1.9.13.tar.gz

Changelog [20130605]

Bugfixes
^^^^^^^^

- offloading cache writes will return the correct status code and not 202
- you can now control the path of temporary files setting the TMPDIR environment variable (this fixes an old issue for users without control over /tmp)
- fixed a compilation error on amqp imperial monitor
- cron commands are correctly escaped when reported in the stats server
- fixed fastcgi parser corner-case bug with big uploads
- fixed support for newest cygwin

New Features
^^^^^^^^^^^^

Offloading responses
********************

Take the following WSGI app:

.. code-block:: python

def application(environ, start_response):
start_response('200 OK', [('Content-Type', 'text/plain')])
return ['u' * 100000000]

it will generate about 100megs of data. 98% of the time the worker spent on the request was on the data transfer. As the whole response
is followed by the end of the request we can offload the data write to a thread and free the worker suddenly (so it will be able to handle a new request).

100megs are a huge value, but even 1MB can cause a dozen of poll()/write() syscalls that blocks your worker for a bunch of milliseconds

Thanks to the 'memory offload' facility added in 1.9.11 implementing it has been very easy.

The offloading happens via the :doc:`Transformations`

.. code-block:: ini

[uwsgi]
socket = :3031
wsgi-file = myapp.py
; offload all of the application writes
route-run = offload:

By default the response is buffered to memory until it reaches 1MB size. After that it will be buffered to disk and the offload engine
will use sendfile().

You can set the limit (in bytes) after disk buffering passing an argument to the offload:

.. code-block:: ini

[uwsgi]
socket = :3031
wsgi-file = myapp.py
; offload all of the application writes (buffer to disk after 1k)
route-run = offload:1024

"offload" MUST BE the last transformation in the chain

.. code-block:: ini

[uwsgi]
socket = :3031
wsgi-file = myapp.py
; gzip the response
route-run = gzip:
; offload all of the application writes (buffer to disk after 1k)
route-run = offload:1024


JWSGI and JVM improvements
**************************

The JVM plugin has been extended to support more objects helper (like ArrayList), while JWSGI can now be used as
a low-level layer to add support for more JVM-based languages.

JRuby integration is the first attempt of such a usage. We have just releases a JWSGI to Rack adapter allowing you tun run
Ruby/Rack apps on top of JRUBY:

https://github.com/unbit/jwsgi-rack


A similar approach for Jython is on work

--touch-signal
**************

A new touch option has been added allowing the rise of a uwsgi signal when a file is touched:

.. code-block:: ini

[uwsgi]
...
; raise signal 17 on /tmp/foobar modifications
touch-signal = /tmp/foobar 17
...

The "pipe" offload engine
*************************

A new offload engine allowing transfer from a socket to the client has been added.

it will be automatically used in the new router_memacached and router_redis plugins


memcached router improvements
*****************************


You can now store responses in memcached (as you can already do with uWSGI caching)

.. code-block:: ini

[uwsgi]
...
route = ^/cacheme memcachedstore:addr=127.0.0.1:11211,key=${REQUEST_URI}
route = ^/cacheme2 memcachedstore:addr=192.168.0.1:11211,key=${REQUEST_URI}foobar
...

obviously you can get them too

.. code-block:: ini

[uwsgi]
...
route-run = memcached:addr=127.0.0.1:11211,key=${REQUEST_URI}
...

The memcached router is now builtin in the default profiles

The new redis router
********************

Based on the memcached router, a redis router has been added. It works in the same way:


.. code-block:: ini

[uwsgi]
...
route = ^/cacheme redisstore:addr=127.0.0.1:6379,key=${REQUEST_URI}
route = ^/cacheme2 redisstore:addr=192.168.0.1:6379,key=${REQUEST_URI}foobar
...

... and get the values

.. code-block:: ini

[uwsgi]
...
route-run = redis:addr=127.0.0.1:6379,key=${REQUEST_URI}
...

The redis router is builtin by default

The "hash" router
*****************

this special routing action allows you to hash a string and return a value from a list (indexed with the hashed key).

Take the following list:

127.0.0.1:11211

192.168.0.1:11222

192.168.0.2:22122

192.168.0.4:11321

and a string:

/foobar

we hash the string /foobar using djb33x algorithm and we apply the modulo 4 (the size of the items list) to the result.

We get "1", so we will get the second items in the list (we are obviously zero-indexed).

Do you recognize the pattern ?

Yes, it is the standard way to distribute items on multiple servers (memcached clients for example uses it from ages).

The hash router exposes this system allowing you to distribute items in you redis/memcached servers or to make other funny things.

This an example usage for redis:

.. code-block:: ini

[uwsgi]
...
; hash the list of servers and return the value in the MYNODE var
route = ^/cacheme_as/(.*) hash:items=127.0.0.1:11211;192.168.0.1:11222;192.168.0.2:22122;192.168.0.4:11321,key=$1,var=MYNODE
; log the result
route = ^/cacheme_as/(.*) log:${MYNODE} is the choosen memcached server !!!
; use MYNODE as the server address
route = ^/cacheme_as/(.*) memcached:addr=${MYNODE},key=$1
...

you can even choose the hashing algo from those supported in uWSGI

.. code-block:: ini

[uwsgi]
...
; hash the list of servers with murmur2 and return the value in the MYNODE var
route = ^/cacheme_as/(.*) hash:algo=murmur2,items=127.0.0.1:11211;192.168.0.1:11222;192.168.0.2:22122;192.168.0.4:11321,key=$1,var=MYNODE
; log the result
route = ^/cacheme_as/(.*) log:${MYNODE} is the choosen memcached server !!!
; use MYNODE as the server address
route = ^/cacheme_as/(.*) memcached:addr=${MYNODE},key=$1
...

the router_hash plugin is compiled-in by default

Availability
^^^^^^^^^^^^

uWSGI 1.9.12 will be available starting from 20130605 at the following url

https://projects.unbit.it/downloads/uwsgi-1.9.12.tar.gz

Changelog [20130526]

Bugfixes
********

* Fixed Python 3 stdout/stderr buffering
* Fixed mule messages (``@mulefunc`` is now reliable)
* Fixed ``SCRIPT_NAME`` handling in dynamic mode
* Fixed X-Sendfile with gzip static mode
* Fixed cache item maximum size with custom block size
* Fixed cache path handling

New features
************

The new high-performance PyPy plugin
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Credits: Maciej Fijalkowski

We are pleased to announce the availability of the new PyPy plugin.

PyPy team has been great in helping us. We hope the uWSGI integration (that exposed new challenges to the PyPy project)
will help PyPy becaming better and better.

Official docs: :doc:`PyPy`

Cron improvements
^^^^^^^^^^^^^^^^^

Credits: Łukasz Mierzwa

Unique crons
------------

You can now avoid overlapping crons. The uWSGI master will track death of a single task, and until its death the same cron
will not be triggered:

.. code-block:: ini

[uwsgi]
unique-cron = -1 -1 -1 -1 -1 my_script.sh

cron2 syntax
------------

A key/value variant of the --cron option is now available:

.. code-block:: ini

[uwsgi]
cron2 = minute=39,hour=23,month=-1,week=-1,day=-1,unique=1,legion=foobar,harakiri=30

harakiri cron
-------------

When using the ``cron2`` option you are allowed to set a harakiri timeout for a cron task. Just add ``harakiri=n`` to the options.

Support for GNU Hurd
^^^^^^^^^^^^^^^^^^^^

Debian GNU/Hurd has been recently released. uWSGI 1.9.11 can be built over it, however very few tests have been made.

The memory offload engine
^^^^^^^^^^^^^^^^^^^^^^^^^

Idea: Stefano Brentegani

When serving content from the cache, a worker could get blocked during transfer from memory to the socket.

A new offload engine named "memory" allows to offload memory transfers. The cache router automatically supports it.
Support for more areas will be added soon.

To enable it just add ``--offload-threads <n>``

New Websockets chat example
^^^^^^^^^^^^^^^^^^^^^^^^^^^

An example websocket chat using Redis has been added to the repository:

https://github.com/unbit/uwsgi/blob/master/tests/websockets_chat.py

Error routes
^^^^^^^^^^^^

You can now define a routing table to be executed as soon as you set the HTTP status code in your plugin.

This allows you to completely modify the response. This is useful for custom error codes.

All of the routing standard options are available (included labels) plus an optimized ``error-route-status``
matching a specific HTTP status code:

.. code-block:: ini

[uwsgi]
error-route-status = 502 redirect:http://unbit.it

Support for corner case usage in wsgi.file_wrapper
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Generally the ``wsgi.file_wrapper`` callable expects a file-like object. PEP 333/3333 reports a special pattern when the object
is not a file (call ``read()`` until the object is consumed). uWSGI now supports this pattern (even if in a hacky way).

HTTP/HTTPS router keepalive improvements
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Credits: André Cruz

When using ``--http-keepalive`` you can now hold the connection open even if the request has a body.


The harakiri routing action
^^^^^^^^^^^^^^^^^^^^^^^^^^^

You can now set a harakiri timer for each request using internal routing:

.. code-block:: ini

[uwsgi]
; set harakiri to 30 seconds for request starting with /slow
route = ^/slow harakiri:30

RPC wrappers
^^^^^^^^^^^^

The RPC plugin has been extended to allows interoperation with other standards.

Currently a simple HTTP wrapper and an XML-RPC one are exposed.

The HTTP simple wrapper works by parsing ``PATH_INFO``.

A ``/foo/bar/test`` call will result in

uwsgi.rpc('foo', 'bar', 'test')

To enable this HTTP mode just set the ``modifier2`` to '2':

.. code-block:: ini

[uwsgi]
http-socket = :9090
http-socket-modifier1 = 173
http-socket-modifier2 = 2
; load the rpc code
import = myrpcfuncs.py

or (to have more control)

.. code-block:: ini

[uwsgi]
http-socket = :9090
route-run = uwsgi:,173,2
; load the rpc code
import = myrpcfuncs.py


The XML-RPC wrapper works in the same way, but it uses the modifier2 value '3'. It requires a libxml2-enabled build of uWSGI.

.. code-block:: ini

[uwsgi]
http-socket = :9090
route-run = uwsgi:,173,3
; load the rpc code
import = myrpcfuncs.py

Then just call it:

.. code-block:: python

proxy = xmlrpclib.ServerProxy("http://localhost:9090')
proxy.hello('foo','bar','test')

You can combine multiple wrappers using routing.

.. code-block:: ini

[uwsgi]
http-socket = :9090
; /xml force xmlrpc wrapper
route = ^/xml uwsgi:,173,3
; fallback to HTTP simple
route-if-not = startswith:${PATH_INFO};/xml uwsgi:,173,2
; load the rpc code
import = myrpcfuncs.py


Availability
************

uWSGI 1.9.11 will be available since 20130526 at:

https://projects.unbit.it/downloads/uwsgi-1.9.11.tar.gz

Changelog [20130511]

Bugfixes
********

* fixed alarm threads during reloads
* fixed uninitialized memory in --touch-* options
* fixed a regression in --attach-daemon

New Features
************

Welcome to gccgo
^^^^^^^^^^^^^^^^

Go support in gcc 4.8 is amazing, thanks to the split-stack feature you can now have goroutines without allocating a whole pthread.

As Go 1.1 will be no no more compatible with uWSGI, gccgo will became the official way for running go apps.

The gccgo plugin is in early stage of development but it is already able to run in preforking mode.

We are heavy working on a true "goroutines" Loop engine. Stay tuned.

Final routes
^^^^^^^^^^^^

You can now run routing rules after a request. Obviously not all of the exposed actions make sense after the request but you should be able
to write even more complex setup.

Check this request limiter based on HTTP response status (a value you can get only after a request):

https://github.com/unbit/uwsgi/blob/master/t/routing/errorlimiter.ini

Availability
************

uWSGI 1.9.10 will be available since 20130511 at the following url:

https://projects.unbit.it/downloads/uwsgi-1.9.10.tar.gz

Changelog [20130508]

Special Warning !!!
*******************

The router_basicauth plugin has changed its default behaviour to return "break" if authorization fails.

The "basicauth-next" action, uses the old behaviour (returning "next")

This new approach should reduce security problems caused by wrong configurations

Bugfixes
********

* do not increment "tx" statistics counter for "unaccountable" plugins
* fixed --backtrace-depth
* fixed cache-sync parsing
* fixed mule farms initialization
* fixed multithreading bug when regexp conditional route is used
* fixed default-app usage in the psgi plugin
* fixed python dynamic mode + threads
* fixed error reporting in corerouter when retry is in place
* correctly report harakiri condition for gateways

New Features
************

The WebDav plugin
^^^^^^^^^^^^^^^^^

WebDav is one of the much requested features for the project. We now have a beta-quality plugin, already supporting
additional standards like the carddav:

https://github.com/unbit/uwsgi/blob/master/t/webdav/carddav.ini

The official modifier is 35, and to mount a simple directory as a webdav shares (for use with windows, gnome...) you only need to
specify the --webdav-mount option:

.. code-block:: ini

[uwsgi]
plugin = webdav
http-socket = :9090
http-socket-modifier1 = 35
webdav-mount = /home/foobar

remember to protect shares:

.. code-block:: ini

[uwsgi]
plugin = webdav,router_basicauth
http-socket = :9090
http-socket-modifier1 = 35
route-run = basicauth:CardDav uWSGI server,unbit:unbit
webdav-mount = /home/foobar

WebDav attributes are stored as filesystem xattr, so be sure to use a filesystem supporting them (ext4, xfs, hfs+...)

LOCK/UNLOCK support is still incomplete

Official docs will be available soon.

Support for Go 1.1 (more or less, sad news for go users...)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Albeit you can successfully embed go 1.1 apps in uWSGI, go 1.1 will be completely fork() unsafe.

That means you are not able to use multiprocessing, the master, mules and so on.

Basically half of the uWSGI features will be no more usable in go apps.

Things could change in the future, but currently our objective is better integration with the gccgo project.

Go 1.0.x will continue to be supported (unless gccgo shows itself as a better alternative)

More to come soon.

Improved async modes
^^^^^^^^^^^^^^^^^^^^

Stackless, Greenlet and Fiber support have been updated to support new async features

The radius plugin
^^^^^^^^^^^^^^^^^

You can now authenticate over radius servers using the router_radius plugin:

.. code-block:: ini

[uwsgi]
plugin = webdav,router_radius
http-socket = :9090
http-socket-modifier1 = 35
route-run = radius:realm=CardDav uWSGI server,server=127.0.0.1:1812
webdav-mount = /home/foobar

The SPNEGO plugin
^^^^^^^^^^^^^^^^^

Another authentication backend, using SPNEGO (kerberos)

.. code-block:: ini

[uwsgi]
plugin = webdav,router_spnego
http-socket = :9090
http-socket-modifier1 = 35
route-run = spnego:HTTP@localhost
webdav-mount = /home/foobar

The plugin is beta quality as it leaks memory (it looks like a bug in MIT-kerberos) and Heimdal implementation does not work.

More reports are wellcomed

The ldap authenticator
^^^^^^^^^^^^^^^^^^^^^^

(Author: Łukasz Mierzwa)

Currently it lacks SASL support. Will be improved soon.

.. code-block:: ini

[uwsgi]
...
plugins = router_ldapauth
route = ^/a ldapauth:LDAP realm,url=ldap://ldap.domain,com;basedn=ou=users,dc=domain.com;binddn=uid=proxy,dc=domain,dc=com;bindpw=password



New internal routing features
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

We removed the GOON action, as it was messy and basically useless with the new authentication approach

The "setscriptname" action has been added to override the internally computed SCRIPT_NAME (not only the var)

The "donotlog" action forces uWSGI to not log the current request

The "regexp" routing conditions has been improved to allows grouping. Now you can easily manipulate strings and adding them as new request VARS:

.. code-block:: ini

[uwsgi]
...
route-if = regexp:${REQUEST_URI};^/(.)oo addvar:PIPPO=$1
route-run = log:PIPPO IS ${PIPPO}

this will take the first char of foo and place in the PIPPO request var

Gevent atexit hook
^^^^^^^^^^^^^^^^^^

uwsgi.atexit hook is now honoured by the gevent plugin (Author: André Cruz)


Streaming transformations
^^^^^^^^^^^^^^^^^^^^^^^^^

Transformations can be applied on the fly (no buffering involved).

Check updated docs: :doc:`Transformations`

The xattr plugin
^^^^^^^^^^^^^^^^

The xattr plugin allows you to reference files extended attributes in the internal routing subsystem:

.. code-block:: ini

[uwsgi]
...
route-run = addvar:MYATTR=user.uwsgi.foo.bar
route-run = log:The attribute is ${xattr[/tmp/foo:MYATTR]}


or (variant with 2 vars)

.. code-block:: ini

[uwsgi]
...
route-run = addvar:MYFILE=/tmp/foo
route-run = addvar:MYATTR=user.uwsgi.foo.bar
route-run = log:The attribute is ${xattr2[MYFILE:MYATTR]}


The airbrake plugin
^^^^^^^^^^^^^^^^^^^

(Author: Łukasz Mierzwa)

Currently at early stage of development allows sending uWSGI exceptions and alarms to airbrake servers.

Official docs will be available soon.

Legion Daemons
^^^^^^^^^^^^^^

(Author: Łukasz Mierzwa)

No, it is not a blackmetal band, it is a new feature of :doc:`Legion` allowing you to run external processes
only when an instance is a lord:

.. code-block:: ini

[uwsgi]

master = true
http = :8081
stats = :2101
wsgi-file = tests/staticfile.py

logdate = true

legion = legion1 225.1.1.1:19678 100 bf-cbc:abc
legion-node = legion1 225.1.1.1:19678

legion-attach-daemon = legion1 memcached -p 10001

legion-smart-attach-daemon = legion1 /tmp/memcached.pid memcached -p 10002 -d -P /tmp/memcached.pid


--touch-exec
^^^^^^^^^^^^

A new "touch" option (like --touch-reload) is available, triggering the execution of a command:

.. code-block:: ini

[uwsgi]
...
touch-exec = /tmp/foobar run_my_script.sh
touch-exec = /var/test/foo.txt run_my_second_script.sh arg1 arg2


Math for cache
^^^^^^^^^^^^^^

You can now use the caching subsystem to store 64bit signed numbers and apply atomic operations on them.

The uwsgi api has been extended with 5 new functions (currently exposed only by the python plugin):

*uwsgi.cache_num(key[,cache]) -> get the 64bit number from the specified item

*uwsgi.cache_inc(key[,amount=1,expires,cache]) -> increment the specified key by the specified amount

*uwsgi.cache_dec(key[,amount=1,expires,cache]) -> deccrement the specified key by the specified amount

*uwsgi.cache_mul(key[,amount=2,expires,cache]) -> multiply the specified key by the specified amount

*uwsgi.cache_div(key[,amount=2,expires,cache]) -> divide the specified key by the specified amount

The new api has been exposed to the routing subsystem, allowing you to implement advanced patterns, like the request limiter:

https://github.com/unbit/uwsgi/blob/master/t/routing/limiter.ini

the example shows hot to limit the request of a single ip to 10 every 30 seconds

The long-term objective of this new feature is being the base for the upcoming metric subsystem

Availability
************

uWSGI 1.9.9 will be availabel since 20130508 at the following url

https://projects.unbit.it/downloads/uwsgi-1.9.9.tar.gz

Changelog [20130423]

Note: this is an "emergency" release fixing 2 regressions causing a crash during reloads and when using async+uGreen

Bugfixes
********

- fixed a crash when reloading the master
- fixed a crash in async mode + uGreen
- the 'mime' routing var requires a request var (not a raw string)

Availability
************

You can download uWSGi 1.9.8 from https://projects.unbit.it/downloads/uwsgi-1.9.8.tar.gz