.. _webserver_evaluation:

====================
Webserver Evaluation
====================

pyLoad supports all kind of webserver that are usable with bottle.py [1]_.
For this reason we evaluted each of them to find the ones that are worth to be supported in pyLoad. The results of this
evaluation make sure pyload can select the most suited webserver with its auto selecting algorithm.

First selection
---------------

The first step was to take a short look at every webserver. For some it was not needed to further inspect them,
since they don't meet our requirements.

================== ==================================================================
Disregarded server Reason
================== ==================================================================
paste              threaded server, no improvement to bundled one
twisted            Too heavy (30 MB RAM min), far more complex as what we need
diesel             Problems with setup, no default packages, Not working in tests
gunicorn           Preforking server, messes many things up in our use-case
gevent             Not usuable with several threads
gae                Google App Engine, not for personal maschines
rocket             threaded server, seems not better than bundled one
waitress           no large improvement to bundled threaded
================== ==================================================================

pyLoad has an threaded server bundled itself. All threaded server that were tested seems not better than this
implementation and thus were not further benchmarked. "flup", known as "fastcgi" in pyload, serves a different
use-case and is not taken into consideration here.

Comparision
-----------

The remaining servers, were evaluated for different criteria. We ran the following benchmark with different options:

    ab -n 15000 -c 1 http://127.0.0.1:8001/login

This benchmark was ran with -c 1, -c 5, as well as -k option to test performance with more concurrency and keep-alive
feature, we use time per request (mean, across all concurrent requests) for comparision.

Additionally we collected RAM usage statistic before (b.) (only 2-3 pages retrieved) and after (a.) the benchmarks were run.
The comparision also includes some notes and available packages or features, especially SSL is of interest.

========== ======== ======== ====== ====== ======= ======= === ============= ================================
Server     RAM (b.) RAM (a.) -c 1   -c 5   -c 1 -k -c 5 -k SSL Packages      Notes
========== ======== ======== ====== ====== ======= ======= === ============= ================================
wsgiref    21.7     22.6     1.240  1.179  1.312   1.513   No  Included
threaded   25.5     28.0     0.912  1.139  0.656   0.784   Yes Included
tornado    23.8     25.9     0.874  0.935  -       -       Yes mac,deb,arch
                                                               freebsd
fapws3     22.3     23.8     0.740  0.733  0.786   0.594   No  pip           Very reliable under load,
                                                                             problem with shutdown, will
                                                                             need patches for integration
meinheld   22.3     23.7     0.622  1.001  1.076   1.388   Yes pip           Segfaults when shutdown
eventlet   25.0     26.0     1.021  1.031  0.755   0.740   Yes mac,deb       Struggles a bit under load
                                                               freebsd       More ram with -k (27.6)

bjoern     21.7     23.2     0.623  0.513  -       -       No  git, freebsd  memory-leak with faulty -k
                                                                             packages out of date
========== ======== ======== ====== ====== ======= ======= === ============= ================================

"wsgiref" is a standard implementation shipped with python and within pyLoad known as builtin.
"threaded" is taken from cherryPy and also included with pyLoad.
The keep-alive implementation of "ab" is not 100% compliant, some server struggle with it.

Conclusion
----------

The wsgiref server is known to show strange performance on some system and is therefore not selected by default anymore.
The included threaded server has all needed functions, including SSL, and is usable without any other packages.
Threaded will be selected in case none of the other server is installed.

Our auto-select will favor RAM usage over performance too choose the most lightweight server as possible.
Activating SSL will decrease the options, many lightweight servers don't include SSL by choice.
They suggest tools like pound [2]_, stunnel [3]_, or any other reverse proxy capable server. Also these that are capable
of SSL suggest using other tools, their SSL performance was not tested here.

pyLoad will select a server in following order:
fapws3 -> meinheld -> bjoern -> tornado -> eventlet

.. rubric:: Footnotes

.. [1] https://bitbucket.org/spoob/pyload/src/127adb41465712548949ea872a5453e4b0b0fbb8/module/lib/bottle.py?at=default#cl-2555
.. [2] http://www.apsis.ch/pound/
.. [3] https://www.stunnel.org/index.html