MultiHoster plugin type, some fixes, new documentation structure

3 files changed, 509 insertions, 0 deletions

diff --git a/docs/api/access_api.rst b/docs/api/access_api.rst
new file mode 100644
index 000000000..efa1ae3fc
--- /dev/null
+++ b/docs/api/access_api.rst
@@ -0,0 +1,122 @@
+.. _access_api:
+
+*********************
+How to access the API
+*********************
+
+pyLoad has a very powerfull API with can be accessed in several ways.
+
+Overview
+--------
+
+First of all, you need to know what you can do with our API. It lets you do all common task like
+retrieving download status, manage queue, manage accounts, modify config and so on.
+
+This document is not intended to explain every function in detail, for a complete listing
+see :class:`Api <module.Api.Api>`.
+
+Of course its possible to access the ``core.api`` attribute in plugins and hooks, but much more
+interesting is the possibillity to call function from different programs written in many different languages.
+
+pyLoad uses thrift as backend and provides its :class:`Api <module.Api.Api>` as service.
+More information about thrift can be found here http://wiki.apache.org/thrift/.
+
+
+Using Thrift
+------------
+
+Every thrift service has to define all data structures and declare every method which should be usable via rpc.
+This file is located at :file:`module/remote/thriftbackend/pyload.thrift`, its very helpful to inform about
+arguments and detailed structure of return types. However it does not contain any information about what the functions does.
+You can also look at it :doc:`here <datatypes>`
+
+Assuming you want to use the API in any other language than python than check if it is
+supported here http://wiki.apache.org/thrift/LibraryFeatures?action=show&redirect=LanguageSupport.
+
+Now install thrift, for instructions see http://wiki.apache.org/thrift/ThriftInstallation.
+If every thing went fine you are ready to generate the method stubs, the command basically looks like this. ::
+
+     $ thrift --gen (language) pyload.thrift
+
+You find now a directory named :file:`gen-(language)`. For instruction how to use the generated files consider the docs
+at the thrift wiki and the examples here http://wiki.apache.org/thrift/ThriftUsage.
+
+
+=======
+Example
+=======
+In case you want to use python, pyload has already all files included to access the api over rpc.
+
+A basic script that prints out some information: ::
+
+    from module.remote.thriftbackend.ThriftClient import ThriftClient, WrongLogin
+
+    try:
+        client = ThriftClient(host="127.0.0.1", port=7227, user="User", password="yourpw")
+    except:
+        print "Login was wrong"
+        exit()
+
+    print "Server version:", client.getServerVersion()
+    print client.statusDownloads()
+    q = client.getQueue()
+    for p in q:
+      data = client.getPackageData(p.pid)
+      print "Package Name: ", data.name
+
+That's all for now, pretty easy isn't it?
+If you still have open questions come around in irc or post them at our pyload forum.
+
+
+Using HTTP/JSON
+---------------
+
+Another maybe easier way, which does not require much setup is to access the JSON Api via HTTP.
+For this reason the webinterface must be enabled.
+
+=====
+Login
+=====
+
+First you need to authenticate, if you using this within the webinterface and the user is logged the API is also accessible,
+since they share the same cookie/session.
+
+However, if you are building a external client and want to authenticate manually
+you have to send your credentials ``username`` and ``password`` as
+POST parameter to ``http://pyload-core/api/login``.
+
+The result will be your session id. If you are using cookies, it will be set and you can use the API now.
+In case you dont't have cookies enabled you can pass the session id as ``session`` POST parameter
+so pyLoad can authenticate you.
+
+===============
+Calling Methods
+===============
+
+In general you can use any method listed at the :class:`Api <module.Api.Api>` documentation, which is also available to
+the thriftbackend.
+
+Access works simply via ``http://pyload-core/api/methodName``, where ``pyload-core`` is the ip address
+or hostname including the webinterface port. By default on local access this would be `localhost:8000`.
+
+The return value will be formatted in JSON, complex data types as dictionaries.
+As mentionted above for a documentation about the return types look at the thrift specification file  :file:`module/remote/thriftbackend/pyload.thrift`.
+
+==================
+Passing parameters
+==================
+
+To pass arguments you have two choices.
+Either use positional arguments, eg ``http://pyload-core/api/getFileData/1``, where 1 is the FileID, or use keyword arguments
+supplied via GET or POST ``http://pyload-core/api/getFileData?fid=1``. You can find the argument names in the :class:`Api <module.Api.Api>`
+documentation.
+
+It is important that *all* arguments are in JSON format. So ``http://pyload-core/api/getFileData/1`` is valid because
+1 represents an integer in json format. On the other hand if the method is expecting strings, this would be correct:
+``http://pyload-core/api/getUserData/"username"/"password"``.
+
+Strings are wrapped in double qoutes, because `"username"` represents a string in json format. It's not limited to strings and intergers,
+every container type like lists and dicts are possible. You usually don't have to convert them. just use a json encoder before using them
+in the HTTP request.
+
+Please note that the data have to be urlencoded at last. (Most libaries will do that automatically)
\ No newline at end of file
diff --git a/docs/api/datatypes.rst b/docs/api/datatypes.rst
new file mode 100644
index 000000000..02e3209cc
--- /dev/null
+++ b/docs/api/datatypes.rst
@@ -0,0 +1,352 @@
+.. _api_datatypes:
+
+***********************
+API Datatype Definition
+***********************
+
+Below you find a copy of :file:`module/remote/thriftbackend/pyload.thrift`, which is used to generate the data structs
+for various languages. It is also a good overview of avaible methods and return data.
+
+.. code-block:: c
+
+     .. [[[cog cog.out(open('module/remote/thriftbackend/pyload.thrift', 'rb').read()) ]]]
+     namespace java org.pyload.thrift
+
+     typedef i32 FileID
+     typedef i32 PackageID
+     typedef i32 ResultID
+     typedef i32 InteractionID
+     typedef list<string> LinkList
+     typedef string PluginName
+     typedef byte Progress
+     typedef byte Priority
+
+
+     enum DownloadStatus {
+       Finished
+       Offline,
+       Online,
+       Queued,
+       Skipped,
+       Waiting,
+       TempOffline,
+       Starting,
+       Failed,
+       Aborted,
+       Decrypting,
+       Custom,
+       Downloading,
+       Processing,
+       Unknown
+     }
+
+     enum Destination {
+       Collector,
+       Queue
+     }
+
+     // types for user interaction
+     // some may only be place holder currently not supported
+     // also all input - output combination are not reasonable, see InteractionManager for further info
+     enum Input {
+       NONE,
+       TEXT,
+       TEXTBOX,
+       PASSWORD,
+       BOOL,   // confirm like, yes or no dialog
+       CLICK,  // for positional captchas
+       CHOICE,  // choice from list
+       MULTIPLE,  // multiple choice from list of elements
+       LIST, // arbitary list of elements
+       TABLE  // table like data structure
+     }
+     // more can be implemented by need
+
+     // this describes the type of the outgoing interaction
+     // ensure they can be logcial or'ed
+     enum Output {
+       CAPTCHA = 1,
+       QUESTION = 2,
+       NOTIFICATION = 4,
+     }
+
+     struct DownloadInfo {
+       1: FileID fid,
+       2: string name,
+       3: i64 speed,
+       4: i32 eta,
+       5: string format_eta,
+       6: i64 bleft,
+       7: i64 size,
+       8: string format_size,
+       9: Progress percent,
+       10: DownloadStatus status,
+       11: string statusmsg,
+       12: string format_wait,
+       13: i64 wait_until,
+       14: PackageID packageID,
+       15: string packageName,
+       16: PluginName plugin,
+     }
+
+     struct ServerStatus {
+       1: bool pause,
+       2: i16 active,
+       3: i16 queue,
+       4: i16 total,
+       5: i64 speed,
+       6: bool download,
+       7: bool reconnect
+     }
+
+     struct FileData {
+       1: FileID fid,
+       2: string url,
+       3: string name,
+       4: PluginName plugin,
+       5: i64 size,
+       6: string format_size,
+       7: DownloadStatus status,
+       8: string statusmsg,
+       9: PackageID packageID,
+       10: string error,
+       11: i16 order
+     }
+
+     struct PackageData {
+       1: PackageID pid,
+       2: string name,
+       3: string folder,
+       4: string site,
+       5: string password,
+       6: Destination dest,
+       7: i16 order,
+       8: optional i16 linksdone,
+       9: optional i64 sizedone,
+       10: optional i64 sizetotal,
+       11: optional i16 linkstotal,
+       12: optional list<FileData> links,
+       13: optional list<FileID> fids
+     }
+
+     struct InteractionTask {
+       1: InteractionID iid,
+       2: Input input,
+       3: list<string> structure,
+       4: list<string> preset,
+       5: Output output,
+       6: list<string> data,
+       7: string title,
+       8: string description,
+       9: string plugin,
+     }
+
+     struct ConfigItem {
+       1: string name,
+       2: string long_name,
+       3: string description,
+       4: string type,
+       5: string default_value,
+       6: string value,
+     }
+
+     struct ConfigSection {
+       1: string name,
+       2: string long_name,
+       3: string description,
+       4: string long_description,
+       5: optional list<ConfigItem> items,
+       6: optional map<string, InteractionTask> handler,
+     }
+
+     struct CaptchaTask {
+       1: i16 tid,
+       2: binary data,
+       3: string type,
+       4: string resultType
+     }
+
+     struct EventInfo {
+       1: string eventname,
+       2: list<string> event_args,
+     }
+
+     struct UserData {
+       1: string name,
+       2: string email,
+       3: i32 role,
+       4: i32 permission,
+       5: string templateName
+     }
+
+     struct AccountInfo {
+       1: PluginName plugin,
+       2: string loginname,
+       3: bool valid,
+       4: i64 validuntil,
+       5: i64 trafficleft,
+       6: i64 maxtraffic,
+       7: bool premium,
+       8: bool activated,
+       9: map<string, string> options,
+     }
+
+     struct ServiceCall {
+         1: PluginName plugin,
+         2: string func,
+         3: string arguments, // empty string or json encoded list
+     }
+
+     struct OnlineStatus {
+         1: string name,
+         2: PluginName plugin,
+         3: string packagename,
+         4: DownloadStatus status,
+         5: i64 size,   // size <= 0 : unknown
+     }
+
+     struct OnlineCheck {
+         1: ResultID rid, // -1 -> nothing more to get
+         2: map<string, OnlineStatus> data, //url to result
+     }
+
+
+     // exceptions
+
+     exception PackageDoesNotExists{
+       1: PackageID pid
+     }
+
+     exception FileDoesNotExists{
+       1: FileID fid
+     }
+
+     exception UserDoesNotExists{
+       1: string user
+     }
+
+     exception ServiceDoesNotExists{
+       1: string plugin
+       2: string func
+     }
+
+     exception ServiceException{
+       1: string msg
+     }
+
+     service Pyload {
+
+       //config
+       string getConfigValue(1: string section, 2: string option),
+       void setConfigValue(1: string section, 2: string option, 3: string value),
+       map<string, ConfigSection> getConfig(),
+       map<PluginName, ConfigSection> getPluginConfig(),
+       ConfigSection configureSection(1: string section),
+
+       // server status
+       void pauseServer(),
+       void unpauseServer(),
+       bool togglePause(),
+       ServerStatus statusServer(),
+       i64 freeSpace(),
+       string getServerVersion(),
+       void kill(),
+       void restart(),
+       list<string> getLog(1: i32 offset),
+       bool isTimeDownload(),
+       bool isTimeReconnect(),
+       bool toggleReconnect(),
+
+       // download preparing
+
+       // packagename - urls
+       map<string, LinkList> generatePackages(1: LinkList links),
+       map<PluginName, LinkList> checkURLs(1: LinkList urls),
+       map<PluginName, LinkList> parseURLs(1: string html, 2: string url),
+
+       // parses results and generates packages
+       OnlineCheck checkOnlineStatus(1: LinkList urls),
+       OnlineCheck checkOnlineStatusContainer(1: LinkList urls, 2: string filename, 3: binary data)
+
+       // poll results from previosly started online check
+       OnlineCheck pollResults(1: ResultID rid),
+
+       // downloads - information
+       list<DownloadInfo> statusDownloads(),
+       PackageData getPackageData(1: PackageID pid) throws (1: PackageDoesNotExists e),
+       PackageData getPackageInfo(1: PackageID pid) throws (1: PackageDoesNotExists e),
+       FileData getFileData(1: FileID fid) throws (1: FileDoesNotExists e),
+       list<PackageData> getQueue(),
+       list<PackageData> getCollector(),
+       list<PackageData> getQueueData(),
+       list<PackageData> getCollectorData(),
+       map<i16, PackageID> getPackageOrder(1: Destination destination),
+       map<i16, FileID> getFileOrder(1: PackageID pid)
+
+       // downloads - adding/deleting
+       list<PackageID> generateAndAddPackages(1: LinkList links, 2: Destination dest),
+       PackageID addPackage(1: string name, 2: LinkList links, 3: Destination dest, 4: string password),
+       void addFiles(1: PackageID pid, 2: LinkList links),
+       void uploadContainer(1: string filename, 2: binary data),
+       void deleteFiles(1: list<FileID> fids),
+       void deletePackages(1: list<PackageID> pids),
+
+       // downloads - modifying
+       void pushToQueue(1: PackageID pid),
+       void pullFromQueue(1: PackageID pid),
+       void restartPackage(1: PackageID pid),
+       void restartFile(1: FileID fid),
+       void recheckPackage(1: PackageID pid),
+       void stopAllDownloads(),
+       void stopDownloads(1: list<FileID> fids),
+       void setPackageName(1: PackageID pid, 2: string name),
+       void movePackage(1: Destination destination, 2: PackageID pid),
+       void moveFiles(1: list<FileID> fids, 2: PackageID pid),
+       void orderPackage(1: PackageID pid, 2: i16 position),
+       void orderFile(1: FileID fid, 2: i16 position),
+       void setPackageData(1: PackageID pid, 2: map<string, string> data) throws (1: PackageDoesNotExists e),
+       list<PackageID> deleteFinished(),
+       void restartFailed(),
+
+       //events
+       list<EventInfo> getEvents(1: string uuid)
+       
+       //accounts
+       list<AccountInfo> getAccounts(1: bool refresh),
+       list<string> getAccountTypes()
+       void updateAccount(1: PluginName plugin, 2: string account, 3: string password, 4: map<string, string> options),
+       void removeAccount(1: PluginName plugin, 2: string account),
+       
+       //auth
+       bool login(1: string username, 2: string password),
+       UserData getUserData(1: string username, 2:string password) throws (1: UserDoesNotExists ex),
+       map<string, UserData> getAllUserData(),
+
+       //services
+
+       // servicename : description
+       map<PluginName, map<string, string>> getServices(),
+       bool hasService(1: PluginName plugin, 2: string func),
+       string call(1: ServiceCall info) throws (1: ServiceDoesNotExists ex, 2: ServiceException e),
+
+
+       //info
+       // {plugin: {name: value}}
+       map<PluginName, map<string,string>> getAllInfo(),
+       map<string, string> getInfoByPlugin(1: PluginName plugin),
+
+       //scheduler
+
+       // TODO
+
+
+       // User interaction
+
+       //captcha
+       bool isCaptchaWaiting(),
+       CaptchaTask getCaptchaTask(1: bool exclusive),
+       string getCaptchaTaskStatus(1: InteractionID tid),
+       void setCaptchaResult(1: InteractionID tid, 2: string result),
+     }
+     .. [[[end]]]
+
diff --git a/docs/api/overview.rst b/docs/api/overview.rst
new file mode 100644
index 000000000..02cee3e0d
--- /dev/null
+++ b/docs/api/overview.rst
@@ -0,0 +1,35 @@
+.. _overview:
+
+=======================================
+API - Application Programming Interface
+=======================================
+
+From Wikipedia, the free encyclopedia [1]_:
+
+    An application programming interface (API) is a source code based specification intended to be used as an interface
+    by software components to communicate with each other. An API may include specifications for routines,
+    data structures, object classes, and variables.
+
+.. rubric:: Motivation
+
+The idea of the centralized pyLoad :class:`Api <module.Api.Api>` is to give uniform access to all integral parts
+and plugins in pyLoad, and furthermore to other clients, written in arbitrary programming languages.
+Most of the :class:`Api <module.Api.Api>` functionality is exposed via RPC [2]_ and accessible via thrift [3]_ or
+simple JSON objects [4]_. In conclusion the :class:`Api <module.Api.Api>` is accessible via many programming language,
+over network from remote maschines and over browser with javascript.
+
+
+.. rubric:: Contents
+
+.. toctree::
+
+    access_api.rst
+    datatypes.rst
+
+
+.. rubric:: Footnotes
+
+.. [1] http://en.wikipedia.org/wiki/Application_programming_interface
+.. [2] http://en.wikipedia.org/wiki/Remote_procedure_call
+.. [3] http://en.wikipedia.org/wiki/Thrift_(protocol)
+.. [4] http://en.wikipedia.org/wiki/Json
\ No newline at end of file