Browse Source

Started to rework the CozyWeboob code

Move it to `infotuyoob`, clean the API and reorganize files.
Phyks (Lucas Verney) 3 years ago
parent
commit
3a75032693
No known key found for this signature in database

+ 28
- 91
README.md View File

@@ -1,30 +1,19 @@
1
-CozyWeboob
2
-==========
1
+infotuyo/infotuyoob
2
+===================
3 3
 
4
-This is an attempt at using [Weboob](http://weboob.org/) as a
5
-[Cozy](http://cozy.io/) [Konnector](https://github.com/cozy-labs/konnectors).
6
-It wraps around Weboob, receiving a JSON description of the modules to fetch
7
-on `stdin` and returning a JSON of the fetched results on `stdout`.
4
+This script wraps around [Weboob](http://weboob.org/) and its modules to
5
+expose an HTTP JSON API, that could be easily requested by
6
+[huginn](https://github.com/cantino/huginn/).
8 7
 
9
-Although the primary goal is to wrap around Weboob to use it in Cozy, this
10
-script might be of interest for anyone willing to wrap around Weboob and
11
-communicate with JSON pipes.
12 8
 
9
+## Installation
13 10
 
14
-## Usage
15
-
16
-First, you need to have Weboob installed on your system.
17
-
18
-## Cozyweboob script
19
-
20
-Typical command-line usage for this script is:
21
-```bash
22
-cat konnectors.json | python -m cozyweboob.__main__
23
-```
24
-where `konnectors.json` is a valid JSON file defining konnectors to be used.
11
+1. Clone this repository
12
+2. Install `weboob` according to [their doc]()
13
+3. Install requirements, `pip install -r requirements.txt`.
25 14
 
26 15
 
27
-## Server script
16
+## Usage
28 17
 
29 18
 Typical command-line usage for this script is:
30 19
 ```bash
@@ -33,78 +22,17 @@ Typical command-line usage for this script is:
33 22
 This script spawns a Bottle webserver, listening on `localhost:8080` (by
34 23
 default).
35 24
 
36
-It exposes a couple of routes:
37
-
38
-* the `/fetch` route, which supports `POST` method to send a valid JSON string
39
-  defining konnectors to be used as the request body. Typical example to send
40
-  it some content is:
41
-
42
-  ```bash
43
-  curl -X POST --data "$(cat konnectors.json)" "http://localhost:8080/"
44
-  ```
45
-  where `konnectors.json` is a valid JSON file defining konnectors to be used.
46
-  Downloaded files will be stored in a temporary directory, and their file URI
47
-  will be passed back in the output JSON. If you do not have a direct access
48
-  to the filesystem, you can use the `/retrieve` endpoint below to retrieve
49
-  such downloaded files through the network.
50
-
51
-* the `/list` route, which will provide you a JSON dump of all the available
52
-  modules, their descriptions and the configuration options you should provide
53
-  them.
54
-
55
-* the `/retrieve` route, which supports `POST` method and a single `path` `POST`
56
-  parameter which is the path to the previously downloaded file to retrieve.
57
-  Note that this route will not delete the temporary file whose content has
58
-  been retrieved, and you should delete it manually.
59
-
60
-* the `/clean` route (`POST` method), which will delete all temporary
61
-  downloaded files. This route will return a JSON list of deleted folders.
62 25
 
63
-**IMPORTANT:** Note this small webserver is **not** production ready and only
64
-here as a proof of concept and to be used in a controlled development
65
-environment. The `/retrieve` route will basically provide anyone to access any
66
-file from your temp directory, which is a real security concern in production.
26
+## API
67 27
 
68
-Note: You can specify the host and port to listen on using the
69
-`COZYWEBOOB_HOST` and `COZYWEBOOB_PORT` environment variables.
28
+For a complete description of the API, see [the API description in the `doc/`
29
+folder](doc/api/v1.md).
70 30
 
71 31
 
72
-## Conversation script
32
+## Capabilities
73 33
 
74
-There is another command-line script available if you would rather communicate
75
-with it in a conversation manner, using `stdin` and `stdout` (typically to
76
-integrate it with Node modules using
77
-[Python-shell](https://github.com/extrabacon/python-shell)). To run it, use:
78
-```bash
79
-./stdin_conversation.py
80
-```
81
-
82
-Then, you can write on `stdin` and fetch the responses from `stdout`.
83
-Available commands are:
84
-* `GET /list` to list all available modules.
85
-* `POST /fetch JSON_PARAMS` where `JSON_PARAMS` is an input JSON for module
86
-  parameters.
87
-  Downloaded files will be stored in a temporary directory, and their file URI
88
-  will be passed back in the output JSON.
89
-* `POST /clean` to clean temporary downloaded files.
90
-* `exit` to quit the script and end the conversation.
91
-
92
-JSON responses are the same one as from the HTTP server script. It is
93
-basically the same script without HTTP encapsulation.
94
-
95
-_Note_: To simplify the script, note that it only supports single line
96
-commands. Then, your `JSON_PARAMS` should be the same single `stdin` line as
97
-the `GET /fetch` part.
98
-
99
-
100
-## Notes concerning all the available scripts
101
-
102
-Using `COZYWEBOOB_ENV=debug`, you can enable debug features for all of these
103
-scripts, which might be useful for development. These features are:
104
-* Logging
105
-* If you pass a blank field in a JSON konnector description
106
-(typically `password: ""`), the script will ask you its value at runtime,
107
-using `getpass`.
34
+`Weboob` exports data according to
35
+[capabilities](http://dev.weboob.org/api/capabilities/index). **TODO**
108 36
 
109 37
 
110 38
 ## Input JSON file
@@ -155,8 +83,18 @@ by Weboob. Detailed informations about these other entires can be found in the
155 83
 All contributions are welcome. Feel free to make a PR :)
156 84
 
157 85
 Python code is currently Python 2, but should be Python 3 compatible as Weboob
158
-is moving towards Python 3. All Python code should be PEP8 compliant. I use
159
-some extra rules, taken from PyLint.
86
+is moving towards Python 3. All Python code should be PEP8 compliant.
87
+
88
+
89
+## CLI interface
90
+
91
+It is possible to run the script from the CLI interface rather than using the
92
+HTTP server, which can be useful for debugging. Typical command-line usage for
93
+this script is:
94
+```bash
95
+cat modules.json | python -m cozyweboob.__main__
96
+```
97
+where `modules.json` is a valid JSON file defining modules to be used.
160 98
 
161 99
 
162 100
 ## License
@@ -167,7 +105,6 @@ explicitly mentionned otherwise.
167 105
 
168 106
 ## Credits
169 107
 
170
-* [Cozy](http://cozy.io/) and the cozy guys on #cozycloud @ freenode
171 108
 * [Weboob](http://weboob.org/) and the weboob guys on #weboob @ freenode
172 109
 * [Kresus](https://github.com/bnjbvr/kresus/) for giving the original idea and
173 110
   base code.

+ 4
- 0
config.py View File

@@ -0,0 +1,4 @@
1
+debug = False
2
+server = "cherrypy"
3
+host = "localhost"
4
+port = 7777

+ 0
- 9
cozyweboob/__init__.py View File

@@ -1,9 +0,0 @@
1
-"""
2
-CozyWeboob main module
3
-"""
4
-from __future__ import absolute_import
5
-
6
-from cozyweboob.WeboobProxy import WeboobProxy
7
-from cozyweboob.__main__ import clean, main_fetch, main
8
-
9
-__all__ = ["WeboobProxy", "clean", "main_fetch", "main"]

+ 0
- 17
cozyweboob/tools/env.py View File

@@ -1,17 +0,0 @@
1
-"""
2
-Helper functions related to environment variables.
3
-"""
4
-import os
5
-
6
-
7
-def is_in_debug_mode():
8
-    """
9
-    Check whether cozyweboob is running in debug mode.
10
-
11
-    Returns:
12
-        true / false
13
-    """
14
-    return (
15
-        "COZYWEBOOB_ENV" in os.environ and
16
-        os.environ["COZYWEBOOB_ENV"] == "debug"
17
-    )

+ 0
- 49
cozyweboob/tools/weboob_tools.py View File

@@ -1,49 +0,0 @@
1
-"""
2
-Helper functions related to Weboob-specific code.
3
-"""
4
-from weboob.tools.value import (ValueBackendPassword, ValueInt, ValueFloat,
5
-                                ValueBool)
6
-
7
-
8
-def Value_to_string(value):
9
-    """
10
-    Convert a Value definition from Weboob to a string describing the field
11
-    type.
12
-
13
-    Args:
14
-        value: A Weboob value definition.
15
-    Returns: A string describing the value type.
16
-    """
17
-    if isinstance(value, ValueBackendPassword):
18
-        return "password"
19
-    elif isinstance(value, ValueInt):
20
-        return "int"
21
-    elif isinstance(value, ValueFloat):
22
-        return "float"
23
-    elif isinstance(value, ValueBool):
24
-        return "bool"
25
-    else:
26
-        return "text"
27
-
28
-
29
-def dictify_config_desc(config):
30
-    """
31
-    Convert a Weboob BackendConfig description to a JSON-serializabe dict.
32
-
33
-    Args:
34
-        config: A Weboob BackendConfig object.
35
-    Returns: A JSON-serializable dict.
36
-    """
37
-    return {
38
-        name: {
39
-            "type": Value_to_string(value),
40
-            "label": value.label,
41
-            "required": value.required,
42
-            "default": value.default,
43
-            "masked": value.masked,
44
-            "regexp": value.regexp,
45
-            "choices": value.choices,
46
-            "tiny": value.tiny
47
-        }
48
-        for name, value in config.items()
49
-    }

+ 53
- 0
doc/api/v1.md View File

@@ -0,0 +1,53 @@
1
+Infotuyo/infotuyoob API v1 documentation
2
+========================================
3
+
4
+This is the documentation for the first version of the JSON API exposed by
5
+`infotuyoob`.
6
+
7
+
8
+## `/api/v1/modules`
9
+
10
+List all the available modules, their descriptions and the configuration
11
+options you can pass them (required and optional ones).
12
+
13
+```
14
+GET /api/v1/modules
15
+
16
+```
17
+
18
+
19
+## `/api/v1/modules/<MODULE>`
20
+
21
+List all the description and the configuration options you can pass to the
22
+specified module (required and optional ones).
23
+
24
+```
25
+GET /api/v1/modules/freemobile
26
+
27
+```
28
+
29
+
30
+## `/api/v1/module/<MODULE>/fetch`
31
+
32
+* the `/fetch` route, which supports `POST` method to send a valid JSON string
33
+  defining konnectors to be used as the request body. Typical example to send
34
+  it some content is:
35
+
36
+  ```bash
37
+  curl -X POST --data "$(cat konnectors.json)" "http://localhost:8080/"
38
+  ```
39
+  where `konnectors.json` is a valid JSON file defining konnectors to be used.
40
+  Downloaded files will be stored in a temporary directory, and their file URI
41
+  will be passed back in the output JSON. If you do not have a direct access
42
+  to the filesystem, you can use the `/retrieve` endpoint below to retrieve
43
+  such downloaded files through the network.
44
+
45
+
46
+## `/api/v1/module/<MODULE>/download`
47
+
48
+**TODO**
49
+
50
+* the `/retrieve` route, which supports `POST` method and a single `path` `POST`
51
+  parameter which is the path to the previously downloaded file to retrieve.
52
+  Note that this route will not delete the temporary file whose content has
53
+  been retrieved, and you should delete it manually.

+ 13
- 0
infotuyoob.py View File

@@ -0,0 +1,13 @@
1
+#!/usr/bin/env python
2
+# coding: utf-8
3
+"""
4
+HTTP server wrapper around weboob
5
+"""
6
+import sys
7
+
8
+from infotuyoob.cmd import main
9
+
10
+if len(sys.argv) > 1:
11
+    main(sys.argv[1])
12
+else:
13
+    main()

cozyweboob/WeboobProxy.py → infotuyoob/WeboobProxy.py View File

@@ -1,22 +1,20 @@
1
-#!/usr/bin/env python2
1
+#!/usr/bin/env python
2
+# coding: utf-8
2 3
 """
3
-Wrapper script around Weboob to be able to use it in combination with Cozy +
4
-Konnectors easily.
4
+Wrapper script around Weboob to be able to use it from an HTTP API.
5 5
 
6 6
 Part of this code comes from [Kresus](https://github.com/bnjbvr/kresus/)
7 7
 written by bnjbvr and released under MIT.
8 8
 """
9
-from __future__ import absolute_import
10
-from __future__ import print_function
9
+from __future__ import absolute_import, unicode_literals, print_function
11 10
 
12 11
 import logging
13 12
 
14 13
 from weboob.core import Weboob
15 14
 from weboob.exceptions import ModuleInstallError
16 15
 
17
-import cozyweboob.tools.weboob_tools as weboob_tools
18
-
19
-from cozyweboob.tools.progress import DummyProgress
16
+from infotuyoob.utils import weboob_utils
17
+from infotuyoob.utils.progress import DummyProgress
20 18
 
21 19
 
22 20
 # Module specific logger
@@ -24,14 +22,6 @@ logger = logging.getLogger(__name__)
24 22
 
25 23
 
26 24
 class WeboobProxy(object):
27
-    """
28
-    Connector is a tool that connects to common websites like bank website,
29
-    phone operator website... and that grabs personal data from there.
30
-    Credentials are required to make this operation.
31
-
32
-    Technically, connectors are weboob backend wrappers.
33
-    """
34
-
35 25
     @staticmethod
36 26
     def version():
37 27
         """
@@ -50,7 +40,7 @@ class WeboobProxy(object):
50 40
         self.weboob = Weboob()
51 41
         self.backend = None
52 42
 
53
-    def install_modules(self, capability=None, name=None):
43
+    def ensure_latest_modules_installed(self, capability=None, name=None):
54 44
         """
55 45
         Ensure latest version of modules is installed.
56 46
 
@@ -83,32 +73,38 @@ class WeboobProxy(object):
83 73
             if infos.is_installed()
84 74
         }
85 75
 
86
-    def list_modules(self, capability=None, name=None):
76
+    def modules_infos(self, capability=None, name=None):
87 77
         """
88
-        Ensure latest version of modules is installed.
78
+        Get description and available configuration options for the specified
79
+        modules.
80
+
81
+        Also ensure latest version of modules is installed.
89 82
 
90 83
         Args:
91
-            capability: Restrict the modules to install to a given capability.
92
-            name: Only install the specified module.
93
-        Returns: The list of installed module infos.
84
+            capability: Restrict the modules to search for to a given
85
+                        capability.
86
+            name: Only do it for the specified module.
87
+        Returns: The list of module infos.
94 88
         """
95 89
         # Update modules and get the latest up to date list
96
-        installed_modules = self.install_modules(
90
+        installed_modules = self.ensure_latest_modules_installed(
97 91
             capability=capability,
98 92
             name=name
99 93
         )
100 94
         # For each module, get back its config options and website base url
101 95
         for module_name in installed_modules:
102
-            module = self.weboob.modules_loader.get_or_load_module(module_name)
103
-            installed_modules[module_name]["config"] = (
104
-                weboob_tools.dictify_config_desc(module.config)
96
+            installed_modules[module_name] = weboob_utils.clean_module_dict(
97
+                installed_modules[module_name],
98
+                self.weboob.modules_loader.get_or_load_module(module_name)
105 99
             )
106
-            installed_modules[module_name]["website"] = module.website
107 100
         return {
108
-            'modules': [
109
-                dict(module, name=name)
110
-                for name, module in installed_modules.items()
111
-            ]
101
+            'modules': {
102
+                name: dict(module)
103
+                for name, module in sorted(
104
+                    installed_modules.items(),
105
+                    key=lambda x: x[0]
106
+                )
107
+            }
112 108
         }
113 109
 
114 110
     def init_backend(self, modulename, parameters):

+ 6
- 0
infotuyoob/__init__.py View File

@@ -0,0 +1,6 @@
1
+#!/usr/bin/env python
2
+# coding: utf-8
3
+
4
+from __future__ import absolute_import, unicode_literals
5
+
6
+__version__ = "0.1"

cozyweboob/__main__.py → infotuyoob/__main__.py View File


cozyweboob/capabilities/CapDocument.py → infotuyoob/capabilities/CapDocument.py View File


cozyweboob/capabilities/__init__.py → infotuyoob/capabilities/__init__.py View File


cozyweboob/capabilities/base.py → infotuyoob/capabilities/base.py View File


+ 63
- 0
infotuyoob/cmd.py View File

@@ -0,0 +1,63 @@
1
+#!/usr/bin/env python
2
+# coding: utf-8
3
+from __future__ import unicode_literals, absolute_import, print_function
4
+
5
+"""
6
+    Main script to launch the webserver
7
+"""
8
+
9
+import logging
10
+
11
+from bottle import run
12
+
13
+from infotuyoob.WeboobProxy import WeboobProxy
14
+from infotuyoob.routes import get_app
15
+from infotuyoob.utils.settings import load_settings
16
+
17
+# Module specific logger
18
+LOGGER = logging.getLogger(__name__)
19
+
20
+
21
+def configured_app(settings_file=None):
22
+    settings = load_settings(settings_file)
23
+
24
+    # Debug only: Set logging level and format
25
+    if settings["debug"]:
26
+        logging.basicConfig(
27
+            format='%(levelname)s: %(message)s',
28
+            level=logging.INFO
29
+        )
30
+    else:
31
+        logging.basicConfig(
32
+            format='%(levelname)s: %(message)s',
33
+            level=logging.ERROR
34
+        )
35
+    # Ensure all modules are installed and up to date before starting the
36
+    # server
37
+    LOGGER.info("Ensuring all modules are installed and up to date.")
38
+    WeboobProxy().ensure_latest_modules_installed()
39
+    LOGGER.info("Starting server.")
40
+
41
+    return settings, get_app()
42
+
43
+
44
+def main(settings_file=None):
45
+    settings, app = configured_app(settings_file)
46
+
47
+    if settings["debug"]:
48
+        run(
49
+            app,
50
+            host=settings["host"],
51
+            port=settings["port"],
52
+            server=settings["server"],
53
+            debug=settings["debug"],
54
+            reload=True
55
+        )
56
+    else:
57
+        run(
58
+            app,
59
+            host=settings["host"],
60
+            port=settings["port"],
61
+            server=settings["server"],
62
+            debug=settings["debug"]
63
+        )

+ 78
- 0
infotuyoob/routes.py View File

@@ -0,0 +1,78 @@
1
+# coding: utf-8
2
+
3
+from __future__ import unicode_literals, absolute_import
4
+
5
+import bottle
6
+
7
+from infotuyoob.utils.jsonwriter import pretty_json
8
+from infotuyoob.WeboobProxy import WeboobProxy
9
+
10
+app = bottle.Bottle()
11
+
12
+@app.route("/")
13
+def index():
14
+    """
15
+    Index view, list APIs.
16
+    """
17
+    return pretty_json({
18
+        "api": {
19
+            "v1": "/api/v1"
20
+        }
21
+    })
22
+
23
+
24
+@app.route("/api/v1")
25
+def api_v1_index():
26
+    """
27
+    Index view of the v1 API, list endpoints.
28
+    """
29
+    return pretty_json({
30
+        "endpoints": {
31
+            "List all modules and configuration": "/api/v1/modules",
32
+            "List configuration for a module": "/api/v1/modules/:module",
33
+            "Update modules": "/api/v1/modules/update",
34
+            "Fetch data from a module": "/api/v1/modules/:module/fetch"
35
+        }
36
+    })
37
+
38
+
39
+@app.route("/api/v1/modules")
40
+def api_v1_modules_infos():
41
+    """
42
+    List all available weboob modules and their configuration options.
43
+    """
44
+    return pretty_json(
45
+        WeboobProxy().modules_infos()
46
+    )
47
+
48
+
49
+@app.route("/api/v1/modules/update")
50
+def api_v1_module_update():
51
+    """
52
+    Update all available weboob modules.
53
+    """
54
+    WeboobProxy().ensure_latest_modules_installed()
55
+    return
56
+
57
+
58
+@app.route("/api/v1/modules/:module_name")
59
+def api_v1_module_infos(module_name):
60
+    """
61
+    List available configuration options for the specified module.
62
+    """
63
+    return pretty_json(
64
+        WeboobProxy().modules_infos(name=module_name)
65
+    )
66
+
67
+
68
+@app.post("/api/v1/modules/:module_name/fetch")
69
+def api_v1_module_fetch():
70
+    """
71
+    Fetch data from the specified module.
72
+    """
73
+    # TODO
74
+    pass
75
+
76
+
77
+def get_app():
78
+    return app

cozyweboob/tools/__init__.py → infotuyoob/utils/__init__.py View File


cozyweboob/tools/jsonwriter.py → infotuyoob/utils/jsonwriter.py View File


cozyweboob/tools/progress.py → infotuyoob/utils/progress.py View File


+ 36
- 0
infotuyoob/utils/settings.py View File

@@ -0,0 +1,36 @@
1
+# coding: utf-8
2
+
3
+from __future__ import unicode_literals
4
+
5
+import json
6
+import os
7
+
8
+
9
+def load_settings(settings_file=None):
10
+    """
11
+    Load current settings, overloading the default ones with user-specified
12
+    ones.
13
+
14
+    Params:
15
+        settings_file: A path to a user-provided settings file.
16
+    Returns: A settings dict.
17
+    """
18
+    settings = {}
19
+
20
+    current_path = os.path.dirname(os.path.realpath(__file__))
21
+    default_settings_path = os.path.join(
22
+        current_path,
23
+        "..",
24
+        "default_settings.json"
25
+    )
26
+    with open(default_settings_path, 'r') as fh:
27
+        settings = json.load(fh)
28
+
29
+    if settings_file is not None:
30
+        with open(settings_file, 'r') as fh:
31
+            settings = dict(
32
+                settings,
33
+                **json.load(fh)
34
+            )
35
+
36
+    return settings

+ 85
- 0
infotuyoob/utils/weboob_utils.py View File

@@ -0,0 +1,85 @@
1
+"""
2
+Helper functions related to Weboob-specific code.
3
+"""
4
+from weboob.tools.value import (ValueBackendPassword, ValueInt, ValueFloat,
5
+                                ValueBool)
6
+
7
+
8
+def Value_to_string(value):
9
+    """
10
+    Convert a Value definition from Weboob to a string describing the field
11
+    type.
12
+
13
+    Args:
14
+        value: A Weboob value definition.
15
+    Returns: A string describing the value type.
16
+    """
17
+    if isinstance(value, ValueBackendPassword):
18
+        return "password"
19
+    elif isinstance(value, ValueInt):
20
+        return "int"
21
+    elif isinstance(value, ValueFloat):
22
+        return "float"
23
+    elif isinstance(value, ValueBool):
24
+        return "bool"
25
+    else:
26
+        return "text"
27
+
28
+
29
+def dictify_config_desc(config):
30
+    """
31
+    Convert a Weboob BackendConfig description to a JSON-serializabe dict.
32
+
33
+    Args:
34
+        config: A Weboob BackendConfig object.
35
+    Returns: A JSON-serializable dict.
36
+    """
37
+    dictified_config = {}
38
+    for name, value in config.items():
39
+        dictified_config[name] = {
40
+            "label": value.label,
41
+            "required": value.required,
42
+            "masked": value.masked
43
+        }
44
+
45
+        if value.choices:
46
+            dictified_config[name]["choices"] = value.choices,
47
+        else:
48
+            dictified_config[name]["type"] = Value_to_string(value)
49
+
50
+        if value.tiny:
51
+            dictified_config[name]["tiny"] = value.tiny
52
+
53
+        if value.default:
54
+            dictified_config[name]["default"] = value.default
55
+
56
+        if value.regexp:
57
+            dictified_config[name]["regexp"] = value.regexp
58
+
59
+    return dictified_config
60
+
61
+
62
+def clean_module_dict(module_infos, loaded_module):
63
+    """
64
+    Extend a module infos dict with elements from the loaded module, and clean
65
+    the fields.
66
+
67
+    Params:
68
+        - module_infos: A dict of module infos, as returned by
69
+                        `repositories.get_module_info`.
70
+        - loaded_module: A loaded module object, as returned by
71
+                        `repositories.get_or_load_module`.
72
+    Returns:
73
+        A cleaned dict containing all possible information about this module.
74
+    """
75
+    cleaned_dict = module_infos
76
+    cleaned_dict["capabilities"] = (
77
+        cleaned_dict["capabilities"].split()
78
+    )
79
+    cleaned_dict["config"] = (
80
+        dictify_config_desc(loaded_module.config)
81
+    )
82
+    cleaned_dict["website"] = loaded_module.website
83
+    del(cleaned_dict["urls"])
84
+
85
+    return cleaned_dict

konnectors.json.sample → modules.json.sample View File


+ 2
- 1
requirements.txt View File

@@ -1 +1,2 @@
1
-bottle
1
+bottle==0.12.9
2
+cherrypy

+ 0
- 98
server.py View File

@@ -1,98 +0,0 @@
1
-#!/usr/bin/env python2
2
-"""
3
-HTTP server wrapper around weboob
4
-"""
5
-import logging
6
-import os
7
-import tempfile
8
-
9
-from bottle import post, request, route, run, static_file
10
-
11
-from cozyweboob import main as cozyweboob
12
-from cozyweboob import clean
13
-from cozyweboob import WeboobProxy
14
-from cozyweboob.tools.env import is_in_debug_mode
15
-from cozyweboob.tools.jsonwriter import pretty_json
16
-
17
-# Module specific logger
18
-logger = logging.getLogger(__name__)
19
-
20
-
21
-@post("/fetch")
22
-def fetch_view():
23
-    """
24
-    Fetch from weboob modules.
25
-    """
26
-    params = request.body.read()
27
-    return pretty_json(cozyweboob(params))
28
-
29
-
30
-@post("/retrieve")
31
-def retrieve_view():
32
-    """
33
-    Retrieve a previously downloaded file from weboob modules.
34
-
35
-    Note: Beware, this route is meant to be used in a controlled development
36
-    environment and can result in leakage of information from your temp
37
-    default directory.
38
-    """
39
-    path = request.forms.get("path")
40
-    return static_file(path.replace(tempfile.gettempdir(), './'),
41
-                       tempfile.gettempdir(),
42
-                       download=True)
43
-
44
-
45
-@post("/clean")
46
-def clean_view():
47
-    """
48
-    Delete all the temporary downloaded files. These are the
49
-    "cozyweboob-*-tmp" folders in your system tmp dir.
50
-    """
51
-    return pretty_json(clean())
52
-
53
-
54
-@route("/list")
55
-def list_view():
56
-    """
57
-    List all available weboob modules and their configuration options.
58
-    """
59
-    proxy = WeboobProxy()
60
-    return pretty_json(proxy.list_modules())
61
-
62
-
63
-def init():
64
-    """
65
-    Init function
66
-    """
67
-    # Debug only: Set logging level and format
68
-    if is_in_debug_mode():
69
-        logging.basicConfig(
70
-            format='%(levelname)s: %(message)s',
71
-            level=logging.INFO
72
-        )
73
-    else:
74
-        logging.basicConfig(
75
-            format='%(levelname)s: %(message)s',
76
-            level=logging.ERROR
77
-        )
78
-    # Ensure all modules are installed and up to date before starting the
79
-    # server
80
-    logger.info("Ensuring all modules are installed and up to date.")
81
-    proxy = WeboobProxy()
82
-    proxy.install_modules()
83
-    logger.info("Starting server.")
84
-
85
-
86
-def main():
87
-    """
88
-    Main function
89
-    """
90
-    init()
91
-    # Get host to listen on
92
-    HOST = os.environ.get("COZYWEBOOB_HOST", "localhost")
93
-    PORT = os.environ.get("COZYWEBOOB_PORT", 8080)
94
-    run(host=HOST, port=PORT, debug=is_in_debug_mode())
95
-
96
-
97
-if __name__ == "__main__":
98
-    main()

+ 0
- 114
stdin_conversation.py View File

@@ -1,114 +0,0 @@
1
-#!/usr/bin/env python2
2
-"""
3
-Wrapper around weboob to be called from mode, in a conversation way, similar to
4
-the
5
-[Python-shell](https://github.com/Birch-san/python-shell/blob/9d8641dc1e55e808ba82d029f9920413ab63206f/test/python/conversation.py)
6
-conversation example.
7
-"""
8
-import logging
9
-import sys
10
-
11
-from cozyweboob import main as cozyweboob
12
-from cozyweboob import clean
13
-from cozyweboob import WeboobProxy
14
-from cozyweboob.tools.env import is_in_debug_mode
15
-from cozyweboob.tools.jsonwriter import json_dump
16
-
17
-# Module specific logger
18
-logger = logging.getLogger(__name__)
19
-
20
-
21
-def fetch_view(params):
22
-    """
23
-    Fetch from weboob modules.
24
-    """
25
-    return json_dump(cozyweboob(params))
26
-
27
-
28
-def list_view():
29
-    """
30
-    List all available weboob modules and their configuration options.
31
-    """
32
-    proxy = WeboobProxy()
33
-    return json_dump(proxy.list_modules())
34
-
35
-
36
-def clean_view():
37
-    """
38
-    Clean temporary downloaded files.
39
-    """
40
-    return json_dump(clean())
41
-
42
-
43
-def process_query(query):
44
-    """
45
-    Process input query on the command-line.
46
-
47
-    Args:
48
-        query: The query received on stdin.
49
-    Returns:
50
-        - A JSON response if a valid query is received.
51
-        - False if should exit.
52
-        - None if invalid query is received.
53
-    """
54
-    query = query.strip()
55
-    if query == "GET /ping":
56
-        return 'UP'
57
-    elif query == "GET /list":
58
-        # List modules view
59
-        logger.info("Calling /list view.")
60
-        return list_view()
61
-    elif query == "POST /clean":
62
-        # Clean view
63
-        logger.info("Calling /clean view")
64
-        return clean_view()
65
-    elif query.startswith("POST /fetch"):
66
-        # Fetch modules view
67
-        logger.info("Calling /fetch view.")
68
-        params = query.split()[2]
69
-        return fetch_view(params)
70
-    elif query == "exit":
71
-        # Exit command
72
-        logger.info("Exiting.")
73
-        return False
74
-    else:
75
-        # Invalid query
76
-        logger.error("Invalid query, exiting: %s" % query)
77
-        sys.exit(1)
78
-
79
-
80
-def main():
81
-    """
82
-    Main function
83
-    """
84
-    # Debug only: Set logging level and format
85
-    if is_in_debug_mode():
86
-        logging.basicConfig(
87
-            format='%(levelname)s: %(message)s',
88
-            level=logging.INFO
89
-        )
90
-    else:
91
-        logging.basicConfig(
92
-            format='%(levelname)s: %(message)s',
93
-            level=logging.ERROR
94
-        )
95
-    # Ensure all modules are installed and up to date before starting the
96
-    # server
97
-    logger.info("Ensuring all modules are installed and up to date.")
98
-    proxy = WeboobProxy()
99
-    proxy.install_modules()
100
-    logger.info("Starting server.")
101
-    while True:
102
-        line = sys.stdin.readline()
103
-        if not line:
104
-            break
105
-        response = process_query(line)
106
-        if response:
107
-            print(response)
108
-        else:
109
-            break
110
-        sys.stdout.flush()
111
-
112
-
113
-if __name__ == "__main__":
114
-    main()