Browse Source

Write some documentation

Phyks (Lucas Verney) 4 years ago
parent
commit
5f47b0ff65

+ 1
- 1
.gitignore View File

@@ -5,4 +5,4 @@ build
5 5
 *.db
6 6
 config/
7 7
 node_modules
8
-flatisfy/web/static/js
8
+flatisfy/web/static/assets

+ 2
- 1
CONTRIBUTING.md View File

@@ -10,7 +10,8 @@
10 10
 * There is a `hooks/pre-commit` file which can be used as a `pre-commit` git
11 11
   hook to check coding style.
12 12
 * Python coding style is PEP8. JS coding style is enforced by `eslint`.
13
-* Some useful `npm` scripts are provided (`build` / `watch` / `lint`)
13
+* Some useful `npm` scripts are provided (`build:{dev,prod}` /
14
+  `watch:{dev,prod}` / `lint`)
14 15
 
15 16
 
16 17
 ## Translating the webapp

+ 15
- 49
README.md View File

@@ -4,6 +4,12 @@ Flatisfy
4 4
 Flatisfy is your new companion to ease your search of a new housing :)
5 5
 
6 6
 
7
+**Note**: This software is under heavy development at the moment, and the
8
+database schema could change at any time. Do not consider it as being
9
+production ready. However, I am currently using it for my own housing search
10
+and it is working fine :)
11
+
12
+
7 13
 It uses [Weboob](http://weboob.org/) to get all the housing posts on most of
8 14
 the websites offering housings posts, and then offers a bunch of pipelines to
9 15
 filter and deduplicate the fetched housings.
@@ -29,60 +35,20 @@ This code is not restricted to handling flats only!
29 35
 1. Clone the repository.
30 36
 2. Install required Python modules: `pip install -r requirements.txt`.
31 37
 3. Init a configuration file: `python -m flatisfy init-config > config.json`.
32
-   Edit it according to your needs (see below).
38
+   Edit it according to your needs (see doc).
33 39
 4. Build the required data files:
34 40
    `python -m flatisfy build-data --config config.json`.
35 41
 5. Use it to `fetch` (and output a filtered JSON list of flats) or `import`
36 42
    (into an SQLite database, for the web visualization) a list of flats
37 43
    matching your criteria.
38
-6. Use `python -m flatisfy serve --config config.json` to serve the web app.
39
-
40
-
41
-## Configuration
42
-
43
-List of configuration options:
44
-
45
-* `data_directory` is the directory in which you want data files to be stored.
46
-  `null` is the default value and means default `XDG` location (typically
47
-  `~/.local/share/flatisfy/`)
48
-* `max_entries` is the maximum number of entries to fetch **per Weboob
49
-  backend** (that is per housing website).
50
-* `passes` is the number of passes to run on the data. First pass is a basic
51
-  filtering and using only the informations from the housings list page.
52
-  Second pass loads any possible information about the filtered flats and does
53
-  better filtering.
54
-* `queries` is a list of queries defined in `flatboob` that should be fetched.
55
-* `database` is an SQLAlchemy URI to a database file. Defaults to `null` which
56
-  means that it will store the database in the default location, in
57
-  `data_directory`.
58
-* `navitia_api_key` is an API token for [Navitia](https://www.navitia.io/)
59
-  which is required to compute travel times.
60
-
61
-### Constraints
62
-
63
-You can specify constraints, under the `constraints` key. The available
64
-constraints are:
65
-
66
-* `area` (in m²), `bedrooms`, `cost` (in currency unit), `rooms`: this is a
67
-  tuple of `(min, max)` values, defining an interval in which the value should
68
-  lie. A `null` value means that any value is within this bound.
69
-* `postal_codes` is a list of allowed postal codes. You should include any
70
-  postal code you want, and especially the postal codes close to the precise
71
-  location you want. You MUST provide some postal codes.
72
-* `time_to` is a dictionary of places to compute travel time to them.
73
-  Typically,
74
-  ```
75
-  "time_to": {
76
-    "foobar": {
77
-        "gps": [LAT, LNG],
78
-        "time": [min, max]
79
-    }
80
-  }
81
-  ```
82
-  means that the housings must be between the `min` and `max` bounds (possibly
83
-  `null`) from the place identified by the GPS coordinates `LAT` and `LNG`
84
-  (latitude and longitude), and we call this place `foobar` in human-readable
85
-  form. Beware that `time` constraints are in **seconds**.
44
+6. Install JS libraries and build the webapp:
45
+   `npm install && npm run build:dev` (use `build:prod` in production).
46
+7. Use `python -m flatisfy serve --config config.json` to serve the web app.
47
+
48
+
49
+## Documentation
50
+
51
+See the [dedicated folder](doc/).
86 52
 
87 53
 
88 54
 ## OpenData

+ 106
- 0
doc/0.getting_started.md View File

@@ -0,0 +1,106 @@
1
+Getting started
2
+===============
3
+
4
+## TL;DR
5
+
6
+1. Clone the repository.
7
+2. Install required Python modules: `pip install -r requirements.txt`.
8
+3. Init a configuration file: `python -m flatisfy init-config > config.json`.
9
+   Edit it according to your needs (see below).
10
+4. Build the required data files:
11
+   `python -m flatisfy build-data --config config.json`.
12
+5. Use it to `fetch` (and output a filtered JSON list of flats) or `import`
13
+   (into an SQLite database, for the web visualization) a list of flats
14
+   matching your criteria.
15
+6. Install JS libraries and build the webapp:
16
+   `npm install && npm run build:dev` (use `build:prod` in production).
17
+7. Use `python -m flatisfy serve --config config.json` to serve the web app.
18
+
19
+
20
+## Available commands
21
+
22
+The available commands are:
23
+
24
+* `init-config` to generate an empty configuration file, either on the `stdin`
25
+  or in the specified file.
26
+* `fetch` to load and filter housings posts and output a JSON dump.
27
+* `filter` to filter a previously fetched list of housings posts, provided as
28
+  a JSON dump.
29
+* `import` to import and filter housing posts into the database.
30
+* `serve` to serve the built-in webapp with the development server. Do not use
31
+  in production.
32
+
33
+
34
+## Configuration
35
+
36
+List of configuration options:
37
+
38
+* `data_directory` is the directory in which you want data files to be stored.
39
+  `null` is the default value and means default `XDG` location (typically
40
+  `~/.local/share/flatisfy/`)
41
+* `max_entries` is the maximum number of entries to fetch.
42
+* `passes` is the number of passes to run on the data. First pass is a basic
43
+  filtering and using only the informations from the housings list page.
44
+  Second pass loads any possible information about the filtered flats and does
45
+  better filtering.
46
+* `database` is an SQLAlchemy URI to a database file. Defaults to `null` which
47
+  means that it will store the database in the default location, in
48
+  `data_directory`.
49
+* `navitia_api_key` is an API token for [Navitia](https://www.navitia.io/)
50
+  which is required to compute travel times.
51
+* `modules_path` is the path to the Weboob modules. It can be `None` if you
52
+  want Weboob to use the locally pip-installed modules (default value).
53
+* `port` is the port on which the development webserver should be
54
+  listening (default to `8080`).
55
+* `host` is the host on which the development webserver should be listening
56
+  (default to `127.0.0.1`).
57
+* `webserver` is a server to use instead of the default Bottle built-in
58
+  webserver, see [Bottle deployment
59
+  doc](http://bottlepy.org/docs/dev/deployment.html).
60
+
61
+_Note:_ In production, you can either use the `serve` command with a reliable
62
+webserver instead of the default Bottle webserver (specifying a `webserver`
63
+value) or use the `wsgi.py` script at the root of the repository to use WSGI.
64
+
65
+
66
+### Constraints
67
+
68
+You should specify some constraints to filter the resulting housings list,
69
+under the `constraints` key. The available constraints are:
70
+
71
+* `type` is the type of housing you want, either `RENT` (to rent), `SALE` (to
72
+  buy) or `SHARING` (for a shared housing).
73
+* `housing_types` is a list of house types you are looking for. Values can be
74
+  `APART` (flat), `HOUSE`, `PARKING`, `LAND`, `OTHER` (everything else) or
75
+  `UNKNOWN` (anything which was not matched with one of the previous
76
+  categories).
77
+* `area` (in m²), `bedrooms`, `cost` (in currency unit), `rooms`: this is a
78
+  tuple of `(min, max)` values, defining an interval in which the value should
79
+  lie. A `null` value means that any value is within this bound.
80
+* `postal_codes` is a list of postal codes. You should include any postal code
81
+  you want, and especially the postal codes close to the precise location you
82
+  want.
83
+* `time_to` is a dictionary of places to compute travel time to them.
84
+  Typically,
85
+
86
+  ```
87
+  "time_to": {
88
+    "foobar": {
89
+        "gps": [LAT, LNG],
90
+        "time": [min, max]
91
+    }
92
+  }
93
+  ```
94
+
95
+  means that the housings must be between the `min` and `max` bounds (possibly
96
+  `null`) from the place identified by the GPS coordinates `LAT` and `LNG`
97
+  (latitude and longitude), and we call this place `foobar` in human-readable
98
+  form. Beware that `time` constraints are in **seconds**.
99
+
100
+
101
+## Building the web assets
102
+
103
+If you want to build the web assets, you can use `npm run build:dev`
104
+(respectively `npm run watch:dev` to build continuously and monitor changes in
105
+source files). You can use `npm run build:prod` (`npm run watch:prod`) to do
106
+the same in production mode (with minification etc).

+ 223
- 0
doc/1.production.md View File

@@ -0,0 +1,223 @@
1
+Use in production environment
2
+=============================
3
+
4
+This is the guide for the manual installation of a "production" instance.
5
+
6
+**IMPORTANT**: At the moment, there is **no** authentication mechanism in
7
+Flatisfy. That is, if you want your instance to be private, you should put
8
+some access control on your own with the webserver. This is explained below in
9
+the case of `Nginx`.
10
+
11
+## Get the app installed
12
+
13
+```bash
14
+# Clone the repository
15
+git clone https://Phyks@git.phyks.me/Phyks/flatisfy.git && cd flatisfy
16
+
17
+# Create a virtualenv
18
+virtualenv .env && source .env/bin/activate
19
+
20
+# Install required Python modules
21
+pip install -r requirements.txt
22
+
23
+# Clone and install weboob
24
+git clone https://git.weboob.org/weboob/devel weboob && cd weboob && python setup.py install && cd ..
25
+
26
+# Install required JS libraries and build the webapp
27
+npm install && npm run build:prod
28
+
29
+# Create a minimal config file
30
+mkdir config && python -m flatisfy init-config > config/config.json
31
+
32
+# Create a data directory to store db and data files
33
+mkdir data
34
+
35
+# Edit the config file according to your needs
36
+$EDITOR config/config.json
37
+
38
+# Build the required data files
39
+python -m flatisfy build-data --config config/config.json -v
40
+
41
+# Run initial import
42
+python -m flatisfy import --config config/config.json -v
43
+```
44
+
45
+_Note_: In the config, you should set `data_directory` to the absolute path of
46
+the `data` directory created below. This directory should be writable by the
47
+user running Flatisfy. You should also set `modules_path` to the absolute path
48
+to the `modules` folder under the previous `weboob` clone. Finally, the last
49
+`import` command can be `cron`-tasked to automatically fetch available
50
+housings posts periodically.
51
+
52
+
53
+## Use an alternative Bottle backend (production)
54
+
55
+This is the simplest option, offers good performances but is less scalable.
56
+You should take care of which user is running `flatisfy`, as there can be some
57
+related security concerns.
58
+
59
+Just choose another [available
60
+webserver](https://bottlepy.org/docs/dev/deployment.html) to use in Bottle
61
+(typically `cherrypi`) and set the `webserver` config option accordingly to
62
+use it.
63
+
64
+### Nginx vhost
65
+
66
+Here is a typical minimal `Nginx` vhost you can use (which provides
67
+HTTP authentication support and SSL):
68
+
69
+```
70
+upstream _flatisfy {
71
+    server 127.0.0.1:PORT;
72
+}
73
+
74
+server {
75
+    listen 443;
76
+    server_name SERVER_NAME;
77
+    root ABSOLUTE_PATH_TO_FLATISFY_GIT_ROOT/flatisfy/web/static;
78
+
79
+    access_log  /var/log/nginx/flatisfy-access.log;
80
+    error_log   /var/log/nginx/flatisfy-error.log warn;
81
+
82
+    ssl                  on;
83
+    ssl_certificate      PATH_TO_SSL_CERT;
84
+    ssl_certificate_key  PATH_TO_SSL_KEY;
85
+
86
+    location / {
87
+        auth_basic "Restricted";
88
+        auth_basic_user_file ABSOLUTE_PATH_TO_FLATISFY_GIT_ROOT/.htpasswd;
89
+
90
+        # Try to serve directly the URI first (static content), fallback on
91
+        # passing it to the Python backend
92
+        try_files $uri @uwsgi;
93
+    }
94
+
95
+    location @uwsgi {
96
+        proxy_pass http://_flatisfy;
97
+    }
98
+}
99
+
100
+
101
+server {
102
+    listen      80;
103
+    server_name SERVER_NAME;
104
+
105
+    root /dev/null;
106
+
107
+    return 301 https://$server_name$request_uri;
108
+}
109
+```
110
+
111
+where  `PORT` (default for Flatisfy is 8080), `SERVER_NAME`,
112
+`ABSOLUTE_PATH_TO_FLATISFY_GIT_ROOT`, `PATH_TO_SSL_CERT` and `PATH_TO_SSL_KEY`
113
+should be replaced by values according to your own setup. You should also set
114
+the `.htpasswd` file with users and credentials.
115
+
116
+_Note_: This vhost is really minimalistic and you should adapt it to your
117
+setup, enforce SSL ciphers for increased security and do such good practices
118
+things.
119
+
120
+
121
+## Use WSGI
122
+
123
+This is the best option in terms of performance and is recommended. It should
124
+offer both the best security and performances.
125
+
126
+### Configure uWSGI
127
+
128
+Assuming you are running Debian stable, you should install `uwsgi`:
129
+
130
+```
131
+apt-get install uwsgi uwsgi-plugin-python
132
+```
133
+
134
+Then, you can create a `/etc/uwsgi/apps-available/flatisfy.ini` with the
135
+following content:
136
+
137
+```ini
138
+[uwsgi]
139
+socket = /run/uwsgi/app/flatisfy/socket
140
+chdir = ABSOLUTE_PATH_TO_FLATISFY_GIT_ROOT
141
+master = true
142
+plugins = python
143
+venv = ABSOLUTE_PATH_TO_FLATISFY_GIT_ROOT/.env
144
+file = wsgi.py
145
+uid = www-data
146
+gid = www-data
147
+```
148
+
149
+where `ABSOLUTE_PATH_TO_FLATISFY_GIT_ROOT` is the absolute path to the root of
150
+the Flatisfy git clone made at the beginning of this document.
151
+
152
+Now, you can enable the app:
153
+
154
+```bash
155
+ln -s /etc/uwsgi/apps-available/flatisfy.ini /etc/uwsgi/apps-enabled/flatisfy.ini
156
+```
157
+
158
+and restart `uWSGI`:
159
+
160
+```bash
161
+systemctl restart uwsgi
162
+```
163
+
164
+_Note_: You should review the `wsgi.py` file at the root of the repository to
165
+check it is matching you setup (and especially is loading the configuration
166
+from the right place, which is `config/config.json` by default).
167
+
168
+
169
+### Nginx vhost
170
+
171
+Here is a typical minimal `Nginx` vhost you can use (which provides
172
+HTTP authentication support and SSL):
173
+
174
+```
175
+upstream _flatisfy {
176
+    server unix:/run/uwsgi/app/flatisfy/socket;
177
+}
178
+
179
+server {
180
+    listen 443;
181
+    server_name SERVER_NAME;
182
+    root ABSOLUTE_PATH_TO_FLATISFY_GIT_ROOT/flatisfy/web/static;
183
+
184
+    access_log  /var/log/nginx/flatisfy-access.log;
185
+    error_log   /var/log/nginx/flatisfy-error.log warn;
186
+
187
+    ssl                  on;
188
+    ssl_certificate      PATH_TO_SSL_CERT;
189
+    ssl_certificate_key  PATH_TO_SSL_KEY;
190
+
191
+    location / {
192
+        auth_basic "Restricted";
193
+        auth_basic_user_file ABSOLUTE_PATH_TO_FLATISFY_GIT_ROOT/.htpasswd;
194
+
195
+        # Try to serve directly the URI first (static content), fallback on
196
+        # passing it to the Python backend
197
+        try_files $uri @uwsgi;
198
+    }
199
+
200
+    location @uwsgi {
201
+        include uwsgi_params;
202
+        uwsgi_pass _flatisfy;
203
+    }
204
+}
205
+
206
+
207
+server {
208
+    listen      80;
209
+    server_name SERVER_NAME;
210
+
211
+    root /dev/null;
212
+
213
+    return 301 https://$server_name$request_uri;
214
+}
215
+```
216
+
217
+where `SERVER_NAME`, `ABSOLUTE_PATH_TO_FLATISFY_GIT_ROOT`, `PATH_TO_SSL_CERT`
218
+and `PATH_TO_SSL_KEY` should be replaced by values according to your own
219
+setup. You should also set the `.htpasswd` file with users and credentials.
220
+
221
+_Note_: This vhost is really minimalistic and you should adapt it to your
222
+setup, enforce SSL ciphers for increased security and do such good practices
223
+things.

+ 4
- 1
flatisfy/web/app.py View File

@@ -83,6 +83,9 @@ def get_app(config):
83 83
     app.route("/", "GET", lambda: _serve_static_file("index.html"))
84 84
 
85 85
     # Static files
86
-    app.route("/static/<filename:path>", "GET", _serve_static_file)
86
+    app.route(
87
+        "/assets/<filename:path>", "GET",
88
+        lambda filename: _serve_static_file("/assets/{}".format(filename))
89
+    )
87 90
 
88 91
     return app

+ 3
- 3
flatisfy/web/js_src/components/flatsmap.vue View File

@@ -42,9 +42,9 @@ export default {
42 42
             },
43 43
             icons: {
44 44
                 flat: L.icon({
45
-                    iconUrl: '/static/js/' + markerUrl,
46
-                    iconRetinaUrl: '/static/js' + marker2XUrl,
47
-                    shadowUrl: '/static/js' + shadowUrl
45
+                    iconUrl: markerUrl,
46
+                    iconRetinaUrl: marker2XUrl,
47
+                    shadowUrl: shadowUrl
48 48
                 }),
49 49
                 place: L.icon.glyph({
50 50
                     prefix: 'fa',

+ 2
- 0
flatisfy/web/js_src/components/flatstable.vue View File

@@ -52,6 +52,8 @@
52 52
 </template>
53 53
 
54 54
 <script>
55
+import "font-awesome-webpack"
56
+
55 57
 export default {
56 58
     props: ['flats'],
57 59
 

+ 2
- 0
flatisfy/web/js_src/views/details.vue View File

@@ -159,6 +159,8 @@
159 159
 </template>
160 160
 
161 161
 <script>
162
+import "font-awesome-webpack"
163
+
162 164
 import FlatsMap from '../components/flatsmap.vue'
163 165
 import Slider from '../components/slider.vue'
164 166
 

+ 1
- 3
flatisfy/web/static/index.html View File

@@ -4,11 +4,9 @@
4 4
         <meta charset="utf-8">
5 5
         <meta name="format-detection" content="telephone=no">
6 6
         <title>Flatisfy</title>
7
-
8
-        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.7.0/css/font-awesome.min.css">
9 7
     </head>
10 8
     <body>
11 9
         <div id="app"></div>
12
-        <script src="static/js/bundle.js"></script>
10
+        <script src="/assets/bundle.js"></script>
13 11
     </body>
14 12
 </html>

BIN
flatisfy/web/static/js/9e9c77db241e8a58da99bf28694c907d.png View File


BIN
flatisfy/web/static/js/a159160a02a1caf189f5008c0d150f36.png View File


BIN
flatisfy/web/static/js/c6477ef24ce2054017fce1a2d8800385.png View File


BIN
flatisfy/web/static/js/d3a5d64a8534322988a4bed1b7dbc8b0.png View File


+ 8
- 2
package.json View File

@@ -10,12 +10,16 @@
10 10
   },
11 11
   "homepage": "https://git.phyks.me/Phyks/flatisfy",
12 12
   "scripts": {
13
-    "build": "webpack --colors --progress",
14
-    "watch": "webpack --colors --progress --watch",
13
+    "build:dev": "webpack --colors --progress",
14
+    "watch:dev": "webpack --colors --progress --watch",
15
+    "build:prod": "NODE_ENV=production webpack --colors --progress -p",
16
+    "watch:prod": "NODE_ENV=production webpack --colors --progress --watch -p",
15 17
     "lint": "eslint --ext .js,.vue ./flatisfy/web/js_src/**"
16 18
   },
17 19
   "dependencies": {
18 20
     "es6-promise": "^4.1.0",
21
+    "font-awesome": "^4.7.0",
22
+    "font-awesome-webpack": "0.0.5-beta.2",
19 23
     "imagesloaded": "^4.1.1",
20 24
     "isomorphic-fetch": "^2.2.1",
21 25
     "isotope-layout": "^3.0.3",
@@ -42,7 +46,9 @@
42 46
     "eslint-plugin-vue": "^2.0.1",
43 47
     "file-loader": "^0.11.1",
44 48
     "image-webpack-loader": "^3.3.0",
49
+    "less": "^2.7.2",
45 50
     "style-loader": "^0.16.1",
51
+    "url-loader": "^0.5.8",
46 52
     "vue-html-loader": "^1.2.4",
47 53
     "vue-loader": "^11.3.4",
48 54
     "vue-template-compiler": "^2.2.6",

+ 11
- 2
webpack.config.js View File

@@ -1,8 +1,9 @@
1 1
 module.exports = {
2 2
     entry: './flatisfy/web/js_src/main.js',
3 3
     output: {
4
-        path: __dirname + '/flatisfy/web/static/js/',
5
-        filename: 'bundle.js'
4
+        path: __dirname + '/flatisfy/web/static/assets/',
5
+        filename: 'bundle.js',
6
+        publicPath: '/assets/'
6 7
     },
7 8
     module: {
8 9
         loaders: [
@@ -43,6 +44,14 @@ module.exports = {
43 44
                         }
44 45
                     }
45 46
                 ]
47
+            },
48
+            {
49
+                test: /\.woff(2)?(\?v=[0-9]\.[0-9]\.[0-9])?$/,
50
+                loader: "url-loader?limit=10000&mimetype=application/font-woff"
51
+            },
52
+            {
53
+                test: /\.(ttf|eot|svg)(\?v=[0-9]\.[0-9]\.[0-9])?$/,
54
+                loader: "file-loader"
46 55
             }
47 56
         ]
48 57
     },