Multiple constraints in config and Tcl data

**Note**: This new feature requires a modification of the SQLite database used by Flatisfy. If you don't care about your current database, you can just remove it and do a reimport. Otherwise, you should do the following modifications manually:

* Add a [flatisfy_constraint](ed7e9dfc1a/flatisfy/models/flat.py (L89)) VARCHAR field in the flats table.
* Create [this table](ed7e9dfc1a/flatisfy/models/public_transport.py (L24-32)) for public transport data and [this one](ed7e9dfc1a/flatisfy/models/postal_code.py (L24-34)) for postal codes data.

Closes #58

See merge request !10

CONTRIBUTING.md

@@ -56,3 +56,20 @@ the list of available backends in
 and update the list of `BACKEND_PRECEDENCES` for deduplication in
 [flatisfy/filters/duplicates.py#L24-31](https://git.phyks.me/Phyks/flatisfy/blob/master/flatisfy/filters/duplicates.py#L24-31).
 Thats' all!
+
+
+## Adding new data files
+
+If you want to add new data files, especially for public transportation stops
+(to cover more cities), please follow these steps:
+
+1. Download and put the **original** file in `flatisfy/data_files`. Please,
+   use the original data file to ease tracking licenses and be able to still
+   have a working pipeline, by letting the user download it and place it in
+   the right place, in case of license conflict.
+2. Mention the added data file and its license in `README.md`, in the
+   dedicated section.
+3. Write a preprocessing function in `flatisfy/data_files/__init__.py`. You
+   can have a look at the existing functions for a model.
+
+Thanks!

README.md

@@ -73,7 +73,8 @@ which covers Paris. If you want to run the script using some other location,
 you might have to change these files by matching datasets.
 
 * [LaPoste Hexasmal](https://datanova.legroupe.laposte.fr/explore/dataset/laposte_hexasmal/?disjunctive.code_commune_insee&disjunctive.nom_de_la_commune&disjunctive.code_postal&disjunctive.libell_d_acheminement&disjunctive.ligne_5) for the list of cities and postal codes in France.
-* [RATP stations](https://data.ratp.fr/explore/dataset/positions-geographiques-des-stations-du-reseau-ratp/table/?disjunctive.stop_name&disjunctive.code_postal&disjunctive.departement) for the list of subway stations with their positions in Paris and nearby areas.
+* [RATP (Paris) stations](https://data.ratp.fr/explore/dataset/positions-geographiques-des-stations-du-reseau-ratp/table/?disjunctive.stop_name&disjunctive.code_postal&disjunctive.departement) for the list of subway/tram/bus stations with their positions in Paris and nearby areas.
+* [Tcl (Lyon) stations](https://download.data.grandlyon.com/wfs/rdata?SERVICE=WFS&VERSION=2.0.0&outputformat=GEOJSON&maxfeatures=4601&request=GetFeature&typename=tcl_sytral.tclarret&SRSNAME=urn:ogc:def:crs:EPSG::4326) for the list of subway/tram/bus stations with their positions in Paris and nearby areas.
 
 Both datasets are licensed under the Open Data Commons Open Database License
 (ODbL): https://opendatacommons.org/licenses/odbl/.

doc/0.getting_started.md

@@ -124,31 +124,14 @@ under the `constraints` key. The available constraints are:
   form. Beware that `time` constraints are in **seconds**.
 
 
+You can think of constraints as "a set of criterias to filter out flats". You
+can specify as many constraints as you want, in the configuration file,
+provided that you name each of them uniquely.
+
+
 ## Building the web assets
 
 If you want to build the web assets, you can use `npm run build:dev`
 (respectively `npm run watch:dev` to build continuously and monitor changes in
 source files). You can use `npm run build:prod` (`npm run watch:prod`) to do
 the same in production mode (with minification etc).
-
-
-## Tips
-
-### Running with multiple configs
-
-Let's say you are looking for a place in different places far apart (e.g. both
-a house in the country for the weekends and a place in a city for the week),
-you can use multiple configuration file (in this case, two configuration
-files) to define everything.
-
-Indeed, `serve` command for the web app only use a subset of the
-configuration, basically only the `data_directory`, `database` URI and so on.
-So, you are free to use any config file for `serve` command and still run
-`import` commands multiple times with different configuration files, for
-different housing queries.
-
-This is kind of a hack on the current system, but is working!
-
-*Note*: You can also use this tip if you are living in a city split across
-multiple postal codes and want to implement constraints such as "close to
-place X if postal code is Y".

flatisfy/__main__.py

@@ -10,11 +10,13 @@ import sys
 
 logging.basicConfig()
 
+# pylint: disable=locally-disabled,wrong-import-position
 import flatisfy.config
 from flatisfy import cmds
 from flatisfy import data
 from flatisfy import fetch
 from flatisfy import tools
+# pylint: enable=locally-disabled,wrong-import-position
 
 
 LOGGER = logging.getLogger("flatisfy")
@@ -53,6 +55,10 @@ def parse_args(argv=None):
         "-vv", action="store_true",
         help="Debug logging output."
     )
+    parent_parser.add_argument(
+        "--constraints", type=str,
+        help="Comma-separated list of constraints to consider."
+    )
 
     # Subcommands
     subparsers = parser.add_subparsers(
@@ -143,55 +149,70 @@ def main():
                          "you run Flatisfy.")
             sys.exit(1)
 
+    # Purge command
+    if args.cmd == "purge":
+        cmds.purge_db(config)
+        return
+
     # Build data files
     try:
+        force = False
+        if args.cmd == "build-data":
+            force = True
+
+        data.preprocess_data(config, force=force)
+        LOGGER.info("Done building data!")
+
         if args.cmd == "build-data":
-            data.preprocess_data(config, force=True)
             sys.exit(0)
-        else:
+    except flatisfy.exceptions.DataBuildError as exc:
-            data.preprocess_data(config)
+        LOGGER.error("%s", exc)
-    except flatisfy.exceptions.DataBuildError:
         sys.exit(1)
 
     # Fetch command
     if args.cmd == "fetch":
         # Fetch and filter flats list
-        flats_list = fetch.fetch_flats_list(config)
+        fetched_flats = fetch.fetch_flats(config)
-        flats_list = cmds.filter_flats(config, flats_list=flats_list,
+        fetched_flats = cmds.filter_fetched_flats(config,
+                                                  fetched_flats=fetched_flats,
                                                   fetch_details=True)["new"]
         # Sort by cost
-        flats_list = tools.sort_list_of_dicts_by(flats_list, "cost")
+        fetched_flats = tools.sort_list_of_dicts_by(fetched_flats, "cost")
 
         print(
-            tools.pretty_json(flats_list)
+            tools.pretty_json(sum(fetched_flats.values(), []))
         )
+        return
     # Filter command
     elif args.cmd == "filter":
         # Load and filter flats list
         if args.input:
-            flats_list = fetch.load_flats_list_from_file(args.input)
+            fetched_flats = fetch.load_flats_from_file(args.input, config)
 
-            flats_list = cmds.filter_flats(config, flats_list=flats_list,
+            fetched_flats = cmds.filter_fetched_flats(
-                                           fetch_details=False)["new"]
+                config,
+                fetched_flats=fetched_flats,
+                fetch_details=False
+            )["new"]
 
             # Sort by cost
-            flats_list = tools.sort_list_of_dicts_by(flats_list, "cost")
+            fetched_flats = tools.sort_list_of_dicts_by(fetched_flats, "cost")
 
             # Output to stdout
             print(
-                tools.pretty_json(flats_list)
+                tools.pretty_json(sum(fetched_flats.values(), []))
             )
         else:
             cmds.import_and_filter(config, load_from_db=True)
+        return
     # Import command
     elif args.cmd == "import":
         cmds.import_and_filter(config, load_from_db=False)
-    # Purge command
+        return
-    elif args.cmd == "purge":
-        cmds.purge_db(config)
     # Serve command
     elif args.cmd == "serve":
         cmds.serve(config)
+        return
 
 
 if __name__ == "__main__":

flatisfy/cmds.py

@@ -10,6 +10,8 @@ import logging
 import flatisfy.filters
 from flatisfy import database
 from flatisfy.models import flat as flat_model
+from flatisfy.models import postal_code as postal_code_model
+from flatisfy.models import public_transport as public_transport_model
 from flatisfy import fetch
 from flatisfy import tools
 from flatisfy.filters import metadata
@@ -19,19 +21,34 @@ from flatisfy.web import app as web_app
 LOGGER = logging.getLogger(__name__)
 
 
-def filter_flats(config, flats_list, fetch_details=True):
+def filter_flats_list(config, constraint_name, flats_list, fetch_details=True):
     """
     Filter the available flats list. Then, filter it according to criteria.
 
     :param config: A config dict.
+    :param constraint_name: The constraint name that the ``flats_list`` should
+    satisfy.
     :param fetch_details: Whether additional details should be fetched between
     the two passes.
     :param flats_list: The initial list of flat objects to filter.
     :return: A dict mapping flat status and list of flat objects.
     """
-    # pylint: disable=locally-disabled,redefined-variable-type
     # Add the flatisfy metadata entry and prepare the flat objects
-    flats_list = metadata.init(flats_list)
+    flats_list = metadata.init(flats_list, constraint_name)
+
+    # Get the associated constraint from config
+    try:
+        constraint = config["constraints"][constraint_name]
+    except KeyError:
+        LOGGER.error(
+            "Missing constraint %s. Skipping filtering for these posts.",
+            constraint_name
+        )
+        return {
+            "new": [],
+            "duplicate": [],
+            "ignored": []
+        }
 
     first_pass_result = collections.defaultdict(list)
     second_pass_result = collections.defaultdict(list)
@@ -40,6 +57,7 @@ def filter_flats(config, flats_list, fetch_details=True):
     # unwanted postings as possible
     if config["passes"] > 0:
         first_pass_result = flatisfy.filters.first_pass(flats_list,
+                                                        constraint,
                                                         config)
     else:
         first_pass_result["new"] = flats_list
@@ -54,7 +72,7 @@ def filter_flats(config, flats_list, fetch_details=True):
     # additional infos
     if config["passes"] > 1:
         second_pass_result = flatisfy.filters.second_pass(
-            first_pass_result["new"], config
+            first_pass_result["new"], constraint, config
         )
     else:
         second_pass_result["new"] = first_pass_result["new"]
@@ -82,6 +100,28 @@ def filter_flats(config, flats_list, fetch_details=True):
     }
 
 
+def filter_fetched_flats(config, fetched_flats, fetch_details=True):
+    """
+    Filter the available flats list. Then, filter it according to criteria.
+
+    :param config: A config dict.
+    :param fetch_details: Whether additional details should be fetched between
+    the two passes.
+    :param fetched_flats: The initial dict mapping constraints to the list of
+    fetched flat objects to filter.
+    :return: A dict mapping constraints to a dict mapping flat status and list
+    of flat objects.
+    """
+    for constraint_name, flats_list in fetched_flats.items():
+        fetched_flats[constraint_name] = filter_flats_list(
+            config,
+            constraint_name,
+            flats_list,
+            fetch_details
+        )
+    return fetched_flats
+
+
 def import_and_filter(config, load_from_db=False):
     """
     Fetch the available flats list. Then, filter it according to criteria.
@@ -94,18 +134,24 @@ def import_and_filter(config, load_from_db=False):
     """
     # Fetch and filter flats list
     if load_from_db:
-        flats_list = fetch.load_flats_list_from_db(config)
+        fetched_flats = fetch.load_flats_from_db(config)
     else:
-        flats_list = fetch.fetch_flats_list(config)
+        fetched_flats = fetch.fetch_flats(config)
     # Do not fetch additional details if we loaded data from the db.
-    flats_list_by_status = filter_flats(config, flats_list=flats_list,
+    flats_by_status = filter_fetched_flats(config, fetched_flats=fetched_flats,
                                            fetch_details=(not load_from_db))
     # Create database connection
     get_session = database.init_db(config["database"], config["search_index"])
 
     LOGGER.info("Merging fetched flats in database...")
+    # Flatten the flats_by_status dict
+    flatten_flats_by_status = collections.defaultdict(list)
+    for flats in flats_by_status.values():
+        for status, flats_list in flats.items():
+            flatten_flats_by_status[status].extend(flats_list)
+
     with get_session() as session:
-        for status, flats_list in flats_list_by_status.items():
+        for status, flats_list in flatten_flats_by_status.items():
             # Build SQLAlchemy Flat model objects for every available flat
             flats_objects = {
                 flat_dict["id"]: flat_model.Flat.from_dict(flat_dict)
@@ -157,6 +203,10 @@ def purge_db(config):
             # Use (slower) deletion by object, to ensure whoosh index is
             # updated
             session.delete(flat)
+        LOGGER.info("Purge all postal codes from the database.")
+        session.query(postal_code_model.PostalCode).delete()
+        LOGGER.info("Purge all public transportations from the database.")
+        session.query(public_transport_model.PublicTransport).delete()
 
 
 def serve(config):

flatisfy/config.py

@@ -23,6 +23,7 @@ from flatisfy import tools
 DEFAULT_CONFIG = {
     # Constraints to match
     "constraints": {
+        "default": {
             "type": None,  # RENT, SALE, SHARING
             "house_types": [],  # List of house types, must be in APART, HOUSE,
                                 # PARKING, LAND, OTHER or UNKNOWN
@@ -34,6 +35,7 @@ DEFAULT_CONFIG = {
             "time_to": {}  # Dict mapping names to {"gps": [lat, lng],
                            #                        "time": (min, max) }
                            # Time is in seconds
+        }
     },
     # Navitia API key
     "navitia_api_key": None,
@@ -94,35 +96,37 @@ def validate_config(config):
         # and use long lines whenever needed, in order to have the full assert
         # message in the log output.
         # pylint: disable=locally-disabled,line-too-long
-        assert "type" in config["constraints"]
-        assert isinstance(config["constraints"]["type"], str)
-        assert config["constraints"]["type"].upper() in ["RENT",
-                                                         "SALE", "SHARING"]
 
-        assert "house_types" in config["constraints"]
+        # Ensure constraints are ok
-        assert config["constraints"]["house_types"]
+        assert len(config["constraints"]) > 0
-        for house_type in config["constraints"]["house_types"]:
+        for constraint in config["constraints"].values():
-            assert house_type.upper() in ["APART", "HOUSE", "PARKING", "LAND",
+            assert "type" in constraint
-                                          "OTHER", "UNKNOWN"]
+            assert isinstance(constraint["type"], str)
+            assert constraint["type"].upper() in ["RENT", "SALE", "SHARING"]
 
-        assert "postal_codes" in config["constraints"]
+            assert "house_types" in constraint
-        assert config["constraints"]["postal_codes"]
+            assert constraint["house_types"]
+            for house_type in constraint["house_types"]:
+                assert house_type.upper() in ["APART", "HOUSE", "PARKING", "LAND", "OTHER", "UNKNOWN"]  # noqa: E501
 
-        assert "area" in config["constraints"]
+            assert "postal_codes" in constraint
-        _check_constraints_bounds(config["constraints"]["area"])
+            assert constraint["postal_codes"]
 
-        assert "cost" in config["constraints"]
+            assert "area" in constraint
-        _check_constraints_bounds(config["constraints"]["cost"])
+            _check_constraints_bounds(constraint["area"])
 
-        assert "rooms" in config["constraints"]
+            assert "cost" in constraint
-        _check_constraints_bounds(config["constraints"]["rooms"])
+            _check_constraints_bounds(constraint["cost"])
 
-        assert "bedrooms" in config["constraints"]
+            assert "rooms" in constraint
-        _check_constraints_bounds(config["constraints"]["bedrooms"])
+            _check_constraints_bounds(constraint["rooms"])
 
-        assert "time_to" in config["constraints"]
+            assert "bedrooms" in constraint
-        assert isinstance(config["constraints"]["time_to"], dict)
+            _check_constraints_bounds(constraint["bedrooms"])
-        for name, item in config["constraints"]["time_to"].items():
+
+            assert "time_to" in constraint
+            assert isinstance(constraint["time_to"], dict)
+            for name, item in constraint["time_to"].items():
                 assert isinstance(name, str)
                 assert "gps" in item
                 assert isinstance(item["gps"], list)
@@ -134,6 +138,7 @@ def validate_config(config):
         assert config["max_entries"] is None or (isinstance(config["max_entries"], int) and config["max_entries"] > 0)  # noqa: E501
 
         assert config["data_directory"] is None or isinstance(config["data_directory"], str)  # noqa: E501
+        assert os.path.isdir(config["data_directory"])
         assert isinstance(config["search_index"], str)
         assert config["modules_path"] is None or isinstance(config["modules_path"], str)  # noqa: E501
 
@@ -206,6 +211,11 @@ def load_config(args=None):
         LOGGER.debug("Using default XDG data directory: %s.",
                      config_data["data_directory"])
 
+    if not os.path.isdir(config_data["data_directory"]):
+        LOGGER.info("Creating data directory according to config: %s",
+                    config_data["data_directory"])
+        os.mkdir(config_data["data_directory"])
+
     if config_data["database"] is None:
         config_data["database"] = "sqlite:///" + os.path.join(
             config_data["data_directory"],
@@ -218,6 +228,18 @@ def load_config(args=None):
             "search_index"
         )
 
+    # Handle constraints filtering
+    if args and getattr(args, "constraints", None) is not None:
+        LOGGER.info(
+            "Filtering constraints from config according to CLI argument."
+        )
+        constraints_filter = args.constraints.split(",")
+        config_data["constraints"] = {
+            k: v
+            for k, v in config_data["constraints"].items()
+            if k in constraints_filter
+        }
+
     config_validation = validate_config(config_data)
     if config_validation is True:
         LOGGER.info("Config has been fully initialized.")

flatisfy/data.py

@@ -5,17 +5,16 @@ the source opendata files.
 """
 from __future__ import absolute_import, print_function, unicode_literals
 
-import collections
-import json
 import logging
-import os
-
 
 import flatisfy.exceptions
 
+from flatisfy import database
+from flatisfy import data_files
+from flatisfy.models.postal_code import PostalCode
+from flatisfy.models.public_transport import PublicTransport
 
 LOGGER = logging.getLogger(__name__)
-MODULE_DIR = os.path.dirname(os.path.realpath(__file__))
 
 # Try to load lru_cache
 try:
@@ -24,7 +23,11 @@ except ImportError:
     try:
         from functools32 import lru_cache
     except ImportError:
-        lru_cache = lambda maxsize=None: lambda func: func
+        def lru_cache(maxsize=None):
+            """
+            Identity implementation of ``lru_cache`` for fallback.
+            """
+            return lambda func: func
         LOGGER.warning(
             "`functools.lru_cache` is not available on your system. Consider "
             "installing `functools32` Python module if using Python2 for "
@@ -32,156 +35,66 @@ except ImportError:
         )
 
 
-def _preprocess_ratp(output_dir):
-    """
-    Build RATP file from the RATP data.
-
-    :param output_dir: Directory in which the output file should reside.
-    :return: ``True`` on successful build, ``False`` otherwise.
-    """
-    ratp_data_raw = []
-    # Load opendata file
-    try:
-        with open(os.path.join(MODULE_DIR, "data_files/ratp.json"), "r") as fh:
-            ratp_data_raw = json.load(fh)
-    except (IOError, ValueError):
-        LOGGER.error("Invalid raw RATP opendata file.")
-        return False
-
-    # Process it
-    ratp_data = collections.defaultdict(list)
-    for item in ratp_data_raw:
-        stop_name = item["fields"]["stop_name"].lower()
-        ratp_data[stop_name].append({
-            "gps": item["fields"]["coord"],
-            "name": item["fields"]["stop_name"]
-        })
-
-    # Output it
-    with open(os.path.join(output_dir, "ratp.json"), "w") as fh:
-        json.dump(ratp_data, fh)
-
-    return True
-
-
-def _preprocess_laposte(output_dir):
-    """
-    Build JSON files from the postal codes data.
-
-    :param output_dir: Directory in which the output file should reside.
-    :return: ``True`` on successful build, ``False`` otherwise.
-    """
-    raw_laposte_data = []
-    # Load opendata file
-    try:
-        with open(
-            os.path.join(MODULE_DIR, "data_files/laposte.json"), "r"
-        ) as fh:
-            raw_laposte_data = json.load(fh)
-    except (IOError, ValueError):
-        LOGGER.error("Invalid raw LaPoste opendata file.")
-        return False
-
-    # Build postal codes to other infos file
-    postal_codes_data = {}
-    for item in raw_laposte_data:
-        try:
-            postal_codes_data[item["fields"]["code_postal"]] = {
-                "gps": item["fields"]["coordonnees_gps"],
-                "nom": item["fields"]["nom_de_la_commune"].title()
-            }
-        except KeyError:
-            LOGGER.info("Missing data for postal code %s, skipping it.",
-                        item["fields"]["code_postal"])
-    with open(os.path.join(output_dir, "postal_codes.json"), "w") as fh:
-        json.dump(postal_codes_data, fh)
-
-    # Build city name to postal codes and other infos file
-    cities_data = {}
-    for item in raw_laposte_data:
-        try:
-            cities_data[item["fields"]["nom_de_la_commune"].title()] = {
-                "gps": item["fields"]["coordonnees_gps"],
-                "postal_code": item["fields"]["code_postal"]
-            }
-        except KeyError:
-            LOGGER.info("Missing data for city %s, skipping it.",
-                        item["fields"]["nom_de_la_commune"])
-    with open(os.path.join(output_dir, "cities.json"), "w") as fh:
-        json.dump(cities_data, fh)
-
-    return True
-
-
-DATA_FILES = {
-    "ratp.json": {
-        "preprocess": _preprocess_ratp,
-        "output": ["ratp.json"]
-    },
-    "laposte.json": {
-        "preprocess": _preprocess_laposte,
-        "output": ["cities.json", "postal_codes.json"]
-    },
-}
-
-
 def preprocess_data(config, force=False):
     """
-    Ensures that all the necessary data files have been built from the raw
+    Ensures that all the necessary data have been inserted in db from the raw
     opendata files.
 
     :params config: A config dictionary.
     :params force: Whether to force rebuild or not.
     """
-    LOGGER.debug("Data directory is %s.", config["data_directory"])
+    # Check if a build is required
-    opendata_directory = os.path.join(config["data_directory"], "opendata")
+    get_session = database.init_db(config["database"], config["search_index"])
-    try:
+    with get_session() as session:
-        LOGGER.info("Ensuring the data directory exists.")
+        is_built = (
-        os.makedirs(opendata_directory)
+            session.query(PublicTransport).count() > 0 and
-        LOGGER.debug("Created opendata directory at %s.", opendata_directory)
+            session.query(PostalCode).count > 0
-    except OSError:
+        )
-        LOGGER.debug("Opendata directory already existed, doing nothing.")
+        if is_built and not force:
+            # No need to rebuild the database, skip
+            return
+        # Otherwise, purge all existing data
+        session.query(PublicTransport).delete()
+        session.query(PostalCode).delete()
 
-    # Build all the necessary data files
+    # Build all opendata files
-    for data_file in DATA_FILES:
+    for preprocess in data_files.PREPROCESSING_FUNCTIONS:
-        # Check if already built
+        data_objects = preprocess()
-        is_built = all(
+        if not data_objects:
-            os.path.isfile(
-                os.path.join(opendata_directory, output)
-            ) for output in DATA_FILES[data_file]["output"]
-        )
-        if not is_built or force:
-            # Build if needed
-            LOGGER.info("Building from {} data.".format(data_file))
-            if not DATA_FILES[data_file]["preprocess"](opendata_directory):
             raise flatisfy.exceptions.DataBuildError(
-                    "Error with {} data.".format(data_file)
+                "Error with %s." % preprocess.__name__
             )
+        with get_session() as session:
+            session.add_all(data_objects)
 
 
 @lru_cache(maxsize=5)
-def load_data(data_type, config):
+def load_data(model, constraint, config):
     """
-    Load a given built data file. This function is memoized.
+    Load data of the specified model from the database. Only load data for the
+    specific areas of the postal codes in config.
 
-    :param data_type: A valid data identifier.
+    :param model: SQLAlchemy model to load.
+    :param constraint: A constraint from configuration to limit the spatial
+    extension of the loaded data.
     :param config: A config dictionary.
-    :return: The loaded data. ``None`` if the query is incorrect.
+    :returns: A list of loaded SQLAlchemy objects from the db
     """
-    opendata_directory = os.path.join(config["data_directory"], "opendata")
+    get_session = database.init_db(config["database"], config["search_index"])
-    datafile_path = os.path.join(opendata_directory, "%s.json" % data_type)
+    results = []
-    data = {}
+    with get_session() as session:
-    try:
+        areas = []
-        with open(datafile_path, "r") as fh:
+        # Get areas to fetch from, using postal codes
-            data = json.load(fh)
+        for postal_code in constraint["postal_codes"]:
-    except IOError:
+            areas.append(data_files.french_postal_codes_to_iso_3166(postal_code))
-        LOGGER.error("No such data file: %s.", datafile_path)
+        # Load data for each area
-        return None
+        areas = list(set(areas))
-    except ValueError:
+        for area in areas:
-        LOGGER.error("Invalid JSON data file: %s.", datafile_path)
+            results.extend(
-        return None
+                session.query(model)
-
+                .filter(model.area == area).all()
-    if not data:
+            )
-        LOGGER.warning("Loading empty data for %s.", data_type)
+        # Expunge loaded data from the session to be able to use them
-
+        # afterwards
-    return data
+        session.expunge_all()
+    return results

flatisfy/data_files/__init__.py

@@ -0,0 +1,175 @@
+# coding : utf-8
+"""
+Preprocessing functions to convert input opendata files into SQLAlchemy objects
+ready to be stored in the database.
+"""
+import json
+import logging
+import os
+
+from flatisfy.models.postal_code import PostalCode
+from flatisfy.models.public_transport import PublicTransport
+
+
+LOGGER = logging.getLogger(__name__)
+MODULE_DIR = os.path.dirname(os.path.realpath(__file__))
+
+
+def french_postal_codes_to_iso_3166(postal_code):
+    """
+    Convert a French postal code to the main subdivision in French this postal
+    code belongs to (ISO 3166-2 code).
+
+    :param postal_code: The postal code to convert.
+    :returns: The ISO 3166-2 code of the subdivision or ``None``.
+    """
+    # Mapping between areas (main subdivisions in French, ISO 3166-2) and
+    # French departements
+    # Taken from Wikipedia data.
+    AREA_TO_DEPARTEMENT = {
+        "FR-ARA": ["01", "03", "07", "15", "26", "38", "42", "43", "63", "69",
+                   "73", "74"],
+        "FR-BFC": ["21", "25", "39", "58", "70", "71", "89", "90"],
+        "FR-BRE": ["22", "29", "35", "44", "56"],
+        "FR-CVL": ["18", "28", "36", "37", "41", "45"],
+        "FR-COR": ["20"],
+        "FR-GES": ["08", "10", "51", "52", "54", "55", "57", "67", "68", "88"],
+        "FR-HDF": ["02", "59", "60", "62", "80"],
+        "FR-IDF": ["75", "77", "78", "91", "92", "93", "94", "95"],
+        "FR-NOR": ["14", "27", "50", "61", "76"],
+        "FR-NAQ": ["16", "17", "19", "23", "24", "33", "40", "47", "64", "79",
+                   "86", "87"],
+        "FR-OCC": ["09", "11", "12", "30", "31", "32", "34", "46", "48", "65",
+                   "66", "81", "82"],
+        "FR-PDL": ["44", "49", "53", "72", "85"],
+        "FR-PAC": ["04", "05", "06", "13", "83", "84"]
+    }
+
+    departement = postal_code[:2]
+    return next(
+        (
+            i
+            for i in AREA_TO_DEPARTEMENT
+            if departement in AREA_TO_DEPARTEMENT[i]
+        ),
+        None
+    )
+
+
+
+def _preprocess_laposte():
+    """
+    Build SQLAlchemy objects from the postal codes data.
+
+    :return: A list of ``PostalCode`` objects to be inserted in database.
+    """
+    data_file = "laposte.json"
+    LOGGER.info("Building from %s data.", data_file)
+
+    raw_laposte_data = []
+    # Load opendata file
+    try:
+        with open(
+            os.path.join(MODULE_DIR, data_file), "r"
+        ) as fh:
+            raw_laposte_data = json.load(fh)
+    except (IOError, ValueError):
+        LOGGER.error("Invalid raw LaPoste opendata file.")
+        return []
+
+    # Build postal codes to other infos file
+    postal_codes_data = []
+    for item in raw_laposte_data:
+        fields = item["fields"]
+        try:
+            area = french_postal_codes_to_iso_3166(fields["code_postal"])
+            if area is None:
+                LOGGER.info(
+                    "No matching area found for postal code %s, skipping it.",
+                    fields["code_postal"]
+                )
+                continue
+
+            postal_codes_data.append(PostalCode(
+                area=area,
+                postal_code=fields["code_postal"],
+                name=fields["nom_de_la_commune"].title(),
+                lat=fields["coordonnees_gps"][0],
+                lng=fields["coordonnees_gps"][1]
+            ))
+        except KeyError:
+            LOGGER.info("Missing data for postal code %s, skipping it.",
+                        fields["code_postal"])
+
+    return postal_codes_data
+
+
+def _preprocess_ratp():
+    """
+    Build SQLAlchemy objects from the RATP data (public transport in Paris,
+    France).
+
+    :return: A list of ``PublicTransport`` objects to be inserted in database.
+    """
+    data_file = "ratp.json"
+    LOGGER.info("Building from %s data.", data_file)
+
+    ratp_data_raw = []
+    # Load opendata file
+    try:
+        with open(os.path.join(MODULE_DIR, data_file), "r") as fh:
+            ratp_data_raw = json.load(fh)
+    except (IOError, ValueError):
+        LOGGER.error("Invalid raw RATP opendata file.")
+        return []
+
+    # Process it
+    ratp_data = []
+    for item in ratp_data_raw:
+        fields = item["fields"]
+        ratp_data.append(PublicTransport(
+            name=fields["stop_name"],
+            area="FR-IDF",
+            lat=fields["coord"][0],
+            lng=fields["coord"][1]
+        ))
+    return ratp_data
+
+
+def _preprocess_tcl():
+    """
+    Build SQLAlchemy objects from the Tcl data (public transport in Lyon,
+    France).
+
+    :return: A list of ``PublicTransport`` objects to be inserted in database.
+    """
+    data_file = "tcl.json"
+    LOGGER.info("Building from %s data.", data_file)
+
+    tcl_data_raw = []
+    # Load opendata file
+    try:
+        with open(os.path.join(MODULE_DIR, data_file), "r") as fh:
+            tcl_data_raw = json.load(fh)
+    except (IOError, ValueError):
+        LOGGER.error("Invalid raw Tcl opendata file.")
+        return []
+
+    # Process it
+    tcl_data = []
+    for item in tcl_data_raw["features"]:
+        tcl_data.append(PublicTransport(
+            name=item["properties"]["nom"],
+            area="FR-ARA",
+            lat=item["geometry"]["coordinates"][1],
+            lng=item["geometry"]["coordinates"][0]
+        ))
+    return tcl_data
+
+
+# List of all the available preprocessing functions. Order can be important.
+PREPROCESSING_FUNCTIONS = [
+    _preprocess_laposte,
+    _preprocess_ratp,
+    _preprocess_tcl
+]

flatisfy/database/__init__.py

@@ -11,7 +11,7 @@ from contextlib import contextmanager
 from sqlalchemy import event, create_engine
 from sqlalchemy.engine import Engine
 from sqlalchemy.orm import sessionmaker
-from sqlalchemy.exc import SQLAlchemyError
+from sqlalchemy.exc import OperationalError, SQLAlchemyError
 
 import flatisfy.models.flat  # noqa: F401
 from flatisfy.database.base import BASE

flatisfy/fetch.py

@@ -4,6 +4,7 @@ This module contains all the code related to fetching and loading flats lists.
 """
 from __future__ import absolute_import, print_function, unicode_literals
 
+import collections
 import itertools
 import json
 import logging
@@ -225,18 +226,20 @@ class WeboobProxy(object):
             return "{}"
 
 
-def fetch_flats_list(config):
+def fetch_flats(config):
     """
     Fetch the available flats using the Flatboob / Weboob config.
 
     :param config: A config dict.
-    :return: A list of all available flats.
+    :return: A dict mapping constraint in config to all available matching
+    flats.
     """
-    flats_list = []
+    fetched_flats = {}
 
+    for constraint_name, constraint in config["constraints"].items():
+        LOGGER.info("Loading flats for constraint %s...", constraint_name)
         with WeboobProxy(config) as weboob_proxy:
-        LOGGER.info("Loading flats...")
+            queries = weboob_proxy.build_queries(constraint)
-        queries = weboob_proxy.build_queries(config["constraints"])
             housing_posts = []
             for query in queries:
                 housing_posts.extend(
@@ -244,10 +247,11 @@ def fetch_flats_list(config):
                 )
         LOGGER.info("Fetched %d flats.", len(housing_posts))
 
-    flats_list = [json.loads(flat) for flat in housing_posts]
+        constraint_flats_list = [json.loads(flat) for flat in housing_posts]
-    flats_list = [WeboobProxy.restore_decimal_fields(flat)
+        constraint_flats_list = [WeboobProxy.restore_decimal_fields(flat)
-                  for flat in flats_list]
+                                 for flat in constraint_flats_list]
-    return flats_list
+        fetched_flats[constraint_name] = constraint_flats_list
+    return fetched_flats
 
 
 def fetch_details(config, flat_id):
@@ -269,12 +273,18 @@ def fetch_details(config, flat_id):
     return flat_details
 
 
-def load_flats_list_from_file(json_file):
+def load_flats_from_file(json_file, config):
     """
     Load a dumped flats list from JSON file.
 
     :param json_file: The file to load housings list from.
-    :return: A list of all the flats in the dump file.
+    :return: A dict mapping constraint in config to all available matching
+    flats.
+
+    .. note::
+        As we do not know which constraint is met by a given flat, all the
+        flats are returned for any available constraint, and they will be
+        filtered out afterwards.
     """
     flats_list = []
     try:
@@ -284,21 +294,24 @@ def load_flats_list_from_file(json_file):
         LOGGER.info("Found %d flats.", len(flats_list))
     except (IOError, ValueError):
         LOGGER.error("File %s is not a valid dump file.", json_file)
-    return flats_list
+    return {
+        constraint_name: flats_list
+        for constraint_name in config["constraints"]
+    }
 
 
-def load_flats_list_from_db(config):
+def load_flats_from_db(config):
     """
     Load flats from database.
 
     :param config: A config dict.
-    :return: A list of all the flats in the database.
+    :return: A dict mapping constraint in config to all available matching
+    flats.
     """
-    flats_list = []
     get_session = database.init_db(config["database"], config["search_index"])
 
+    loaded_flats = collections.defaultdict(list)
     with get_session() as session:
-        # TODO: Better serialization
+        for flat in session.query(flat_model.Flat).all():
-        flats_list = [flat.json_api_repr()
+            loaded_flats[flat.flatisfy_constraint].append(flat.json_api_repr())
-                      for flat in session.query(flat_model.Flat).all()]
+    return loaded_flats
-    return flats_list

flatisfy/filters/__init__.py

@@ -16,7 +16,7 @@ from flatisfy.filters import metadata
 LOGGER = logging.getLogger(__name__)
 
 
-def refine_with_housing_criteria(flats_list, config):
+def refine_with_housing_criteria(flats_list, constraint, config):
     """
     Filter a list of flats according to criteria.
 
@@ -25,6 +25,7 @@ def refine_with_housing_criteria(flats_list, config):
     user criteria, and avoid exposing unwanted flats.
 
     :param flats_list: A list of flats dict to filter.
+    :param constraint: The constraint that the ``flats_list`` should satisfy.
     :param config: A config dict.
     :return: A tuple of flats to keep and flats to delete.
     """
@@ -37,7 +38,7 @@ def refine_with_housing_criteria(flats_list, config):
         postal_code = flat["flatisfy"].get("postal_code", None)
         if (
                 postal_code and
-                postal_code not in config["constraints"]["postal_codes"]
+                postal_code not in constraint["postal_codes"]
         ):
             LOGGER.info("Postal code for flat %s is out of range.", flat["id"])
             is_ok[i] = is_ok[i] and False
@@ -47,7 +48,7 @@ def refine_with_housing_criteria(flats_list, config):
             time = time["time"]
             is_within_interval = tools.is_within_interval(
                 time,
-                *(config["constraints"]["time_to"][place_name]["time"])
+                *(constraint["time_to"][place_name]["time"])
             )
             if not is_within_interval:
                 LOGGER.info("Flat %s is too far from place %s: %ds.",
@@ -56,7 +57,7 @@ def refine_with_housing_criteria(flats_list, config):
 
         # Check other fields
         for field in ["area", "cost", "rooms", "bedrooms"]:
-            interval = config["constraints"][field]
+            interval = constraint[field]
             is_within_interval = tools.is_within_interval(
                 flat.get(field, None),
                 *interval
@@ -80,7 +81,7 @@ def refine_with_housing_criteria(flats_list, config):
     )
 
 
-def first_pass(flats_list, config):
+def first_pass(flats_list, constraint, config):
     """
     First filtering pass.
 
@@ -89,6 +90,7 @@ def first_pass(flats_list, config):
     only request more data for the remaining housings.
 
     :param flats_list: A list of flats dict to filter.
+    :param constraint: The constraint that the ``flats_list`` should satisfy.
     :param config: A config dict.
     :return: A dict mapping flat status and list of flat objects.
     """
@@ -108,11 +110,12 @@ def first_pass(flats_list, config):
     )
 
     # Guess the postal codes
-    flats_list = metadata.guess_postal_code(flats_list, config)
+    flats_list = metadata.guess_postal_code(flats_list, constraint, config)
     # Try to match with stations
-    flats_list = metadata.guess_stations(flats_list, config)
+    flats_list = metadata.guess_stations(flats_list, constraint, config)
     # Remove returned housing posts that do not match criteria
-    flats_list, ignored_list = refine_with_housing_criteria(flats_list, config)
+    flats_list, ignored_list = refine_with_housing_criteria(flats_list,
+                                                            constraint, config)
 
     return {
         "new": flats_list,
@@ -121,7 +124,7 @@ def first_pass(flats_list, config):
     }
 
 
-def second_pass(flats_list, config):
+def second_pass(flats_list, constraint, config):
     """
     Second filtering pass.
 
@@ -133,6 +136,7 @@ def second_pass(flats_list, config):
     possible from the fetched housings.
 
     :param flats_list: A list of flats dict to filter.
+    :param constraint: The constraint that the ``flats_list`` should satisfy.
     :param config: A config dict.
     :return: A dict mapping flat status and list of flat objects.
     """
@@ -141,16 +145,17 @@ def second_pass(flats_list, config):
     # left and we already tried to find postal code and nearby stations.
 
     # Confirm postal code
-    flats_list = metadata.guess_postal_code(flats_list, config)
+    flats_list = metadata.guess_postal_code(flats_list, constraint, config)
 
     # Better match with stations (confirm and check better)
-    flats_list = metadata.guess_stations(flats_list, config)
+    flats_list = metadata.guess_stations(flats_list, constraint, config)
 
     # Compute travel time to specified points
-    flats_list = metadata.compute_travel_times(flats_list, config)
+    flats_list = metadata.compute_travel_times(flats_list, constraint, config)
 
     # Remove returned housing posts that do not match criteria
-    flats_list, ignored_list = refine_with_housing_criteria(flats_list, config)
+    flats_list, ignored_list = refine_with_housing_criteria(flats_list,
+                                                            constraint, config)
 
     return {
         "new": flats_list,

flatisfy/filters/metadata.py

@@ -12,24 +12,29 @@ import re
 
 from flatisfy import data
 from flatisfy import tools
+from flatisfy.models.postal_code import PostalCode
+from flatisfy.models.public_transport import PublicTransport
 
 
 LOGGER = logging.getLogger(__name__)
 
 
-def init(flats_list):
+def init(flats_list, constraint):
     """
     Create a flatisfy key containing a dict of metadata fetched by flatisfy for
     each flat in the list. Also perform some basic transform on flat objects to
     prepare for the metadata fetching.
 
     :param flats_list: A list of flats dict.
+    :param constraint: The constraint that the ``flats_list`` should satisfy.
     :return: The updated list
     """
     for flat in flats_list:
         # Init flatisfy key
         if "flatisfy" not in flat:
             flat["flatisfy"] = {}
+        if "constraint" not in flat["flatisfy"]:
+            flat["flatisfy"]["constraint"] = constraint
         # Move url key to urls
         if "urls" not in flat:
             if "url" in flat:
@@ -117,11 +122,12 @@ def fuzzy_match(query, choices, limit=3, threshold=75):
     return matches
 
 
-def guess_postal_code(flats_list, config, distance_threshold=20000):
+def guess_postal_code(flats_list, constraint, config, distance_threshold=20000):
     """
     Try to guess the postal code from the location of the flats.
 
     :param flats_list: A list of flats dict.
+    :param constraint: The constraint that the ``flats_list`` should satisfy.
     :param config: A config dict.
     :param distance_threshold: Maximum distance in meters between the
     constraint postal codes (from config) and the one found by this function,
@@ -130,8 +136,7 @@ def guess_postal_code(flats_list, config, distance_threshold=20000):
     :return: An updated list of flats dict with guessed postal code.
     """
     opendata = {
-        "cities": data.load_data("cities", config),
+        "postal_codes": data.load_data(PostalCode, constraint, config)
-        "postal_codes": data.load_data("postal_codes", config)
     }
 
     for flat in flats_list:
@@ -155,7 +160,8 @@ def guess_postal_code(flats_list, config, distance_threshold=20000):
             postal_code = postal_code.group(0)
 
             # Check the postal code is within the db
-            assert postal_code in opendata["postal_codes"]
+            assert postal_code in [x.postal_code
+                                   for x in opendata["postal_codes"]]
 
             LOGGER.info(
                 "Found postal code in location field for flat %s: %s.",
@@ -165,10 +171,11 @@ def guess_postal_code(flats_list, config, distance_threshold=20000):
             postal_code = None
 
         # If not found, try to find a city
+        cities = {x.name: x for x in opendata["postal_codes"]}
         if not postal_code:
             matched_city = fuzzy_match(
                 location,
-                opendata["cities"].keys(),
+                cities.keys(),
                 limit=1
             )
             if matched_city:
@@ -176,7 +183,7 @@ def guess_postal_code(flats_list, config, distance_threshold=20000):
                 matched_city = matched_city[0]
                 matched_city_name = matched_city[0]
                 postal_code = (
-                    opendata["cities"][matched_city_name]["postal_code"]
+                    cities[matched_city_name].postal_code
                 )
                 LOGGER.info(
                     ("Found postal code in location field through city lookup "
@@ -189,10 +196,18 @@ def guess_postal_code(flats_list, config, distance_threshold=20000):
         if postal_code and distance_threshold:
             distance = min(
                 tools.distance(
-                    opendata["postal_codes"][postal_code]["gps"],
+                    next(
-                    opendata["postal_codes"][constraint]["gps"],
+                        (x.lat, x.lng)
+                        for x in opendata["postal_codes"]
+                        if x.postal_code == postal_code
+                    ),
+                    next(
+                        (x.lat, x.lng)
+                        for x in opendata["postal_codes"]
+                        if x.postal_code == constraint_postal_code
                     )
-                for constraint in config["constraints"]["postal_codes"]
+                )
+                for constraint_postal_code in constraint["postal_codes"]
             )
 
             if distance > distance_threshold:
@@ -218,11 +233,12 @@ def guess_postal_code(flats_list, config, distance_threshold=20000):
     return flats_list
 
 
-def guess_stations(flats_list, config, distance_threshold=1500):
+def guess_stations(flats_list, constraint, config, distance_threshold=1500):
     """
     Try to match the station field with a list of available stations nearby.
 
     :param flats_list: A list of flats dict.
+    :param constraint: The constraint that the ``flats_list`` should satisfy.
     :param config: A config dict.
     :param distance_threshold: Maximum distance (in meters) between the center
     of the postal code and the station to consider it ok.
@@ -230,8 +246,8 @@ def guess_stations(flats_list, config, distance_threshold=1500):
     :return: An updated list of flats dict with guessed nearby stations.
     """
     opendata = {
-        "postal_codes": data.load_data("postal_codes", config),
+        "postal_codes": data.load_data(PostalCode, constraint, config),
-        "stations": data.load_data("ratp", config)
+        "stations": data.load_data(PublicTransport, constraint, config)
     }
 
     for flat in flats_list:
@@ -247,7 +263,7 @@ def guess_stations(flats_list, config, distance_threshold=1500):
 
         matched_stations = fuzzy_match(
             flat_station,
-            opendata["stations"].keys(),
+            [x.name for x in opendata["stations"]],
             limit=10,
             threshold=50
         )
@@ -259,53 +275,62 @@ def guess_stations(flats_list, config, distance_threshold=1500):
         if postal_code:
             # If there is a postal code, check that the matched station is
             # closed to it
-            postal_code_gps = opendata["postal_codes"][postal_code]["gps"]
+            postal_code_gps = next(
+                (x.lat, x.lng)
+                for x in opendata["postal_codes"]
+                if x.postal_code == postal_code
+            )
             for station in matched_stations:
-                # opendata["stations"] is a dict mapping station names to list
+                # Note that multiple stations with the same name exist in a
-                # of coordinates, for efficiency. Note that multiple stations
+                # city, hence the list of stations objects for a given matching
-                # with the same name exist in a city, hence the list of
+                # station name.
-                # coordinates.
+                stations_objects = [
-                for station_data in opendata["stations"][station[0]]:
+                    x for x in opendata["stations"] if x.name == station[0]
-                    distance = tools.distance(station_data["gps"],
+                ]
-                                              postal_code_gps)
+                for station_data in stations_objects:
+                    distance = tools.distance(
+                        (station_data.lat, station_data.lng),
+                        postal_code_gps
+                    )
                     if distance < distance_threshold:
                         # If at least one of the coordinates for a given
                         # station is close enough, that's ok and we can add
                         # the station
                         good_matched_stations.append({
                             "key": station[0],
-                            "name": station_data["name"],
+                            "name": station_data.name,
                             "confidence": station[1],
-                            "gps": station_data["gps"]
+                            "gps": (station_data.lat, station_data.lng)
                         })
                         break
-                    LOGGER.debug(
+                    LOGGER.info(
-                        "Station %s is too far from flat %s, discarding it.",
+                        ("Station %s is too far from flat %s (%dm > %dm), "
-                        station[0], flat["id"]
+                         "discarding it."),
+                        station[0],
+                        flat["id"],
+                        int(distance),
+                        int(distance_threshold)
                     )
         else:
             LOGGER.info(
-                ("No postal code for flat %s, keeping all the matched "
+                "No postal code for flat %s, skipping stations detection.",
-                 "stations with half confidence."),
                 flat["id"]
             )
-            # Otherwise, we keep every matching station but with half
-            # confidence
-            good_matched_stations = [
-                {
-                    "name": station[0],
-                    "confidence": station[1] * 0.5,
-                    "gps": station_gps
-                }
-                for station in matched_stations
-                for station_gps in opendata["stations"][station[0]]
-            ]
 
-        # Store matched stations and the associated confidence
+        if not good_matched_stations:
+            # No stations found, log it and cotninue with next housing
             LOGGER.info(
-            "Found stations for flat %s: %s.",
+                "No stations found for flat %s, matching %s.",
                 flat["id"],
-            ", ".join(x["name"] for x in good_matched_stations)
+                flat["station"]
+            )
+            continue
+
+        LOGGER.info(
+            "Found stations for flat %s: %s (matching %s).",
+            flat["id"],
+            ", ".join(x["name"] for x in good_matched_stations),
+            flat["station"]
         )
 
         # If some stations were already filled in and the result is different,
@@ -335,12 +360,13 @@ def guess_stations(flats_list, config, distance_threshold=1500):
     return flats_list
 
 
-def compute_travel_times(flats_list, config):
+def compute_travel_times(flats_list, constraint, config):
     """
     Compute the travel time between each flat and the points listed in the
     constraints.
 
     :param flats_list: A list of flats dict.
+    :param constraint: The constraint that the ``flats_list`` should satisfy.
     :param config: A config dict.
 
     :return: An updated list of flats dict with computed travel times.
@@ -363,7 +389,7 @@ def compute_travel_times(flats_list, config):
 
         # For each place, loop over the stations close to the flat, and find
         # the minimum travel time.
-        for place_name, place in config["constraints"]["time_to"].items():
+        for place_name, place in constraint["time_to"].items():
             time_to_place = None
             for station in flat["flatisfy"]["matched_stations"]:
                 time_from_station = tools.get_travel_time_between(

flatisfy/models/flat.py

@@ -86,6 +86,7 @@ class Flat(BASE):
     flatisfy_stations = Column(MagicJSON)
     flatisfy_postal_code = Column(String)
     flatisfy_time_to = Column(MagicJSON)
+    flatisfy_constraint = Column(String)
 
     # Status
     status = Column(Enum(FlatStatus), default=FlatStatus.new)
@@ -108,6 +109,9 @@ class Flat(BASE):
             flat_dict["flatisfy_time_to"] = (
                 flat_dict["flatisfy"].get("time_to", {})
             )
+            flat_dict["flatisfy_constraint"] = (
+                flat_dict["flatisfy"].get("constraint", "default")
+            )
             del flat_dict["flatisfy"]
 
         # Handle utilities field

flatisfy/models/postal_code.py

@@ -0,0 +1,37 @@
+# coding: utf-8
+"""
+This modules defines an SQLAlchemy ORM model for a postal code opendata.
+"""
+# pylint: disable=locally-disabled,invalid-name,too-few-public-methods
+from __future__ import absolute_import, print_function, unicode_literals
+
+import logging
+
+from sqlalchemy import (
+    Column, Float, Integer, String, UniqueConstraint
+)
+
+from flatisfy.database.base import BASE
+
+
+LOGGER = logging.getLogger(__name__)
+
+
+class PostalCode(BASE):
+    """
+    SQLAlchemy ORM model to store a postal code opendata.
+    """
+    __tablename__ = "postal_codes"
+
+    id = Column(Integer, primary_key=True)
+    # Area is an identifier to prevent loading unnecessary stops. For now it is
+    # following ISO 3166-2.
+    area = Column(String, index=True)
+    postal_code = Column(String, index=True)
+    name = Column(String, index=True)
+    lat = Column(Float)
+    lng = Column(Float)
+    UniqueConstraint("postal_code", "name")
+
+    def __repr__(self):
+        return "<PostalCode(id=%s)>" % self.id

flatisfy/models/public_transport.py

@@ -0,0 +1,35 @@
+# coding: utf-8
+"""
+This modules defines an SQLAlchemy ORM model for public transport opendata.
+"""
+# pylint: disable=locally-disabled,invalid-name,too-few-public-methods
+from __future__ import absolute_import, print_function, unicode_literals
+
+import logging
+
+from sqlalchemy import (
+    Column, Float, Integer, String
+)
+
+from flatisfy.database.base import BASE
+
+
+LOGGER = logging.getLogger(__name__)
+
+
+class PublicTransport(BASE):
+    """
+    SQLAlchemy ORM model to store public transport opendata.
+    """
+    __tablename__ = "public_transports"
+
+    id = Column(Integer, primary_key=True)
+    # Area is an identifier to prevent loading unnecessary stops. For now it is
+    # following ISO 3166-2.
+    area = Column(String, index=True)
+    name = Column(String)
+    lat = Column(Float)
+    lng = Column(Float)
+
+    def __repr__(self):
+        return "<PublicTransport(id=%s)>" % self.id

flatisfy/tools.py

@@ -298,5 +298,4 @@ def get_travel_time_between(latlng_from, latlng_to, config):
             "time": time,
             "sections": sections
         }
-    else:
     return None

flatisfy/web/app.py

@@ -69,7 +69,8 @@ def get_app(config):
     # API v1 routes
     app.route("/api/v1/", "GET", api_routes.index_v1)
 
-    app.route("/api/v1/time_to/places", "GET", api_routes.time_to_places_v1)
+    app.route("/api/v1/time_to_places", "GET",
+              api_routes.time_to_places_v1)
 
     app.route("/api/v1/flats", "GET", api_routes.flats_v1)
     app.route("/api/v1/flats/status/:status", "GET",

flatisfy/web/js_src/api/index.js

@@ -107,7 +107,7 @@ export const updateFlatNotation = function (flatId, newNotation, callback) {
 }
 
 export const getTimeToPlaces = function (callback) {
-    fetch('/api/v1/time_to/places', { credentials: 'same-origin' })
+    fetch('/api/v1/time_to_places', { credentials: 'same-origin' })
     .then(function (response) {
         return response.json()
     }).then(function (json) {

flatisfy/web/js_src/store/getters.js

@@ -54,5 +54,18 @@ export default {
         return markers
     },
 
-    allTimeToPlaces: state => state.timeToPlaces
+    allTimeToPlaces: state => {
+        let places = {}
+        Object.keys(state.timeToPlaces).forEach(constraint => {
+            let constraintTimeToPlaces = state.timeToPlaces[constraint]
+            Object.keys(constraintTimeToPlaces).forEach(name =>
+                places[name] = constraintTimeToPlaces[name]
+            )
+        })
+        return places
+    },
+
+    timeToPlaces: (state, getters) => (constraint_name) => {
+        return state.timeToPlaces[constraint_name]
+    },
 }

flatisfy/web/js_src/views/details.vue

@@ -222,12 +222,12 @@ export default {
         flatMarkers () {
             return this.$store.getters.flatsMarkers(this.$router, flat => flat.id === this.$route.params.id)
         },
-        timeToPlaces () {
-            return this.$store.getters.allTimeToPlaces
-        },
         flat () {
             return this.$store.getters.flat(this.$route.params.id)
         },
+        timeToPlaces () {
+            return this.$store.getters.timeToPlaces(this.flat.flatisfy_constraint)
+        },
         notation () {
             if (this.overloadNotation) {
                 return this.overloadNotation

flatisfy/web/routes/api.py

@@ -12,6 +12,7 @@ import bottle
 
 import flatisfy.data
 from flatisfy.models import flat as flat_model
+from flatisfy.models.postal_code import PostalCode
 
 # TODO: Flat post-processing code should be factorized
 
@@ -38,7 +39,11 @@ def flats_v1(config, db):
 
     :return: The available flats objects in a JSON ``data`` dict.
     """
-    postal_codes = flatisfy.data.load_data("postal_codes", config)
+    postal_codes = {}
+    for constraint_name, constraint in config["constraints"].items():
+        postal_codes[constraint_name] = flatisfy.data.load_data(
+            PostalCode, constraint, config
+        )
 
     flats = [
         flat.json_api_repr()
@@ -46,14 +51,20 @@ def flats_v1(config, db):
     ]
 
     for flat in flats:
-        if flat["flatisfy_postal_code"]:
+        try:
-            postal_code_data = postal_codes[flat["flatisfy_postal_code"]]
+            assert flat["flatisfy_postal_code"]
+
+            postal_code_data = next(
+                x
+                for x in postal_codes.get(flat["flatisfy_constraint"], [])
+                if x.postal_code == flat["flatisfy_postal_code"]
+            )
             flat["flatisfy_postal_code"] = {
                 "postal_code": flat["flatisfy_postal_code"],
-                "name": postal_code_data["nom"],
+                "name": postal_code_data.name,
-                "gps": postal_code_data["gps"]
+                "gps": (postal_code_data.lat, postal_code_data.lng)
             }
-        else:
+        except (AssertionError, StopIteration):
             flat["flatisfy_postal_code"] = {}
 
     return {
@@ -94,8 +105,6 @@ def flat_v1(flat_id, config, db):
 
     :return: The flat object in a JSON ``data`` dict.
     """
-    postal_codes = flatisfy.data.load_data("postal_codes", config)
-
     flat = db.query(flat_model.Flat).filter_by(id=flat_id).first()
 
     if not flat:
@@ -103,14 +112,25 @@ def flat_v1(flat_id, config, db):
 
     flat = flat.json_api_repr()
 
-    if flat["flatisfy_postal_code"]:
+    try:
-        postal_code_data = postal_codes[flat["flatisfy_postal_code"]]
+        assert flat["flatisfy_postal_code"]
+
+        constraint = config["constraints"].get(flat["flatisfy_constraint"],
+                                               None)
+        assert constraint is not None
+        postal_codes = flatisfy.data.load_data(PostalCode, constraint, config)
+
+        postal_code_data = next(
+            x
+            for x in postal_codes
+            if x.postal_code == flat["flatisfy_postal_code"]
+        )
         flat["flatisfy_postal_code"] = {
             "postal_code": flat["flatisfy_postal_code"],
-            "name": postal_code_data["nom"],
+            "name": postal_code_data.name,
-            "gps": postal_code_data["gps"]
+            "gps": (postal_code_data.lat, postal_code_data.lng)
         }
-    else:
+    except (AssertionError, StopIteration):
         flat["flatisfy_postal_code"] = {}
 
     return {
@@ -206,14 +226,16 @@ def time_to_places_v1(config):
     """
     API v1 route to fetch the details of the places to compute time to.
 
-        GET /api/v1/time_to/places
+        GET /api/v1/time_to_places
 
     :return: The JSON dump of the places to compute time to (dict of places
     names mapped to GPS coordinates).
     """
-    places = {
+    places = {}
+    for constraint_name, constraint in config["constraints"].items():
+        places[constraint_name] = {
             k: v["gps"]
-        for k, v in config["constraints"]["time_to"].items()
+            for k, v in constraint["time_to"].items()
         }
     return {
         "data": places
@@ -231,7 +253,11 @@ def search_v1(db, config):
 
     :return: The matching flat objects in a JSON ``data`` dict.
     """
-    postal_codes = flatisfy.data.load_data("postal_codes", config)
+    postal_codes = {}
+    for constraint_name, constraint in config["constraints"].items():
+        postal_codes[constraint_name] = flatisfy.data.load_data(
+            PostalCode, constraint, config
+        )
 
     try:
         query = json.load(bottle.request.body)["query"]
@@ -245,14 +271,20 @@ def search_v1(db, config):
     ]
 
     for flat in flats:
-        if flat["flatisfy_postal_code"]:
+        try:
-            postal_code_data = postal_codes[flat["flatisfy_postal_code"]]
+            assert flat["flatisfy_postal_code"]
+
+            postal_code_data = next(
+                x
+                for x in postal_codes.get(flat["flatisfy_constraint"], [])
+                if x.postal_code == flat["flatisfy_postal_code"]
+            )
             flat["flatisfy_postal_code"] = {
                 "postal_code": flat["flatisfy_postal_code"],
-                "name": postal_code_data["nom"],
+                "name": postal_code_data.name,
-                "gps": postal_code_data["gps"]
+                "gps": (postal_code_data.lat, postal_code_data.lng)
             }
-        else:
+        except (AssertionError, StopIteration):
             flat["flatisfy_postal_code"] = {}
 
     return {