phyks/flatisfy
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Multiple constraints in config and Tcl data

Browse Source 
**Note**: This new feature requires a modification of the SQLite database used by Flatisfy. If you don't care about your current database, you can just remove it and do a reimport. Otherwise, you should do the following modifications manually:

* Add a [flatisfy_constraint](ed7e9dfc1a/flatisfy/models/flat.py (L89)) VARCHAR field in the flats table.
* Create [this table](ed7e9dfc1a/flatisfy/models/public_transport.py (L24-32)) for public transport data and [this one](ed7e9dfc1a/flatisfy/models/postal_code.py (L24-34)) for postal codes data.

Closes #58

See merge request !10
This commit is contained in:
Lucas Verney 2017-06-19 16:12:17 +02:00
commit edb7b822d2
22 changed files with 5302 additions and 349 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

17
CONTRIBUTING.md
View File

@ -56,3 +56,20 @@ the list of available backends in
and update the list of `BACKEND_PRECEDENCES` for deduplication in
[flatisfy/filters/duplicates.py#L24-31](https://git.phyks.me/Phyks/flatisfy/blob/master/flatisfy/filters/duplicates.py#L24-31).
Thats' all!
## Adding new data files
If you want to add new data files, especially for public transportation stops
(to cover more cities), please follow these steps:
1. Download and put the **original** file in `flatisfy/data_files`. Please,
use the original data file to ease tracking licenses and be able to still
have a working pipeline, by letting the user download it and place it in
the right place, in case of license conflict.
2. Mention the added data file and its license in `README.md`, in the
dedicated section.
3. Write a preprocessing function in `flatisfy/data_files/__init__.py`. You
can have a look at the existing functions for a model.
Thanks!

3
README.md
View File

@ -73,7 +73,8 @@ which covers Paris. If you want to run the script using some other location,
you might have to change these files by matching datasets.
* [LaPoste Hexasmal](https://datanova.legroupe.laposte.fr/explore/dataset/laposte_hexasmal/?disjunctive.code_commune_insee&disjunctive.nom_de_la_commune&disjunctive.code_postal&disjunctive.libell_d_acheminement&disjunctive.ligne_5) for the list of cities and postal codes in France.
* [RATP stations](https://data.ratp.fr/explore/dataset/positions-geographiques-des-stations-du-reseau-ratp/table/?disjunctive.stop_name&disjunctive.code_postal&disjunctive.departement) for the list of subway stations with their positions in Paris and nearby areas.
* [RATP (Paris) stations](https://data.ratp.fr/explore/dataset/positions-geographiques-des-stations-du-reseau-ratp/table/?disjunctive.stop_name&disjunctive.code_postal&disjunctive.departement) for the list of subway/tram/bus stations with their positions in Paris and nearby areas.
* [Tcl (Lyon) stations](https://download.data.grandlyon.com/wfs/rdata?SERVICE=WFS&VERSION=2.0.0&outputformat=GEOJSON&maxfeatures=4601&request=GetFeature&typename=tcl_sytral.tclarret&SRSNAME=urn:ogc:def:crs:EPSG::4326) for the list of subway/tram/bus stations with their positions in Paris and nearby areas.
Both datasets are licensed under the Open Data Commons Open Database License
(ODbL): https://opendatacommons.org/licenses/odbl/.

27
doc/0.getting_started.md
View File

@ -124,31 +124,14 @@ under the `constraints` key. The available constraints are:
form. Beware that `time` constraints are in **seconds**.
You can think of constraints as "a set of criterias to filter out flats". You
can specify as many constraints as you want, in the configuration file,
provided that you name each of them uniquely.
## Building the web assets
If you want to build the web assets, you can use `npm run build:dev`
(respectively `npm run watch:dev` to build continuously and monitor changes in
source files). You can use `npm run build:prod` (`npm run watch:prod`) to do
the same in production mode (with minification etc).
## Tips
### Running with multiple configs
Let's say you are looking for a place in different places far apart (e.g. both
a house in the country for the weekends and a place in a city for the week),
you can use multiple configuration file (in this case, two configuration
files) to define everything.
Indeed, `serve` command for the web app only use a subset of the
configuration, basically only the `data_directory`, `database` URI and so on.
So, you are free to use any config file for `serve` command and still run
`import` commands multiple times with different configuration files, for
different housing queries.
This is kind of a hack on the current system, but is working!
*Note*: You can also use this tip if you are living in a city split across
multiple postal codes and want to implement constraints such as "close to
place X if postal code is Y".

53
flatisfy/__main__.py
View File

@ -10,11 +10,13 @@ import sys
logging.basicConfig()
# pylint: disable=locally-disabled,wrong-import-position
import flatisfy.config
from flatisfy import cmds
from flatisfy import data
from flatisfy import fetch
from flatisfy import tools
# pylint: enable=locally-disabled,wrong-import-position
LOGGER = logging.getLogger("flatisfy")
@ -53,6 +55,10 @@ def parse_args(argv=None):
"-vv", action="store_true",
help="Debug logging output."
)
parent_parser.add_argument(
"--constraints", type=str,
help="Comma-separated list of constraints to consider."
)
# Subcommands
subparsers = parser.add_subparsers(
@ -143,55 +149,70 @@ def main():
"you run Flatisfy.")
sys.exit(1)
# Purge command
if args.cmd == "purge":
cmds.purge_db(config)
return
# Build data files
try:
force = False
if args.cmd == "build-data":
force = True
data.preprocess_data(config, force=force)
LOGGER.info("Done building data!")
if args.cmd == "build-data":
data.preprocess_data(config, force=True)
sys.exit(0)
else:
data.preprocess_data(config)
except flatisfy.exceptions.DataBuildError:
except flatisfy.exceptions.DataBuildError as exc:
LOGGER.error("%s", exc)
sys.exit(1)
# Fetch command
if args.cmd == "fetch":
# Fetch and filter flats list
flats_list = fetch.fetch_flats_list(config)
flats_list = cmds.filter_flats(config, flats_list=flats_list,
fetched_flats = fetch.fetch_flats(config)
fetched_flats = cmds.filter_fetched_flats(config,
fetched_flats=fetched_flats,
fetch_details=True)["new"]
# Sort by cost
flats_list = tools.sort_list_of_dicts_by(flats_list, "cost")
fetched_flats = tools.sort_list_of_dicts_by(fetched_flats, "cost")
print(
tools.pretty_json(flats_list)
tools.pretty_json(sum(fetched_flats.values(), []))
)
return
# Filter command
elif args.cmd == "filter":
# Load and filter flats list
if args.input:
flats_list = fetch.load_flats_list_from_file(args.input)
fetched_flats = fetch.load_flats_from_file(args.input, config)
flats_list = cmds.filter_flats(config, flats_list=flats_list,
fetch_details=False)["new"]
fetched_flats = cmds.filter_fetched_flats(
config,
fetched_flats=fetched_flats,
fetch_details=False
)["new"]
# Sort by cost
flats_list = tools.sort_list_of_dicts_by(flats_list, "cost")
fetched_flats = tools.sort_list_of_dicts_by(fetched_flats, "cost")
# Output to stdout
print(
tools.pretty_json(flats_list)
tools.pretty_json(sum(fetched_flats.values(), []))
)
else:
cmds.import_and_filter(config, load_from_db=True)
return
# Import command
elif args.cmd == "import":
cmds.import_and_filter(config, load_from_db=False)
# Purge command
elif args.cmd == "purge":
cmds.purge_db(config)
return
# Serve command
elif args.cmd == "serve":
cmds.serve(config)
return
if __name__ == "__main__":

66
flatisfy/cmds.py
View File

@ -10,6 +10,8 @@ import logging
import flatisfy.filters
from flatisfy import database
from flatisfy.models import flat as flat_model
from flatisfy.models import postal_code as postal_code_model
from flatisfy.models import public_transport as public_transport_model
from flatisfy import fetch
from flatisfy import tools
from flatisfy.filters import metadata
@ -19,19 +21,34 @@ from flatisfy.web import app as web_app
LOGGER = logging.getLogger(__name__)
def filter_flats(config, flats_list, fetch_details=True):
def filter_flats_list(config, constraint_name, flats_list, fetch_details=True):
"""
Filter the available flats list. Then, filter it according to criteria.
:param config: A config dict.
:param constraint_name: The constraint name that the ``flats_list`` should
satisfy.
:param fetch_details: Whether additional details should be fetched between
the two passes.
:param flats_list: The initial list of flat objects to filter.
:return: A dict mapping flat status and list of flat objects.
"""
# pylint: disable=locally-disabled,redefined-variable-type
# Add the flatisfy metadata entry and prepare the flat objects
flats_list = metadata.init(flats_list)
flats_list = metadata.init(flats_list, constraint_name)
# Get the associated constraint from config
try:
constraint = config["constraints"][constraint_name]
except KeyError:
LOGGER.error(
"Missing constraint %s. Skipping filtering for these posts.",
constraint_name
)
return {
"new": [],
"duplicate": [],
"ignored": []
}
first_pass_result = collections.defaultdict(list)
second_pass_result = collections.defaultdict(list)
@ -40,6 +57,7 @@ def filter_flats(config, flats_list, fetch_details=True):
# unwanted postings as possible
if config["passes"] > 0:
first_pass_result = flatisfy.filters.first_pass(flats_list,
constraint,
config)
else:
first_pass_result["new"] = flats_list
@ -54,7 +72,7 @@ def filter_flats(config, flats_list, fetch_details=True):
# additional infos
if config["passes"] > 1:
second_pass_result = flatisfy.filters.second_pass(
first_pass_result["new"], config
first_pass_result["new"], constraint, config
)
else:
second_pass_result["new"] = first_pass_result["new"]
@ -82,6 +100,28 @@ def filter_flats(config, flats_list, fetch_details=True):
}
def filter_fetched_flats(config, fetched_flats, fetch_details=True):
"""
Filter the available flats list. Then, filter it according to criteria.
:param config: A config dict.
:param fetch_details: Whether additional details should be fetched between
the two passes.
:param fetched_flats: The initial dict mapping constraints to the list of
fetched flat objects to filter.
:return: A dict mapping constraints to a dict mapping flat status and list
of flat objects.
"""
for constraint_name, flats_list in fetched_flats.items():
fetched_flats[constraint_name] = filter_flats_list(
config,
constraint_name,
flats_list,
fetch_details
)
return fetched_flats
def import_and_filter(config, load_from_db=False):
"""
Fetch the available flats list. Then, filter it according to criteria.
@ -94,18 +134,24 @@ def import_and_filter(config, load_from_db=False):
"""
# Fetch and filter flats list
if load_from_db:
flats_list = fetch.load_flats_list_from_db(config)
fetched_flats = fetch.load_flats_from_db(config)
else:
flats_list = fetch.fetch_flats_list(config)
fetched_flats = fetch.fetch_flats(config)
# Do not fetch additional details if we loaded data from the db.
flats_list_by_status = filter_flats(config, flats_list=flats_list,
flats_by_status = filter_fetched_flats(config, fetched_flats=fetched_flats,
fetch_details=(not load_from_db))
# Create database connection
get_session = database.init_db(config["database"], config["search_index"])
LOGGER.info("Merging fetched flats in database...")
# Flatten the flats_by_status dict
flatten_flats_by_status = collections.defaultdict(list)
for flats in flats_by_status.values():
for status, flats_list in flats.items():
flatten_flats_by_status[status].extend(flats_list)
with get_session() as session:
for status, flats_list in flats_list_by_status.items():
for status, flats_list in flatten_flats_by_status.items():
# Build SQLAlchemy Flat model objects for every available flat
flats_objects = {
flat_dict["id"]: flat_model.Flat.from_dict(flat_dict)
@ -157,6 +203,10 @@ def purge_db(config):
# Use (slower) deletion by object, to ensure whoosh index is
# updated
session.delete(flat)
LOGGER.info("Purge all postal codes from the database.")
session.query(postal_code_model.PostalCode).delete()
LOGGER.info("Purge all public transportations from the database.")
session.query(public_transport_model.PublicTransport).delete()
def serve(config):

66
flatisfy/config.py
View File

@ -23,6 +23,7 @@ from flatisfy import tools
DEFAULT_CONFIG = {
# Constraints to match
"constraints": {
"default": {
"type": None, # RENT, SALE, SHARING
"house_types": [], # List of house types, must be in APART, HOUSE,
# PARKING, LAND, OTHER or UNKNOWN
@ -34,6 +35,7 @@ DEFAULT_CONFIG = {
"time_to": {} # Dict mapping names to {"gps": [lat, lng],
# "time": (min, max) }
# Time is in seconds
}
},
# Navitia API key
"navitia_api_key": None,
@ -94,35 +96,37 @@ def validate_config(config):
# and use long lines whenever needed, in order to have the full assert
# message in the log output.
# pylint: disable=locally-disabled,line-too-long
assert "type" in config["constraints"]
assert isinstance(config["constraints"]["type"], str)
assert config["constraints"]["type"].upper() in ["RENT",
"SALE", "SHARING"]
assert "house_types" in config["constraints"]
assert config["constraints"]["house_types"]
for house_type in config["constraints"]["house_types"]:
assert house_type.upper() in ["APART", "HOUSE", "PARKING", "LAND",
"OTHER", "UNKNOWN"]
# Ensure constraints are ok
assert len(config["constraints"]) > 0
for constraint in config["constraints"].values():
assert "type" in constraint
assert isinstance(constraint["type"], str)
assert constraint["type"].upper() in ["RENT", "SALE", "SHARING"]
assert "postal_codes" in config["constraints"]
assert config["constraints"]["postal_codes"]
assert "house_types" in constraint
assert constraint["house_types"]
for house_type in constraint["house_types"]:
assert house_type.upper() in ["APART", "HOUSE", "PARKING", "LAND", "OTHER", "UNKNOWN"] # noqa: E501
assert "area" in config["constraints"]
_check_constraints_bounds(config["constraints"]["area"])
assert "postal_codes" in constraint
assert constraint["postal_codes"]
assert "cost" in config["constraints"]
_check_constraints_bounds(config["constraints"]["cost"])
assert "area" in constraint
_check_constraints_bounds(constraint["area"])
assert "rooms" in config["constraints"]
_check_constraints_bounds(config["constraints"]["rooms"])
assert "cost" in constraint
_check_constraints_bounds(constraint["cost"])
assert "bedrooms" in config["constraints"]
_check_constraints_bounds(config["constraints"]["bedrooms"])
assert "rooms" in constraint
_check_constraints_bounds(constraint["rooms"])
assert "time_to" in config["constraints"]
assert isinstance(config["constraints"]["time_to"], dict)
for name, item in config["constraints"]["time_to"].items():
assert "bedrooms" in constraint
_check_constraints_bounds(constraint["bedrooms"])
assert "time_to" in constraint
assert isinstance(constraint["time_to"], dict)
for name, item in constraint["time_to"].items():
assert isinstance(name, str)
assert "gps" in item
assert isinstance(item["gps"], list)
@ -134,6 +138,7 @@ def validate_config(config):
assert config["max_entries"] is None or (isinstance(config["max_entries"], int) and config["max_entries"] > 0) # noqa: E501
assert config["data_directory"] is None or isinstance(config["data_directory"], str) # noqa: E501
assert os.path.isdir(config["data_directory"])
assert isinstance(config["search_index"], str)
assert config["modules_path"] is None or isinstance(config["modules_path"], str) # noqa: E501
@ -206,6 +211,11 @@ def load_config(args=None):
LOGGER.debug("Using default XDG data directory: %s.",
config_data["data_directory"])
if not os.path.isdir(config_data["data_directory"]):
LOGGER.info("Creating data directory according to config: %s",
config_data["data_directory"])
os.mkdir(config_data["data_directory"])
if config_data["database"] is None:
config_data["database"] = "sqlite:///" + os.path.join(
config_data["data_directory"],
@ -218,6 +228,18 @@ def load_config(args=None):
"search_index"
)
# Handle constraints filtering
if args and getattr(args, "constraints", None) is not None:
LOGGER.info(
"Filtering constraints from config according to CLI argument."
)
constraints_filter = args.constraints.split(",")
config_data["constraints"] = {
k: v
for k, v in config_data["constraints"].items()
if k in constraints_filter
}
config_validation = validate_config(config_data)
if config_validation is True:
LOGGER.info("Config has been fully initialized.")

197
flatisfy/data.py
View File

@ -5,17 +5,16 @@ the source opendata files.
"""
from __future__ import absolute_import, print_function, unicode_literals
import collections
import json
import logging
import os
import flatisfy.exceptions
from flatisfy import database
from flatisfy import data_files
from flatisfy.models.postal_code import PostalCode
from flatisfy.models.public_transport import PublicTransport
LOGGER = logging.getLogger(__name__)
MODULE_DIR = os.path.dirname(os.path.realpath(__file__))
# Try to load lru_cache
try:
@ -24,7 +23,11 @@ except ImportError:
try:
from functools32 import lru_cache
except ImportError:
lru_cache = lambda maxsize=None: lambda func: func
def lru_cache(maxsize=None):
"""
Identity implementation of ``lru_cache`` for fallback.
"""
return lambda func: func
LOGGER.warning(
"`functools.lru_cache` is not available on your system. Consider "
"installing `functools32` Python module if using Python2 for "
@ -32,156 +35,66 @@ except ImportError:
)
def _preprocess_ratp(output_dir):
"""
Build RATP file from the RATP data.
:param output_dir: Directory in which the output file should reside.
:return: ``True`` on successful build, ``False`` otherwise.
"""
ratp_data_raw = []
# Load opendata file
try:
with open(os.path.join(MODULE_DIR, "data_files/ratp.json"), "r") as fh:
ratp_data_raw = json.load(fh)
except (IOError, ValueError):
LOGGER.error("Invalid raw RATP opendata file.")
return False
# Process it
ratp_data = collections.defaultdict(list)
for item in ratp_data_raw:
stop_name = item["fields"]["stop_name"].lower()
ratp_data[stop_name].append({
"gps": item["fields"]["coord"],
"name": item["fields"]["stop_name"]
})
# Output it
with open(os.path.join(output_dir, "ratp.json"), "w") as fh:
json.dump(ratp_data, fh)
return True
def _preprocess_laposte(output_dir):
"""
Build JSON files from the postal codes data.
:param output_dir: Directory in which the output file should reside.
:return: ``True`` on successful build, ``False`` otherwise.
"""
raw_laposte_data = []
# Load opendata file
try:
with open(
os.path.join(MODULE_DIR, "data_files/laposte.json"), "r"
) as fh:
raw_laposte_data = json.load(fh)
except (IOError, ValueError):
LOGGER.error("Invalid raw LaPoste opendata file.")
return False
# Build postal codes to other infos file
postal_codes_data = {}
for item in raw_laposte_data:
try:
postal_codes_data[item["fields"]["code_postal"]] = {
"gps": item["fields"]["coordonnees_gps"],
"nom": item["fields"]["nom_de_la_commune"].title()
}
except KeyError:
LOGGER.info("Missing data for postal code %s, skipping it.",
item["fields"]["code_postal"])
with open(os.path.join(output_dir, "postal_codes.json"), "w") as fh:
json.dump(postal_codes_data, fh)
# Build city name to postal codes and other infos file
cities_data = {}
for item in raw_laposte_data:
try:
cities_data[item["fields"]["nom_de_la_commune"].title()] = {
"gps": item["fields"]["coordonnees_gps"],
"postal_code": item["fields"]["code_postal"]
}
except KeyError:
LOGGER.info("Missing data for city %s, skipping it.",
item["fields"]["nom_de_la_commune"])
with open(os.path.join(output_dir, "cities.json"), "w") as fh:
json.dump(cities_data, fh)
return True
DATA_FILES = {
"ratp.json": {
"preprocess": _preprocess_ratp,
"output": ["ratp.json"]
},
"laposte.json": {
"preprocess": _preprocess_laposte,
"output": ["cities.json", "postal_codes.json"]
},
}
def preprocess_data(config, force=False):
"""
Ensures that all the necessary data files have been built from the raw
Ensures that all the necessary data have been inserted in db from the raw
opendata files.
:params config: A config dictionary.
:params force: Whether to force rebuild or not.
"""
LOGGER.debug("Data directory is %s.", config["data_directory"])
opendata_directory = os.path.join(config["data_directory"], "opendata")
try:
LOGGER.info("Ensuring the data directory exists.")
os.makedirs(opendata_directory)
LOGGER.debug("Created opendata directory at %s.", opendata_directory)
except OSError:
LOGGER.debug("Opendata directory already existed, doing nothing.")
# Check if a build is required
get_session = database.init_db(config["database"], config["search_index"])
with get_session() as session:
is_built = (
session.query(PublicTransport).count() > 0 and
session.query(PostalCode).count > 0
)
if is_built and not force:
# No need to rebuild the database, skip
return
# Otherwise, purge all existing data
session.query(PublicTransport).delete()
session.query(PostalCode).delete()
# Build all the necessary data files
for data_file in DATA_FILES:
# Check if already built
is_built = all(
os.path.isfile(
os.path.join(opendata_directory, output)
) for output in DATA_FILES[data_file]["output"]
)
if not is_built or force:
# Build if needed
LOGGER.info("Building from {} data.".format(data_file))
if not DATA_FILES[data_file]["preprocess"](opendata_directory):
# Build all opendata files
for preprocess in data_files.PREPROCESSING_FUNCTIONS:
data_objects = preprocess()
if not data_objects:
raise flatisfy.exceptions.DataBuildError(
"Error with {} data.".format(data_file)
"Error with %s." % preprocess.__name__
)
with get_session() as session:
session.add_all(data_objects)
@lru_cache(maxsize=5)
def load_data(data_type, config):
def load_data(model, constraint, config):
"""
Load a given built data file. This function is memoized.
Load data of the specified model from the database. Only load data for the
specific areas of the postal codes in config.
:param data_type: A valid data identifier.
:param model: SQLAlchemy model to load.
:param constraint: A constraint from configuration to limit the spatial
extension of the loaded data.
:param config: A config dictionary.
:return: The loaded data. ``None`` if the query is incorrect.
:returns: A list of loaded SQLAlchemy objects from the db
"""
opendata_directory = os.path.join(config["data_directory"], "opendata")
datafile_path = os.path.join(opendata_directory, "%s.json" % data_type)
data = {}
try:
with open(datafile_path, "r") as fh:
data = json.load(fh)
except IOError:
LOGGER.error("No such data file: %s.", datafile_path)
return None
except ValueError:
LOGGER.error("Invalid JSON data file: %s.", datafile_path)
return None
if not data:
LOGGER.warning("Loading empty data for %s.", data_type)
return data
get_session = database.init_db(config["database"], config["search_index"])
results = []
with get_session() as session:
areas = []
# Get areas to fetch from, using postal codes
for postal_code in constraint["postal_codes"]:
areas.append(data_files.french_postal_codes_to_iso_3166(postal_code))
# Load data for each area
areas = list(set(areas))
for area in areas:
results.extend(
session.query(model)
.filter(model.area == area).all()
)
# Expunge loaded data from the session to be able to use them
# afterwards
session.expunge_all()
return results

175
flatisfy/data_files/__init__.py Normal file
View File

@ -0,0 +1,175 @@
# coding : utf-8
"""
Preprocessing functions to convert input opendata files into SQLAlchemy objects
ready to be stored in the database.
"""
import json
import logging
import os
from flatisfy.models.postal_code import PostalCode
from flatisfy.models.public_transport import PublicTransport
LOGGER = logging.getLogger(__name__)
MODULE_DIR = os.path.dirname(os.path.realpath(__file__))
def french_postal_codes_to_iso_3166(postal_code):
"""
Convert a French postal code to the main subdivision in French this postal
code belongs to (ISO 3166-2 code).
:param postal_code: The postal code to convert.
:returns: The ISO 3166-2 code of the subdivision or ``None``.
"""
# Mapping between areas (main subdivisions in French, ISO 3166-2) and
# French departements
# Taken from Wikipedia data.
AREA_TO_DEPARTEMENT = {
"FR-ARA": ["01", "03", "07", "15", "26", "38", "42", "43", "63", "69",
"73", "74"],
"FR-BFC": ["21", "25", "39", "58", "70", "71", "89", "90"],
"FR-BRE": ["22", "29", "35", "44", "56"],
"FR-CVL": ["18", "28", "36", "37", "41", "45"],
"FR-COR": ["20"],
"FR-GES": ["08", "10", "51", "52", "54", "55", "57", "67", "68", "88"],
"FR-HDF": ["02", "59", "60", "62", "80"],
"FR-IDF": ["75", "77", "78", "91", "92", "93", "94", "95"],
"FR-NOR": ["14", "27", "50", "61", "76"],
"FR-NAQ": ["16", "17", "19", "23", "24", "33", "40", "47", "64", "79",
"86", "87"],
"FR-OCC": ["09", "11", "12", "30", "31", "32", "34", "46", "48", "65",
"66", "81", "82"],
"FR-PDL": ["44", "49", "53", "72", "85"],
"FR-PAC": ["04", "05", "06", "13", "83", "84"]
}
departement = postal_code[:2]
return next(
(
i
for i in AREA_TO_DEPARTEMENT
if departement in AREA_TO_DEPARTEMENT[i]
),
None
)
def _preprocess_laposte():
"""
Build SQLAlchemy objects from the postal codes data.
:return: A list of ``PostalCode`` objects to be inserted in database.
"""
data_file = "laposte.json"
LOGGER.info("Building from %s data.", data_file)
raw_laposte_data = []
# Load opendata file
try:
with open(
os.path.join(MODULE_DIR, data_file), "r"
) as fh:
raw_laposte_data = json.load(fh)
except (IOError, ValueError):
LOGGER.error("Invalid raw LaPoste opendata file.")
return []
# Build postal codes to other infos file
postal_codes_data = []
for item in raw_laposte_data:
fields = item["fields"]
try:
area = french_postal_codes_to_iso_3166(fields["code_postal"])
if area is None:
LOGGER.info(
"No matching area found for postal code %s, skipping it.",
fields["code_postal"]
)
continue
postal_codes_data.append(PostalCode(
area=area,
postal_code=fields["code_postal"],
name=fields["nom_de_la_commune"].title(),
lat=fields["coordonnees_gps"][0],
lng=fields["coordonnees_gps"][1]
))
except KeyError:
LOGGER.info("Missing data for postal code %s, skipping it.",
fields["code_postal"])
return postal_codes_data
def _preprocess_ratp():
"""
Build SQLAlchemy objects from the RATP data (public transport in Paris,
France).
:return: A list of ``PublicTransport`` objects to be inserted in database.
"""
data_file = "ratp.json"
LOGGER.info("Building from %s data.", data_file)
ratp_data_raw = []
# Load opendata file
try:
with open(os.path.join(MODULE_DIR, data_file), "r") as fh:
ratp_data_raw = json.load(fh)
except (IOError, ValueError):
LOGGER.error("Invalid raw RATP opendata file.")
return []
# Process it
ratp_data = []
for item in ratp_data_raw:
fields = item["fields"]
ratp_data.append(PublicTransport(
name=fields["stop_name"],
area="FR-IDF",
lat=fields["coord"][0],
lng=fields["coord"][1]
))
return ratp_data
def _preprocess_tcl():
"""
Build SQLAlchemy objects from the Tcl data (public transport in Lyon,
France).
:return: A list of ``PublicTransport`` objects to be inserted in database.
"""
data_file = "tcl.json"
LOGGER.info("Building from %s data.", data_file)
tcl_data_raw = []
# Load opendata file
try:
with open(os.path.join(MODULE_DIR, data_file), "r") as fh:
tcl_data_raw = json.load(fh)
except (IOError, ValueError):
LOGGER.error("Invalid raw Tcl opendata file.")
return []
# Process it
tcl_data = []
for item in tcl_data_raw["features"]:
tcl_data.append(PublicTransport(
name=item["properties"]["nom"],
area="FR-ARA",
lat=item["geometry"]["coordinates"][1],
lng=item["geometry"]["coordinates"][0]
))
return tcl_data
# List of all the available preprocessing functions. Order can be important.
PREPROCESSING_FUNCTIONS = [
_preprocess_laposte,
_preprocess_ratp,
_preprocess_tcl
]

4606
flatisfy/data_files/tcl.json Normal file
View File

File diff suppressed because it is too large Load Diff

2
flatisfy/database/__init__.py
View File

@ -11,7 +11,7 @@ from contextlib import contextmanager
from sqlalchemy import event, create_engine
from sqlalchemy.engine import Engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.exc import SQLAlchemyError
from sqlalchemy.exc import OperationalError, SQLAlchemyError
import flatisfy.models.flat # noqa: F401
from flatisfy.database.base import BASE

51
flatisfy/fetch.py
View File

@ -4,6 +4,7 @@ This module contains all the code related to fetching and loading flats lists.
"""
from __future__ import absolute_import, print_function, unicode_literals
import collections
import itertools
import json
import logging
@ -225,18 +226,20 @@ class WeboobProxy(object):
return "{}"
def fetch_flats_list(config):
def fetch_flats(config):
"""
Fetch the available flats using the Flatboob / Weboob config.
:param config: A config dict.
:return: A list of all available flats.
:return: A dict mapping constraint in config to all available matching
flats.
"""
flats_list = []
fetched_flats = {}
for constraint_name, constraint in config["constraints"].items():
LOGGER.info("Loading flats for constraint %s...", constraint_name)
with WeboobProxy(config) as weboob_proxy:
LOGGER.info("Loading flats...")
queries = weboob_proxy.build_queries(config["constraints"])
queries = weboob_proxy.build_queries(constraint)
housing_posts = []
for query in queries:
housing_posts.extend(
@ -244,10 +247,11 @@ def fetch_flats_list(config):
)
LOGGER.info("Fetched %d flats.", len(housing_posts))
flats_list = [json.loads(flat) for flat in housing_posts]
flats_list = [WeboobProxy.restore_decimal_fields(flat)
for flat in flats_list]
return flats_list
constraint_flats_list = [json.loads(flat) for flat in housing_posts]
constraint_flats_list = [WeboobProxy.restore_decimal_fields(flat)
for flat in constraint_flats_list]
fetched_flats[constraint_name] = constraint_flats_list
return fetched_flats
def fetch_details(config, flat_id):
@ -269,12 +273,18 @@ def fetch_details(config, flat_id):
return flat_details
def load_flats_list_from_file(json_file):
def load_flats_from_file(json_file, config):
"""
Load a dumped flats list from JSON file.
:param json_file: The file to load housings list from.
:return: A list of all the flats in the dump file.
:return: A dict mapping constraint in config to all available matching
flats.
.. note::
As we do not know which constraint is met by a given flat, all the
flats are returned for any available constraint, and they will be
filtered out afterwards.
"""
flats_list = []
try:
@ -284,21 +294,24 @@ def load_flats_list_from_file(json_file):
LOGGER.info("Found %d flats.", len(flats_list))
except (IOError, ValueError):
LOGGER.error("File %s is not a valid dump file.", json_file)
return flats_list
return {
constraint_name: flats_list
for constraint_name in config["constraints"]
}
def load_flats_list_from_db(config):
def load_flats_from_db(config):
"""
Load flats from database.
:param config: A config dict.
:return: A list of all the flats in the database.
:return: A dict mapping constraint in config to all available matching
flats.
"""
flats_list = []
get_session = database.init_db(config["database"], config["search_index"])
loaded_flats = collections.defaultdict(list)
with get_session() as session:
# TODO: Better serialization
flats_list = [flat.json_api_repr()
for flat in session.query(flat_model.Flat).all()]
return flats_list
for flat in session.query(flat_model.Flat).all():
loaded_flats[flat.flatisfy_constraint].append(flat.json_api_repr())
return loaded_flats

31
flatisfy/filters/__init__.py
View File

@ -16,7 +16,7 @@ from flatisfy.filters import metadata
LOGGER = logging.getLogger(__name__)
def refine_with_housing_criteria(flats_list, config):
def refine_with_housing_criteria(flats_list, constraint, config):
"""
Filter a list of flats according to criteria.
@ -25,6 +25,7 @@ def refine_with_housing_criteria(flats_list, config):
user criteria, and avoid exposing unwanted flats.
:param flats_list: A list of flats dict to filter.
:param constraint: The constraint that the ``flats_list`` should satisfy.
:param config: A config dict.
:return: A tuple of flats to keep and flats to delete.
"""
@ -37,7 +38,7 @@ def refine_with_housing_criteria(flats_list, config):
postal_code = flat["flatisfy"].get("postal_code", None)
if (
postal_code and
postal_code not in config["constraints"]["postal_codes"]
postal_code not in constraint["postal_codes"]
):
LOGGER.info("Postal code for flat %s is out of range.", flat["id"])
is_ok[i] = is_ok[i] and False
@ -47,7 +48,7 @@ def refine_with_housing_criteria(flats_list, config):
time = time["time"]
is_within_interval = tools.is_within_interval(
time,
*(config["constraints"]["time_to"][place_name]["time"])
*(constraint["time_to"][place_name]["time"])
)
if not is_within_interval:
LOGGER.info("Flat %s is too far from place %s: %ds.",
@ -56,7 +57,7 @@ def refine_with_housing_criteria(flats_list, config):
# Check other fields
for field in ["area", "cost", "rooms", "bedrooms"]:
interval = config["constraints"][field]
interval = constraint[field]
is_within_interval = tools.is_within_interval(
flat.get(field, None),
*interval
@ -80,7 +81,7 @@ def refine_with_housing_criteria(flats_list, config):
)
def first_pass(flats_list, config):
def first_pass(flats_list, constraint, config):
"""
First filtering pass.
@ -89,6 +90,7 @@ def first_pass(flats_list, config):
only request more data for the remaining housings.
:param flats_list: A list of flats dict to filter.
:param constraint: The constraint that the ``flats_list`` should satisfy.
:param config: A config dict.
:return: A dict mapping flat status and list of flat objects.
"""
@ -108,11 +110,12 @@ def first_pass(flats_list, config):
)
# Guess the postal codes
flats_list = metadata.guess_postal_code(flats_list, config)
flats_list = metadata.guess_postal_code(flats_list, constraint, config)
# Try to match with stations
flats_list = metadata.guess_stations(flats_list, config)
flats_list = metadata.guess_stations(flats_list, constraint, config)
# Remove returned housing posts that do not match criteria
flats_list, ignored_list = refine_with_housing_criteria(flats_list, config)
flats_list, ignored_list = refine_with_housing_criteria(flats_list,
constraint, config)
return {
"new": flats_list,
@ -121,7 +124,7 @@ def first_pass(flats_list, config):
}
def second_pass(flats_list, config):
def second_pass(flats_list, constraint, config):
"""
Second filtering pass.
@ -133,6 +136,7 @@ def second_pass(flats_list, config):
possible from the fetched housings.
:param flats_list: A list of flats dict to filter.
:param constraint: The constraint that the ``flats_list`` should satisfy.
:param config: A config dict.
:return: A dict mapping flat status and list of flat objects.
"""
@ -141,16 +145,17 @@ def second_pass(flats_list, config):
# left and we already tried to find postal code and nearby stations.
# Confirm postal code
flats_list = metadata.guess_postal_code(flats_list, config)
flats_list = metadata.guess_postal_code(flats_list, constraint, config)
# Better match with stations (confirm and check better)
flats_list = metadata.guess_stations(flats_list, config)
flats_list = metadata.guess_stations(flats_list, constraint, config)
# Compute travel time to specified points
flats_list = metadata.compute_travel_times(flats_list, config)
flats_list = metadata.compute_travel_times(flats_list, constraint, config)
# Remove returned housing posts that do not match criteria
flats_list, ignored_list = refine_with_housing_criteria(flats_list, config)
flats_list, ignored_list = refine_with_housing_criteria(flats_list,
constraint, config)
return {
"new": flats_list,

116
flatisfy/filters/metadata.py
View File

@ -12,24 +12,29 @@ import re
from flatisfy import data
from flatisfy import tools
from flatisfy.models.postal_code import PostalCode
from flatisfy.models.public_transport import PublicTransport
LOGGER = logging.getLogger(__name__)
def init(flats_list):
def init(flats_list, constraint):
"""
Create a flatisfy key containing a dict of metadata fetched by flatisfy for
each flat in the list. Also perform some basic transform on flat objects to
prepare for the metadata fetching.
:param flats_list: A list of flats dict.
:param constraint: The constraint that the ``flats_list`` should satisfy.
:return: The updated list
"""
for flat in flats_list:
# Init flatisfy key
if "flatisfy" not in flat:
flat["flatisfy"] = {}
if "constraint" not in flat["flatisfy"]:
flat["flatisfy"]["constraint"] = constraint
# Move url key to urls
if "urls" not in flat:
if "url" in flat:
@ -117,11 +122,12 @@ def fuzzy_match(query, choices, limit=3, threshold=75):
return matches
def guess_postal_code(flats_list, config, distance_threshold=20000):
def guess_postal_code(flats_list, constraint, config, distance_threshold=20000):
"""
Try to guess the postal code from the location of the flats.
:param flats_list: A list of flats dict.
:param constraint: The constraint that the ``flats_list`` should satisfy.
:param config: A config dict.
:param distance_threshold: Maximum distance in meters between the
constraint postal codes (from config) and the one found by this function,
@ -130,8 +136,7 @@ def guess_postal_code(flats_list, config, distance_threshold=20000):
:return: An updated list of flats dict with guessed postal code.
"""
opendata = {
"cities": data.load_data("cities", config),
"postal_codes": data.load_data("postal_codes", config)
"postal_codes": data.load_data(PostalCode, constraint, config)
}
for flat in flats_list:
@ -155,7 +160,8 @@ def guess_postal_code(flats_list, config, distance_threshold=20000):
postal_code = postal_code.group(0)
# Check the postal code is within the db
assert postal_code in opendata["postal_codes"]
assert postal_code in [x.postal_code
for x in opendata["postal_codes"]]
LOGGER.info(
"Found postal code in location field for flat %s: %s.",
@ -165,10 +171,11 @@ def guess_postal_code(flats_list, config, distance_threshold=20000):
postal_code = None
# If not found, try to find a city
cities = {x.name: x for x in opendata["postal_codes"]}
if not postal_code:
matched_city = fuzzy_match(
location,
opendata["cities"].keys(),
cities.keys(),
limit=1
)
if matched_city:
@ -176,7 +183,7 @@ def guess_postal_code(flats_list, config, distance_threshold=20000):
matched_city = matched_city[0]
matched_city_name = matched_city[0]
postal_code = (
opendata["cities"][matched_city_name]["postal_code"]
cities[matched_city_name].postal_code
)
LOGGER.info(
("Found postal code in location field through city lookup "
@ -189,10 +196,18 @@ def guess_postal_code(flats_list, config, distance_threshold=20000):
if postal_code and distance_threshold:
distance = min(
tools.distance(
opendata["postal_codes"][postal_code]["gps"],
opendata["postal_codes"][constraint]["gps"],
next(
(x.lat, x.lng)
for x in opendata["postal_codes"]
if x.postal_code == postal_code
),
next(
(x.lat, x.lng)
for x in opendata["postal_codes"]
if x.postal_code == constraint_postal_code
)
for constraint in config["constraints"]["postal_codes"]
)
for constraint_postal_code in constraint["postal_codes"]
)
if distance > distance_threshold:
@ -218,11 +233,12 @@ def guess_postal_code(flats_list, config, distance_threshold=20000):
return flats_list
def guess_stations(flats_list, config, distance_threshold=1500):
def guess_stations(flats_list, constraint, config, distance_threshold=1500):
"""
Try to match the station field with a list of available stations nearby.
:param flats_list: A list of flats dict.
:param constraint: The constraint that the ``flats_list`` should satisfy.
:param config: A config dict.
:param distance_threshold: Maximum distance (in meters) between the center
of the postal code and the station to consider it ok.
@ -230,8 +246,8 @@ def guess_stations(flats_list, config, distance_threshold=1500):
:return: An updated list of flats dict with guessed nearby stations.
"""
opendata = {
"postal_codes": data.load_data("postal_codes", config),
"stations": data.load_data("ratp", config)
"postal_codes": data.load_data(PostalCode, constraint, config),
"stations": data.load_data(PublicTransport, constraint, config)
}
for flat in flats_list:
@ -247,7 +263,7 @@ def guess_stations(flats_list, config, distance_threshold=1500):
matched_stations = fuzzy_match(
flat_station,
opendata["stations"].keys(),
[x.name for x in opendata["stations"]],
limit=10,
threshold=50
)
@ -259,53 +275,62 @@ def guess_stations(flats_list, config, distance_threshold=1500):
if postal_code:
# If there is a postal code, check that the matched station is
# closed to it
postal_code_gps = opendata["postal_codes"][postal_code]["gps"]
postal_code_gps = next(
(x.lat, x.lng)
for x in opendata["postal_codes"]
if x.postal_code == postal_code
)
for station in matched_stations:
# opendata["stations"] is a dict mapping station names to list
# of coordinates, for efficiency. Note that multiple stations
# with the same name exist in a city, hence the list of
# coordinates.
for station_data in opendata["stations"][station[0]]:
distance = tools.distance(station_data["gps"],
postal_code_gps)
# Note that multiple stations with the same name exist in a
# city, hence the list of stations objects for a given matching
# station name.
stations_objects = [
x for x in opendata["stations"] if x.name == station[0]
]
for station_data in stations_objects:
distance = tools.distance(
(station_data.lat, station_data.lng),
postal_code_gps
)
if distance < distance_threshold:
# If at least one of the coordinates for a given
# station is close enough, that's ok and we can add
# the station
good_matched_stations.append({
"key": station[0],
"name": station_data["name"],
"name": station_data.name,
"confidence": station[1],
"gps": station_data["gps"]
"gps": (station_data.lat, station_data.lng)
})
break
LOGGER.debug(
"Station %s is too far from flat %s, discarding it.",
station[0], flat["id"]
LOGGER.info(
("Station %s is too far from flat %s (%dm > %dm), "
"discarding it."),
station[0],
flat["id"],
int(distance),
int(distance_threshold)
)
else:
LOGGER.info(
("No postal code for flat %s, keeping all the matched "
"stations with half confidence."),
"No postal code for flat %s, skipping stations detection.",
flat["id"]
)
# Otherwise, we keep every matching station but with half
# confidence
good_matched_stations = [
{
"name": station[0],
"confidence": station[1] * 0.5,
"gps": station_gps
}
for station in matched_stations
for station_gps in opendata["stations"][station[0]]
]
# Store matched stations and the associated confidence
if not good_matched_stations:
# No stations found, log it and cotninue with next housing
LOGGER.info(
"Found stations for flat %s: %s.",
"No stations found for flat %s, matching %s.",
flat["id"],
", ".join(x["name"] for x in good_matched_stations)
flat["station"]
)
continue
LOGGER.info(
"Found stations for flat %s: %s (matching %s).",
flat["id"],
", ".join(x["name"] for x in good_matched_stations),
flat["station"]
)
# If some stations were already filled in and the result is different,
@ -335,12 +360,13 @@ def guess_stations(flats_list, config, distance_threshold=1500):
return flats_list
def compute_travel_times(flats_list, config):
def compute_travel_times(flats_list, constraint, config):
"""
Compute the travel time between each flat and the points listed in the
constraints.
:param flats_list: A list of flats dict.
:param constraint: The constraint that the ``flats_list`` should satisfy.
:param config: A config dict.
:return: An updated list of flats dict with computed travel times.
@ -363,7 +389,7 @@ def compute_travel_times(flats_list, config):
# For each place, loop over the stations close to the flat, and find
# the minimum travel time.
for place_name, place in config["constraints"]["time_to"].items():
for place_name, place in constraint["time_to"].items():
time_to_place = None
for station in flat["flatisfy"]["matched_stations"]:
time_from_station = tools.get_travel_time_between(

4
flatisfy/models/flat.py
View File

@ -86,6 +86,7 @@ class Flat(BASE):
flatisfy_stations = Column(MagicJSON)
flatisfy_postal_code = Column(String)
flatisfy_time_to = Column(MagicJSON)
flatisfy_constraint = Column(String)
# Status
status = Column(Enum(FlatStatus), default=FlatStatus.new)
@ -108,6 +109,9 @@ class Flat(BASE):
flat_dict["flatisfy_time_to"] = (
flat_dict["flatisfy"].get("time_to", {})
)
flat_dict["flatisfy_constraint"] = (
flat_dict["flatisfy"].get("constraint", "default")
)
del flat_dict["flatisfy"]
# Handle utilities field

37
flatisfy/models/postal_code.py Normal file
View File

@ -0,0 +1,37 @@
# coding: utf-8
"""
This modules defines an SQLAlchemy ORM model for a postal code opendata.
"""
# pylint: disable=locally-disabled,invalid-name,too-few-public-methods
from __future__ import absolute_import, print_function, unicode_literals
import logging
from sqlalchemy import (
Column, Float, Integer, String, UniqueConstraint
)
from flatisfy.database.base import BASE
LOGGER = logging.getLogger(__name__)
class PostalCode(BASE):
"""
SQLAlchemy ORM model to store a postal code opendata.
"""
__tablename__ = "postal_codes"
id = Column(Integer, primary_key=True)
# Area is an identifier to prevent loading unnecessary stops. For now it is
# following ISO 3166-2.
area = Column(String, index=True)
postal_code = Column(String, index=True)
name = Column(String, index=True)
lat = Column(Float)
lng = Column(Float)
UniqueConstraint("postal_code", "name")
def __repr__(self):
return "<PostalCode(id=%s)>" % self.id

35
flatisfy/models/public_transport.py Normal file
View File

@ -0,0 +1,35 @@
# coding: utf-8
"""
This modules defines an SQLAlchemy ORM model for public transport opendata.
"""
# pylint: disable=locally-disabled,invalid-name,too-few-public-methods
from __future__ import absolute_import, print_function, unicode_literals
import logging
from sqlalchemy import (
Column, Float, Integer, String
)
from flatisfy.database.base import BASE
LOGGER = logging.getLogger(__name__)
class PublicTransport(BASE):
"""
SQLAlchemy ORM model to store public transport opendata.
"""
__tablename__ = "public_transports"
id = Column(Integer, primary_key=True)
# Area is an identifier to prevent loading unnecessary stops. For now it is
# following ISO 3166-2.
area = Column(String, index=True)
name = Column(String)
lat = Column(Float)
lng = Column(Float)
def __repr__(self):
return "<PublicTransport(id=%s)>" % self.id

1
flatisfy/tools.py
View File

@ -298,5 +298,4 @@ def get_travel_time_between(latlng_from, latlng_to, config):
"time": time,
"sections": sections
}
else:
return None

3
flatisfy/web/app.py
View File

@ -69,7 +69,8 @@ def get_app(config):
# API v1 routes
app.route("/api/v1/", "GET", api_routes.index_v1)
app.route("/api/v1/time_to/places", "GET", api_routes.time_to_places_v1)
app.route("/api/v1/time_to_places", "GET",
api_routes.time_to_places_v1)
app.route("/api/v1/flats", "GET", api_routes.flats_v1)
app.route("/api/v1/flats/status/:status", "GET",

2
flatisfy/web/js_src/api/index.js
View File

@ -107,7 +107,7 @@ export const updateFlatNotation = function (flatId, newNotation, callback) {
}
export const getTimeToPlaces = function (callback) {
fetch('/api/v1/time_to/places', { credentials: 'same-origin' })
fetch('/api/v1/time_to_places', { credentials: 'same-origin' })
.then(function (response) {
return response.json()
}).then(function (json) {

15
flatisfy/web/js_src/store/getters.js
View File

@ -54,5 +54,18 @@ export default {
return markers
},
allTimeToPlaces: state => state.timeToPlaces
allTimeToPlaces: state => {
let places = {}
Object.keys(state.timeToPlaces).forEach(constraint => {
let constraintTimeToPlaces = state.timeToPlaces[constraint]
Object.keys(constraintTimeToPlaces).forEach(name =>
places[name] = constraintTimeToPlaces[name]
)
})
return places
},
timeToPlaces: (state, getters) => (constraint_name) => {
return state.timeToPlaces[constraint_name]
},
}

6
flatisfy/web/js_src/views/details.vue
View File

@ -222,12 +222,12 @@ export default {
flatMarkers () {
return this.$store.getters.flatsMarkers(this.$router, flat => flat.id === this.$route.params.id)
},
timeToPlaces () {
return this.$store.getters.allTimeToPlaces
},
flat () {
return this.$store.getters.flat(this.$route.params.id)
},
timeToPlaces () {
return this.$store.getters.timeToPlaces(this.flat.flatisfy_constraint)
},
notation () {
if (this.overloadNotation) {
return this.overloadNotation

76
flatisfy/web/routes/api.py
View File

@ -12,6 +12,7 @@ import bottle
import flatisfy.data
from flatisfy.models import flat as flat_model
from flatisfy.models.postal_code import PostalCode
# TODO: Flat post-processing code should be factorized
@ -38,7 +39,11 @@ def flats_v1(config, db):
:return: The available flats objects in a JSON ``data`` dict.
"""
postal_codes = flatisfy.data.load_data("postal_codes", config)
postal_codes = {}
for constraint_name, constraint in config["constraints"].items():
postal_codes[constraint_name] = flatisfy.data.load_data(
PostalCode, constraint, config
)
flats = [
flat.json_api_repr()
@ -46,14 +51,20 @@ def flats_v1(config, db):
]
for flat in flats:
if flat["flatisfy_postal_code"]:
postal_code_data = postal_codes[flat["flatisfy_postal_code"]]
try:
assert flat["flatisfy_postal_code"]
postal_code_data = next(
x
for x in postal_codes.get(flat["flatisfy_constraint"], [])
if x.postal_code == flat["flatisfy_postal_code"]
)
flat["flatisfy_postal_code"] = {
"postal_code": flat["flatisfy_postal_code"],
"name": postal_code_data["nom"],
"gps": postal_code_data["gps"]
"name": postal_code_data.name,
"gps": (postal_code_data.lat, postal_code_data.lng)
}
else:
except (AssertionError, StopIteration):
flat["flatisfy_postal_code"] = {}
return {
@ -94,8 +105,6 @@ def flat_v1(flat_id, config, db):
:return: The flat object in a JSON ``data`` dict.
"""
postal_codes = flatisfy.data.load_data("postal_codes", config)
flat = db.query(flat_model.Flat).filter_by(id=flat_id).first()
if not flat:
@ -103,14 +112,25 @@ def flat_v1(flat_id, config, db):
flat = flat.json_api_repr()
if flat["flatisfy_postal_code"]:
postal_code_data = postal_codes[flat["flatisfy_postal_code"]]
try:
assert flat["flatisfy_postal_code"]
constraint = config["constraints"].get(flat["flatisfy_constraint"],
None)
assert constraint is not None
postal_codes = flatisfy.data.load_data(PostalCode, constraint, config)
postal_code_data = next(
x
for x in postal_codes
if x.postal_code == flat["flatisfy_postal_code"]
)
flat["flatisfy_postal_code"] = {
"postal_code": flat["flatisfy_postal_code"],
"name": postal_code_data["nom"],
"gps": postal_code_data["gps"]
"name": postal_code_data.name,
"gps": (postal_code_data.lat, postal_code_data.lng)
}
else:
except (AssertionError, StopIteration):
flat["flatisfy_postal_code"] = {}
return {
@ -206,14 +226,16 @@ def time_to_places_v1(config):
"""
API v1 route to fetch the details of the places to compute time to.
GET /api/v1/time_to/places
GET /api/v1/time_to_places
:return: The JSON dump of the places to compute time to (dict of places
names mapped to GPS coordinates).
"""
places = {
places = {}
for constraint_name, constraint in config["constraints"].items():
places[constraint_name] = {
k: v["gps"]
for k, v in config["constraints"]["time_to"].items()
for k, v in constraint["time_to"].items()
}
return {
"data": places
@ -231,7 +253,11 @@ def search_v1(db, config):
:return: The matching flat objects in a JSON ``data`` dict.
"""
postal_codes = flatisfy.data.load_data("postal_codes", config)
postal_codes = {}
for constraint_name, constraint in config["constraints"].items():
postal_codes[constraint_name] = flatisfy.data.load_data(
PostalCode, constraint, config
)
try:
query = json.load(bottle.request.body)["query"]
@ -245,14 +271,20 @@ def search_v1(db, config):
]
for flat in flats:
if flat["flatisfy_postal_code"]:
postal_code_data = postal_codes[flat["flatisfy_postal_code"]]
try:
assert flat["flatisfy_postal_code"]
postal_code_data = next(
x
for x in postal_codes.get(flat["flatisfy_constraint"], [])
if x.postal_code == flat["flatisfy_postal_code"]
)
flat["flatisfy_postal_code"] = {
"postal_code": flat["flatisfy_postal_code"],
"name": postal_code_data["nom"],
"gps": postal_code_data["gps"]
"name": postal_code_data.name,
"gps": (postal_code_data.lat, postal_code_data.lng)
}
else:
except (AssertionError, StopIteration):
flat["flatisfy_postal_code"] = {}
return {