Multiple constraints in config and Tcl data
**Note**: This new feature requires a modification of the SQLite database used by Flatisfy. If you don't care about your current database, you can just remove it and do a reimport. Otherwise, you should do the following modifications manually: * Add a [flatisfy_constraint](ed7e9dfc1a/flatisfy/models/flat.py (L89)
) VARCHAR field in the flats table. * Create [this table](ed7e9dfc1a/flatisfy/models/public_transport.py (L24-32)
) for public transport data and [this one](ed7e9dfc1a/flatisfy/models/postal_code.py (L24-34)
) for postal codes data. Closes #58 See merge request !10
This commit is contained in:
commit
edb7b822d2
@ -56,3 +56,20 @@ the list of available backends in
|
||||
and update the list of `BACKEND_PRECEDENCES` for deduplication in
|
||||
[flatisfy/filters/duplicates.py#L24-31](https://git.phyks.me/Phyks/flatisfy/blob/master/flatisfy/filters/duplicates.py#L24-31).
|
||||
Thats' all!
|
||||
|
||||
|
||||
## Adding new data files
|
||||
|
||||
If you want to add new data files, especially for public transportation stops
|
||||
(to cover more cities), please follow these steps:
|
||||
|
||||
1. Download and put the **original** file in `flatisfy/data_files`. Please,
|
||||
use the original data file to ease tracking licenses and be able to still
|
||||
have a working pipeline, by letting the user download it and place it in
|
||||
the right place, in case of license conflict.
|
||||
2. Mention the added data file and its license in `README.md`, in the
|
||||
dedicated section.
|
||||
3. Write a preprocessing function in `flatisfy/data_files/__init__.py`. You
|
||||
can have a look at the existing functions for a model.
|
||||
|
||||
Thanks!
|
||||
|
@ -73,7 +73,8 @@ which covers Paris. If you want to run the script using some other location,
|
||||
you might have to change these files by matching datasets.
|
||||
|
||||
* [LaPoste Hexasmal](https://datanova.legroupe.laposte.fr/explore/dataset/laposte_hexasmal/?disjunctive.code_commune_insee&disjunctive.nom_de_la_commune&disjunctive.code_postal&disjunctive.libell_d_acheminement&disjunctive.ligne_5) for the list of cities and postal codes in France.
|
||||
* [RATP stations](https://data.ratp.fr/explore/dataset/positions-geographiques-des-stations-du-reseau-ratp/table/?disjunctive.stop_name&disjunctive.code_postal&disjunctive.departement) for the list of subway stations with their positions in Paris and nearby areas.
|
||||
* [RATP (Paris) stations](https://data.ratp.fr/explore/dataset/positions-geographiques-des-stations-du-reseau-ratp/table/?disjunctive.stop_name&disjunctive.code_postal&disjunctive.departement) for the list of subway/tram/bus stations with their positions in Paris and nearby areas.
|
||||
* [Tcl (Lyon) stations](https://download.data.grandlyon.com/wfs/rdata?SERVICE=WFS&VERSION=2.0.0&outputformat=GEOJSON&maxfeatures=4601&request=GetFeature&typename=tcl_sytral.tclarret&SRSNAME=urn:ogc:def:crs:EPSG::4326) for the list of subway/tram/bus stations with their positions in Paris and nearby areas.
|
||||
|
||||
Both datasets are licensed under the Open Data Commons Open Database License
|
||||
(ODbL): https://opendatacommons.org/licenses/odbl/.
|
||||
|
@ -124,31 +124,14 @@ under the `constraints` key. The available constraints are:
|
||||
form. Beware that `time` constraints are in **seconds**.
|
||||
|
||||
|
||||
You can think of constraints as "a set of criterias to filter out flats". You
|
||||
can specify as many constraints as you want, in the configuration file,
|
||||
provided that you name each of them uniquely.
|
||||
|
||||
|
||||
## Building the web assets
|
||||
|
||||
If you want to build the web assets, you can use `npm run build:dev`
|
||||
(respectively `npm run watch:dev` to build continuously and monitor changes in
|
||||
source files). You can use `npm run build:prod` (`npm run watch:prod`) to do
|
||||
the same in production mode (with minification etc).
|
||||
|
||||
|
||||
## Tips
|
||||
|
||||
### Running with multiple configs
|
||||
|
||||
Let's say you are looking for a place in different places far apart (e.g. both
|
||||
a house in the country for the weekends and a place in a city for the week),
|
||||
you can use multiple configuration file (in this case, two configuration
|
||||
files) to define everything.
|
||||
|
||||
Indeed, `serve` command for the web app only use a subset of the
|
||||
configuration, basically only the `data_directory`, `database` URI and so on.
|
||||
So, you are free to use any config file for `serve` command and still run
|
||||
`import` commands multiple times with different configuration files, for
|
||||
different housing queries.
|
||||
|
||||
This is kind of a hack on the current system, but is working!
|
||||
|
||||
*Note*: You can also use this tip if you are living in a city split across
|
||||
multiple postal codes and want to implement constraints such as "close to
|
||||
place X if postal code is Y".
|
||||
|
@ -10,11 +10,13 @@ import sys
|
||||
|
||||
logging.basicConfig()
|
||||
|
||||
# pylint: disable=locally-disabled,wrong-import-position
|
||||
import flatisfy.config
|
||||
from flatisfy import cmds
|
||||
from flatisfy import data
|
||||
from flatisfy import fetch
|
||||
from flatisfy import tools
|
||||
# pylint: enable=locally-disabled,wrong-import-position
|
||||
|
||||
|
||||
LOGGER = logging.getLogger("flatisfy")
|
||||
@ -53,6 +55,10 @@ def parse_args(argv=None):
|
||||
"-vv", action="store_true",
|
||||
help="Debug logging output."
|
||||
)
|
||||
parent_parser.add_argument(
|
||||
"--constraints", type=str,
|
||||
help="Comma-separated list of constraints to consider."
|
||||
)
|
||||
|
||||
# Subcommands
|
||||
subparsers = parser.add_subparsers(
|
||||
@ -143,55 +149,70 @@ def main():
|
||||
"you run Flatisfy.")
|
||||
sys.exit(1)
|
||||
|
||||
# Purge command
|
||||
if args.cmd == "purge":
|
||||
cmds.purge_db(config)
|
||||
return
|
||||
|
||||
# Build data files
|
||||
try:
|
||||
force = False
|
||||
if args.cmd == "build-data":
|
||||
force = True
|
||||
|
||||
data.preprocess_data(config, force=force)
|
||||
LOGGER.info("Done building data!")
|
||||
|
||||
if args.cmd == "build-data":
|
||||
data.preprocess_data(config, force=True)
|
||||
sys.exit(0)
|
||||
else:
|
||||
data.preprocess_data(config)
|
||||
except flatisfy.exceptions.DataBuildError:
|
||||
except flatisfy.exceptions.DataBuildError as exc:
|
||||
LOGGER.error("%s", exc)
|
||||
sys.exit(1)
|
||||
|
||||
# Fetch command
|
||||
if args.cmd == "fetch":
|
||||
# Fetch and filter flats list
|
||||
flats_list = fetch.fetch_flats_list(config)
|
||||
flats_list = cmds.filter_flats(config, flats_list=flats_list,
|
||||
fetch_details=True)["new"]
|
||||
fetched_flats = fetch.fetch_flats(config)
|
||||
fetched_flats = cmds.filter_fetched_flats(config,
|
||||
fetched_flats=fetched_flats,
|
||||
fetch_details=True)["new"]
|
||||
# Sort by cost
|
||||
flats_list = tools.sort_list_of_dicts_by(flats_list, "cost")
|
||||
fetched_flats = tools.sort_list_of_dicts_by(fetched_flats, "cost")
|
||||
|
||||
print(
|
||||
tools.pretty_json(flats_list)
|
||||
tools.pretty_json(sum(fetched_flats.values(), []))
|
||||
)
|
||||
return
|
||||
# Filter command
|
||||
elif args.cmd == "filter":
|
||||
# Load and filter flats list
|
||||
if args.input:
|
||||
flats_list = fetch.load_flats_list_from_file(args.input)
|
||||
fetched_flats = fetch.load_flats_from_file(args.input, config)
|
||||
|
||||
flats_list = cmds.filter_flats(config, flats_list=flats_list,
|
||||
fetch_details=False)["new"]
|
||||
fetched_flats = cmds.filter_fetched_flats(
|
||||
config,
|
||||
fetched_flats=fetched_flats,
|
||||
fetch_details=False
|
||||
)["new"]
|
||||
|
||||
# Sort by cost
|
||||
flats_list = tools.sort_list_of_dicts_by(flats_list, "cost")
|
||||
fetched_flats = tools.sort_list_of_dicts_by(fetched_flats, "cost")
|
||||
|
||||
# Output to stdout
|
||||
print(
|
||||
tools.pretty_json(flats_list)
|
||||
tools.pretty_json(sum(fetched_flats.values(), []))
|
||||
)
|
||||
else:
|
||||
cmds.import_and_filter(config, load_from_db=True)
|
||||
return
|
||||
# Import command
|
||||
elif args.cmd == "import":
|
||||
cmds.import_and_filter(config, load_from_db=False)
|
||||
# Purge command
|
||||
elif args.cmd == "purge":
|
||||
cmds.purge_db(config)
|
||||
return
|
||||
# Serve command
|
||||
elif args.cmd == "serve":
|
||||
cmds.serve(config)
|
||||
return
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
@ -10,6 +10,8 @@ import logging
|
||||
import flatisfy.filters
|
||||
from flatisfy import database
|
||||
from flatisfy.models import flat as flat_model
|
||||
from flatisfy.models import postal_code as postal_code_model
|
||||
from flatisfy.models import public_transport as public_transport_model
|
||||
from flatisfy import fetch
|
||||
from flatisfy import tools
|
||||
from flatisfy.filters import metadata
|
||||
@ -19,19 +21,34 @@ from flatisfy.web import app as web_app
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def filter_flats(config, flats_list, fetch_details=True):
|
||||
def filter_flats_list(config, constraint_name, flats_list, fetch_details=True):
|
||||
"""
|
||||
Filter the available flats list. Then, filter it according to criteria.
|
||||
|
||||
:param config: A config dict.
|
||||
:param constraint_name: The constraint name that the ``flats_list`` should
|
||||
satisfy.
|
||||
:param fetch_details: Whether additional details should be fetched between
|
||||
the two passes.
|
||||
:param flats_list: The initial list of flat objects to filter.
|
||||
:return: A dict mapping flat status and list of flat objects.
|
||||
"""
|
||||
# pylint: disable=locally-disabled,redefined-variable-type
|
||||
# Add the flatisfy metadata entry and prepare the flat objects
|
||||
flats_list = metadata.init(flats_list)
|
||||
flats_list = metadata.init(flats_list, constraint_name)
|
||||
|
||||
# Get the associated constraint from config
|
||||
try:
|
||||
constraint = config["constraints"][constraint_name]
|
||||
except KeyError:
|
||||
LOGGER.error(
|
||||
"Missing constraint %s. Skipping filtering for these posts.",
|
||||
constraint_name
|
||||
)
|
||||
return {
|
||||
"new": [],
|
||||
"duplicate": [],
|
||||
"ignored": []
|
||||
}
|
||||
|
||||
first_pass_result = collections.defaultdict(list)
|
||||
second_pass_result = collections.defaultdict(list)
|
||||
@ -40,6 +57,7 @@ def filter_flats(config, flats_list, fetch_details=True):
|
||||
# unwanted postings as possible
|
||||
if config["passes"] > 0:
|
||||
first_pass_result = flatisfy.filters.first_pass(flats_list,
|
||||
constraint,
|
||||
config)
|
||||
else:
|
||||
first_pass_result["new"] = flats_list
|
||||
@ -54,7 +72,7 @@ def filter_flats(config, flats_list, fetch_details=True):
|
||||
# additional infos
|
||||
if config["passes"] > 1:
|
||||
second_pass_result = flatisfy.filters.second_pass(
|
||||
first_pass_result["new"], config
|
||||
first_pass_result["new"], constraint, config
|
||||
)
|
||||
else:
|
||||
second_pass_result["new"] = first_pass_result["new"]
|
||||
@ -82,6 +100,28 @@ def filter_flats(config, flats_list, fetch_details=True):
|
||||
}
|
||||
|
||||
|
||||
def filter_fetched_flats(config, fetched_flats, fetch_details=True):
|
||||
"""
|
||||
Filter the available flats list. Then, filter it according to criteria.
|
||||
|
||||
:param config: A config dict.
|
||||
:param fetch_details: Whether additional details should be fetched between
|
||||
the two passes.
|
||||
:param fetched_flats: The initial dict mapping constraints to the list of
|
||||
fetched flat objects to filter.
|
||||
:return: A dict mapping constraints to a dict mapping flat status and list
|
||||
of flat objects.
|
||||
"""
|
||||
for constraint_name, flats_list in fetched_flats.items():
|
||||
fetched_flats[constraint_name] = filter_flats_list(
|
||||
config,
|
||||
constraint_name,
|
||||
flats_list,
|
||||
fetch_details
|
||||
)
|
||||
return fetched_flats
|
||||
|
||||
|
||||
def import_and_filter(config, load_from_db=False):
|
||||
"""
|
||||
Fetch the available flats list. Then, filter it according to criteria.
|
||||
@ -94,18 +134,24 @@ def import_and_filter(config, load_from_db=False):
|
||||
"""
|
||||
# Fetch and filter flats list
|
||||
if load_from_db:
|
||||
flats_list = fetch.load_flats_list_from_db(config)
|
||||
fetched_flats = fetch.load_flats_from_db(config)
|
||||
else:
|
||||
flats_list = fetch.fetch_flats_list(config)
|
||||
fetched_flats = fetch.fetch_flats(config)
|
||||
# Do not fetch additional details if we loaded data from the db.
|
||||
flats_list_by_status = filter_flats(config, flats_list=flats_list,
|
||||
fetch_details=(not load_from_db))
|
||||
flats_by_status = filter_fetched_flats(config, fetched_flats=fetched_flats,
|
||||
fetch_details=(not load_from_db))
|
||||
# Create database connection
|
||||
get_session = database.init_db(config["database"], config["search_index"])
|
||||
|
||||
LOGGER.info("Merging fetched flats in database...")
|
||||
# Flatten the flats_by_status dict
|
||||
flatten_flats_by_status = collections.defaultdict(list)
|
||||
for flats in flats_by_status.values():
|
||||
for status, flats_list in flats.items():
|
||||
flatten_flats_by_status[status].extend(flats_list)
|
||||
|
||||
with get_session() as session:
|
||||
for status, flats_list in flats_list_by_status.items():
|
||||
for status, flats_list in flatten_flats_by_status.items():
|
||||
# Build SQLAlchemy Flat model objects for every available flat
|
||||
flats_objects = {
|
||||
flat_dict["id"]: flat_model.Flat.from_dict(flat_dict)
|
||||
@ -157,6 +203,10 @@ def purge_db(config):
|
||||
# Use (slower) deletion by object, to ensure whoosh index is
|
||||
# updated
|
||||
session.delete(flat)
|
||||
LOGGER.info("Purge all postal codes from the database.")
|
||||
session.query(postal_code_model.PostalCode).delete()
|
||||
LOGGER.info("Purge all public transportations from the database.")
|
||||
session.query(public_transport_model.PublicTransport).delete()
|
||||
|
||||
|
||||
def serve(config):
|
||||
|
@ -23,17 +23,19 @@ from flatisfy import tools
|
||||
DEFAULT_CONFIG = {
|
||||
# Constraints to match
|
||||
"constraints": {
|
||||
"type": None, # RENT, SALE, SHARING
|
||||
"house_types": [], # List of house types, must be in APART, HOUSE,
|
||||
# PARKING, LAND, OTHER or UNKNOWN
|
||||
"postal_codes": [], # List of postal codes
|
||||
"area": (None, None), # (min, max) in m^2
|
||||
"cost": (None, None), # (min, max) in currency unit
|
||||
"rooms": (None, None), # (min, max)
|
||||
"bedrooms": (None, None), # (min, max)
|
||||
"time_to": {} # Dict mapping names to {"gps": [lat, lng],
|
||||
# "time": (min, max) }
|
||||
# Time is in seconds
|
||||
"default": {
|
||||
"type": None, # RENT, SALE, SHARING
|
||||
"house_types": [], # List of house types, must be in APART, HOUSE,
|
||||
# PARKING, LAND, OTHER or UNKNOWN
|
||||
"postal_codes": [], # List of postal codes
|
||||
"area": (None, None), # (min, max) in m^2
|
||||
"cost": (None, None), # (min, max) in currency unit
|
||||
"rooms": (None, None), # (min, max)
|
||||
"bedrooms": (None, None), # (min, max)
|
||||
"time_to": {} # Dict mapping names to {"gps": [lat, lng],
|
||||
# "time": (min, max) }
|
||||
# Time is in seconds
|
||||
}
|
||||
},
|
||||
# Navitia API key
|
||||
"navitia_api_key": None,
|
||||
@ -94,46 +96,49 @@ def validate_config(config):
|
||||
# and use long lines whenever needed, in order to have the full assert
|
||||
# message in the log output.
|
||||
# pylint: disable=locally-disabled,line-too-long
|
||||
assert "type" in config["constraints"]
|
||||
assert isinstance(config["constraints"]["type"], str)
|
||||
assert config["constraints"]["type"].upper() in ["RENT",
|
||||
"SALE", "SHARING"]
|
||||
|
||||
assert "house_types" in config["constraints"]
|
||||
assert config["constraints"]["house_types"]
|
||||
for house_type in config["constraints"]["house_types"]:
|
||||
assert house_type.upper() in ["APART", "HOUSE", "PARKING", "LAND",
|
||||
"OTHER", "UNKNOWN"]
|
||||
# Ensure constraints are ok
|
||||
assert len(config["constraints"]) > 0
|
||||
for constraint in config["constraints"].values():
|
||||
assert "type" in constraint
|
||||
assert isinstance(constraint["type"], str)
|
||||
assert constraint["type"].upper() in ["RENT", "SALE", "SHARING"]
|
||||
|
||||
assert "postal_codes" in config["constraints"]
|
||||
assert config["constraints"]["postal_codes"]
|
||||
assert "house_types" in constraint
|
||||
assert constraint["house_types"]
|
||||
for house_type in constraint["house_types"]:
|
||||
assert house_type.upper() in ["APART", "HOUSE", "PARKING", "LAND", "OTHER", "UNKNOWN"] # noqa: E501
|
||||
|
||||
assert "area" in config["constraints"]
|
||||
_check_constraints_bounds(config["constraints"]["area"])
|
||||
assert "postal_codes" in constraint
|
||||
assert constraint["postal_codes"]
|
||||
|
||||
assert "cost" in config["constraints"]
|
||||
_check_constraints_bounds(config["constraints"]["cost"])
|
||||
assert "area" in constraint
|
||||
_check_constraints_bounds(constraint["area"])
|
||||
|
||||
assert "rooms" in config["constraints"]
|
||||
_check_constraints_bounds(config["constraints"]["rooms"])
|
||||
assert "cost" in constraint
|
||||
_check_constraints_bounds(constraint["cost"])
|
||||
|
||||
assert "bedrooms" in config["constraints"]
|
||||
_check_constraints_bounds(config["constraints"]["bedrooms"])
|
||||
assert "rooms" in constraint
|
||||
_check_constraints_bounds(constraint["rooms"])
|
||||
|
||||
assert "time_to" in config["constraints"]
|
||||
assert isinstance(config["constraints"]["time_to"], dict)
|
||||
for name, item in config["constraints"]["time_to"].items():
|
||||
assert isinstance(name, str)
|
||||
assert "gps" in item
|
||||
assert isinstance(item["gps"], list)
|
||||
assert len(item["gps"]) == 2
|
||||
assert "time" in item
|
||||
_check_constraints_bounds(item["time"])
|
||||
assert "bedrooms" in constraint
|
||||
_check_constraints_bounds(constraint["bedrooms"])
|
||||
|
||||
assert "time_to" in constraint
|
||||
assert isinstance(constraint["time_to"], dict)
|
||||
for name, item in constraint["time_to"].items():
|
||||
assert isinstance(name, str)
|
||||
assert "gps" in item
|
||||
assert isinstance(item["gps"], list)
|
||||
assert len(item["gps"]) == 2
|
||||
assert "time" in item
|
||||
_check_constraints_bounds(item["time"])
|
||||
|
||||
assert config["passes"] in [0, 1, 2, 3]
|
||||
assert config["max_entries"] is None or (isinstance(config["max_entries"], int) and config["max_entries"] > 0) # noqa: E501
|
||||
|
||||
assert config["data_directory"] is None or isinstance(config["data_directory"], str) # noqa: E501
|
||||
assert os.path.isdir(config["data_directory"])
|
||||
assert isinstance(config["search_index"], str)
|
||||
assert config["modules_path"] is None or isinstance(config["modules_path"], str) # noqa: E501
|
||||
|
||||
@ -206,6 +211,11 @@ def load_config(args=None):
|
||||
LOGGER.debug("Using default XDG data directory: %s.",
|
||||
config_data["data_directory"])
|
||||
|
||||
if not os.path.isdir(config_data["data_directory"]):
|
||||
LOGGER.info("Creating data directory according to config: %s",
|
||||
config_data["data_directory"])
|
||||
os.mkdir(config_data["data_directory"])
|
||||
|
||||
if config_data["database"] is None:
|
||||
config_data["database"] = "sqlite:///" + os.path.join(
|
||||
config_data["data_directory"],
|
||||
@ -218,6 +228,18 @@ def load_config(args=None):
|
||||
"search_index"
|
||||
)
|
||||
|
||||
# Handle constraints filtering
|
||||
if args and getattr(args, "constraints", None) is not None:
|
||||
LOGGER.info(
|
||||
"Filtering constraints from config according to CLI argument."
|
||||
)
|
||||
constraints_filter = args.constraints.split(",")
|
||||
config_data["constraints"] = {
|
||||
k: v
|
||||
for k, v in config_data["constraints"].items()
|
||||
if k in constraints_filter
|
||||
}
|
||||
|
||||
config_validation = validate_config(config_data)
|
||||
if config_validation is True:
|
||||
LOGGER.info("Config has been fully initialized.")
|
||||
|
201
flatisfy/data.py
201
flatisfy/data.py
@ -5,17 +5,16 @@ the source opendata files.
|
||||
"""
|
||||
from __future__ import absolute_import, print_function, unicode_literals
|
||||
|
||||
import collections
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
|
||||
|
||||
import flatisfy.exceptions
|
||||
|
||||
from flatisfy import database
|
||||
from flatisfy import data_files
|
||||
from flatisfy.models.postal_code import PostalCode
|
||||
from flatisfy.models.public_transport import PublicTransport
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
MODULE_DIR = os.path.dirname(os.path.realpath(__file__))
|
||||
|
||||
# Try to load lru_cache
|
||||
try:
|
||||
@ -24,7 +23,11 @@ except ImportError:
|
||||
try:
|
||||
from functools32 import lru_cache
|
||||
except ImportError:
|
||||
lru_cache = lambda maxsize=None: lambda func: func
|
||||
def lru_cache(maxsize=None):
|
||||
"""
|
||||
Identity implementation of ``lru_cache`` for fallback.
|
||||
"""
|
||||
return lambda func: func
|
||||
LOGGER.warning(
|
||||
"`functools.lru_cache` is not available on your system. Consider "
|
||||
"installing `functools32` Python module if using Python2 for "
|
||||
@ -32,156 +35,66 @@ except ImportError:
|
||||
)
|
||||
|
||||
|
||||
def _preprocess_ratp(output_dir):
|
||||
"""
|
||||
Build RATP file from the RATP data.
|
||||
|
||||
:param output_dir: Directory in which the output file should reside.
|
||||
:return: ``True`` on successful build, ``False`` otherwise.
|
||||
"""
|
||||
ratp_data_raw = []
|
||||
# Load opendata file
|
||||
try:
|
||||
with open(os.path.join(MODULE_DIR, "data_files/ratp.json"), "r") as fh:
|
||||
ratp_data_raw = json.load(fh)
|
||||
except (IOError, ValueError):
|
||||
LOGGER.error("Invalid raw RATP opendata file.")
|
||||
return False
|
||||
|
||||
# Process it
|
||||
ratp_data = collections.defaultdict(list)
|
||||
for item in ratp_data_raw:
|
||||
stop_name = item["fields"]["stop_name"].lower()
|
||||
ratp_data[stop_name].append({
|
||||
"gps": item["fields"]["coord"],
|
||||
"name": item["fields"]["stop_name"]
|
||||
})
|
||||
|
||||
# Output it
|
||||
with open(os.path.join(output_dir, "ratp.json"), "w") as fh:
|
||||
json.dump(ratp_data, fh)
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def _preprocess_laposte(output_dir):
|
||||
"""
|
||||
Build JSON files from the postal codes data.
|
||||
|
||||
:param output_dir: Directory in which the output file should reside.
|
||||
:return: ``True`` on successful build, ``False`` otherwise.
|
||||
"""
|
||||
raw_laposte_data = []
|
||||
# Load opendata file
|
||||
try:
|
||||
with open(
|
||||
os.path.join(MODULE_DIR, "data_files/laposte.json"), "r"
|
||||
) as fh:
|
||||
raw_laposte_data = json.load(fh)
|
||||
except (IOError, ValueError):
|
||||
LOGGER.error("Invalid raw LaPoste opendata file.")
|
||||
return False
|
||||
|
||||
# Build postal codes to other infos file
|
||||
postal_codes_data = {}
|
||||
for item in raw_laposte_data:
|
||||
try:
|
||||
postal_codes_data[item["fields"]["code_postal"]] = {
|
||||
"gps": item["fields"]["coordonnees_gps"],
|
||||
"nom": item["fields"]["nom_de_la_commune"].title()
|
||||
}
|
||||
except KeyError:
|
||||
LOGGER.info("Missing data for postal code %s, skipping it.",
|
||||
item["fields"]["code_postal"])
|
||||
with open(os.path.join(output_dir, "postal_codes.json"), "w") as fh:
|
||||
json.dump(postal_codes_data, fh)
|
||||
|
||||
# Build city name to postal codes and other infos file
|
||||
cities_data = {}
|
||||
for item in raw_laposte_data:
|
||||
try:
|
||||
cities_data[item["fields"]["nom_de_la_commune"].title()] = {
|
||||
"gps": item["fields"]["coordonnees_gps"],
|
||||
"postal_code": item["fields"]["code_postal"]
|
||||
}
|
||||
except KeyError:
|
||||
LOGGER.info("Missing data for city %s, skipping it.",
|
||||
item["fields"]["nom_de_la_commune"])
|
||||
with open(os.path.join(output_dir, "cities.json"), "w") as fh:
|
||||
json.dump(cities_data, fh)
|
||||
|
||||
return True
|
||||
|
||||
|
||||
DATA_FILES = {
|
||||
"ratp.json": {
|
||||
"preprocess": _preprocess_ratp,
|
||||
"output": ["ratp.json"]
|
||||
},
|
||||
"laposte.json": {
|
||||
"preprocess": _preprocess_laposte,
|
||||
"output": ["cities.json", "postal_codes.json"]
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def preprocess_data(config, force=False):
|
||||
"""
|
||||
Ensures that all the necessary data files have been built from the raw
|
||||
Ensures that all the necessary data have been inserted in db from the raw
|
||||
opendata files.
|
||||
|
||||
:params config: A config dictionary.
|
||||
:params force: Whether to force rebuild or not.
|
||||
"""
|
||||
LOGGER.debug("Data directory is %s.", config["data_directory"])
|
||||
opendata_directory = os.path.join(config["data_directory"], "opendata")
|
||||
try:
|
||||
LOGGER.info("Ensuring the data directory exists.")
|
||||
os.makedirs(opendata_directory)
|
||||
LOGGER.debug("Created opendata directory at %s.", opendata_directory)
|
||||
except OSError:
|
||||
LOGGER.debug("Opendata directory already existed, doing nothing.")
|
||||
|
||||
# Build all the necessary data files
|
||||
for data_file in DATA_FILES:
|
||||
# Check if already built
|
||||
is_built = all(
|
||||
os.path.isfile(
|
||||
os.path.join(opendata_directory, output)
|
||||
) for output in DATA_FILES[data_file]["output"]
|
||||
# Check if a build is required
|
||||
get_session = database.init_db(config["database"], config["search_index"])
|
||||
with get_session() as session:
|
||||
is_built = (
|
||||
session.query(PublicTransport).count() > 0 and
|
||||
session.query(PostalCode).count > 0
|
||||
)
|
||||
if not is_built or force:
|
||||
# Build if needed
|
||||
LOGGER.info("Building from {} data.".format(data_file))
|
||||
if not DATA_FILES[data_file]["preprocess"](opendata_directory):
|
||||
raise flatisfy.exceptions.DataBuildError(
|
||||
"Error with {} data.".format(data_file)
|
||||
)
|
||||
if is_built and not force:
|
||||
# No need to rebuild the database, skip
|
||||
return
|
||||
# Otherwise, purge all existing data
|
||||
session.query(PublicTransport).delete()
|
||||
session.query(PostalCode).delete()
|
||||
|
||||
# Build all opendata files
|
||||
for preprocess in data_files.PREPROCESSING_FUNCTIONS:
|
||||
data_objects = preprocess()
|
||||
if not data_objects:
|
||||
raise flatisfy.exceptions.DataBuildError(
|
||||
"Error with %s." % preprocess.__name__
|
||||
)
|
||||
with get_session() as session:
|
||||
session.add_all(data_objects)
|
||||
|
||||
|
||||
@lru_cache(maxsize=5)
|
||||
def load_data(data_type, config):
|
||||
def load_data(model, constraint, config):
|
||||
"""
|
||||
Load a given built data file. This function is memoized.
|
||||
Load data of the specified model from the database. Only load data for the
|
||||
specific areas of the postal codes in config.
|
||||
|
||||
:param data_type: A valid data identifier.
|
||||
:param model: SQLAlchemy model to load.
|
||||
:param constraint: A constraint from configuration to limit the spatial
|
||||
extension of the loaded data.
|
||||
:param config: A config dictionary.
|
||||
:return: The loaded data. ``None`` if the query is incorrect.
|
||||
:returns: A list of loaded SQLAlchemy objects from the db
|
||||
"""
|
||||
opendata_directory = os.path.join(config["data_directory"], "opendata")
|
||||
datafile_path = os.path.join(opendata_directory, "%s.json" % data_type)
|
||||
data = {}
|
||||
try:
|
||||
with open(datafile_path, "r") as fh:
|
||||
data = json.load(fh)
|
||||
except IOError:
|
||||
LOGGER.error("No such data file: %s.", datafile_path)
|
||||
return None
|
||||
except ValueError:
|
||||
LOGGER.error("Invalid JSON data file: %s.", datafile_path)
|
||||
return None
|
||||
|
||||
if not data:
|
||||
LOGGER.warning("Loading empty data for %s.", data_type)
|
||||
|
||||
return data
|
||||
get_session = database.init_db(config["database"], config["search_index"])
|
||||
results = []
|
||||
with get_session() as session:
|
||||
areas = []
|
||||
# Get areas to fetch from, using postal codes
|
||||
for postal_code in constraint["postal_codes"]:
|
||||
areas.append(data_files.french_postal_codes_to_iso_3166(postal_code))
|
||||
# Load data for each area
|
||||
areas = list(set(areas))
|
||||
for area in areas:
|
||||
results.extend(
|
||||
session.query(model)
|
||||
.filter(model.area == area).all()
|
||||
)
|
||||
# Expunge loaded data from the session to be able to use them
|
||||
# afterwards
|
||||
session.expunge_all()
|
||||
return results
|
||||
|
175
flatisfy/data_files/__init__.py
Normal file
175
flatisfy/data_files/__init__.py
Normal file
@ -0,0 +1,175 @@
|
||||
# coding : utf-8
|
||||
"""
|
||||
Preprocessing functions to convert input opendata files into SQLAlchemy objects
|
||||
ready to be stored in the database.
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
|
||||
from flatisfy.models.postal_code import PostalCode
|
||||
from flatisfy.models.public_transport import PublicTransport
|
||||
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
MODULE_DIR = os.path.dirname(os.path.realpath(__file__))
|
||||
|
||||
|
||||
def french_postal_codes_to_iso_3166(postal_code):
|
||||
"""
|
||||
Convert a French postal code to the main subdivision in French this postal
|
||||
code belongs to (ISO 3166-2 code).
|
||||
|
||||
:param postal_code: The postal code to convert.
|
||||
:returns: The ISO 3166-2 code of the subdivision or ``None``.
|
||||
"""
|
||||
# Mapping between areas (main subdivisions in French, ISO 3166-2) and
|
||||
# French departements
|
||||
# Taken from Wikipedia data.
|
||||
AREA_TO_DEPARTEMENT = {
|
||||
"FR-ARA": ["01", "03", "07", "15", "26", "38", "42", "43", "63", "69",
|
||||
"73", "74"],
|
||||
"FR-BFC": ["21", "25", "39", "58", "70", "71", "89", "90"],
|
||||
"FR-BRE": ["22", "29", "35", "44", "56"],
|
||||
"FR-CVL": ["18", "28", "36", "37", "41", "45"],
|
||||
"FR-COR": ["20"],
|
||||
"FR-GES": ["08", "10", "51", "52", "54", "55", "57", "67", "68", "88"],
|
||||
"FR-HDF": ["02", "59", "60", "62", "80"],
|
||||
"FR-IDF": ["75", "77", "78", "91", "92", "93", "94", "95"],
|
||||
"FR-NOR": ["14", "27", "50", "61", "76"],
|
||||
"FR-NAQ": ["16", "17", "19", "23", "24", "33", "40", "47", "64", "79",
|
||||
"86", "87"],
|
||||
"FR-OCC": ["09", "11", "12", "30", "31", "32", "34", "46", "48", "65",
|
||||
"66", "81", "82"],
|
||||
"FR-PDL": ["44", "49", "53", "72", "85"],
|
||||
"FR-PAC": ["04", "05", "06", "13", "83", "84"]
|
||||
}
|
||||
|
||||
departement = postal_code[:2]
|
||||
return next(
|
||||
(
|
||||
i
|
||||
for i in AREA_TO_DEPARTEMENT
|
||||
if departement in AREA_TO_DEPARTEMENT[i]
|
||||
),
|
||||
None
|
||||
)
|
||||
|
||||
|
||||
|
||||
def _preprocess_laposte():
|
||||
"""
|
||||
Build SQLAlchemy objects from the postal codes data.
|
||||
|
||||
:return: A list of ``PostalCode`` objects to be inserted in database.
|
||||
"""
|
||||
data_file = "laposte.json"
|
||||
LOGGER.info("Building from %s data.", data_file)
|
||||
|
||||
raw_laposte_data = []
|
||||
# Load opendata file
|
||||
try:
|
||||
with open(
|
||||
os.path.join(MODULE_DIR, data_file), "r"
|
||||
) as fh:
|
||||
raw_laposte_data = json.load(fh)
|
||||
except (IOError, ValueError):
|
||||
LOGGER.error("Invalid raw LaPoste opendata file.")
|
||||
return []
|
||||
|
||||
# Build postal codes to other infos file
|
||||
postal_codes_data = []
|
||||
for item in raw_laposte_data:
|
||||
fields = item["fields"]
|
||||
try:
|
||||
area = french_postal_codes_to_iso_3166(fields["code_postal"])
|
||||
if area is None:
|
||||
LOGGER.info(
|
||||
"No matching area found for postal code %s, skipping it.",
|
||||
fields["code_postal"]
|
||||
)
|
||||
continue
|
||||
|
||||
postal_codes_data.append(PostalCode(
|
||||
area=area,
|
||||
postal_code=fields["code_postal"],
|
||||
name=fields["nom_de_la_commune"].title(),
|
||||
lat=fields["coordonnees_gps"][0],
|
||||
lng=fields["coordonnees_gps"][1]
|
||||
))
|
||||
except KeyError:
|
||||
LOGGER.info("Missing data for postal code %s, skipping it.",
|
||||
fields["code_postal"])
|
||||
|
||||
return postal_codes_data
|
||||
|
||||
|
||||
def _preprocess_ratp():
|
||||
"""
|
||||
Build SQLAlchemy objects from the RATP data (public transport in Paris,
|
||||
France).
|
||||
|
||||
:return: A list of ``PublicTransport`` objects to be inserted in database.
|
||||
"""
|
||||
data_file = "ratp.json"
|
||||
LOGGER.info("Building from %s data.", data_file)
|
||||
|
||||
ratp_data_raw = []
|
||||
# Load opendata file
|
||||
try:
|
||||
with open(os.path.join(MODULE_DIR, data_file), "r") as fh:
|
||||
ratp_data_raw = json.load(fh)
|
||||
except (IOError, ValueError):
|
||||
LOGGER.error("Invalid raw RATP opendata file.")
|
||||
return []
|
||||
|
||||
# Process it
|
||||
ratp_data = []
|
||||
for item in ratp_data_raw:
|
||||
fields = item["fields"]
|
||||
ratp_data.append(PublicTransport(
|
||||
name=fields["stop_name"],
|
||||
area="FR-IDF",
|
||||
lat=fields["coord"][0],
|
||||
lng=fields["coord"][1]
|
||||
))
|
||||
return ratp_data
|
||||
|
||||
|
||||
def _preprocess_tcl():
|
||||
"""
|
||||
Build SQLAlchemy objects from the Tcl data (public transport in Lyon,
|
||||
France).
|
||||
|
||||
:return: A list of ``PublicTransport`` objects to be inserted in database.
|
||||
"""
|
||||
data_file = "tcl.json"
|
||||
LOGGER.info("Building from %s data.", data_file)
|
||||
|
||||
tcl_data_raw = []
|
||||
# Load opendata file
|
||||
try:
|
||||
with open(os.path.join(MODULE_DIR, data_file), "r") as fh:
|
||||
tcl_data_raw = json.load(fh)
|
||||
except (IOError, ValueError):
|
||||
LOGGER.error("Invalid raw Tcl opendata file.")
|
||||
return []
|
||||
|
||||
# Process it
|
||||
tcl_data = []
|
||||
for item in tcl_data_raw["features"]:
|
||||
tcl_data.append(PublicTransport(
|
||||
name=item["properties"]["nom"],
|
||||
area="FR-ARA",
|
||||
lat=item["geometry"]["coordinates"][1],
|
||||
lng=item["geometry"]["coordinates"][0]
|
||||
))
|
||||
return tcl_data
|
||||
|
||||
|
||||
# List of all the available preprocessing functions. Order can be important.
|
||||
PREPROCESSING_FUNCTIONS = [
|
||||
_preprocess_laposte,
|
||||
_preprocess_ratp,
|
||||
_preprocess_tcl
|
||||
]
|
4606
flatisfy/data_files/tcl.json
Normal file
4606
flatisfy/data_files/tcl.json
Normal file
File diff suppressed because it is too large
Load Diff
@ -11,7 +11,7 @@ from contextlib import contextmanager
|
||||
from sqlalchemy import event, create_engine
|
||||
from sqlalchemy.engine import Engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
from sqlalchemy.exc import SQLAlchemyError
|
||||
from sqlalchemy.exc import OperationalError, SQLAlchemyError
|
||||
|
||||
import flatisfy.models.flat # noqa: F401
|
||||
from flatisfy.database.base import BASE
|
||||
|
@ -4,6 +4,7 @@ This module contains all the code related to fetching and loading flats lists.
|
||||
"""
|
||||
from __future__ import absolute_import, print_function, unicode_literals
|
||||
|
||||
import collections
|
||||
import itertools
|
||||
import json
|
||||
import logging
|
||||
@ -225,29 +226,32 @@ class WeboobProxy(object):
|
||||
return "{}"
|
||||
|
||||
|
||||
def fetch_flats_list(config):
|
||||
def fetch_flats(config):
|
||||
"""
|
||||
Fetch the available flats using the Flatboob / Weboob config.
|
||||
|
||||
:param config: A config dict.
|
||||
:return: A list of all available flats.
|
||||
:return: A dict mapping constraint in config to all available matching
|
||||
flats.
|
||||
"""
|
||||
flats_list = []
|
||||
fetched_flats = {}
|
||||
|
||||
with WeboobProxy(config) as weboob_proxy:
|
||||
LOGGER.info("Loading flats...")
|
||||
queries = weboob_proxy.build_queries(config["constraints"])
|
||||
housing_posts = []
|
||||
for query in queries:
|
||||
housing_posts.extend(
|
||||
weboob_proxy.query(query, config["max_entries"])
|
||||
)
|
||||
for constraint_name, constraint in config["constraints"].items():
|
||||
LOGGER.info("Loading flats for constraint %s...", constraint_name)
|
||||
with WeboobProxy(config) as weboob_proxy:
|
||||
queries = weboob_proxy.build_queries(constraint)
|
||||
housing_posts = []
|
||||
for query in queries:
|
||||
housing_posts.extend(
|
||||
weboob_proxy.query(query, config["max_entries"])
|
||||
)
|
||||
LOGGER.info("Fetched %d flats.", len(housing_posts))
|
||||
|
||||
flats_list = [json.loads(flat) for flat in housing_posts]
|
||||
flats_list = [WeboobProxy.restore_decimal_fields(flat)
|
||||
for flat in flats_list]
|
||||
return flats_list
|
||||
constraint_flats_list = [json.loads(flat) for flat in housing_posts]
|
||||
constraint_flats_list = [WeboobProxy.restore_decimal_fields(flat)
|
||||
for flat in constraint_flats_list]
|
||||
fetched_flats[constraint_name] = constraint_flats_list
|
||||
return fetched_flats
|
||||
|
||||
|
||||
def fetch_details(config, flat_id):
|
||||
@ -269,12 +273,18 @@ def fetch_details(config, flat_id):
|
||||
return flat_details
|
||||
|
||||
|
||||
def load_flats_list_from_file(json_file):
|
||||
def load_flats_from_file(json_file, config):
|
||||
"""
|
||||
Load a dumped flats list from JSON file.
|
||||
|
||||
:param json_file: The file to load housings list from.
|
||||
:return: A list of all the flats in the dump file.
|
||||
:return: A dict mapping constraint in config to all available matching
|
||||
flats.
|
||||
|
||||
.. note::
|
||||
As we do not know which constraint is met by a given flat, all the
|
||||
flats are returned for any available constraint, and they will be
|
||||
filtered out afterwards.
|
||||
"""
|
||||
flats_list = []
|
||||
try:
|
||||
@ -284,21 +294,24 @@ def load_flats_list_from_file(json_file):
|
||||
LOGGER.info("Found %d flats.", len(flats_list))
|
||||
except (IOError, ValueError):
|
||||
LOGGER.error("File %s is not a valid dump file.", json_file)
|
||||
return flats_list
|
||||
return {
|
||||
constraint_name: flats_list
|
||||
for constraint_name in config["constraints"]
|
||||
}
|
||||
|
||||
|
||||
def load_flats_list_from_db(config):
|
||||
def load_flats_from_db(config):
|
||||
"""
|
||||
Load flats from database.
|
||||
|
||||
:param config: A config dict.
|
||||
:return: A list of all the flats in the database.
|
||||
:return: A dict mapping constraint in config to all available matching
|
||||
flats.
|
||||
"""
|
||||
flats_list = []
|
||||
get_session = database.init_db(config["database"], config["search_index"])
|
||||
|
||||
loaded_flats = collections.defaultdict(list)
|
||||
with get_session() as session:
|
||||
# TODO: Better serialization
|
||||
flats_list = [flat.json_api_repr()
|
||||
for flat in session.query(flat_model.Flat).all()]
|
||||
return flats_list
|
||||
for flat in session.query(flat_model.Flat).all():
|
||||
loaded_flats[flat.flatisfy_constraint].append(flat.json_api_repr())
|
||||
return loaded_flats
|
||||
|
@ -16,7 +16,7 @@ from flatisfy.filters import metadata
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def refine_with_housing_criteria(flats_list, config):
|
||||
def refine_with_housing_criteria(flats_list, constraint, config):
|
||||
"""
|
||||
Filter a list of flats according to criteria.
|
||||
|
||||
@ -25,6 +25,7 @@ def refine_with_housing_criteria(flats_list, config):
|
||||
user criteria, and avoid exposing unwanted flats.
|
||||
|
||||
:param flats_list: A list of flats dict to filter.
|
||||
:param constraint: The constraint that the ``flats_list`` should satisfy.
|
||||
:param config: A config dict.
|
||||
:return: A tuple of flats to keep and flats to delete.
|
||||
"""
|
||||
@ -37,7 +38,7 @@ def refine_with_housing_criteria(flats_list, config):
|
||||
postal_code = flat["flatisfy"].get("postal_code", None)
|
||||
if (
|
||||
postal_code and
|
||||
postal_code not in config["constraints"]["postal_codes"]
|
||||
postal_code not in constraint["postal_codes"]
|
||||
):
|
||||
LOGGER.info("Postal code for flat %s is out of range.", flat["id"])
|
||||
is_ok[i] = is_ok[i] and False
|
||||
@ -47,7 +48,7 @@ def refine_with_housing_criteria(flats_list, config):
|
||||
time = time["time"]
|
||||
is_within_interval = tools.is_within_interval(
|
||||
time,
|
||||
*(config["constraints"]["time_to"][place_name]["time"])
|
||||
*(constraint["time_to"][place_name]["time"])
|
||||
)
|
||||
if not is_within_interval:
|
||||
LOGGER.info("Flat %s is too far from place %s: %ds.",
|
||||
@ -56,7 +57,7 @@ def refine_with_housing_criteria(flats_list, config):
|
||||
|
||||
# Check other fields
|
||||
for field in ["area", "cost", "rooms", "bedrooms"]:
|
||||
interval = config["constraints"][field]
|
||||
interval = constraint[field]
|
||||
is_within_interval = tools.is_within_interval(
|
||||
flat.get(field, None),
|
||||
*interval
|
||||
@ -80,7 +81,7 @@ def refine_with_housing_criteria(flats_list, config):
|
||||
)
|
||||
|
||||
|
||||
def first_pass(flats_list, config):
|
||||
def first_pass(flats_list, constraint, config):
|
||||
"""
|
||||
First filtering pass.
|
||||
|
||||
@ -89,6 +90,7 @@ def first_pass(flats_list, config):
|
||||
only request more data for the remaining housings.
|
||||
|
||||
:param flats_list: A list of flats dict to filter.
|
||||
:param constraint: The constraint that the ``flats_list`` should satisfy.
|
||||
:param config: A config dict.
|
||||
:return: A dict mapping flat status and list of flat objects.
|
||||
"""
|
||||
@ -108,11 +110,12 @@ def first_pass(flats_list, config):
|
||||
)
|
||||
|
||||
# Guess the postal codes
|
||||
flats_list = metadata.guess_postal_code(flats_list, config)
|
||||
flats_list = metadata.guess_postal_code(flats_list, constraint, config)
|
||||
# Try to match with stations
|
||||
flats_list = metadata.guess_stations(flats_list, config)
|
||||
flats_list = metadata.guess_stations(flats_list, constraint, config)
|
||||
# Remove returned housing posts that do not match criteria
|
||||
flats_list, ignored_list = refine_with_housing_criteria(flats_list, config)
|
||||
flats_list, ignored_list = refine_with_housing_criteria(flats_list,
|
||||
constraint, config)
|
||||
|
||||
return {
|
||||
"new": flats_list,
|
||||
@ -121,7 +124,7 @@ def first_pass(flats_list, config):
|
||||
}
|
||||
|
||||
|
||||
def second_pass(flats_list, config):
|
||||
def second_pass(flats_list, constraint, config):
|
||||
"""
|
||||
Second filtering pass.
|
||||
|
||||
@ -133,6 +136,7 @@ def second_pass(flats_list, config):
|
||||
possible from the fetched housings.
|
||||
|
||||
:param flats_list: A list of flats dict to filter.
|
||||
:param constraint: The constraint that the ``flats_list`` should satisfy.
|
||||
:param config: A config dict.
|
||||
:return: A dict mapping flat status and list of flat objects.
|
||||
"""
|
||||
@ -141,16 +145,17 @@ def second_pass(flats_list, config):
|
||||
# left and we already tried to find postal code and nearby stations.
|
||||
|
||||
# Confirm postal code
|
||||
flats_list = metadata.guess_postal_code(flats_list, config)
|
||||
flats_list = metadata.guess_postal_code(flats_list, constraint, config)
|
||||
|
||||
# Better match with stations (confirm and check better)
|
||||
flats_list = metadata.guess_stations(flats_list, config)
|
||||
flats_list = metadata.guess_stations(flats_list, constraint, config)
|
||||
|
||||
# Compute travel time to specified points
|
||||
flats_list = metadata.compute_travel_times(flats_list, config)
|
||||
flats_list = metadata.compute_travel_times(flats_list, constraint, config)
|
||||
|
||||
# Remove returned housing posts that do not match criteria
|
||||
flats_list, ignored_list = refine_with_housing_criteria(flats_list, config)
|
||||
flats_list, ignored_list = refine_with_housing_criteria(flats_list,
|
||||
constraint, config)
|
||||
|
||||
return {
|
||||
"new": flats_list,
|
||||
|
@ -12,24 +12,29 @@ import re
|
||||
|
||||
from flatisfy import data
|
||||
from flatisfy import tools
|
||||
from flatisfy.models.postal_code import PostalCode
|
||||
from flatisfy.models.public_transport import PublicTransport
|
||||
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def init(flats_list):
|
||||
def init(flats_list, constraint):
|
||||
"""
|
||||
Create a flatisfy key containing a dict of metadata fetched by flatisfy for
|
||||
each flat in the list. Also perform some basic transform on flat objects to
|
||||
prepare for the metadata fetching.
|
||||
|
||||
:param flats_list: A list of flats dict.
|
||||
:param constraint: The constraint that the ``flats_list`` should satisfy.
|
||||
:return: The updated list
|
||||
"""
|
||||
for flat in flats_list:
|
||||
# Init flatisfy key
|
||||
if "flatisfy" not in flat:
|
||||
flat["flatisfy"] = {}
|
||||
if "constraint" not in flat["flatisfy"]:
|
||||
flat["flatisfy"]["constraint"] = constraint
|
||||
# Move url key to urls
|
||||
if "urls" not in flat:
|
||||
if "url" in flat:
|
||||
@ -117,11 +122,12 @@ def fuzzy_match(query, choices, limit=3, threshold=75):
|
||||
return matches
|
||||
|
||||
|
||||
def guess_postal_code(flats_list, config, distance_threshold=20000):
|
||||
def guess_postal_code(flats_list, constraint, config, distance_threshold=20000):
|
||||
"""
|
||||
Try to guess the postal code from the location of the flats.
|
||||
|
||||
:param flats_list: A list of flats dict.
|
||||
:param constraint: The constraint that the ``flats_list`` should satisfy.
|
||||
:param config: A config dict.
|
||||
:param distance_threshold: Maximum distance in meters between the
|
||||
constraint postal codes (from config) and the one found by this function,
|
||||
@ -130,8 +136,7 @@ def guess_postal_code(flats_list, config, distance_threshold=20000):
|
||||
:return: An updated list of flats dict with guessed postal code.
|
||||
"""
|
||||
opendata = {
|
||||
"cities": data.load_data("cities", config),
|
||||
"postal_codes": data.load_data("postal_codes", config)
|
||||
"postal_codes": data.load_data(PostalCode, constraint, config)
|
||||
}
|
||||
|
||||
for flat in flats_list:
|
||||
@ -155,7 +160,8 @@ def guess_postal_code(flats_list, config, distance_threshold=20000):
|
||||
postal_code = postal_code.group(0)
|
||||
|
||||
# Check the postal code is within the db
|
||||
assert postal_code in opendata["postal_codes"]
|
||||
assert postal_code in [x.postal_code
|
||||
for x in opendata["postal_codes"]]
|
||||
|
||||
LOGGER.info(
|
||||
"Found postal code in location field for flat %s: %s.",
|
||||
@ -165,10 +171,11 @@ def guess_postal_code(flats_list, config, distance_threshold=20000):
|
||||
postal_code = None
|
||||
|
||||
# If not found, try to find a city
|
||||
cities = {x.name: x for x in opendata["postal_codes"]}
|
||||
if not postal_code:
|
||||
matched_city = fuzzy_match(
|
||||
location,
|
||||
opendata["cities"].keys(),
|
||||
cities.keys(),
|
||||
limit=1
|
||||
)
|
||||
if matched_city:
|
||||
@ -176,7 +183,7 @@ def guess_postal_code(flats_list, config, distance_threshold=20000):
|
||||
matched_city = matched_city[0]
|
||||
matched_city_name = matched_city[0]
|
||||
postal_code = (
|
||||
opendata["cities"][matched_city_name]["postal_code"]
|
||||
cities[matched_city_name].postal_code
|
||||
)
|
||||
LOGGER.info(
|
||||
("Found postal code in location field through city lookup "
|
||||
@ -189,10 +196,18 @@ def guess_postal_code(flats_list, config, distance_threshold=20000):
|
||||
if postal_code and distance_threshold:
|
||||
distance = min(
|
||||
tools.distance(
|
||||
opendata["postal_codes"][postal_code]["gps"],
|
||||
opendata["postal_codes"][constraint]["gps"],
|
||||
next(
|
||||
(x.lat, x.lng)
|
||||
for x in opendata["postal_codes"]
|
||||
if x.postal_code == postal_code
|
||||
),
|
||||
next(
|
||||
(x.lat, x.lng)
|
||||
for x in opendata["postal_codes"]
|
||||
if x.postal_code == constraint_postal_code
|
||||
)
|
||||
)
|
||||
for constraint in config["constraints"]["postal_codes"]
|
||||
for constraint_postal_code in constraint["postal_codes"]
|
||||
)
|
||||
|
||||
if distance > distance_threshold:
|
||||
@ -218,11 +233,12 @@ def guess_postal_code(flats_list, config, distance_threshold=20000):
|
||||
return flats_list
|
||||
|
||||
|
||||
def guess_stations(flats_list, config, distance_threshold=1500):
|
||||
def guess_stations(flats_list, constraint, config, distance_threshold=1500):
|
||||
"""
|
||||
Try to match the station field with a list of available stations nearby.
|
||||
|
||||
:param flats_list: A list of flats dict.
|
||||
:param constraint: The constraint that the ``flats_list`` should satisfy.
|
||||
:param config: A config dict.
|
||||
:param distance_threshold: Maximum distance (in meters) between the center
|
||||
of the postal code and the station to consider it ok.
|
||||
@ -230,8 +246,8 @@ def guess_stations(flats_list, config, distance_threshold=1500):
|
||||
:return: An updated list of flats dict with guessed nearby stations.
|
||||
"""
|
||||
opendata = {
|
||||
"postal_codes": data.load_data("postal_codes", config),
|
||||
"stations": data.load_data("ratp", config)
|
||||
"postal_codes": data.load_data(PostalCode, constraint, config),
|
||||
"stations": data.load_data(PublicTransport, constraint, config)
|
||||
}
|
||||
|
||||
for flat in flats_list:
|
||||
@ -247,7 +263,7 @@ def guess_stations(flats_list, config, distance_threshold=1500):
|
||||
|
||||
matched_stations = fuzzy_match(
|
||||
flat_station,
|
||||
opendata["stations"].keys(),
|
||||
[x.name for x in opendata["stations"]],
|
||||
limit=10,
|
||||
threshold=50
|
||||
)
|
||||
@ -259,53 +275,62 @@ def guess_stations(flats_list, config, distance_threshold=1500):
|
||||
if postal_code:
|
||||
# If there is a postal code, check that the matched station is
|
||||
# closed to it
|
||||
postal_code_gps = opendata["postal_codes"][postal_code]["gps"]
|
||||
postal_code_gps = next(
|
||||
(x.lat, x.lng)
|
||||
for x in opendata["postal_codes"]
|
||||
if x.postal_code == postal_code
|
||||
)
|
||||
for station in matched_stations:
|
||||
# opendata["stations"] is a dict mapping station names to list
|
||||
# of coordinates, for efficiency. Note that multiple stations
|
||||
# with the same name exist in a city, hence the list of
|
||||
# coordinates.
|
||||
for station_data in opendata["stations"][station[0]]:
|
||||
distance = tools.distance(station_data["gps"],
|
||||
postal_code_gps)
|
||||
# Note that multiple stations with the same name exist in a
|
||||
# city, hence the list of stations objects for a given matching
|
||||
# station name.
|
||||
stations_objects = [
|
||||
x for x in opendata["stations"] if x.name == station[0]
|
||||
]
|
||||
for station_data in stations_objects:
|
||||
distance = tools.distance(
|
||||
(station_data.lat, station_data.lng),
|
||||
postal_code_gps
|
||||
)
|
||||
if distance < distance_threshold:
|
||||
# If at least one of the coordinates for a given
|
||||
# station is close enough, that's ok and we can add
|
||||
# the station
|
||||
good_matched_stations.append({
|
||||
"key": station[0],
|
||||
"name": station_data["name"],
|
||||
"name": station_data.name,
|
||||
"confidence": station[1],
|
||||
"gps": station_data["gps"]
|
||||
"gps": (station_data.lat, station_data.lng)
|
||||
})
|
||||
break
|
||||
LOGGER.debug(
|
||||
"Station %s is too far from flat %s, discarding it.",
|
||||
station[0], flat["id"]
|
||||
LOGGER.info(
|
||||
("Station %s is too far from flat %s (%dm > %dm), "
|
||||
"discarding it."),
|
||||
station[0],
|
||||
flat["id"],
|
||||
int(distance),
|
||||
int(distance_threshold)
|
||||
)
|
||||
else:
|
||||
LOGGER.info(
|
||||
("No postal code for flat %s, keeping all the matched "
|
||||
"stations with half confidence."),
|
||||
"No postal code for flat %s, skipping stations detection.",
|
||||
flat["id"]
|
||||
)
|
||||
# Otherwise, we keep every matching station but with half
|
||||
# confidence
|
||||
good_matched_stations = [
|
||||
{
|
||||
"name": station[0],
|
||||
"confidence": station[1] * 0.5,
|
||||
"gps": station_gps
|
||||
}
|
||||
for station in matched_stations
|
||||
for station_gps in opendata["stations"][station[0]]
|
||||
]
|
||||
|
||||
# Store matched stations and the associated confidence
|
||||
if not good_matched_stations:
|
||||
# No stations found, log it and cotninue with next housing
|
||||
LOGGER.info(
|
||||
"No stations found for flat %s, matching %s.",
|
||||
flat["id"],
|
||||
flat["station"]
|
||||
)
|
||||
continue
|
||||
|
||||
LOGGER.info(
|
||||
"Found stations for flat %s: %s.",
|
||||
"Found stations for flat %s: %s (matching %s).",
|
||||
flat["id"],
|
||||
", ".join(x["name"] for x in good_matched_stations)
|
||||
", ".join(x["name"] for x in good_matched_stations),
|
||||
flat["station"]
|
||||
)
|
||||
|
||||
# If some stations were already filled in and the result is different,
|
||||
@ -335,12 +360,13 @@ def guess_stations(flats_list, config, distance_threshold=1500):
|
||||
return flats_list
|
||||
|
||||
|
||||
def compute_travel_times(flats_list, config):
|
||||
def compute_travel_times(flats_list, constraint, config):
|
||||
"""
|
||||
Compute the travel time between each flat and the points listed in the
|
||||
constraints.
|
||||
|
||||
:param flats_list: A list of flats dict.
|
||||
:param constraint: The constraint that the ``flats_list`` should satisfy.
|
||||
:param config: A config dict.
|
||||
|
||||
:return: An updated list of flats dict with computed travel times.
|
||||
@ -363,7 +389,7 @@ def compute_travel_times(flats_list, config):
|
||||
|
||||
# For each place, loop over the stations close to the flat, and find
|
||||
# the minimum travel time.
|
||||
for place_name, place in config["constraints"]["time_to"].items():
|
||||
for place_name, place in constraint["time_to"].items():
|
||||
time_to_place = None
|
||||
for station in flat["flatisfy"]["matched_stations"]:
|
||||
time_from_station = tools.get_travel_time_between(
|
||||
|
@ -86,6 +86,7 @@ class Flat(BASE):
|
||||
flatisfy_stations = Column(MagicJSON)
|
||||
flatisfy_postal_code = Column(String)
|
||||
flatisfy_time_to = Column(MagicJSON)
|
||||
flatisfy_constraint = Column(String)
|
||||
|
||||
# Status
|
||||
status = Column(Enum(FlatStatus), default=FlatStatus.new)
|
||||
@ -108,6 +109,9 @@ class Flat(BASE):
|
||||
flat_dict["flatisfy_time_to"] = (
|
||||
flat_dict["flatisfy"].get("time_to", {})
|
||||
)
|
||||
flat_dict["flatisfy_constraint"] = (
|
||||
flat_dict["flatisfy"].get("constraint", "default")
|
||||
)
|
||||
del flat_dict["flatisfy"]
|
||||
|
||||
# Handle utilities field
|
||||
|
37
flatisfy/models/postal_code.py
Normal file
37
flatisfy/models/postal_code.py
Normal file
@ -0,0 +1,37 @@
|
||||
# coding: utf-8
|
||||
"""
|
||||
This modules defines an SQLAlchemy ORM model for a postal code opendata.
|
||||
"""
|
||||
# pylint: disable=locally-disabled,invalid-name,too-few-public-methods
|
||||
from __future__ import absolute_import, print_function, unicode_literals
|
||||
|
||||
import logging
|
||||
|
||||
from sqlalchemy import (
|
||||
Column, Float, Integer, String, UniqueConstraint
|
||||
)
|
||||
|
||||
from flatisfy.database.base import BASE
|
||||
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class PostalCode(BASE):
|
||||
"""
|
||||
SQLAlchemy ORM model to store a postal code opendata.
|
||||
"""
|
||||
__tablename__ = "postal_codes"
|
||||
|
||||
id = Column(Integer, primary_key=True)
|
||||
# Area is an identifier to prevent loading unnecessary stops. For now it is
|
||||
# following ISO 3166-2.
|
||||
area = Column(String, index=True)
|
||||
postal_code = Column(String, index=True)
|
||||
name = Column(String, index=True)
|
||||
lat = Column(Float)
|
||||
lng = Column(Float)
|
||||
UniqueConstraint("postal_code", "name")
|
||||
|
||||
def __repr__(self):
|
||||
return "<PostalCode(id=%s)>" % self.id
|
35
flatisfy/models/public_transport.py
Normal file
35
flatisfy/models/public_transport.py
Normal file
@ -0,0 +1,35 @@
|
||||
# coding: utf-8
|
||||
"""
|
||||
This modules defines an SQLAlchemy ORM model for public transport opendata.
|
||||
"""
|
||||
# pylint: disable=locally-disabled,invalid-name,too-few-public-methods
|
||||
from __future__ import absolute_import, print_function, unicode_literals
|
||||
|
||||
import logging
|
||||
|
||||
from sqlalchemy import (
|
||||
Column, Float, Integer, String
|
||||
)
|
||||
|
||||
from flatisfy.database.base import BASE
|
||||
|
||||
|
||||
LOGGER = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class PublicTransport(BASE):
|
||||
"""
|
||||
SQLAlchemy ORM model to store public transport opendata.
|
||||
"""
|
||||
__tablename__ = "public_transports"
|
||||
|
||||
id = Column(Integer, primary_key=True)
|
||||
# Area is an identifier to prevent loading unnecessary stops. For now it is
|
||||
# following ISO 3166-2.
|
||||
area = Column(String, index=True)
|
||||
name = Column(String)
|
||||
lat = Column(Float)
|
||||
lng = Column(Float)
|
||||
|
||||
def __repr__(self):
|
||||
return "<PublicTransport(id=%s)>" % self.id
|
@ -298,5 +298,4 @@ def get_travel_time_between(latlng_from, latlng_to, config):
|
||||
"time": time,
|
||||
"sections": sections
|
||||
}
|
||||
else:
|
||||
return None
|
||||
return None
|
||||
|
@ -69,7 +69,8 @@ def get_app(config):
|
||||
# API v1 routes
|
||||
app.route("/api/v1/", "GET", api_routes.index_v1)
|
||||
|
||||
app.route("/api/v1/time_to/places", "GET", api_routes.time_to_places_v1)
|
||||
app.route("/api/v1/time_to_places", "GET",
|
||||
api_routes.time_to_places_v1)
|
||||
|
||||
app.route("/api/v1/flats", "GET", api_routes.flats_v1)
|
||||
app.route("/api/v1/flats/status/:status", "GET",
|
||||
|
@ -107,7 +107,7 @@ export const updateFlatNotation = function (flatId, newNotation, callback) {
|
||||
}
|
||||
|
||||
export const getTimeToPlaces = function (callback) {
|
||||
fetch('/api/v1/time_to/places', { credentials: 'same-origin' })
|
||||
fetch('/api/v1/time_to_places', { credentials: 'same-origin' })
|
||||
.then(function (response) {
|
||||
return response.json()
|
||||
}).then(function (json) {
|
||||
|
@ -5,7 +5,7 @@ export default {
|
||||
|
||||
flat: (state, getters) => id => state.flats.find(flat => flat.id === id),
|
||||
|
||||
isLoading: state => state.loading > 0,
|
||||
isLoading: state => state.loading > 0,
|
||||
|
||||
postalCodesFlatsBuckets: (state, getters) => filter => {
|
||||
const postalCodeBuckets = {}
|
||||
@ -54,5 +54,18 @@ export default {
|
||||
return markers
|
||||
},
|
||||
|
||||
allTimeToPlaces: state => state.timeToPlaces
|
||||
allTimeToPlaces: state => {
|
||||
let places = {}
|
||||
Object.keys(state.timeToPlaces).forEach(constraint => {
|
||||
let constraintTimeToPlaces = state.timeToPlaces[constraint]
|
||||
Object.keys(constraintTimeToPlaces).forEach(name =>
|
||||
places[name] = constraintTimeToPlaces[name]
|
||||
)
|
||||
})
|
||||
return places
|
||||
},
|
||||
|
||||
timeToPlaces: (state, getters) => (constraint_name) => {
|
||||
return state.timeToPlaces[constraint_name]
|
||||
},
|
||||
}
|
||||
|
@ -222,12 +222,12 @@ export default {
|
||||
flatMarkers () {
|
||||
return this.$store.getters.flatsMarkers(this.$router, flat => flat.id === this.$route.params.id)
|
||||
},
|
||||
timeToPlaces () {
|
||||
return this.$store.getters.allTimeToPlaces
|
||||
},
|
||||
flat () {
|
||||
return this.$store.getters.flat(this.$route.params.id)
|
||||
},
|
||||
timeToPlaces () {
|
||||
return this.$store.getters.timeToPlaces(this.flat.flatisfy_constraint)
|
||||
},
|
||||
notation () {
|
||||
if (this.overloadNotation) {
|
||||
return this.overloadNotation
|
||||
|
@ -12,6 +12,7 @@ import bottle
|
||||
|
||||
import flatisfy.data
|
||||
from flatisfy.models import flat as flat_model
|
||||
from flatisfy.models.postal_code import PostalCode
|
||||
|
||||
# TODO: Flat post-processing code should be factorized
|
||||
|
||||
@ -38,7 +39,11 @@ def flats_v1(config, db):
|
||||
|
||||
:return: The available flats objects in a JSON ``data`` dict.
|
||||
"""
|
||||
postal_codes = flatisfy.data.load_data("postal_codes", config)
|
||||
postal_codes = {}
|
||||
for constraint_name, constraint in config["constraints"].items():
|
||||
postal_codes[constraint_name] = flatisfy.data.load_data(
|
||||
PostalCode, constraint, config
|
||||
)
|
||||
|
||||
flats = [
|
||||
flat.json_api_repr()
|
||||
@ -46,14 +51,20 @@ def flats_v1(config, db):
|
||||
]
|
||||
|
||||
for flat in flats:
|
||||
if flat["flatisfy_postal_code"]:
|
||||
postal_code_data = postal_codes[flat["flatisfy_postal_code"]]
|
||||
try:
|
||||
assert flat["flatisfy_postal_code"]
|
||||
|
||||
postal_code_data = next(
|
||||
x
|
||||
for x in postal_codes.get(flat["flatisfy_constraint"], [])
|
||||
if x.postal_code == flat["flatisfy_postal_code"]
|
||||
)
|
||||
flat["flatisfy_postal_code"] = {
|
||||
"postal_code": flat["flatisfy_postal_code"],
|
||||
"name": postal_code_data["nom"],
|
||||
"gps": postal_code_data["gps"]
|
||||
"name": postal_code_data.name,
|
||||
"gps": (postal_code_data.lat, postal_code_data.lng)
|
||||
}
|
||||
else:
|
||||
except (AssertionError, StopIteration):
|
||||
flat["flatisfy_postal_code"] = {}
|
||||
|
||||
return {
|
||||
@ -94,8 +105,6 @@ def flat_v1(flat_id, config, db):
|
||||
|
||||
:return: The flat object in a JSON ``data`` dict.
|
||||
"""
|
||||
postal_codes = flatisfy.data.load_data("postal_codes", config)
|
||||
|
||||
flat = db.query(flat_model.Flat).filter_by(id=flat_id).first()
|
||||
|
||||
if not flat:
|
||||
@ -103,14 +112,25 @@ def flat_v1(flat_id, config, db):
|
||||
|
||||
flat = flat.json_api_repr()
|
||||
|
||||
if flat["flatisfy_postal_code"]:
|
||||
postal_code_data = postal_codes[flat["flatisfy_postal_code"]]
|
||||
try:
|
||||
assert flat["flatisfy_postal_code"]
|
||||
|
||||
constraint = config["constraints"].get(flat["flatisfy_constraint"],
|
||||
None)
|
||||
assert constraint is not None
|
||||
postal_codes = flatisfy.data.load_data(PostalCode, constraint, config)
|
||||
|
||||
postal_code_data = next(
|
||||
x
|
||||
for x in postal_codes
|
||||
if x.postal_code == flat["flatisfy_postal_code"]
|
||||
)
|
||||
flat["flatisfy_postal_code"] = {
|
||||
"postal_code": flat["flatisfy_postal_code"],
|
||||
"name": postal_code_data["nom"],
|
||||
"gps": postal_code_data["gps"]
|
||||
"name": postal_code_data.name,
|
||||
"gps": (postal_code_data.lat, postal_code_data.lng)
|
||||
}
|
||||
else:
|
||||
except (AssertionError, StopIteration):
|
||||
flat["flatisfy_postal_code"] = {}
|
||||
|
||||
return {
|
||||
@ -206,15 +226,17 @@ def time_to_places_v1(config):
|
||||
"""
|
||||
API v1 route to fetch the details of the places to compute time to.
|
||||
|
||||
GET /api/v1/time_to/places
|
||||
GET /api/v1/time_to_places
|
||||
|
||||
:return: The JSON dump of the places to compute time to (dict of places
|
||||
names mapped to GPS coordinates).
|
||||
"""
|
||||
places = {
|
||||
k: v["gps"]
|
||||
for k, v in config["constraints"]["time_to"].items()
|
||||
}
|
||||
places = {}
|
||||
for constraint_name, constraint in config["constraints"].items():
|
||||
places[constraint_name] = {
|
||||
k: v["gps"]
|
||||
for k, v in constraint["time_to"].items()
|
||||
}
|
||||
return {
|
||||
"data": places
|
||||
}
|
||||
@ -231,7 +253,11 @@ def search_v1(db, config):
|
||||
|
||||
:return: The matching flat objects in a JSON ``data`` dict.
|
||||
"""
|
||||
postal_codes = flatisfy.data.load_data("postal_codes", config)
|
||||
postal_codes = {}
|
||||
for constraint_name, constraint in config["constraints"].items():
|
||||
postal_codes[constraint_name] = flatisfy.data.load_data(
|
||||
PostalCode, constraint, config
|
||||
)
|
||||
|
||||
try:
|
||||
query = json.load(bottle.request.body)["query"]
|
||||
@ -245,14 +271,20 @@ def search_v1(db, config):
|
||||
]
|
||||
|
||||
for flat in flats:
|
||||
if flat["flatisfy_postal_code"]:
|
||||
postal_code_data = postal_codes[flat["flatisfy_postal_code"]]
|
||||
try:
|
||||
assert flat["flatisfy_postal_code"]
|
||||
|
||||
postal_code_data = next(
|
||||
x
|
||||
for x in postal_codes.get(flat["flatisfy_constraint"], [])
|
||||
if x.postal_code == flat["flatisfy_postal_code"]
|
||||
)
|
||||
flat["flatisfy_postal_code"] = {
|
||||
"postal_code": flat["flatisfy_postal_code"],
|
||||
"name": postal_code_data["nom"],
|
||||
"gps": postal_code_data["gps"]
|
||||
"name": postal_code_data.name,
|
||||
"gps": (postal_code_data.lat, postal_code_data.lng)
|
||||
}
|
||||
else:
|
||||
except (AssertionError, StopIteration):
|
||||
flat["flatisfy_postal_code"] = {}
|
||||
|
||||
return {
|
||||
|
Loading…
Reference in New Issue
Block a user