Pipeline ID: activities/social_services/social_service_reports
Pipeline ID: activities/social_services/social_service_suppliers
Pipeline ID: activities/social_services/social_service_tenders
Pipeline ID: activities/social_services/social_services
Pipeline ID: bank_of_israel/bank_of_israel
Pipeline ID: budget/municipal/fetcher
Pipeline ID: budget/national/changes/explanations/all
Pipeline ID: budget/national/changes/original/committee-zipfile
Pipeline ID: budget/national/changes/original/current-year-fixes
Pipeline ID: budget/national/changes/original/national-budget-changes
Dependency unsuccessful :Cannot run until dependency "./budget/national/changes/processed/transactions" is successfully executed
Pipeline ID: budget/national/changes/processed/national-budget-changes-aggregated
extract-change-groups Traceback (most recent call last): File "/datapackage_pipelines_budgetkey/pipelines/budget/national/changes/processed/extract-change-groups.py", line 253, in <module> main() File "/datapackage_pipelines_budgetkey/pipelines/budget/national/changes/processed/extract-change-groups.py", line 250, in main spew(update_datapackage(datapackage), process_resources(res_iter)) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew for rec in res: File "/datapackage_pipelines_budgetkey/pipelines/budget/national/changes/processed/extract-change-groups.py", line 232, in process_resource for transaction in get_transactions(get_changes(rows)): File "/datapackage_pipelines_budgetkey/pipelines/budget/national/changes/processed/extract-change-groups.py", line 193, in assign_transactions changes = changes_by_trcode[trcode] KeyError: '2011/73-009'
Pipeline ID: budget/national/changes/processed/transactions
We get the data the Ministry of Finance publishes in data.gov.il every year. It comes in XLS format, with one row per TAKANA and phase (original, approved, executed). In this pipeline we create from each triplet a single row that has all the data.
Pipeline ID: budget/national/original/national-budgets
This pipeline joins the different phases of the budget (allocated, revised and executed). In the original file there's a separate row for each of the phases. We like it better as a single row with all phase info. Another thing this pipeline does is to rename the column titles - to more friendly English names. Also, it create rows for all hierarchies - where upper hierarchies (2, 4 & 6 digits) are plain aggregations of the 8-digit items they contain.
Pipeline ID: budget/national/processed/aggregated-yearly
Pipeline ID: budget/national/processed/category-explanations
This pipeline joins budget items that span across years.
Pipeline ID: budget/national/processed/connected-items-explained
This pipeline joins budget items that span across years.
Pipeline ID: budget/national/processed/connected-national-budgets
Pipeline ID: budget/national/processed/just-the-total
Pipeline ID: budget/national/processed/roof-names
This pipeline joins the budget data to itself so that each item has a list of its immediate children.
Pipeline ID: budget/national/processed/with-extras
Pipeline ID: budgetkey/analysis/spending/publisher_entity_analysis
Pipeline ID: budgetkey/analysis/spending/publisher_foa_analysis
dump.to_sql ERROR :Traceback (most recent call last): ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context ERROR :self.dialect.do_execute( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute ERROR :cursor.execute(statement, parameters) ERROR :psycopg2.errors ERROR :. ERROR :InternalError_ ERROR :: ERROR :could not open relation with OID 487608360 ERROR :The above exception was the direct cause of the following exception: ERROR :Traceback (most recent call last): ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/dump/to_sql.py", line 15, in <module> ERROR :spew_flow(flow(ctx.parameters), ctx) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 68, in spew ERROR :for res in resources_iterator: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 68, in <genexpr> ERROR :res_iter = (it if isinstance(it, ResourceWrapper) else ResourceWrapper(res, it) ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/dumper_base.py", line 82, in process_resources ERROR :ret = self.process_resource( ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/to_sql.py", line 108, in process_resource ERROR :storage.delete('') ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_sql/storage.py", line 183, in delete ERROR :self.__reflect() ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_sql/storage.py", line 278, in __reflect ERROR :self.__metadata.reflect(only=only, bind=self.__engine) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 5752, in reflect ERROR :_reflect_info = insp._get_reflection_info( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 2018, in _get_reflection_info ERROR :check_constraints=run( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 1994, in run ERROR :res = meth(filter_names=_fn, **kw) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 1457, in get_multi_check_constraints ERROR :self.dialect.get_multi_check_constraints( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/base.py", line 4677, in get_multi_check_constraints ERROR :result = connection.execute(query, params) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1412, in execute ERROR :return meth( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 483, in _execute_on_connection ERROR :return connection._execute_clauseelement( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1635, in _execute_clauseelement ERROR :ret = self._execute_context( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1844, in _execute_context ERROR :return self._exec_single_context( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1984, in _exec_single_context ERROR :self._handle_dbapi_exception( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2339, in _handle_dbapi_exception ERROR :raise sqlalchemy_exception.with_traceback(exc_info[2]) from e ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context ERROR :self.dialect.do_execute( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute ERROR :cursor.execute(statement, parameters) ERROR :sqlalchemy.exc ERROR :. ERROR :InternalError ERROR :: ERROR :(psycopg2.errors.InternalError_) could not open relation with OID 487608360 [SQL: SELECT pg_catalog.pg_class.relname, pg_catalog.pg_constraint.conname, CASE WHEN (pg_catalog.pg_constraint.oid IS NOT NULL) THEN pg_catalog.pg_get_constraintdef(pg_catalog.pg_constraint.oid, %(pg_get_constraintdef_1)s) END AS anon_1, pg_catalog.pg_description.description FROM pg_catalog.pg_class LEFT OUTER JOIN pg_catalog.pg_constraint ON pg_catalog.pg_class.oid = pg_catalog.pg_constraint.conrelid AND pg_catalog.pg_constraint.contype = %(contype_1)s LEFT OUTER JOIN pg_catalog.pg_description ON pg_catalog.pg_description.objoid = pg_catalog.pg_constraint.oid JOIN pg_catalog.pg_namespace ON pg_catalog.pg_namespace.oid = pg_catalog.pg_class.relnamespace WHERE pg_catalog.pg_class.relkind = ANY (ARRAY[%(param_1)s, %(param_2)s, %(param_3)s]) AND pg_catalog.pg_table_is_visible(pg_catalog.pg_class.oid) AND pg_catalog.pg_namespace.nspname != %(nspname_1)s ORDER BY pg_catalog.pg_class.relname, pg_catalog.pg_constraint.conname] [parameters: {'pg_get_constraintdef_1': True, 'contype_1': 'c', 'param_1': 'r', 'param_2': 'p', 'param_3': 'f', 'nspname_1': 'pg_catalog'}] (Background on this error at: https://sqlalche.me/e/20/2j85)
Pipeline ID: budgetkey/analysis/spending/united
Pipeline ID: budgetkey/budget-functional-aggregates
Pipeline ID: budgetkey/documents
dump_to_es ERROR :DUMP TO ES ERROR 'any' ERROR :TB Traceback (most recent call last): File "/datapackage_pipelines_budgetkey/processors/dump_to_es.py", line 146, in <module> spew_flow(flow(ctx.parameters), ctx) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ spew(self.datapackage, self.resource_iterator, stats=self.stats) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 68, in spew for res in resources_iterator: File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 68, in <genexpr> res_iter = (it if isinstance(it, ResourceWrapper) else ResourceWrapper(res, it) File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/dumper_base.py", line 82, in process_resources ret = self.process_resource( File "/usr/local/lib/python3.9/site-packages/dataflows_elasticsearch/processors/dump_to_elasticsearch.py", line 69, in process_resource storage.create(index_name, res.descriptor['schema'], File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/storage.py", line 123, in create self.put_mapping(bucket, descriptor, index_name, mapping_generator_cls) File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/storage.py", line 79, in put_mapping mapping = mappers.descriptor_to_mapping( File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 139, in descriptor_to_mapping mapping_gen.generate_from_schema(descriptor) File "/datapackage_pipelines_budgetkey/processors/dump_to_es.py", line 55, in generate_from_schema super(BoostingMappingGenerator, self).generate_from_schema(schema) File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 126, in generate_from_schema self._update_properties(properties, schema) File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 116, in _update_properties dict( File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 117, in <genexpr> cls._convert_field(f, prefix) File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 109, in _convert_field prop = cls._convert_type(schema_type, field, prefix) File "/datapackage_pipelines_budgetkey/processors/dump_to_es.py", line 39, in _convert_type prop = super(BoostingMappingGenerator, cls)._convert_type(schema_type, field, prefix) File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 64, in _convert_type prop = { KeyError: 'any' ERROR :Traceback (most recent call last): ERROR :File "/datapackage_pipelines_budgetkey/processors/dump_to_es.py", line 146, in <module> ERROR :spew_flow(flow(ctx.parameters), ctx) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 68, in spew ERROR :for res in resources_iterator: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 68, in <genexpr> ERROR :res_iter = (it if isinstance(it, ResourceWrapper) else ResourceWrapper(res, it) ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/dumper_base.py", line 82, in process_resources ERROR :ret = self.process_resource( ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows_elasticsearch/processors/dump_to_elasticsearch.py", line 69, in process_resource ERROR :storage.create(index_name, res.descriptor['schema'], ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/storage.py", line 123, in create ERROR :self.put_mapping(bucket, descriptor, index_name, mapping_generator_cls) ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/storage.py", line 79, in put_mapping ERROR :mapping = mappers.descriptor_to_mapping( ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 139, in descriptor_to_mapping ERROR :mapping_gen.generate_from_schema(descriptor) ERROR :File "/datapackage_pipelines_budgetkey/processors/dump_to_es.py", line 55, in generate_from_schema ERROR :super(BoostingMappingGenerator, self).generate_from_schema(schema) ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 126, in generate_from_schema ERROR :self._update_properties(properties, schema) ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 116, in _update_properties ERROR :dict( ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 117, in <genexpr> ERROR :cls._convert_field(f, prefix) ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 109, in _convert_field ERROR :prop = cls._convert_type(schema_type, field, prefix) ERROR :File "/datapackage_pipelines_budgetkey/processors/dump_to_es.py", line 39, in _convert_type ERROR :prop = super(BoostingMappingGenerator, cls)._convert_type(schema_type, field, prefix) ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 64, in _convert_type ERROR :prop = { ERROR :KeyError ERROR :: ERROR :'any'
Pipeline ID: budgetkey/elasticsearch/index_activities
Pipeline ID: budgetkey/elasticsearch/index_budget
Invalid dependency :Cannot run until dependency passes validation: ./procurement/calls_for_bids/calls-for-bids-all From ./procurement/calls_for_bids/calls-for-bids-all :['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/calls_for_bids/jobiz-calls-for-bids'] From ./procurement/calls_for_bids/calls-for-bids-all :['From ./procurement/calls_for_bids/jobiz-calls-for-bids', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/calls_for_bids/jobiz-all']] From ./procurement/calls_for_bids/calls-for-bids-all :['From ./procurement/calls_for_bids/jobiz-calls-for-bids', ['From ./procurement/calls_for_bids/jobiz-all', ['Dependency unsuccessful', 'Cannot run until dependency "./procurement/calls_for_bids/jobiz-scraper" is successfully executed']]]
Pipeline ID: budgetkey/elasticsearch/index_calls_for_bids
Pipeline ID: budgetkey/elasticsearch/index_contract_spending
Pipeline ID: budgetkey/elasticsearch/index_entities
dump_to_es ERROR :DUMP TO ES ERROR ApiError(413, 'None') ERROR :TB Traceback (most recent call last): File "/datapackage_pipelines_budgetkey/processors/dump_to_es.py", line 146, in <module> spew_flow(flow(ctx.parameters), ctx) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ spew(self.datapackage, self.resource_iterator, stats=self.stats) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew for rec in res: File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/dumper_base.py", line 69, in row_counter for row in iterator: File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/storage.py", line 226, in write for result, row in iter: File "/usr/local/lib/python3.9/site-packages/elasticsearch/helpers/actions.py", line 453, in streaming_bulk for data, (ok, info) in zip( File "/usr/local/lib/python3.9/site-packages/elasticsearch/helpers/actions.py", line 359, in _process_bulk_chunk yield from gen File "/usr/local/lib/python3.9/site-packages/elasticsearch/helpers/actions.py", line 293, in _process_bulk_chunk_error raise error File "/usr/local/lib/python3.9/site-packages/elasticsearch/helpers/actions.py", line 343, in _process_bulk_chunk resp = client.bulk(*args, operations=bulk_actions, **kwargs) # type: ignore[arg-type] File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/utils.py", line 452, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/__init__.py", line 827, in bulk return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/_base.py", line 271, in perform_request response = self._perform_request( File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/_base.py", line 351, in _perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(413, 'None') ERROR :Traceback (most recent call last): ERROR :File "/datapackage_pipelines_budgetkey/processors/dump_to_es.py", line 146, in <module> ERROR :spew_flow(flow(ctx.parameters), ctx) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew ERROR :for rec in res: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/dumper_base.py", line 69, in row_counter ERROR :for row in iterator: ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/storage.py", line 226, in write ERROR :for result, row in iter: ERROR :File "/usr/local/lib/python3.9/site-packages/elasticsearch/helpers/actions.py", line 453, in streaming_bulk ERROR :for data, (ok, info) in zip( ERROR :File "/usr/local/lib/python3.9/site-packages/elasticsearch/helpers/actions.py", line 359, in _process_bulk_chunk ERROR :yield from gen ERROR :File "/usr/local/lib/python3.9/site-packages/elasticsearch/helpers/actions.py", line 293, in _process_bulk_chunk_error ERROR :raise error ERROR :File "/usr/local/lib/python3.9/site-packages/elasticsearch/helpers/actions.py", line 343, in _process_bulk_chunk ERROR :resp = client.bulk(*args, operations=bulk_actions, **kwargs) # type: ignore[arg-type] ERROR :File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/utils.py", line 452, in wrapped ERROR :return api(*args, **kwargs) ERROR :File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/__init__.py", line 827, in bulk ERROR :return self.perform_request( # type: ignore[return-value] ERROR :File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/_base.py", line 271, in perform_request ERROR :response = self._perform_request( ERROR :File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/_base.py", line 351, in _perform_request ERROR :raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( ERROR :elasticsearch ERROR :. ERROR :ApiError ERROR :: ERROR :ApiError(413, 'None')
Pipeline ID: budgetkey/elasticsearch/index_gov_decisions
Pipeline ID: budgetkey/elasticsearch/index_muni_budgets
Pipeline ID: budgetkey/elasticsearch/index_muni_tenders
Pipeline ID: budgetkey/elasticsearch/index_national_budget_changes
Invalid dependency :Cannot run until dependency passes validation: ./people/aggregated From ./people/aggregated :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/all'] From ./people/aggregated :['From ./people/all', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/all-appointments']] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/media/themarker/themarker" is successfully executed']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]]] From ./people/aggregated :['From ./people/all', ['Dependency unsuccessful', 'Cannot run until dependency "./people/political_candidates/election_candidates21" is successfully executed']]
Pipeline ID: budgetkey/elasticsearch/index_people
Pipeline ID: budgetkey/elasticsearch/index_reports
Dependency unsuccessful :Cannot run until dependency "./supports/criteria/support-criteria" is successfully executed
Pipeline ID: budgetkey/elasticsearch/index_support_criteria
Pipeline ID: budgetkey/elasticsearch/index_supports
Pipeline ID: budgetkey/elasticsearch/index_tenders
Pipeline ID: budgetkey/elasticsearch/index_units
Dependency unsuccessful :Cannot run until dependency "./budgetkey/elasticsearch/index_budget" is successfully executed Dependency unsuccessful :Cannot run until dependency "./budgetkey/elasticsearch/index_gov_decisions" is successfully executed
Pipeline ID: budgetkey/elasticsearch/sitemaps
Pipeline ID: budgetkey/emails/emails
Pipeline ID: donations/candidates
Pipeline ID: donations/parties
get_transactions Traceback (most recent call last): File "/datapackage_pipelines_budgetkey/pipelines/donations/get_transactions.py", line 150, in <module> spew(dp, process_resources(res_iter)) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew for rec in res: File "/datapackage_pipelines_budgetkey/pipelines/donations/get_transactions.py", line 109, in get_transactions for resp in self.get_for_candidate(cid): File "/datapackage_pipelines_budgetkey/pipelines/donations/get_transactions.py", line 95, in get_for_candidate resp = self.get_for_range(cid, range_start, range_end) File "/datapackage_pipelines_budgetkey/pipelines/donations/get_transactions.py", line 70, in get_for_range assert len(resp) == 6 AssertionError
Pipeline ID: donations/transactions
flow Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 13, in <module> flow = flow_module.flow(parameters, datapackage, resources, ctx.stats) File "/datapackage_pipelines_budgetkey/pipelines/education/tochniyot/scraper.py", line 103, in flow return Flow(scrape(), File "/datapackage_pipelines_budgetkey/pipelines/education/tochniyot/scraper.py", line 65, in scrape education_programs_count = get_education_programs_count() File "/datapackage_pipelines_budgetkey/pipelines/education/tochniyot/scraper.py", line 56, in get_education_programs_count res = send_tochniyot_request() File "/datapackage_pipelines_budgetkey/pipelines/education/tochniyot/scraper.py", line 80, in send_tochniyot_request resp.raise_for_status() File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://apps.education.gov.il/TyhNet/ClientWs/TochnitCh.asmx/IturTochnitChByMeafyenim
Pipeline ID: education/tochniyot/tochniyot
Pipeline ID: entities/all
Pipeline ID: entities/associations/guidestar/guidestar
Pipeline ID: entities/associations/guidestar/guidestar-scraper
רשימת כלל העמותות הפעילות, מתעדכן שבועית.
Pipeline ID: entities/associations/registrar/registry
Pipeline ID: entities/companies/registrar/details
רשימת כלל החברות הפעילות, מתעדכן שבועית.
Pipeline ID: entities/companies/registrar/scraper
רשימת אגודות שיתופיות, מתוך ממשק נתונים של מאגר של משרד הכלכלה / האגף לאיגוד שיתופי, מתעדכן שבועית
Pipeline ID: entities/cooperatives/cooperatives
Pipeline ID: entities/endowments/endowments
Dirty dependency :Cannot run until dependency is executed: ./entities/all
Pipeline ID: entities/entities
Dirty dependency :Cannot run until dependency is executed: ./entities/all
Pipeline ID: entities/fingerprints
Pipeline ID: entities/foi_orgs/foi_orgs
scraper Traceback (most recent call last): File "/datapackage_pipelines_budgetkey/pipelines/entities/moital/scraper.py", line 79, in <module> spew(datapackage, [scrape()]) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew for rec in res: File "/datapackage_pipelines_budgetkey/pipelines/entities/moital/scraper.py", line 32, in scrape desired_capabilities=DesiredCapabilities.PHANTOMJS) AttributeError: type object 'DesiredCapabilities' has no attribute 'PHANTOMJS'
Pipeline ID: entities/moital/moital_service_providers
רשימת אגודות עותומניות, מתוך תשובה לבקשת חופש מידע
Pipeline ID: entities/ottoman/ottoman-association-registry
Pipeline ID: entities/partnerships/partnerships
stream_remote_resources Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn conn = connection.create_connection( File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 72, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/usr/local/lib/python3.9/socket.py", line 954, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -2] Name or service not known During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 714, in urlopen httplib_response = self._make_request( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 415, in _make_request conn.request(method, url, **httplib_request_kw) File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 244, in request super(HTTPConnection, self).request(method, url, body=body, headers=headers) File "/usr/local/lib/python3.9/http/client.py", line 1285, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/local/lib/python3.9/http/client.py", line 1331, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/local/lib/python3.9/http/client.py", line 1280, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/local/lib/python3.9/http/client.py", line 1040, in _send_output self.send(msg) File "/usr/local/lib/python3.9/http/client.py", line 980, in send self.connect() File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 205, in connect conn = self._new_conn() File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fa94d43b6a0>: Failed to establish a new connection: [Errno -2] Name or service not known During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 798, in urlopen retries = retries.increment( File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='employment.molsa.gov.il', port=80): Max retries exceeded with url: /Employment/SafetyAndHealth/Enforcement/Documents/%D7%A6%D7%95%D7%95%D7%99%20%D7%91%D7%98%D7%99%D7%97%D7%95%D7%AA.xlsx (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa94d43b6a0>: Failed to establish a new connection: [Errno -2] Name or service not known')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/tabulator/loaders/remote.py", line 63, in load bytes = _RemoteStream(source, self.__http_session, self.__http_timeout).open() File "/usr/local/lib/python3.9/site-packages/tabulator/loaders/remote.py", line 121, in open self.seek(0) File "/usr/local/lib/python3.9/site-packages/tabulator/loaders/remote.py", line 142, in seek self.__response = self.__session.get(self.__source, stream=True, timeout=self.__timeout) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='employment.molsa.gov.il', port=80): Max retries exceeded with url: /Employment/SafetyAndHealth/Enforcement/Documents/%D7%A6%D7%95%D7%95%D7%99%20%D7%91%D7%98%D7%99%D7%97%D7%95%D7%AA.xlsx (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa94d43b6a0>: Failed to establish a new connection: [Errno -2] Name or service not known')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/stream_remote_resources.py", line 244, in <module> rows = stream_reader(resource, url, ignore_missing or url == "", limit_rows, File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/stream_remote_resources.py", line 199, in stream_reader schema, headers, columns, stream, close = get_opener(url, _resource)() File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/stream_remote_resources.py", line 175, in opener _stream.open() File "/usr/local/lib/python3.9/site-packages/tabulator/stream.py", line 425, in open self.__parser.open(source, encoding=self.__encoding) File "/usr/local/lib/python3.9/site-packages/tabulator/parsers/xlsx.py", line 81, in open source_bytes = self.__loader.load(source, mode="b", encoding=encoding) File "/usr/local/lib/python3.9/site-packages/tabulator/loaders/remote.py", line 72, in load raise exceptions.HTTPError(str(exception)) tabulator.exceptions.HTTPError: HTTPConnectionPool(host='employment.molsa.gov.il', port=80): Max retries exceeded with url: /Employment/SafetyAndHealth/Enforcement/Documents/%D7%A6%D7%95%D7%95%D7%99%20%D7%91%D7%98%D7%99%D7%97%D7%95%D7%AA.xlsx (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa94d43b6a0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
Pipeline ID: entities/safety-orders/safety-orders
Dependency unsuccessful :Cannot run until dependency "./entities/safety-orders/safety-orders" is successfully executed
Pipeline ID: entities/safety-orders/safety-orders-by-entity
Pipeline ID: entities/special/registry
Pipeline ID: ezvonot/ezvonot_accounts
Pipeline ID: ezvonot/ezvonot_cases
Pipeline ID: ezvonot/ezvonot_crontab
Pipeline ID: ezvonot/ezvonot_decisions
Pipeline ID: ezvonot/ezvonot_decisions_join
Pipeline ID: ezvonot/ezvonot_deposits
Pipeline ID: ezvonot/ezvonot_purposes
Dependency unsuccessful :Cannot run until dependency "./facilities/labor/facilities_labor" is successfully executed
Pipeline ID: facilities/all/facilities_all
Pipeline ID: facilities/education/facilities_education
flow Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 466, in _make_request six.raise_from(e, None) File "<string>", line 3, in raise_from File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 461, in _make_request httplib_response = conn.getresponse() File "/usr/local/lib/python3.9/http/client.py", line 1377, in getresponse response.begin() File "/usr/local/lib/python3.9/http/client.py", line 320, in begin version, status, reason = self._read_status() File "/usr/local/lib/python3.9/http/client.py", line 281, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/usr/local/lib/python3.9/socket.py", line 704, in readinto return self._sock.recv_into(b) File "/usr/local/lib/python3.9/ssl.py", line 1242, in recv_into return self.read(nbytes, buffer) File "/usr/local/lib/python3.9/ssl.py", line 1100, in read return self._sslobj.read(len, buffer) socket.timeout: The read operation timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 798, in urlopen retries = retries.increment( File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 550, in increment raise six.reraise(type(error), error, _stacktrace) File "/usr/local/lib/python3.9/site-packages/urllib3/packages/six.py", line 770, in reraise raise value File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 714, in urlopen httplib_response = self._make_request( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 468, in _make_request self._raise_timeout(err=e, url=url, timeout_value=read_timeout) File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 357, in _raise_timeout raise ReadTimeoutError( urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='daycareclasssearch.labor.gov.il', port=443): Read timed out. (read timeout=30) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 13, in <module> flow = flow_module.flow(parameters, datapackage, resources, ctx.stats) File "/datapackage_pipelines_budgetkey/pipelines/facilities/labor/mol.py", line 71, in flow scrape(), File "/datapackage_pipelines_budgetkey/pipelines/facilities/labor/mol.py", line 42, in scrape get_data(), File "/datapackage_pipelines_budgetkey/pipelines/facilities/labor/mol.py", line 25, in get_data response = requests.post(url, headers=headers, json=post_data, timeout=30) File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 115, in post return request("post", url, data=data, json=json, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 532, in send raise ReadTimeout(e, request=request) requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='daycareclasssearch.labor.gov.il', port=443): Read timed out. (read timeout=30)
Pipeline ID: facilities/labor/facilities_labor
Pipeline ID: facilities/welfare/facilities_welfare
Pipeline ID: government_decisions/government_decisions
Pipeline ID: government_decisions/scraper
Pipeline ID: lamas/municipal-data
Pipeline ID: lamas/municipality-to-city
Invalid dependency :Cannot run until dependency passes validation: ./maya/maya-complete-notification-list From ./maya/maya-complete-notification-list :['Dependency unsuccessful', 'Cannot run until dependency "./maya/maya-notification-parse-updates" is successfully executed']
Pipeline ID: maya/maya-bonds-and-options-holdings-changes
Invalid dependency :Cannot run until dependency passes validation: ./maya/maya-complete-notification-list From ./maya/maya-complete-notification-list :['Dependency unsuccessful', 'Cannot run until dependency "./maya/maya-notification-parse-updates" is successfully executed']
Pipeline ID: maya/maya-company-director-appointments
Invalid dependency :Cannot run until dependency passes validation: ./maya/maya-complete-notification-list From ./maya/maya-complete-notification-list :['Dependency unsuccessful', 'Cannot run until dependency "./maya/maya-notification-parse-updates" is successfully executed']
Pipeline ID: maya/maya-company-officer-list
Invalid dependency :Cannot run until dependency passes validation: ./maya/maya-complete-notification-list From ./maya/maya-complete-notification-list :['Dependency unsuccessful', 'Cannot run until dependency "./maya/maya-notification-parse-updates" is successfully executed']
Pipeline ID: maya/maya-company-stakeholder-list
Invalid dependency :Cannot run until dependency passes validation: ./maya/maya-complete-notification-list From ./maya/maya-complete-notification-list :['Dependency unsuccessful', 'Cannot run until dependency "./maya/maya-notification-parse-updates" is successfully executed']
Pipeline ID: maya/maya-company-vip-appointments
Dependency unsuccessful :Cannot run until dependency "./maya/maya-notification-parse-updates" is successfully executed
Pipeline ID: maya/maya-complete-notification-list
Invalid dependency :Cannot run until dependency passes validation: ./maya/maya-complete-notification-list From ./maya/maya-complete-notification-list :['Dependency unsuccessful', 'Cannot run until dependency "./maya/maya-notification-parse-updates" is successfully executed']
Pipeline ID: maya/maya-holdings-changes
flow ERROR :Traceback (most recent call last): ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 15, in <module> ERROR :spew_flow(flow, ctx) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew ERROR :for rec in res: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource ERROR :yield from self.func(resource) ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/maya_notification_parse_updates.py", line 91, in parse_notification ERROR :for row in rows: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource ERROR :yield from self.func(resource) ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/maya_notification_parse_updates.py", line 66, in store_on_s3 ERROR :for row in rows: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource ERROR :yield from self.func(resource) ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/maya_notification_parse_updates.py", line 111, in limiter ERROR :for row in rows: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource ERROR :yield from self.func(resource) ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/maya_notification_parse_updates.py", line 101, in remove_already_parsed ERROR :all_existing_ids = get_all_existing_ids() ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/maya_notification_parse_updates.py", line 32, in get_all_existing_ids ERROR :rows = engine.execute(text("SELECT s3_object_name FROM maya_notifications where parser_version=:v"), v=PARSER_VERSION) ERROR :TypeError ERROR :: ERROR :execute() got an unexpected keyword argument 'v'
Pipeline ID: maya/maya-notification-parse-updates
Invalid dependency :Cannot run until dependency passes validation: ./maya/maya-complete-notification-list From ./maya/maya-complete-notification-list :['Dependency unsuccessful', 'Cannot run until dependency "./maya/maya-notification-parse-updates" is successfully executed']
Pipeline ID: maya/maya-reported-academic-degrees
Invalid dependency :Cannot run until dependency passes validation: ./maya/maya-complete-notification-list From ./maya/maya-complete-notification-list :['Dependency unsuccessful', 'Cannot run until dependency "./maya/maya-notification-parse-updates" is successfully executed']
Pipeline ID: maya/maya-reported-work-record
Dependency unsuccessful :Cannot run until dependency "./maya/scrape-maya-tase-companies" is successfully executed
Pipeline ID: maya/maya-tase-companies-current-management
Pipeline ID: maya/scrape-maya-notification-list
dump_to_path ERROR :Data Package validation error: Descriptor validation error: [] is too short at "schema/fields" in descriptor and at "properties/schema/properties/fields/minItems" in profile Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/dump_to_path.py", line 22, in <module> with ingest() as ctx: File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 186, in ingest params, datapackage, resource_iterator = _ingest(debug=debug) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 44, in _ingest datapackage, resource_iterator, dependency_dp = process_input(sys.stdin, validate, debug) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/input_processor.py", line 88, in process_input datapackage.validate(dp_to_validate) File "/usr/local/lib/python3.9/site-packages/datapackage/validate.py", line 28, in validate Package(descriptor, strict=True) File "/usr/local/lib/python3.9/site-packages/datapackage/package.py", line 126, in __init__ self.__build() File "/usr/local/lib/python3.9/site-packages/datapackage/package.py", line 534, in __build updated_resource = Resource(descriptor, File "/usr/local/lib/python3.9/site-packages/datapackage/resource.py", line 90, in __init__ self.__build() File "/usr/local/lib/python3.9/site-packages/datapackage/resource.py", line 648, in __build raise exception File "/usr/local/lib/python3.9/site-packages/datapackage/resource.py", line 643, in __build self.__profile.validate(self.__current_descriptor) File "/usr/local/lib/python3.9/site-packages/datapackage/profile.py", line 89, in validate raise exceptions.ValidationError(message, errors=errors) datapackage.exceptions.ValidationError: There are 1 validation errors (see exception.errors)
Pipeline ID: maya/scrape-maya-tase-companies
Invalid dependency :Cannot run until dependency passes validation: ./people/all From ./people/all :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/all-appointments'] From ./people/all :['From ./people/company_appointments/all-appointments', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/media/themarker/themarker" is successfully executed']] From ./people/all :['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations']] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]] From ./people/all :['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations']] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]] From ./people/all :['Dependency unsuccessful', 'Cannot run until dependency "./people/political_candidates/election_candidates21" is successfully executed']
Pipeline ID: people/aggregated
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/all-appointments From ./people/company_appointments/all-appointments :['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/media/themarker/themarker" is successfully executed'] From ./people/company_appointments/all-appointments :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations'] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-nominations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]] From ./people/company_appointments/all-appointments :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations'] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-resignations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]] Dependency unsuccessful :Cannot run until dependency "./people/political_candidates/election_candidates21" is successfully executed
Pipeline ID: people/all
Pipeline ID: people/association_founders/association-founders
Dependency unsuccessful :Cannot run until dependency "./people/company_appointments/media/themarker/themarker" is successfully executed Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations From ./people/company_appointments/maya/maya-nominations :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list'] From ./people/company_appointments/maya/maya-nominations :['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']] From ./people/company_appointments/maya/maya-nominations :['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]] Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations From ./people/company_appointments/maya/maya-resignations :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list'] From ./people/company_appointments/maya/maya-resignations :['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']] From ./people/company_appointments/maya/maya-resignations :['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]
Pipeline ID: people/company_appointments/all-appointments
Dependency unsuccessful :Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed
Pipeline ID: people/company_appointments/maya/collect-maya-forms-debug-stats
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations From ./people/company_appointments/maya/maya-nominations :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list'] From ./people/company_appointments/maya/maya-nominations :['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']] From ./people/company_appointments/maya/maya-nominations :['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]] Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations From ./people/company_appointments/maya/maya-resignations :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list'] From ./people/company_appointments/maya/maya-resignations :['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']] From ./people/company_appointments/maya/maya-resignations :['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]
Pipeline ID: people/company_appointments/maya/correlate
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates From ./people/company_appointments/maya/maya-notification-updates :['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']
Pipeline ID: people/company_appointments/maya/maya-list
Name, job and company description.
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list From ./people/company_appointments/maya/maya-list :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates'] From ./people/company_appointments/maya/maya-list :['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]
Pipeline ID: people/company_appointments/maya/maya-nominations
scrape-notification-list Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 971, in json return complexjson.loads(self.text, **kwargs) File "/usr/local/lib/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/datapackage_pipelines_budgetkey/pipelines/people/company_appointments/maya/scrape-notification-list.py", line 55, in _maya_api_call return res.json() File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 975, in json raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/datapackage_pipelines_budgetkey/pipelines/people/company_appointments/maya/scrape-notification-list.py", line 130, in <module> spew(datapackage, [collect()]) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew for rec in res: File "/datapackage_pipelines_budgetkey/pipelines/people/company_appointments/maya/scrape-notification-list.py", line 113, in collect yield from _collect_date_range(year_start, year_end) File "/datapackage_pipelines_budgetkey/pipelines/people/company_appointments/maya/scrape-notification-list.py", line 75, in _collect_date_range res = _maya_api_call(date_from, date_to, current_page) File "/datapackage_pipelines_budgetkey/pipelines/people/company_appointments/maya/scrape-notification-list.py", line 57, in _maya_api_call raise Exception("Failed to Call Maya API for date_from:{} date_to:{} page_num:{}".format(date_from, date_to, page_num)) from e Exception: Failed to Call Maya API for date_from:2006-01-01 date_to:2007-01-01 page_num:0
Pipeline ID: people/company_appointments/maya/maya-notification-list
id, date, company, notification_type, s3_object_name, url
Dependency unsuccessful :Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed
Pipeline ID: people/company_appointments/maya/maya-notification-updates
Name, job and company description.
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list From ./people/company_appointments/maya/maya-list :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates'] From ./people/company_appointments/maya/maya-list :['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]
Pipeline ID: people/company_appointments/maya/maya-resignations
רשימת כל המינויים והחברות מאתר כלכליסט
Pipeline ID: people/company_appointments/media/calcalist/calcalist
Name, job and company description.
stream_remote_resources Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/tabulator/loaders/remote.py", line 63, in load bytes = _RemoteStream(source, self.__http_session, self.__http_timeout).open() File "/usr/local/lib/python3.9/site-packages/tabulator/loaders/remote.py", line 121, in open self.seek(0) File "/usr/local/lib/python3.9/site-packages/tabulator/loaders/remote.py", line 143, in seek self.__response.raise_for_status() File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://www.themarker.com/st/inter/DB/tm/2015/minuyim30.xlsx During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/stream_remote_resources.py", line 244, in <module> rows = stream_reader(resource, url, ignore_missing or url == "", limit_rows, File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/stream_remote_resources.py", line 199, in stream_reader schema, headers, columns, stream, close = get_opener(url, _resource)() File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/stream_remote_resources.py", line 175, in opener _stream.open() File "/usr/local/lib/python3.9/site-packages/tabulator/stream.py", line 425, in open self.__parser.open(source, encoding=self.__encoding) File "/usr/local/lib/python3.9/site-packages/tabulator/parsers/xlsx.py", line 81, in open source_bytes = self.__loader.load(source, mode="b", encoding=encoding) File "/usr/local/lib/python3.9/site-packages/tabulator/loaders/remote.py", line 72, in load raise exceptions.HTTPError(str(exception)) tabulator.exceptions.HTTPError: 404 Client Error: Not Found for url: https://www.themarker.com/st/inter/DB/tm/2015/minuyim30.xlsx
Pipeline ID: people/company_appointments/media/themarker/themarker
dump_to_sql ERROR :Traceback (most recent call last): ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context ERROR :self.dialect.do_execute( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute ERROR :cursor.execute(statement, parameters) ERROR :psycopg2.errors ERROR :. ERROR :InternalError_ ERROR :: ERROR :could not open relation with OID 487689883 ERROR :The above exception was the direct cause of the following exception: ERROR :Traceback (most recent call last): ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/dump_to_sql.py", line 19, in <module> ERROR :spew_flow(flow(ctx.parameters), ctx) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 68, in spew ERROR :for res in resources_iterator: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 68, in <genexpr> ERROR :res_iter = (it if isinstance(it, ResourceWrapper) else ResourceWrapper(res, it) ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/dumper_base.py", line 82, in process_resources ERROR :ret = self.process_resource( ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/to_sql.py", line 108, in process_resource ERROR :storage.delete('') ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_sql/storage.py", line 183, in delete ERROR :self.__reflect() ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_sql/storage.py", line 278, in __reflect ERROR :self.__metadata.reflect(only=only, bind=self.__engine) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 5752, in reflect ERROR :_reflect_info = insp._get_reflection_info( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 2018, in _get_reflection_info ERROR :check_constraints=run( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 1994, in run ERROR :res = meth(filter_names=_fn, **kw) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 1457, in get_multi_check_constraints ERROR :self.dialect.get_multi_check_constraints( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/base.py", line 4677, in get_multi_check_constraints ERROR :result = connection.execute(query, params) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1412, in execute ERROR :return meth( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 483, in _execute_on_connection ERROR :return connection._execute_clauseelement( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1635, in _execute_clauseelement ERROR :ret = self._execute_context( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1844, in _execute_context ERROR :return self._exec_single_context( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1984, in _exec_single_context ERROR :self._handle_dbapi_exception( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2339, in _handle_dbapi_exception ERROR :raise sqlalchemy_exception.with_traceback(exc_info[2]) from e ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context ERROR :self.dialect.do_execute( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute ERROR :cursor.execute(statement, parameters) ERROR :sqlalchemy.exc ERROR :. ERROR :InternalError ERROR :: ERROR :(psycopg2.errors.InternalError_) could not open relation with OID 487689883 [SQL: SELECT pg_catalog.pg_class.relname, pg_catalog.pg_constraint.conname, CASE WHEN (pg_catalog.pg_constraint.oid IS NOT NULL) THEN pg_catalog.pg_get_constraintdef(pg_catalog.pg_constraint.oid, %(pg_get_constraintdef_1)s) END AS anon_1, pg_catalog.pg_description.description FROM pg_catalog.pg_class LEFT OUTER JOIN pg_catalog.pg_constraint ON pg_catalog.pg_class.oid = pg_catalog.pg_constraint.conrelid AND pg_catalog.pg_constraint.contype = %(contype_1)s LEFT OUTER JOIN pg_catalog.pg_description ON pg_catalog.pg_description.objoid = pg_catalog.pg_constraint.oid JOIN pg_catalog.pg_namespace ON pg_catalog.pg_namespace.oid = pg_catalog.pg_class.relnamespace WHERE pg_catalog.pg_class.relkind = ANY (ARRAY[%(param_1)s, %(param_2)s, %(param_3)s]) AND pg_catalog.pg_table_is_visible(pg_catalog.pg_class.oid) AND pg_catalog.pg_namespace.nspname != %(nspname_1)s ORDER BY pg_catalog.pg_class.relname, pg_catalog.pg_constraint.conname] [parameters: {'pg_get_constraintdef_1': True, 'contype_1': 'c', 'param_1': 'r', 'param_2': 'p', 'param_3': 'f', 'nspname_1': 'pg_catalog'}] (Background on this error at: https://sqlalche.me/e/20/2j85)
Pipeline ID: people/company_appointments/state_owned/correlate
Pipeline ID: people/company_appointments/state_owned/parse_pdf_resources
columns: date, url
Pipeline ID: people/company_appointments/state_owned/reports
Pipeline ID: people/company_appointments/state_owned/state-owned-nominations
Pipeline ID: people/municipal_appointments/municipal_representitives
flow Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 79, in _process self.datapackage = self.process_datapackage(self.datapackage) File "/usr/local/lib/python3.9/site-packages/dataflows/processors/set_type.py", line 71, in process_datapackage assert added, 'Failed to find field {} in schema'.format(self.name) AssertionError: Failed to find field re.compile('^party_letter$') in schema The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 15, in <module> spew_flow(flow, ctx) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/utilities/flow_utils.py", line 46, in spew_flow datastream = flow.datastream() File "/usr/local/lib/python3.9/site-packages/dataflows/base/flow.py", line 19, in datastream return self._chain(ds)._process() File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process datastream = self.source._process() File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process datastream = self.source._process() File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process datastream = self.source._process() [Previous line repeated 4 more times] File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 86, in _process self.raise_exception(exception) File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 96, in raise_exception raise error from cause dataflows.base.exceptions.ProcessorError: Errored in processor set_type in position #3: Failed to find field re.compile('^party_letter$') in schema
Pipeline ID: people/political_candidates/election_candidates21
Pipeline ID: people/political_donations/political_donations
Pipeline ID: people/procurement/procurement-individuals
Invalid dependency :Cannot run until dependency passes validation: ./procurement/calls_for_bids/jobiz-calls-for-bids From ./procurement/calls_for_bids/jobiz-calls-for-bids :['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/calls_for_bids/jobiz-all'] From ./procurement/calls_for_bids/jobiz-calls-for-bids :['From ./procurement/calls_for_bids/jobiz-all', ['Dependency unsuccessful', 'Cannot run until dependency "./procurement/calls_for_bids/jobiz-scraper" is successfully executed']]
Pipeline ID: procurement/calls_for_bids/calls-for-bids-all
flow Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 79, in _process self.datapackage = self.process_datapackage(self.datapackage) File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/iterable_loader.py", line 112, in process_datapackage raise self.exc File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/iterable_loader.py", line 60, in describe sample = list(itertools.islice(self.iterable, self.SAMPLE_SIZE)) File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/iterable_loader.py", line 89, in handle_iterable for x in self.iterable: File "/datapackage_pipelines_budgetkey/pipelines/procurement/calls_for_bids/m_tmicha.py", line 47, in m_tmicha_scraper assert total > 0 AssertionError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 15, in <module> spew_flow(flow, ctx) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/utilities/flow_utils.py", line 46, in spew_flow datastream = flow.datastream() File "/usr/local/lib/python3.9/site-packages/dataflows/base/flow.py", line 19, in datastream return self._chain(ds)._process() File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process datastream = self.source._process() File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process datastream = self.source._process() File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 86, in _process self.raise_exception(exception) File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 96, in raise_exception raise error from cause dataflows.base.exceptions.ProcessorError: Errored in processor iterable_loader in position #1:
Pipeline ID: procurement/calls_for_bids/calls-for-bids-moh
Pipeline ID: procurement/calls_for_bids/gov-publications
Pipeline ID: procurement/calls_for_bids/gov-publications-ministry-of-immigration
Pipeline ID: procurement/calls_for_bids/gov-publications2
Dependency unsuccessful :Cannot run until dependency "./procurement/calls_for_bids/jobiz-scraper" is successfully executed
Pipeline ID: procurement/calls_for_bids/jobiz-all
Invalid dependency :Cannot run until dependency passes validation: ./procurement/calls_for_bids/jobiz-all From ./procurement/calls_for_bids/jobiz-all :['Dependency unsuccessful', 'Cannot run until dependency "./procurement/calls_for_bids/jobiz-scraper" is successfully executed']
Pipeline ID: procurement/calls_for_bids/jobiz-calls-for-bids
flow Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn conn = connection.create_connection( File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 72, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/usr/local/lib/python3.9/socket.py", line 954, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -5] No address associated with hostname During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 714, in urlopen httplib_response = self._make_request( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 403, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1053, in _validate_conn conn.connect() File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 363, in connect self.sock = conn = self._new_conn() File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f9a31e4f280>: Failed to establish a new connection: [Errno -5] No address associated with hostname During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 798, in urlopen retries = retries.increment( File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='jobiz.gov.il', port=443): Max retries exceeded with url: /ajax/results/%D7%94%D7%95%D7%93%D7%A2%D7%94%20%D7%A6%D7%99%D7%91%D7%95%D7%A8%D7%99%D7%AA/0?ie=0&typeie=%D7%94%D7%95%D7%93%D7%A2%D7%94+%D7%A6%D7%99%D7%91%D7%95%D7%A8%D7%99%D7%AA&search=Array (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9a31e4f280>: Failed to establish a new connection: [Errno -5] No address associated with hostname')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 79, in _process self.datapackage = self.process_datapackage(self.datapackage) File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/iterable_loader.py", line 112, in process_datapackage raise self.exc File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/iterable_loader.py", line 60, in describe sample = list(itertools.islice(self.iterable, self.SAMPLE_SIZE)) File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/iterable_loader.py", line 89, in handle_iterable for x in self.iterable: File "/datapackage_pipelines_budgetkey/pipelines/procurement/calls_for_bids/jobiz.py", line 19, in fetch_results content = requests.get(URL.format(index)).json() File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 73, in get return request("get", url, params=params, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='jobiz.gov.il', port=443): Max retries exceeded with url: /ajax/results/%D7%94%D7%95%D7%93%D7%A2%D7%94%20%D7%A6%D7%99%D7%91%D7%95%D7%A8%D7%99%D7%AA/0?ie=0&typeie=%D7%94%D7%95%D7%93%D7%A2%D7%94+%D7%A6%D7%99%D7%91%D7%95%D7%A8%D7%99%D7%AA&search=Array (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9a31e4f280>: Failed to establish a new connection: [Errno -5] No address associated with hostname')) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 15, in <module> spew_flow(flow, ctx) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/utilities/flow_utils.py", line 46, in spew_flow datastream = flow.datastream() File "/usr/local/lib/python3.9/site-packages/dataflows/base/flow.py", line 19, in datastream return self._chain(ds)._process() File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process datastream = self.source._process() File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process datastream = self.source._process() File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process datastream = self.source._process() [Previous line repeated 5 more times] File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 86, in _process self.raise_exception(exception) File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 96, in raise_exception raise error from cause dataflows.base.exceptions.ProcessorError: Errored in processor iterable_loader in position #1: HTTPSConnectionPool(host='jobiz.gov.il', port=443): Max retries exceeded with url: /ajax/results/%D7%94%D7%95%D7%93%D7%A2%D7%94%20%D7%A6%D7%99%D7%91%D7%95%D7%A8%D7%99%D7%AA/0?ie=0&typeie=%D7%94%D7%95%D7%93%D7%A2%D7%94+%D7%A6%D7%99%D7%91%D7%95%D7%A8%D7%99%D7%AA&search=Array (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9a31e4f280>: Failed to establish a new connection: [Errno -5] No address associated with hostname'))
Pipeline ID: procurement/calls_for_bids/jobiz-scraper
Pipeline ID: procurement/municipal/fetcher
אנו סוכמים מידע מתוך הדוחות הרבעוניים לתוך טבלה המכילה את המידע העדכני ביותר.
Pipeline ID: procurement/spending/latest-contract-spending
Pipeline ID: procurement/spending/quarterly-contract-spending-report-uris
כל משרדי הממשלה ויחידות הסמך (שמחוברות למערכת מרכבה) צריכים לפרסם אחת לרבעון (תוך 45 יום מסוף הרבעון) דו״ח התקשרויות לפי פורמט מוגדר היטב הממומש כדו״ח מובנה במערכת המרכבה. כל המשרדים מדווחים עצמאית והדו״חות עולים לאתר היחידה לחופש המידע, משם אנו אוספים אותם. התוצר של השרשר הזה הוא פירוט כל הדיווחים של כל המשרדים.
Pipeline ID: procurement/spending/quarterly-contract-spending-reports
Pipeline ID: procurement/spending/quarterly-contract-spending-reports-data
Pipeline ID: procurement/spending/spending-by-entity
Pipeline ID: procurement/tenders/all
dump.to_sql ERROR :Traceback (most recent call last): ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context ERROR :self.dialect.do_execute( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute ERROR :cursor.execute(statement, parameters) ERROR :psycopg2.errors ERROR :. ERROR :InternalError_ ERROR :: ERROR :could not open relation with OID 487689883 ERROR :The above exception was the direct cause of the following exception: ERROR :Traceback (most recent call last): ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/dump/to_sql.py", line 15, in <module> ERROR :spew_flow(flow(ctx.parameters), ctx) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 68, in spew ERROR :for res in resources_iterator: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 68, in <genexpr> ERROR :res_iter = (it if isinstance(it, ResourceWrapper) else ResourceWrapper(res, it) ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/dumper_base.py", line 82, in process_resources ERROR :ret = self.process_resource( ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/to_sql.py", line 108, in process_resource ERROR :storage.delete('') ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_sql/storage.py", line 183, in delete ERROR :self.__reflect() ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_sql/storage.py", line 278, in __reflect ERROR :self.__metadata.reflect(only=only, bind=self.__engine) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 5752, in reflect ERROR :_reflect_info = insp._get_reflection_info( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 2018, in _get_reflection_info ERROR :check_constraints=run( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 1994, in run ERROR :res = meth(filter_names=_fn, **kw) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 1457, in get_multi_check_constraints ERROR :self.dialect.get_multi_check_constraints( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/base.py", line 4677, in get_multi_check_constraints ERROR :result = connection.execute(query, params) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1412, in execute ERROR :return meth( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 483, in _execute_on_connection ERROR :return connection._execute_clauseelement( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1635, in _execute_clauseelement ERROR :ret = self._execute_context( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1844, in _execute_context ERROR :return self._exec_single_context( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1984, in _exec_single_context ERROR :self._handle_dbapi_exception( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2339, in _handle_dbapi_exception ERROR :raise sqlalchemy_exception.with_traceback(exc_info[2]) from e ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context ERROR :self.dialect.do_execute( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute ERROR :cursor.execute(statement, parameters) ERROR :sqlalchemy.exc ERROR :. ERROR :InternalError ERROR :: ERROR :(psycopg2.errors.InternalError_) could not open relation with OID 487689883 [SQL: SELECT pg_catalog.pg_class.relname, pg_catalog.pg_constraint.conname, CASE WHEN (pg_catalog.pg_constraint.oid IS NOT NULL) THEN pg_catalog.pg_get_constraintdef(pg_catalog.pg_constraint.oid, %(pg_get_constraintdef_1)s) END AS anon_1, pg_catalog.pg_description.description FROM pg_catalog.pg_class LEFT OUTER JOIN pg_catalog.pg_constraint ON pg_catalog.pg_class.oid = pg_catalog.pg_constraint.conrelid AND pg_catalog.pg_constraint.contype = %(contype_1)s LEFT OUTER JOIN pg_catalog.pg_description ON pg_catalog.pg_description.objoid = pg_catalog.pg_constraint.oid JOIN pg_catalog.pg_namespace ON pg_catalog.pg_namespace.oid = pg_catalog.pg_class.relnamespace WHERE pg_catalog.pg_class.relkind = ANY (ARRAY[%(param_1)s, %(param_2)s, %(param_3)s]) AND pg_catalog.pg_table_is_visible(pg_catalog.pg_class.oid) AND pg_catalog.pg_namespace.nspname != %(nspname_1)s ORDER BY pg_catalog.pg_class.relname, pg_catalog.pg_constraint.conname] [parameters: {'pg_get_constraintdef_1': True, 'contype_1': 'c', 'param_1': 'r', 'param_2': 'p', 'param_3': 'f', 'nspname_1': 'pg_catalog'}] (Background on this error at: https://sqlalche.me/e/20/2j85)
Pipeline ID: procurement/tenders/processed
Pipeline ID: reports/all
Pipeline ID: reports/ngos/ngo-activity-report
Pipeline ID: reports/ngos/ngo-district-report
Pipeline ID: simpledb/process_simple_dbs
Pipeline ID: supports/all
dump.to_sql ERROR :Traceback (most recent call last): ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context ERROR :self.dialect.do_execute( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute ERROR :cursor.execute(statement, parameters) ERROR :psycopg2.errors ERROR :. ERROR :InternalError_ ERROR :: ERROR :could not open relation with OID 487689883 ERROR :The above exception was the direct cause of the following exception: ERROR :Traceback (most recent call last): ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/dump/to_sql.py", line 15, in <module> ERROR :spew_flow(flow(ctx.parameters), ctx) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 68, in spew ERROR :for res in resources_iterator: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 68, in <genexpr> ERROR :res_iter = (it if isinstance(it, ResourceWrapper) else ResourceWrapper(res, it) ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/dumper_base.py", line 82, in process_resources ERROR :ret = self.process_resource( ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/to_sql.py", line 108, in process_resource ERROR :storage.delete('') ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_sql/storage.py", line 183, in delete ERROR :self.__reflect() ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_sql/storage.py", line 278, in __reflect ERROR :self.__metadata.reflect(only=only, bind=self.__engine) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 5752, in reflect ERROR :_reflect_info = insp._get_reflection_info( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 2018, in _get_reflection_info ERROR :check_constraints=run( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 1994, in run ERROR :res = meth(filter_names=_fn, **kw) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 1457, in get_multi_check_constraints ERROR :self.dialect.get_multi_check_constraints( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/base.py", line 4677, in get_multi_check_constraints ERROR :result = connection.execute(query, params) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1412, in execute ERROR :return meth( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 483, in _execute_on_connection ERROR :return connection._execute_clauseelement( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1635, in _execute_clauseelement ERROR :ret = self._execute_context( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1844, in _execute_context ERROR :return self._exec_single_context( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1984, in _exec_single_context ERROR :self._handle_dbapi_exception( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2339, in _handle_dbapi_exception ERROR :raise sqlalchemy_exception.with_traceback(exc_info[2]) from e ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context ERROR :self.dialect.do_execute( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute ERROR :cursor.execute(statement, parameters) ERROR :sqlalchemy.exc ERROR :. ERROR :InternalError ERROR :: ERROR :(psycopg2.errors.InternalError_) could not open relation with OID 487689883 [SQL: SELECT pg_catalog.pg_class.relname, pg_catalog.pg_constraint.conname, CASE WHEN (pg_catalog.pg_constraint.oid IS NOT NULL) THEN pg_catalog.pg_get_constraintdef(pg_catalog.pg_constraint.oid, %(pg_get_constraintdef_1)s) END AS anon_1, pg_catalog.pg_description.description FROM pg_catalog.pg_class LEFT OUTER JOIN pg_catalog.pg_constraint ON pg_catalog.pg_class.oid = pg_catalog.pg_constraint.conrelid AND pg_catalog.pg_constraint.contype = %(contype_1)s LEFT OUTER JOIN pg_catalog.pg_description ON pg_catalog.pg_description.objoid = pg_catalog.pg_constraint.oid JOIN pg_catalog.pg_namespace ON pg_catalog.pg_namespace.oid = pg_catalog.pg_class.relnamespace WHERE pg_catalog.pg_class.relkind = ANY (ARRAY[%(param_1)s, %(param_2)s, %(param_3)s]) AND pg_catalog.pg_table_is_visible(pg_catalog.pg_class.oid) AND pg_catalog.pg_namespace.nspname != %(nspname_1)s ORDER BY pg_catalog.pg_class.relname, pg_catalog.pg_constraint.conname] [parameters: {'pg_get_constraintdef_1': True, 'contype_1': 'c', 'param_1': 'r', 'param_2': 'p', 'param_3': 'f', 'nspname_1': 'pg_catalog'}] (Background on this error at: https://sqlalche.me/e/20/2j85)
Pipeline ID: supports/by-payment-year
Pipeline ID: supports/by-request-year
sample Traceback (most recent call last): File "/datapackage_pipelines_budgetkey/processors/sample.py", line 11, in <module> res_name = parameters.get('resource', datapackage['resources'][0]['name']) IndexError: list index out of range
Pipeline ID: supports/criteria/support-criteria
Pipeline ID: supports/criteria/support-criteria-doj
Pipeline ID: supports/supports-2004
Pipeline ID: supports/supports-2005
Pipeline ID: supports/supports-2006
Pipeline ID: supports/supports-2007
Pipeline ID: supports/supports-2008
Pipeline ID: supports/supports-2009
Pipeline ID: supports/supports-2010
Pipeline ID: supports/supports-2011
Pipeline ID: supports/supports-2012
Pipeline ID: supports/supports-2013
Pipeline ID: supports/supports-2014
Pipeline ID: supports/supports-2015
Pipeline ID: supports/supports-2016
Pipeline ID: supports/supports-2017
Pipeline ID: supports/supports-2018
Pipeline ID: supports/supports-2019
Pipeline ID: supports/supports-2020
Pipeline ID: supports/supports-2021
Pipeline ID: supports/supports-2022
flow Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 13, in <module> flow = flow_module.flow(parameters, datapackage, resources, ctx.stats) File "/datapackage_pipelines_budgetkey/pipelines/supports/scraper.py", line 180, in flow DF.load(wrapper(year), format='csv', File "/datapackage_pipelines_budgetkey/pipelines/supports/scraper.py", line 21, in wrapper return scraper(gcd, year) File "/datapackage_pipelines_budgetkey/pipelines/supports/scraper.py", line 142, in scraper chart = get_chart(driver) File "/datapackage_pipelines_budgetkey/pipelines/supports/scraper.py", line 34, in get_chart chart = WebDriverWait(driver, 60).until( File "/usr/local/lib/python3.9/site-packages/selenium/webdriver/support/wait.py", line 87, in until raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message: Stacktrace: #0 0x5c6a851a6199 <unknown> WARNING :FAILED to teardown google-chrome
Pipeline ID: supports/supports-2023
Pipeline ID: supports/supports-2024
Pipeline ID: supports/supports-2025
Pipeline ID: supports/supports-by-entity
Pipeline ID: supports/with-entities
Pipeline ID: units/social_services/social_services
Dependency unsuccessful :Cannot run until dependency "./budget/national/changes/processed/transactions" is successfully executed
Pipeline ID: budget/national/changes/processed/national-budget-changes-aggregated
Invalid dependency :Cannot run until dependency passes validation: ./procurement/calls_for_bids/calls-for-bids-all From ./procurement/calls_for_bids/calls-for-bids-all :['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/calls_for_bids/jobiz-calls-for-bids'] From ./procurement/calls_for_bids/calls-for-bids-all :['From ./procurement/calls_for_bids/jobiz-calls-for-bids', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/calls_for_bids/jobiz-all']] From ./procurement/calls_for_bids/calls-for-bids-all :['From ./procurement/calls_for_bids/jobiz-calls-for-bids', ['From ./procurement/calls_for_bids/jobiz-all', ['Dependency unsuccessful', 'Cannot run until dependency "./procurement/calls_for_bids/jobiz-scraper" is successfully executed']]]
Pipeline ID: budgetkey/elasticsearch/index_calls_for_bids
Invalid dependency :Cannot run until dependency passes validation: ./people/aggregated From ./people/aggregated :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/all'] From ./people/aggregated :['From ./people/all', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/all-appointments']] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/media/themarker/themarker" is successfully executed']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]]] From ./people/aggregated :['From ./people/all', ['Dependency unsuccessful', 'Cannot run until dependency "./people/political_candidates/election_candidates21" is successfully executed']]
Pipeline ID: budgetkey/elasticsearch/index_people
Dependency unsuccessful :Cannot run until dependency "./supports/criteria/support-criteria" is successfully executed
Pipeline ID: budgetkey/elasticsearch/index_support_criteria
Dependency unsuccessful :Cannot run until dependency "./budgetkey/elasticsearch/index_budget" is successfully executed Dependency unsuccessful :Cannot run until dependency "./budgetkey/elasticsearch/index_gov_decisions" is successfully executed
Pipeline ID: budgetkey/elasticsearch/sitemaps
Dirty dependency :Cannot run until dependency is executed: ./entities/all
Pipeline ID: entities/entities
Dirty dependency :Cannot run until dependency is executed: ./entities/all
Pipeline ID: entities/fingerprints
Dependency unsuccessful :Cannot run until dependency "./entities/safety-orders/safety-orders" is successfully executed
Pipeline ID: entities/safety-orders/safety-orders-by-entity
Dependency unsuccessful :Cannot run until dependency "./facilities/labor/facilities_labor" is successfully executed
Pipeline ID: facilities/all/facilities_all
Invalid dependency :Cannot run until dependency passes validation: ./maya/maya-complete-notification-list From ./maya/maya-complete-notification-list :['Dependency unsuccessful', 'Cannot run until dependency "./maya/maya-notification-parse-updates" is successfully executed']
Pipeline ID: maya/maya-bonds-and-options-holdings-changes
Invalid dependency :Cannot run until dependency passes validation: ./maya/maya-complete-notification-list From ./maya/maya-complete-notification-list :['Dependency unsuccessful', 'Cannot run until dependency "./maya/maya-notification-parse-updates" is successfully executed']
Pipeline ID: maya/maya-company-director-appointments
Invalid dependency :Cannot run until dependency passes validation: ./maya/maya-complete-notification-list From ./maya/maya-complete-notification-list :['Dependency unsuccessful', 'Cannot run until dependency "./maya/maya-notification-parse-updates" is successfully executed']
Pipeline ID: maya/maya-company-officer-list
Invalid dependency :Cannot run until dependency passes validation: ./maya/maya-complete-notification-list From ./maya/maya-complete-notification-list :['Dependency unsuccessful', 'Cannot run until dependency "./maya/maya-notification-parse-updates" is successfully executed']
Pipeline ID: maya/maya-company-stakeholder-list
Invalid dependency :Cannot run until dependency passes validation: ./maya/maya-complete-notification-list From ./maya/maya-complete-notification-list :['Dependency unsuccessful', 'Cannot run until dependency "./maya/maya-notification-parse-updates" is successfully executed']
Pipeline ID: maya/maya-company-vip-appointments
Dependency unsuccessful :Cannot run until dependency "./maya/maya-notification-parse-updates" is successfully executed
Pipeline ID: maya/maya-complete-notification-list
Invalid dependency :Cannot run until dependency passes validation: ./maya/maya-complete-notification-list From ./maya/maya-complete-notification-list :['Dependency unsuccessful', 'Cannot run until dependency "./maya/maya-notification-parse-updates" is successfully executed']
Pipeline ID: maya/maya-holdings-changes
Invalid dependency :Cannot run until dependency passes validation: ./maya/maya-complete-notification-list From ./maya/maya-complete-notification-list :['Dependency unsuccessful', 'Cannot run until dependency "./maya/maya-notification-parse-updates" is successfully executed']
Pipeline ID: maya/maya-reported-academic-degrees
Invalid dependency :Cannot run until dependency passes validation: ./maya/maya-complete-notification-list From ./maya/maya-complete-notification-list :['Dependency unsuccessful', 'Cannot run until dependency "./maya/maya-notification-parse-updates" is successfully executed']
Pipeline ID: maya/maya-reported-work-record
Dependency unsuccessful :Cannot run until dependency "./maya/scrape-maya-tase-companies" is successfully executed
Pipeline ID: maya/maya-tase-companies-current-management
Invalid dependency :Cannot run until dependency passes validation: ./people/all From ./people/all :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/all-appointments'] From ./people/all :['From ./people/company_appointments/all-appointments', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/media/themarker/themarker" is successfully executed']] From ./people/all :['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations']] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]] From ./people/all :['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations']] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]] From ./people/all :['Dependency unsuccessful', 'Cannot run until dependency "./people/political_candidates/election_candidates21" is successfully executed']
Pipeline ID: people/aggregated
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/all-appointments From ./people/company_appointments/all-appointments :['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/media/themarker/themarker" is successfully executed'] From ./people/company_appointments/all-appointments :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations'] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-nominations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]] From ./people/company_appointments/all-appointments :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations'] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-resignations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]] Dependency unsuccessful :Cannot run until dependency "./people/political_candidates/election_candidates21" is successfully executed
Pipeline ID: people/all
Dependency unsuccessful :Cannot run until dependency "./people/company_appointments/media/themarker/themarker" is successfully executed Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations From ./people/company_appointments/maya/maya-nominations :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list'] From ./people/company_appointments/maya/maya-nominations :['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']] From ./people/company_appointments/maya/maya-nominations :['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]] Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations From ./people/company_appointments/maya/maya-resignations :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list'] From ./people/company_appointments/maya/maya-resignations :['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']] From ./people/company_appointments/maya/maya-resignations :['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]
Pipeline ID: people/company_appointments/all-appointments
Dependency unsuccessful :Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed
Pipeline ID: people/company_appointments/maya/collect-maya-forms-debug-stats
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations From ./people/company_appointments/maya/maya-nominations :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list'] From ./people/company_appointments/maya/maya-nominations :['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']] From ./people/company_appointments/maya/maya-nominations :['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]] Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations From ./people/company_appointments/maya/maya-resignations :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list'] From ./people/company_appointments/maya/maya-resignations :['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']] From ./people/company_appointments/maya/maya-resignations :['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]
Pipeline ID: people/company_appointments/maya/correlate
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates From ./people/company_appointments/maya/maya-notification-updates :['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']
Pipeline ID: people/company_appointments/maya/maya-list
Name, job and company description.
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list From ./people/company_appointments/maya/maya-list :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates'] From ./people/company_appointments/maya/maya-list :['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]
Pipeline ID: people/company_appointments/maya/maya-nominations
id, date, company, notification_type, s3_object_name, url
Dependency unsuccessful :Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed
Pipeline ID: people/company_appointments/maya/maya-notification-updates
Name, job and company description.
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list From ./people/company_appointments/maya/maya-list :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates'] From ./people/company_appointments/maya/maya-list :['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]
Pipeline ID: people/company_appointments/maya/maya-resignations
Invalid dependency :Cannot run until dependency passes validation: ./procurement/calls_for_bids/jobiz-calls-for-bids From ./procurement/calls_for_bids/jobiz-calls-for-bids :['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/calls_for_bids/jobiz-all'] From ./procurement/calls_for_bids/jobiz-calls-for-bids :['From ./procurement/calls_for_bids/jobiz-all', ['Dependency unsuccessful', 'Cannot run until dependency "./procurement/calls_for_bids/jobiz-scraper" is successfully executed']]
Pipeline ID: procurement/calls_for_bids/calls-for-bids-all
Dependency unsuccessful :Cannot run until dependency "./procurement/calls_for_bids/jobiz-scraper" is successfully executed
Pipeline ID: procurement/calls_for_bids/jobiz-all
Invalid dependency :Cannot run until dependency passes validation: ./procurement/calls_for_bids/jobiz-all From ./procurement/calls_for_bids/jobiz-all :['Dependency unsuccessful', 'Cannot run until dependency "./procurement/calls_for_bids/jobiz-scraper" is successfully executed']
Pipeline ID: procurement/calls_for_bids/jobiz-calls-for-bids
This pipeline joins budget items that span across years.
Pipeline ID: budget/national/processed/connected-national-budgets
Pipeline ID: budgetkey/documents
Pipeline ID: simpledb/process_simple_dbs
This pipeline joins the different phases of the budget (allocated, revised and executed). In the original file there's a separate row for each of the phases. We like it better as a single row with all phase info. Another thing this pipeline does is to rename the column titles - to more friendly English names. Also, it create rows for all hierarchies - where upper hierarchies (2, 4 & 6 digits) are plain aggregations of the 8-digit items they contain.
Pipeline ID: budget/national/processed/aggregated-yearly
Pipeline ID: budget/national/processed/just-the-total
Pipeline ID: budget/national/processed/roof-names
Pipeline ID: budgetkey/elasticsearch/index_budget
Pipeline ID: budgetkey/emails/emails
Pipeline ID: supports/by-request-year
extract-change-groups Traceback (most recent call last): File "/datapackage_pipelines_budgetkey/pipelines/budget/national/changes/processed/extract-change-groups.py", line 253, in <module> main() File "/datapackage_pipelines_budgetkey/pipelines/budget/national/changes/processed/extract-change-groups.py", line 250, in main spew(update_datapackage(datapackage), process_resources(res_iter)) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew for rec in res: File "/datapackage_pipelines_budgetkey/pipelines/budget/national/changes/processed/extract-change-groups.py", line 232, in process_resource for transaction in get_transactions(get_changes(rows)): File "/datapackage_pipelines_budgetkey/pipelines/budget/national/changes/processed/extract-change-groups.py", line 193, in assign_transactions changes = changes_by_trcode[trcode] KeyError: '2011/73-009'
Pipeline ID: budget/national/changes/processed/transactions
dump.to_sql ERROR :Traceback (most recent call last): ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context ERROR :self.dialect.do_execute( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute ERROR :cursor.execute(statement, parameters) ERROR :psycopg2.errors ERROR :. ERROR :InternalError_ ERROR :: ERROR :could not open relation with OID 487608360 ERROR :The above exception was the direct cause of the following exception: ERROR :Traceback (most recent call last): ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/dump/to_sql.py", line 15, in <module> ERROR :spew_flow(flow(ctx.parameters), ctx) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 68, in spew ERROR :for res in resources_iterator: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 68, in <genexpr> ERROR :res_iter = (it if isinstance(it, ResourceWrapper) else ResourceWrapper(res, it) ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/dumper_base.py", line 82, in process_resources ERROR :ret = self.process_resource( ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/to_sql.py", line 108, in process_resource ERROR :storage.delete('') ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_sql/storage.py", line 183, in delete ERROR :self.__reflect() ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_sql/storage.py", line 278, in __reflect ERROR :self.__metadata.reflect(only=only, bind=self.__engine) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 5752, in reflect ERROR :_reflect_info = insp._get_reflection_info( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 2018, in _get_reflection_info ERROR :check_constraints=run( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 1994, in run ERROR :res = meth(filter_names=_fn, **kw) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 1457, in get_multi_check_constraints ERROR :self.dialect.get_multi_check_constraints( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/base.py", line 4677, in get_multi_check_constraints ERROR :result = connection.execute(query, params) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1412, in execute ERROR :return meth( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 483, in _execute_on_connection ERROR :return connection._execute_clauseelement( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1635, in _execute_clauseelement ERROR :ret = self._execute_context( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1844, in _execute_context ERROR :return self._exec_single_context( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1984, in _exec_single_context ERROR :self._handle_dbapi_exception( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2339, in _handle_dbapi_exception ERROR :raise sqlalchemy_exception.with_traceback(exc_info[2]) from e ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context ERROR :self.dialect.do_execute( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute ERROR :cursor.execute(statement, parameters) ERROR :sqlalchemy.exc ERROR :. ERROR :InternalError ERROR :: ERROR :(psycopg2.errors.InternalError_) could not open relation with OID 487608360 [SQL: SELECT pg_catalog.pg_class.relname, pg_catalog.pg_constraint.conname, CASE WHEN (pg_catalog.pg_constraint.oid IS NOT NULL) THEN pg_catalog.pg_get_constraintdef(pg_catalog.pg_constraint.oid, %(pg_get_constraintdef_1)s) END AS anon_1, pg_catalog.pg_description.description FROM pg_catalog.pg_class LEFT OUTER JOIN pg_catalog.pg_constraint ON pg_catalog.pg_class.oid = pg_catalog.pg_constraint.conrelid AND pg_catalog.pg_constraint.contype = %(contype_1)s LEFT OUTER JOIN pg_catalog.pg_description ON pg_catalog.pg_description.objoid = pg_catalog.pg_constraint.oid JOIN pg_catalog.pg_namespace ON pg_catalog.pg_namespace.oid = pg_catalog.pg_class.relnamespace WHERE pg_catalog.pg_class.relkind = ANY (ARRAY[%(param_1)s, %(param_2)s, %(param_3)s]) AND pg_catalog.pg_table_is_visible(pg_catalog.pg_class.oid) AND pg_catalog.pg_namespace.nspname != %(nspname_1)s ORDER BY pg_catalog.pg_class.relname, pg_catalog.pg_constraint.conname] [parameters: {'pg_get_constraintdef_1': True, 'contype_1': 'c', 'param_1': 'r', 'param_2': 'p', 'param_3': 'f', 'nspname_1': 'pg_catalog'}] (Background on this error at: https://sqlalche.me/e/20/2j85)
Pipeline ID: budgetkey/analysis/spending/united
dump_to_es ERROR :DUMP TO ES ERROR 'any' ERROR :TB Traceback (most recent call last): File "/datapackage_pipelines_budgetkey/processors/dump_to_es.py", line 146, in <module> spew_flow(flow(ctx.parameters), ctx) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ spew(self.datapackage, self.resource_iterator, stats=self.stats) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 68, in spew for res in resources_iterator: File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 68, in <genexpr> res_iter = (it if isinstance(it, ResourceWrapper) else ResourceWrapper(res, it) File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/dumper_base.py", line 82, in process_resources ret = self.process_resource( File "/usr/local/lib/python3.9/site-packages/dataflows_elasticsearch/processors/dump_to_elasticsearch.py", line 69, in process_resource storage.create(index_name, res.descriptor['schema'], File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/storage.py", line 123, in create self.put_mapping(bucket, descriptor, index_name, mapping_generator_cls) File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/storage.py", line 79, in put_mapping mapping = mappers.descriptor_to_mapping( File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 139, in descriptor_to_mapping mapping_gen.generate_from_schema(descriptor) File "/datapackage_pipelines_budgetkey/processors/dump_to_es.py", line 55, in generate_from_schema super(BoostingMappingGenerator, self).generate_from_schema(schema) File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 126, in generate_from_schema self._update_properties(properties, schema) File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 116, in _update_properties dict( File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 117, in <genexpr> cls._convert_field(f, prefix) File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 109, in _convert_field prop = cls._convert_type(schema_type, field, prefix) File "/datapackage_pipelines_budgetkey/processors/dump_to_es.py", line 39, in _convert_type prop = super(BoostingMappingGenerator, cls)._convert_type(schema_type, field, prefix) File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 64, in _convert_type prop = { KeyError: 'any' ERROR :Traceback (most recent call last): ERROR :File "/datapackage_pipelines_budgetkey/processors/dump_to_es.py", line 146, in <module> ERROR :spew_flow(flow(ctx.parameters), ctx) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 68, in spew ERROR :for res in resources_iterator: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 68, in <genexpr> ERROR :res_iter = (it if isinstance(it, ResourceWrapper) else ResourceWrapper(res, it) ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/dumper_base.py", line 82, in process_resources ERROR :ret = self.process_resource( ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows_elasticsearch/processors/dump_to_elasticsearch.py", line 69, in process_resource ERROR :storage.create(index_name, res.descriptor['schema'], ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/storage.py", line 123, in create ERROR :self.put_mapping(bucket, descriptor, index_name, mapping_generator_cls) ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/storage.py", line 79, in put_mapping ERROR :mapping = mappers.descriptor_to_mapping( ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 139, in descriptor_to_mapping ERROR :mapping_gen.generate_from_schema(descriptor) ERROR :File "/datapackage_pipelines_budgetkey/processors/dump_to_es.py", line 55, in generate_from_schema ERROR :super(BoostingMappingGenerator, self).generate_from_schema(schema) ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 126, in generate_from_schema ERROR :self._update_properties(properties, schema) ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 116, in _update_properties ERROR :dict( ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 117, in <genexpr> ERROR :cls._convert_field(f, prefix) ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 109, in _convert_field ERROR :prop = cls._convert_type(schema_type, field, prefix) ERROR :File "/datapackage_pipelines_budgetkey/processors/dump_to_es.py", line 39, in _convert_type ERROR :prop = super(BoostingMappingGenerator, cls)._convert_type(schema_type, field, prefix) ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/mappers.py", line 64, in _convert_type ERROR :prop = { ERROR :KeyError ERROR :: ERROR :'any'
Pipeline ID: budgetkey/elasticsearch/index_activities
dump_to_es ERROR :DUMP TO ES ERROR ApiError(413, 'None') ERROR :TB Traceback (most recent call last): File "/datapackage_pipelines_budgetkey/processors/dump_to_es.py", line 146, in <module> spew_flow(flow(ctx.parameters), ctx) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ spew(self.datapackage, self.resource_iterator, stats=self.stats) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew for rec in res: File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/dumper_base.py", line 69, in row_counter for row in iterator: File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/storage.py", line 226, in write for result, row in iter: File "/usr/local/lib/python3.9/site-packages/elasticsearch/helpers/actions.py", line 453, in streaming_bulk for data, (ok, info) in zip( File "/usr/local/lib/python3.9/site-packages/elasticsearch/helpers/actions.py", line 359, in _process_bulk_chunk yield from gen File "/usr/local/lib/python3.9/site-packages/elasticsearch/helpers/actions.py", line 293, in _process_bulk_chunk_error raise error File "/usr/local/lib/python3.9/site-packages/elasticsearch/helpers/actions.py", line 343, in _process_bulk_chunk resp = client.bulk(*args, operations=bulk_actions, **kwargs) # type: ignore[arg-type] File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/utils.py", line 452, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/__init__.py", line 827, in bulk return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/_base.py", line 271, in perform_request response = self._perform_request( File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/_base.py", line 351, in _perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(413, 'None') ERROR :Traceback (most recent call last): ERROR :File "/datapackage_pipelines_budgetkey/processors/dump_to_es.py", line 146, in <module> ERROR :spew_flow(flow(ctx.parameters), ctx) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew ERROR :for rec in res: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/dumper_base.py", line 69, in row_counter ERROR :for row in iterator: ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_elasticsearch/storage.py", line 226, in write ERROR :for result, row in iter: ERROR :File "/usr/local/lib/python3.9/site-packages/elasticsearch/helpers/actions.py", line 453, in streaming_bulk ERROR :for data, (ok, info) in zip( ERROR :File "/usr/local/lib/python3.9/site-packages/elasticsearch/helpers/actions.py", line 359, in _process_bulk_chunk ERROR :yield from gen ERROR :File "/usr/local/lib/python3.9/site-packages/elasticsearch/helpers/actions.py", line 293, in _process_bulk_chunk_error ERROR :raise error ERROR :File "/usr/local/lib/python3.9/site-packages/elasticsearch/helpers/actions.py", line 343, in _process_bulk_chunk ERROR :resp = client.bulk(*args, operations=bulk_actions, **kwargs) # type: ignore[arg-type] ERROR :File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/utils.py", line 452, in wrapped ERROR :return api(*args, **kwargs) ERROR :File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/__init__.py", line 827, in bulk ERROR :return self.perform_request( # type: ignore[return-value] ERROR :File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/_base.py", line 271, in perform_request ERROR :response = self._perform_request( ERROR :File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/_base.py", line 351, in _perform_request ERROR :raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( ERROR :elasticsearch ERROR :. ERROR :ApiError ERROR :: ERROR :ApiError(413, 'None')
Pipeline ID: budgetkey/elasticsearch/index_gov_decisions
get_transactions Traceback (most recent call last): File "/datapackage_pipelines_budgetkey/pipelines/donations/get_transactions.py", line 150, in <module> spew(dp, process_resources(res_iter)) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew for rec in res: File "/datapackage_pipelines_budgetkey/pipelines/donations/get_transactions.py", line 109, in get_transactions for resp in self.get_for_candidate(cid): File "/datapackage_pipelines_budgetkey/pipelines/donations/get_transactions.py", line 95, in get_for_candidate resp = self.get_for_range(cid, range_start, range_end) File "/datapackage_pipelines_budgetkey/pipelines/donations/get_transactions.py", line 70, in get_for_range assert len(resp) == 6 AssertionError
Pipeline ID: donations/transactions
flow Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 13, in <module> flow = flow_module.flow(parameters, datapackage, resources, ctx.stats) File "/datapackage_pipelines_budgetkey/pipelines/education/tochniyot/scraper.py", line 103, in flow return Flow(scrape(), File "/datapackage_pipelines_budgetkey/pipelines/education/tochniyot/scraper.py", line 65, in scrape education_programs_count = get_education_programs_count() File "/datapackage_pipelines_budgetkey/pipelines/education/tochniyot/scraper.py", line 56, in get_education_programs_count res = send_tochniyot_request() File "/datapackage_pipelines_budgetkey/pipelines/education/tochniyot/scraper.py", line 80, in send_tochniyot_request resp.raise_for_status() File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://apps.education.gov.il/TyhNet/ClientWs/TochnitCh.asmx/IturTochnitChByMeafyenim
Pipeline ID: education/tochniyot/tochniyot
scraper Traceback (most recent call last): File "/datapackage_pipelines_budgetkey/pipelines/entities/moital/scraper.py", line 79, in <module> spew(datapackage, [scrape()]) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew for rec in res: File "/datapackage_pipelines_budgetkey/pipelines/entities/moital/scraper.py", line 32, in scrape desired_capabilities=DesiredCapabilities.PHANTOMJS) AttributeError: type object 'DesiredCapabilities' has no attribute 'PHANTOMJS'
Pipeline ID: entities/moital/moital_service_providers
stream_remote_resources Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn conn = connection.create_connection( File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 72, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/usr/local/lib/python3.9/socket.py", line 954, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -2] Name or service not known During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 714, in urlopen httplib_response = self._make_request( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 415, in _make_request conn.request(method, url, **httplib_request_kw) File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 244, in request super(HTTPConnection, self).request(method, url, body=body, headers=headers) File "/usr/local/lib/python3.9/http/client.py", line 1285, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/local/lib/python3.9/http/client.py", line 1331, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/local/lib/python3.9/http/client.py", line 1280, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/local/lib/python3.9/http/client.py", line 1040, in _send_output self.send(msg) File "/usr/local/lib/python3.9/http/client.py", line 980, in send self.connect() File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 205, in connect conn = self._new_conn() File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fa94d43b6a0>: Failed to establish a new connection: [Errno -2] Name or service not known During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 798, in urlopen retries = retries.increment( File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='employment.molsa.gov.il', port=80): Max retries exceeded with url: /Employment/SafetyAndHealth/Enforcement/Documents/%D7%A6%D7%95%D7%95%D7%99%20%D7%91%D7%98%D7%99%D7%97%D7%95%D7%AA.xlsx (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa94d43b6a0>: Failed to establish a new connection: [Errno -2] Name or service not known')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/tabulator/loaders/remote.py", line 63, in load bytes = _RemoteStream(source, self.__http_session, self.__http_timeout).open() File "/usr/local/lib/python3.9/site-packages/tabulator/loaders/remote.py", line 121, in open self.seek(0) File "/usr/local/lib/python3.9/site-packages/tabulator/loaders/remote.py", line 142, in seek self.__response = self.__session.get(self.__source, stream=True, timeout=self.__timeout) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 602, in get return self.request("GET", url, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='employment.molsa.gov.il', port=80): Max retries exceeded with url: /Employment/SafetyAndHealth/Enforcement/Documents/%D7%A6%D7%95%D7%95%D7%99%20%D7%91%D7%98%D7%99%D7%97%D7%95%D7%AA.xlsx (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa94d43b6a0>: Failed to establish a new connection: [Errno -2] Name or service not known')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/stream_remote_resources.py", line 244, in <module> rows = stream_reader(resource, url, ignore_missing or url == "", limit_rows, File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/stream_remote_resources.py", line 199, in stream_reader schema, headers, columns, stream, close = get_opener(url, _resource)() File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/stream_remote_resources.py", line 175, in opener _stream.open() File "/usr/local/lib/python3.9/site-packages/tabulator/stream.py", line 425, in open self.__parser.open(source, encoding=self.__encoding) File "/usr/local/lib/python3.9/site-packages/tabulator/parsers/xlsx.py", line 81, in open source_bytes = self.__loader.load(source, mode="b", encoding=encoding) File "/usr/local/lib/python3.9/site-packages/tabulator/loaders/remote.py", line 72, in load raise exceptions.HTTPError(str(exception)) tabulator.exceptions.HTTPError: HTTPConnectionPool(host='employment.molsa.gov.il', port=80): Max retries exceeded with url: /Employment/SafetyAndHealth/Enforcement/Documents/%D7%A6%D7%95%D7%95%D7%99%20%D7%91%D7%98%D7%99%D7%97%D7%95%D7%AA.xlsx (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa94d43b6a0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
Pipeline ID: entities/safety-orders/safety-orders
flow Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 466, in _make_request six.raise_from(e, None) File "<string>", line 3, in raise_from File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 461, in _make_request httplib_response = conn.getresponse() File "/usr/local/lib/python3.9/http/client.py", line 1377, in getresponse response.begin() File "/usr/local/lib/python3.9/http/client.py", line 320, in begin version, status, reason = self._read_status() File "/usr/local/lib/python3.9/http/client.py", line 281, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/usr/local/lib/python3.9/socket.py", line 704, in readinto return self._sock.recv_into(b) File "/usr/local/lib/python3.9/ssl.py", line 1242, in recv_into return self.read(nbytes, buffer) File "/usr/local/lib/python3.9/ssl.py", line 1100, in read return self._sslobj.read(len, buffer) socket.timeout: The read operation timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 798, in urlopen retries = retries.increment( File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 550, in increment raise six.reraise(type(error), error, _stacktrace) File "/usr/local/lib/python3.9/site-packages/urllib3/packages/six.py", line 770, in reraise raise value File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 714, in urlopen httplib_response = self._make_request( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 468, in _make_request self._raise_timeout(err=e, url=url, timeout_value=read_timeout) File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 357, in _raise_timeout raise ReadTimeoutError( urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='daycareclasssearch.labor.gov.il', port=443): Read timed out. (read timeout=30) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 13, in <module> flow = flow_module.flow(parameters, datapackage, resources, ctx.stats) File "/datapackage_pipelines_budgetkey/pipelines/facilities/labor/mol.py", line 71, in flow scrape(), File "/datapackage_pipelines_budgetkey/pipelines/facilities/labor/mol.py", line 42, in scrape get_data(), File "/datapackage_pipelines_budgetkey/pipelines/facilities/labor/mol.py", line 25, in get_data response = requests.post(url, headers=headers, json=post_data, timeout=30) File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 115, in post return request("post", url, data=data, json=json, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 532, in send raise ReadTimeout(e, request=request) requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='daycareclasssearch.labor.gov.il', port=443): Read timed out. (read timeout=30)
Pipeline ID: facilities/labor/facilities_labor
flow ERROR :Traceback (most recent call last): ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 15, in <module> ERROR :spew_flow(flow, ctx) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew ERROR :for rec in res: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource ERROR :yield from self.func(resource) ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/maya_notification_parse_updates.py", line 91, in parse_notification ERROR :for row in rows: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource ERROR :yield from self.func(resource) ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/maya_notification_parse_updates.py", line 66, in store_on_s3 ERROR :for row in rows: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource ERROR :yield from self.func(resource) ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/maya_notification_parse_updates.py", line 111, in limiter ERROR :for row in rows: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource ERROR :yield from self.func(resource) ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/maya_notification_parse_updates.py", line 101, in remove_already_parsed ERROR :all_existing_ids = get_all_existing_ids() ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/maya_notification_parse_updates.py", line 32, in get_all_existing_ids ERROR :rows = engine.execute(text("SELECT s3_object_name FROM maya_notifications where parser_version=:v"), v=PARSER_VERSION) ERROR :TypeError ERROR :: ERROR :execute() got an unexpected keyword argument 'v'
Pipeline ID: maya/maya-notification-parse-updates
dump_to_path ERROR :Data Package validation error: Descriptor validation error: [] is too short at "schema/fields" in descriptor and at "properties/schema/properties/fields/minItems" in profile Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/dump_to_path.py", line 22, in <module> with ingest() as ctx: File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 186, in ingest params, datapackage, resource_iterator = _ingest(debug=debug) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 44, in _ingest datapackage, resource_iterator, dependency_dp = process_input(sys.stdin, validate, debug) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/input_processor.py", line 88, in process_input datapackage.validate(dp_to_validate) File "/usr/local/lib/python3.9/site-packages/datapackage/validate.py", line 28, in validate Package(descriptor, strict=True) File "/usr/local/lib/python3.9/site-packages/datapackage/package.py", line 126, in __init__ self.__build() File "/usr/local/lib/python3.9/site-packages/datapackage/package.py", line 534, in __build updated_resource = Resource(descriptor, File "/usr/local/lib/python3.9/site-packages/datapackage/resource.py", line 90, in __init__ self.__build() File "/usr/local/lib/python3.9/site-packages/datapackage/resource.py", line 648, in __build raise exception File "/usr/local/lib/python3.9/site-packages/datapackage/resource.py", line 643, in __build self.__profile.validate(self.__current_descriptor) File "/usr/local/lib/python3.9/site-packages/datapackage/profile.py", line 89, in validate raise exceptions.ValidationError(message, errors=errors) datapackage.exceptions.ValidationError: There are 1 validation errors (see exception.errors)
Pipeline ID: maya/scrape-maya-tase-companies
scrape-notification-list Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 971, in json return complexjson.loads(self.text, **kwargs) File "/usr/local/lib/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/datapackage_pipelines_budgetkey/pipelines/people/company_appointments/maya/scrape-notification-list.py", line 55, in _maya_api_call return res.json() File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 975, in json raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/datapackage_pipelines_budgetkey/pipelines/people/company_appointments/maya/scrape-notification-list.py", line 130, in <module> spew(datapackage, [collect()]) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew for rec in res: File "/datapackage_pipelines_budgetkey/pipelines/people/company_appointments/maya/scrape-notification-list.py", line 113, in collect yield from _collect_date_range(year_start, year_end) File "/datapackage_pipelines_budgetkey/pipelines/people/company_appointments/maya/scrape-notification-list.py", line 75, in _collect_date_range res = _maya_api_call(date_from, date_to, current_page) File "/datapackage_pipelines_budgetkey/pipelines/people/company_appointments/maya/scrape-notification-list.py", line 57, in _maya_api_call raise Exception("Failed to Call Maya API for date_from:{} date_to:{} page_num:{}".format(date_from, date_to, page_num)) from e Exception: Failed to Call Maya API for date_from:2006-01-01 date_to:2007-01-01 page_num:0
Pipeline ID: people/company_appointments/maya/maya-notification-list
Name, job and company description.
stream_remote_resources Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/tabulator/loaders/remote.py", line 63, in load bytes = _RemoteStream(source, self.__http_session, self.__http_timeout).open() File "/usr/local/lib/python3.9/site-packages/tabulator/loaders/remote.py", line 121, in open self.seek(0) File "/usr/local/lib/python3.9/site-packages/tabulator/loaders/remote.py", line 143, in seek self.__response.raise_for_status() File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://www.themarker.com/st/inter/DB/tm/2015/minuyim30.xlsx During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/stream_remote_resources.py", line 244, in <module> rows = stream_reader(resource, url, ignore_missing or url == "", limit_rows, File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/stream_remote_resources.py", line 199, in stream_reader schema, headers, columns, stream, close = get_opener(url, _resource)() File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/stream_remote_resources.py", line 175, in opener _stream.open() File "/usr/local/lib/python3.9/site-packages/tabulator/stream.py", line 425, in open self.__parser.open(source, encoding=self.__encoding) File "/usr/local/lib/python3.9/site-packages/tabulator/parsers/xlsx.py", line 81, in open source_bytes = self.__loader.load(source, mode="b", encoding=encoding) File "/usr/local/lib/python3.9/site-packages/tabulator/loaders/remote.py", line 72, in load raise exceptions.HTTPError(str(exception)) tabulator.exceptions.HTTPError: 404 Client Error: Not Found for url: https://www.themarker.com/st/inter/DB/tm/2015/minuyim30.xlsx
Pipeline ID: people/company_appointments/media/themarker/themarker
dump_to_sql ERROR :Traceback (most recent call last): ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context ERROR :self.dialect.do_execute( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute ERROR :cursor.execute(statement, parameters) ERROR :psycopg2.errors ERROR :. ERROR :InternalError_ ERROR :: ERROR :could not open relation with OID 487689883 ERROR :The above exception was the direct cause of the following exception: ERROR :Traceback (most recent call last): ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/dump_to_sql.py", line 19, in <module> ERROR :spew_flow(flow(ctx.parameters), ctx) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 68, in spew ERROR :for res in resources_iterator: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 68, in <genexpr> ERROR :res_iter = (it if isinstance(it, ResourceWrapper) else ResourceWrapper(res, it) ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/dumper_base.py", line 82, in process_resources ERROR :ret = self.process_resource( ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/to_sql.py", line 108, in process_resource ERROR :storage.delete('') ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_sql/storage.py", line 183, in delete ERROR :self.__reflect() ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_sql/storage.py", line 278, in __reflect ERROR :self.__metadata.reflect(only=only, bind=self.__engine) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 5752, in reflect ERROR :_reflect_info = insp._get_reflection_info( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 2018, in _get_reflection_info ERROR :check_constraints=run( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 1994, in run ERROR :res = meth(filter_names=_fn, **kw) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 1457, in get_multi_check_constraints ERROR :self.dialect.get_multi_check_constraints( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/base.py", line 4677, in get_multi_check_constraints ERROR :result = connection.execute(query, params) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1412, in execute ERROR :return meth( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 483, in _execute_on_connection ERROR :return connection._execute_clauseelement( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1635, in _execute_clauseelement ERROR :ret = self._execute_context( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1844, in _execute_context ERROR :return self._exec_single_context( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1984, in _exec_single_context ERROR :self._handle_dbapi_exception( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2339, in _handle_dbapi_exception ERROR :raise sqlalchemy_exception.with_traceback(exc_info[2]) from e ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context ERROR :self.dialect.do_execute( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute ERROR :cursor.execute(statement, parameters) ERROR :sqlalchemy.exc ERROR :. ERROR :InternalError ERROR :: ERROR :(psycopg2.errors.InternalError_) could not open relation with OID 487689883 [SQL: SELECT pg_catalog.pg_class.relname, pg_catalog.pg_constraint.conname, CASE WHEN (pg_catalog.pg_constraint.oid IS NOT NULL) THEN pg_catalog.pg_get_constraintdef(pg_catalog.pg_constraint.oid, %(pg_get_constraintdef_1)s) END AS anon_1, pg_catalog.pg_description.description FROM pg_catalog.pg_class LEFT OUTER JOIN pg_catalog.pg_constraint ON pg_catalog.pg_class.oid = pg_catalog.pg_constraint.conrelid AND pg_catalog.pg_constraint.contype = %(contype_1)s LEFT OUTER JOIN pg_catalog.pg_description ON pg_catalog.pg_description.objoid = pg_catalog.pg_constraint.oid JOIN pg_catalog.pg_namespace ON pg_catalog.pg_namespace.oid = pg_catalog.pg_class.relnamespace WHERE pg_catalog.pg_class.relkind = ANY (ARRAY[%(param_1)s, %(param_2)s, %(param_3)s]) AND pg_catalog.pg_table_is_visible(pg_catalog.pg_class.oid) AND pg_catalog.pg_namespace.nspname != %(nspname_1)s ORDER BY pg_catalog.pg_class.relname, pg_catalog.pg_constraint.conname] [parameters: {'pg_get_constraintdef_1': True, 'contype_1': 'c', 'param_1': 'r', 'param_2': 'p', 'param_3': 'f', 'nspname_1': 'pg_catalog'}] (Background on this error at: https://sqlalche.me/e/20/2j85)
Pipeline ID: people/company_appointments/state_owned/correlate
flow Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 79, in _process self.datapackage = self.process_datapackage(self.datapackage) File "/usr/local/lib/python3.9/site-packages/dataflows/processors/set_type.py", line 71, in process_datapackage assert added, 'Failed to find field {} in schema'.format(self.name) AssertionError: Failed to find field re.compile('^party_letter$') in schema The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 15, in <module> spew_flow(flow, ctx) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/utilities/flow_utils.py", line 46, in spew_flow datastream = flow.datastream() File "/usr/local/lib/python3.9/site-packages/dataflows/base/flow.py", line 19, in datastream return self._chain(ds)._process() File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process datastream = self.source._process() File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process datastream = self.source._process() File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process datastream = self.source._process() [Previous line repeated 4 more times] File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 86, in _process self.raise_exception(exception) File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 96, in raise_exception raise error from cause dataflows.base.exceptions.ProcessorError: Errored in processor set_type in position #3: Failed to find field re.compile('^party_letter$') in schema
Pipeline ID: people/political_candidates/election_candidates21
flow Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 79, in _process self.datapackage = self.process_datapackage(self.datapackage) File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/iterable_loader.py", line 112, in process_datapackage raise self.exc File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/iterable_loader.py", line 60, in describe sample = list(itertools.islice(self.iterable, self.SAMPLE_SIZE)) File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/iterable_loader.py", line 89, in handle_iterable for x in self.iterable: File "/datapackage_pipelines_budgetkey/pipelines/procurement/calls_for_bids/m_tmicha.py", line 47, in m_tmicha_scraper assert total > 0 AssertionError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 15, in <module> spew_flow(flow, ctx) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/utilities/flow_utils.py", line 46, in spew_flow datastream = flow.datastream() File "/usr/local/lib/python3.9/site-packages/dataflows/base/flow.py", line 19, in datastream return self._chain(ds)._process() File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process datastream = self.source._process() File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process datastream = self.source._process() File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 86, in _process self.raise_exception(exception) File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 96, in raise_exception raise error from cause dataflows.base.exceptions.ProcessorError: Errored in processor iterable_loader in position #1:
Pipeline ID: procurement/calls_for_bids/calls-for-bids-moh
flow Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn conn = connection.create_connection( File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 72, in create_connection for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): File "/usr/local/lib/python3.9/socket.py", line 954, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -5] No address associated with hostname During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 714, in urlopen httplib_response = self._make_request( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 403, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1053, in _validate_conn conn.connect() File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 363, in connect self.sock = conn = self._new_conn() File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f9a31e4f280>: Failed to establish a new connection: [Errno -5] No address associated with hostname During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 798, in urlopen retries = retries.increment( File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='jobiz.gov.il', port=443): Max retries exceeded with url: /ajax/results/%D7%94%D7%95%D7%93%D7%A2%D7%94%20%D7%A6%D7%99%D7%91%D7%95%D7%A8%D7%99%D7%AA/0?ie=0&typeie=%D7%94%D7%95%D7%93%D7%A2%D7%94+%D7%A6%D7%99%D7%91%D7%95%D7%A8%D7%99%D7%AA&search=Array (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9a31e4f280>: Failed to establish a new connection: [Errno -5] No address associated with hostname')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 79, in _process self.datapackage = self.process_datapackage(self.datapackage) File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/iterable_loader.py", line 112, in process_datapackage raise self.exc File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/iterable_loader.py", line 60, in describe sample = list(itertools.islice(self.iterable, self.SAMPLE_SIZE)) File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/iterable_loader.py", line 89, in handle_iterable for x in self.iterable: File "/datapackage_pipelines_budgetkey/pipelines/procurement/calls_for_bids/jobiz.py", line 19, in fetch_results content = requests.get(URL.format(index)).json() File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 73, in get return request("get", url, params=params, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='jobiz.gov.il', port=443): Max retries exceeded with url: /ajax/results/%D7%94%D7%95%D7%93%D7%A2%D7%94%20%D7%A6%D7%99%D7%91%D7%95%D7%A8%D7%99%D7%AA/0?ie=0&typeie=%D7%94%D7%95%D7%93%D7%A2%D7%94+%D7%A6%D7%99%D7%91%D7%95%D7%A8%D7%99%D7%AA&search=Array (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9a31e4f280>: Failed to establish a new connection: [Errno -5] No address associated with hostname')) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 15, in <module> spew_flow(flow, ctx) File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/utilities/flow_utils.py", line 46, in spew_flow datastream = flow.datastream() File "/usr/local/lib/python3.9/site-packages/dataflows/base/flow.py", line 19, in datastream return self._chain(ds)._process() File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process datastream = self.source._process() File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process datastream = self.source._process() File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process datastream = self.source._process() [Previous line repeated 5 more times] File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 86, in _process self.raise_exception(exception) File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 96, in raise_exception raise error from cause dataflows.base.exceptions.ProcessorError: Errored in processor iterable_loader in position #1: HTTPSConnectionPool(host='jobiz.gov.il', port=443): Max retries exceeded with url: /ajax/results/%D7%94%D7%95%D7%93%D7%A2%D7%94%20%D7%A6%D7%99%D7%91%D7%95%D7%A8%D7%99%D7%AA/0?ie=0&typeie=%D7%94%D7%95%D7%93%D7%A2%D7%94+%D7%A6%D7%99%D7%91%D7%95%D7%A8%D7%99%D7%AA&search=Array (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9a31e4f280>: Failed to establish a new connection: [Errno -5] No address associated with hostname'))
Pipeline ID: procurement/calls_for_bids/jobiz-scraper
dump.to_sql ERROR :Traceback (most recent call last): ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context ERROR :self.dialect.do_execute( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute ERROR :cursor.execute(statement, parameters) ERROR :psycopg2.errors ERROR :. ERROR :InternalError_ ERROR :: ERROR :could not open relation with OID 487689883 ERROR :The above exception was the direct cause of the following exception: ERROR :Traceback (most recent call last): ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/dump/to_sql.py", line 15, in <module> ERROR :spew_flow(flow(ctx.parameters), ctx) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 68, in spew ERROR :for res in resources_iterator: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 68, in <genexpr> ERROR :res_iter = (it if isinstance(it, ResourceWrapper) else ResourceWrapper(res, it) ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/dumper_base.py", line 82, in process_resources ERROR :ret = self.process_resource( ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/to_sql.py", line 108, in process_resource ERROR :storage.delete('') ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_sql/storage.py", line 183, in delete ERROR :self.__reflect() ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_sql/storage.py", line 278, in __reflect ERROR :self.__metadata.reflect(only=only, bind=self.__engine) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 5752, in reflect ERROR :_reflect_info = insp._get_reflection_info( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 2018, in _get_reflection_info ERROR :check_constraints=run( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 1994, in run ERROR :res = meth(filter_names=_fn, **kw) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 1457, in get_multi_check_constraints ERROR :self.dialect.get_multi_check_constraints( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/base.py", line 4677, in get_multi_check_constraints ERROR :result = connection.execute(query, params) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1412, in execute ERROR :return meth( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 483, in _execute_on_connection ERROR :return connection._execute_clauseelement( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1635, in _execute_clauseelement ERROR :ret = self._execute_context( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1844, in _execute_context ERROR :return self._exec_single_context( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1984, in _exec_single_context ERROR :self._handle_dbapi_exception( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2339, in _handle_dbapi_exception ERROR :raise sqlalchemy_exception.with_traceback(exc_info[2]) from e ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context ERROR :self.dialect.do_execute( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute ERROR :cursor.execute(statement, parameters) ERROR :sqlalchemy.exc ERROR :. ERROR :InternalError ERROR :: ERROR :(psycopg2.errors.InternalError_) could not open relation with OID 487689883 [SQL: SELECT pg_catalog.pg_class.relname, pg_catalog.pg_constraint.conname, CASE WHEN (pg_catalog.pg_constraint.oid IS NOT NULL) THEN pg_catalog.pg_get_constraintdef(pg_catalog.pg_constraint.oid, %(pg_get_constraintdef_1)s) END AS anon_1, pg_catalog.pg_description.description FROM pg_catalog.pg_class LEFT OUTER JOIN pg_catalog.pg_constraint ON pg_catalog.pg_class.oid = pg_catalog.pg_constraint.conrelid AND pg_catalog.pg_constraint.contype = %(contype_1)s LEFT OUTER JOIN pg_catalog.pg_description ON pg_catalog.pg_description.objoid = pg_catalog.pg_constraint.oid JOIN pg_catalog.pg_namespace ON pg_catalog.pg_namespace.oid = pg_catalog.pg_class.relnamespace WHERE pg_catalog.pg_class.relkind = ANY (ARRAY[%(param_1)s, %(param_2)s, %(param_3)s]) AND pg_catalog.pg_table_is_visible(pg_catalog.pg_class.oid) AND pg_catalog.pg_namespace.nspname != %(nspname_1)s ORDER BY pg_catalog.pg_class.relname, pg_catalog.pg_constraint.conname] [parameters: {'pg_get_constraintdef_1': True, 'contype_1': 'c', 'param_1': 'r', 'param_2': 'p', 'param_3': 'f', 'nspname_1': 'pg_catalog'}] (Background on this error at: https://sqlalche.me/e/20/2j85)
Pipeline ID: procurement/tenders/processed
dump.to_sql ERROR :Traceback (most recent call last): ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context ERROR :self.dialect.do_execute( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute ERROR :cursor.execute(statement, parameters) ERROR :psycopg2.errors ERROR :. ERROR :InternalError_ ERROR :: ERROR :could not open relation with OID 487689883 ERROR :The above exception was the direct cause of the following exception: ERROR :Traceback (most recent call last): ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/dump/to_sql.py", line 15, in <module> ERROR :spew_flow(flow(ctx.parameters), ctx) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 68, in spew ERROR :for res in resources_iterator: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 68, in <genexpr> ERROR :res_iter = (it if isinstance(it, ResourceWrapper) else ResourceWrapper(res, it) ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/dumper_base.py", line 82, in process_resources ERROR :ret = self.process_resource( ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/to_sql.py", line 108, in process_resource ERROR :storage.delete('') ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_sql/storage.py", line 183, in delete ERROR :self.__reflect() ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_sql/storage.py", line 278, in __reflect ERROR :self.__metadata.reflect(only=only, bind=self.__engine) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/schema.py", line 5752, in reflect ERROR :_reflect_info = insp._get_reflection_info( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 2018, in _get_reflection_info ERROR :check_constraints=run( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 1994, in run ERROR :res = meth(filter_names=_fn, **kw) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/reflection.py", line 1457, in get_multi_check_constraints ERROR :self.dialect.get_multi_check_constraints( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/base.py", line 4677, in get_multi_check_constraints ERROR :result = connection.execute(query, params) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1412, in execute ERROR :return meth( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 483, in _execute_on_connection ERROR :return connection._execute_clauseelement( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1635, in _execute_clauseelement ERROR :ret = self._execute_context( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1844, in _execute_context ERROR :return self._exec_single_context( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1984, in _exec_single_context ERROR :self._handle_dbapi_exception( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2339, in _handle_dbapi_exception ERROR :raise sqlalchemy_exception.with_traceback(exc_info[2]) from e ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context ERROR :self.dialect.do_execute( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute ERROR :cursor.execute(statement, parameters) ERROR :sqlalchemy.exc ERROR :. ERROR :InternalError ERROR :: ERROR :(psycopg2.errors.InternalError_) could not open relation with OID 487689883 [SQL: SELECT pg_catalog.pg_class.relname, pg_catalog.pg_constraint.conname, CASE WHEN (pg_catalog.pg_constraint.oid IS NOT NULL) THEN pg_catalog.pg_get_constraintdef(pg_catalog.pg_constraint.oid, %(pg_get_constraintdef_1)s) END AS anon_1, pg_catalog.pg_description.description FROM pg_catalog.pg_class LEFT OUTER JOIN pg_catalog.pg_constraint ON pg_catalog.pg_class.oid = pg_catalog.pg_constraint.conrelid AND pg_catalog.pg_constraint.contype = %(contype_1)s LEFT OUTER JOIN pg_catalog.pg_description ON pg_catalog.pg_description.objoid = pg_catalog.pg_constraint.oid JOIN pg_catalog.pg_namespace ON pg_catalog.pg_namespace.oid = pg_catalog.pg_class.relnamespace WHERE pg_catalog.pg_class.relkind = ANY (ARRAY[%(param_1)s, %(param_2)s, %(param_3)s]) AND pg_catalog.pg_table_is_visible(pg_catalog.pg_class.oid) AND pg_catalog.pg_namespace.nspname != %(nspname_1)s ORDER BY pg_catalog.pg_class.relname, pg_catalog.pg_constraint.conname] [parameters: {'pg_get_constraintdef_1': True, 'contype_1': 'c', 'param_1': 'r', 'param_2': 'p', 'param_3': 'f', 'nspname_1': 'pg_catalog'}] (Background on this error at: https://sqlalche.me/e/20/2j85)
Pipeline ID: supports/by-payment-year
sample Traceback (most recent call last): File "/datapackage_pipelines_budgetkey/processors/sample.py", line 11, in <module> res_name = parameters.get('resource', datapackage['resources'][0]['name']) IndexError: list index out of range
Pipeline ID: supports/criteria/support-criteria
flow Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 13, in <module> flow = flow_module.flow(parameters, datapackage, resources, ctx.stats) File "/datapackage_pipelines_budgetkey/pipelines/supports/scraper.py", line 180, in flow DF.load(wrapper(year), format='csv', File "/datapackage_pipelines_budgetkey/pipelines/supports/scraper.py", line 21, in wrapper return scraper(gcd, year) File "/datapackage_pipelines_budgetkey/pipelines/supports/scraper.py", line 142, in scraper chart = get_chart(driver) File "/datapackage_pipelines_budgetkey/pipelines/supports/scraper.py", line 34, in get_chart chart = WebDriverWait(driver, 60).until( File "/usr/local/lib/python3.9/site-packages/selenium/webdriver/support/wait.py", line 87, in until raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message: Stacktrace: #0 0x5c6a851a6199 <unknown> WARNING :FAILED to teardown google-chrome
Pipeline ID: supports/supports-2023
Pipeline ID: activities/social_services/social_service_reports
Pipeline ID: activities/social_services/social_service_suppliers
Pipeline ID: activities/social_services/social_service_tenders
Pipeline ID: activities/social_services/social_services
Pipeline ID: bank_of_israel/bank_of_israel
Pipeline ID: budget/municipal/fetcher
Pipeline ID: budget/national/changes/explanations/all
Pipeline ID: budget/national/changes/original/committee-zipfile
Pipeline ID: budget/national/changes/original/current-year-fixes
Pipeline ID: budget/national/changes/original/national-budget-changes
We get the data the Ministry of Finance publishes in data.gov.il every year. It comes in XLS format, with one row per TAKANA and phase (original, approved, executed). In this pipeline we create from each triplet a single row that has all the data.
Pipeline ID: budget/national/original/national-budgets
Pipeline ID: budget/national/processed/category-explanations
This pipeline joins budget items that span across years.
Pipeline ID: budget/national/processed/connected-items-explained
This pipeline joins the budget data to itself so that each item has a list of its immediate children.
Pipeline ID: budget/national/processed/with-extras
Pipeline ID: budgetkey/analysis/spending/publisher_entity_analysis
Pipeline ID: budgetkey/analysis/spending/publisher_foa_analysis
Pipeline ID: budgetkey/budget-functional-aggregates
Pipeline ID: budgetkey/elasticsearch/index_contract_spending
Pipeline ID: budgetkey/elasticsearch/index_entities
Pipeline ID: budgetkey/elasticsearch/index_muni_budgets
Pipeline ID: budgetkey/elasticsearch/index_muni_tenders
Pipeline ID: budgetkey/elasticsearch/index_national_budget_changes
Pipeline ID: budgetkey/elasticsearch/index_reports
Pipeline ID: budgetkey/elasticsearch/index_supports
Pipeline ID: budgetkey/elasticsearch/index_tenders
Pipeline ID: budgetkey/elasticsearch/index_units
Pipeline ID: donations/candidates
Pipeline ID: donations/parties
Pipeline ID: entities/associations/guidestar/guidestar
Pipeline ID: entities/associations/guidestar/guidestar-scraper
רשימת כלל העמותות הפעילות, מתעדכן שבועית.
Pipeline ID: entities/associations/registrar/registry
Pipeline ID: entities/companies/registrar/details
רשימת כלל החברות הפעילות, מתעדכן שבועית.
Pipeline ID: entities/companies/registrar/scraper
רשימת אגודות שיתופיות, מתוך ממשק נתונים של מאגר של משרד הכלכלה / האגף לאיגוד שיתופי, מתעדכן שבועית
Pipeline ID: entities/cooperatives/cooperatives
Pipeline ID: entities/endowments/endowments
Pipeline ID: entities/foi_orgs/foi_orgs
רשימת אגודות עותומניות, מתוך תשובה לבקשת חופש מידע
Pipeline ID: entities/ottoman/ottoman-association-registry
Pipeline ID: entities/partnerships/partnerships
Pipeline ID: entities/special/registry
Pipeline ID: ezvonot/ezvonot_accounts
Pipeline ID: ezvonot/ezvonot_cases
Pipeline ID: ezvonot/ezvonot_crontab
Pipeline ID: ezvonot/ezvonot_decisions
Pipeline ID: ezvonot/ezvonot_decisions_join
Pipeline ID: ezvonot/ezvonot_deposits
Pipeline ID: ezvonot/ezvonot_purposes
Pipeline ID: facilities/education/facilities_education
Pipeline ID: facilities/welfare/facilities_welfare
Pipeline ID: government_decisions/government_decisions
Pipeline ID: government_decisions/scraper
Pipeline ID: lamas/municipal-data
Pipeline ID: lamas/municipality-to-city
Pipeline ID: maya/scrape-maya-notification-list
Pipeline ID: people/association_founders/association-founders
רשימת כל המינויים והחברות מאתר כלכליסט
Pipeline ID: people/company_appointments/media/calcalist/calcalist
Pipeline ID: people/company_appointments/state_owned/parse_pdf_resources
columns: date, url
Pipeline ID: people/company_appointments/state_owned/reports
Pipeline ID: people/company_appointments/state_owned/state-owned-nominations
Pipeline ID: people/municipal_appointments/municipal_representitives
Pipeline ID: people/political_donations/political_donations
Pipeline ID: people/procurement/procurement-individuals
Pipeline ID: procurement/calls_for_bids/gov-publications
Pipeline ID: procurement/calls_for_bids/gov-publications-ministry-of-immigration
Pipeline ID: procurement/calls_for_bids/gov-publications2
Pipeline ID: procurement/municipal/fetcher
אנו סוכמים מידע מתוך הדוחות הרבעוניים לתוך טבלה המכילה את המידע העדכני ביותר.
Pipeline ID: procurement/spending/latest-contract-spending
Pipeline ID: procurement/spending/quarterly-contract-spending-report-uris
כל משרדי הממשלה ויחידות הסמך (שמחוברות למערכת מרכבה) צריכים לפרסם אחת לרבעון (תוך 45 יום מסוף הרבעון) דו״ח התקשרויות לפי פורמט מוגדר היטב הממומש כדו״ח מובנה במערכת המרכבה. כל המשרדים מדווחים עצמאית והדו״חות עולים לאתר היחידה לחופש המידע, משם אנו אוספים אותם. התוצר של השרשר הזה הוא פירוט כל הדיווחים של כל המשרדים.
Pipeline ID: procurement/spending/quarterly-contract-spending-reports
Pipeline ID: procurement/spending/quarterly-contract-spending-reports-data
Pipeline ID: procurement/spending/spending-by-entity
Pipeline ID: procurement/tenders/all
Pipeline ID: reports/all
Pipeline ID: reports/ngos/ngo-activity-report
Pipeline ID: reports/ngos/ngo-district-report
Pipeline ID: supports/all
Pipeline ID: supports/criteria/support-criteria-doj
Pipeline ID: supports/supports-2004
Pipeline ID: supports/supports-2005
Pipeline ID: supports/supports-2006
Pipeline ID: supports/supports-2007
Pipeline ID: supports/supports-2008
Pipeline ID: supports/supports-2009
Pipeline ID: supports/supports-2010
Pipeline ID: supports/supports-2011
Pipeline ID: supports/supports-2012
Pipeline ID: supports/supports-2013
Pipeline ID: supports/supports-2014
Pipeline ID: supports/supports-2015
Pipeline ID: supports/supports-2016
Pipeline ID: supports/supports-2017
Pipeline ID: supports/supports-2018
Pipeline ID: supports/supports-2019
Pipeline ID: supports/supports-2020
Pipeline ID: supports/supports-2021
Pipeline ID: supports/supports-2022
Pipeline ID: supports/supports-2024
Pipeline ID: supports/supports-2025
Pipeline ID: supports/supports-by-entity
Pipeline ID: supports/with-entities
Pipeline ID: units/social_services/social_services