Pipeline ID: budget/municipal/fetcher
explanation_fetcher
Traceback (most recent call last):
File "/datapackage_pipelines_budgetkey/pipelines/budget/national/changes/explanations/explanation_fetcher.py", line 35, in get_explanations
assert resp.status_code < 400
AssertionError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/datapackage_pipelines_budgetkey/pipelines/budget/national/changes/explanations/explanation_fetcher.py", line 82, in <module>
spew(dp, itertools.chain(res_iter, [get_explanations(parameters['url'])]))
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew
for rec in res:
File "/datapackage_pipelines_budgetkey/pipelines/budget/national/changes/explanations/explanation_fetcher.py", line 45, in get_explanations
archive = gcl.download(url)
File "/datapackage_pipelines_budgetkey/common/google_chrome.py", line 165, in download
assert False, 'Failed to download file, %r' % downloads
AssertionError: Failed to download file, ['']
DEBUG :[chan 14] Max packet in: 32768 bytes
DEBUG :[chan 14] Max packet out: 32768 bytes
DEBUG :Secsh channel 14 opened.
DEBUG :[chan 14] Sesch channel 14 request ok
DEBUG :[chan 14] EOF received (14)
DEBUG :[chan 14] EOF sent (14)
DEBUG :Dropping user packet because connection is dead.
Pipeline ID: budget/national/changes/explanations/all
flow
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 13, in <module>
flow = flow_module.flow(parameters, datapackage, resources, ctx.stats)
File "/datapackage_pipelines_budgetkey/pipelines/budget/national/changes/original/committee-zipfile.py", line 14, in flow
archive_ = gcl.download(WEIRD_ZIP_FILE, outfile=f'p{YEAR}.zip')
File "/datapackage_pipelines_budgetkey/common/google_chrome.py", line 165, in download
assert False, 'Failed to download file, %r' % downloads
AssertionError: Failed to download file, ['']
DEBUG :[chan 14] Max packet in: 32768 bytes
DEBUG :[chan 14] Max packet out: 32768 bytes
DEBUG :Secsh channel 14 opened.
DEBUG :[chan 14] Sesch channel 14 request ok
DEBUG :[chan 14] EOF received (14)
DEBUG :[chan 14] EOF sent (14)
DEBUG :Dropping user packet because connection is dead.
Pipeline ID: budget/national/changes/original/committee-zipfile
Pipeline ID: budget/national/changes/original/current-year-fixes
Pipeline ID: budget/national/changes/original/national-budget-changes
Dependency unsuccessful :Cannot run until dependency "./budget/national/changes/explanations/all" is successfully executed
Pipeline ID: budget/national/changes/processed/national-budget-changes-aggregated
Pipeline ID: budget/national/changes/processed/transactions
We get the data the Ministry of Finance publishes in data.gov.il every year. It comes in XLS format, with one row per TAKANA and phase (original, approved, executed). In this pipeline we create from each triplet a single row that has all the data.
Pipeline ID: budget/national/original/national-budgets
This pipeline joins the different phases of the budget (allocated, revised and executed). In the original file there's a separate row for each of the phases. We like it better as a single row with all phase info. Another thing this pipeline does is to rename the column titles - to more friendly English names. Also, it create rows for all hierarchies - where upper hierarchies (2, 4 & 6 digits) are plain aggregations of the 8-digit items they contain.
Pipeline ID: budget/national/processed/aggregated-yearly
Pipeline ID: budget/national/processed/category-explanations
This pipeline joins budget items that span across years.
Pipeline ID: budget/national/processed/connected-items-explained
This pipeline joins budget items that span across years.
Pipeline ID: budget/national/processed/connected-national-budgets
Pipeline ID: budget/national/processed/just-the-total
Pipeline ID: budget/national/processed/roof-names
This pipeline joins the budget data to itself so that each item has a list of its immediate children.
Pipeline ID: budget/national/processed/with-extras
Pipeline ID: budgetkey/analysis/spending/publisher_entity_analysis
Pipeline ID: budgetkey/analysis/spending/publisher_foa_analysis
Pipeline ID: budgetkey/analysis/spending/united
Pipeline ID: budgetkey/budget-functional-aggregates
Pipeline ID: budgetkey/documents
Pipeline ID: budgetkey/elasticsearch/index_activities
dump.to_sql ERROR :Traceback (most recent call last): ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/dump/to_sql.py", line 15, in <module> ERROR :spew_flow(flow(ctx.parameters), ctx) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew ERROR :for rec in res: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/dumper_base.py", line 69, in row_counter ERROR :for row in iterator: ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_sql/writer.py", line 49, in write ERROR :ret = self.__update(connection, keyed_row) ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_sql/writer.py", line 100, in __update ERROR :res = connection.execute(expr) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1412, in execute ERROR :return meth( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 483, in _execute_on_connection ERROR :return connection._execute_clauseelement( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1627, in _execute_clauseelement ERROR :compiled_sql, extracted_params, cache_hit = elem._compile_w_cache( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 671, in _compile_w_cache ERROR :compiled_sql = self._compiler( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 288, in _compiler ERROR :return dialect.statement_compiler(dialect, self, **kw) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1425, in __init__ ERROR :Compiled.__init__(self, dialect, statement, **kwargs) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 866, in __init__ ERROR :self.string = self.process(self.statement, **compile_kwargs) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 911, in process ERROR :return obj._compiler_dispatch(self, **kwargs) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 143, in _compiler_dispatch ERROR :return meth(self, **kw) # type: ignore # noqa: E501 ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 6040, in visit_update ERROR :crud_params_struct = crud._get_crud_params( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/crud.py", line 335, in _get_crud_params ERROR :raise exc.CompileError( ERROR :sqlalchemy.exc ERROR :. ERROR :CompileError ERROR :: ERROR :Unconsumed column names: code_level6, code_level4, title_level4, code_level2, code_level8, title_level2, title_level6, budget_kind_title, title_level8, budget_kind_code
Pipeline ID: budgetkey/elasticsearch/index_budget
Invalid dependency :Cannot run until dependency passes validation: ./procurement/calls_for_bids/calls-for-bids-all From ./procurement/calls_for_bids/calls-for-bids-all :['Dependency unsuccessful', 'Cannot run until dependency "./procurement/calls_for_bids/gov-publications" is successfully executed'] From ./procurement/calls_for_bids/calls-for-bids-all :['Dependency unsuccessful', 'Cannot run until dependency "./procurement/calls_for_bids/gov-publications2" is successfully executed'] From ./procurement/calls_for_bids/calls-for-bids-all :['Dependency unsuccessful', 'Cannot run until dependency "./procurement/calls_for_bids/gov-publications-ministry-of-immigration" is successfully executed']
Pipeline ID: budgetkey/elasticsearch/index_calls_for_bids
Invalid dependency :Cannot run until dependency passes validation: ./procurement/spending/latest-contract-spending From ./procurement/spending/latest-contract-spending :['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports-data'] From ./procurement/spending/latest-contract-spending :['From ./procurement/spending/quarterly-contract-spending-reports-data', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports']] From ./procurement/spending/latest-contract-spending :['From ./procurement/spending/quarterly-contract-spending-reports-data', ['From ./procurement/spending/quarterly-contract-spending-reports', ['Dependency unsuccessful', 'Cannot run until dependency "./procurement/spending/quarterly-contract-spending-report-uris" is successfully executed']]]
Pipeline ID: budgetkey/elasticsearch/index_contract_spending
Invalid dependency :Cannot run until dependency passes validation: ./entities/entities From ./entities/entities :['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/spending-by-entity'] From ./entities/entities :['From ./procurement/spending/spending-by-entity', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/latest-contract-spending']] From ./entities/entities :['From ./procurement/spending/spending-by-entity', ['From ./procurement/spending/latest-contract-spending', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports-data']]] From ./entities/entities :['From ./procurement/spending/spending-by-entity', ['From ./procurement/spending/latest-contract-spending', ['From ./procurement/spending/quarterly-contract-spending-reports-data', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports']]]] From ./entities/entities :['From ./procurement/spending/spending-by-entity', ['From ./procurement/spending/latest-contract-spending', ['From ./procurement/spending/quarterly-contract-spending-reports-data', ['From ./procurement/spending/quarterly-contract-spending-reports', ['Dependency unsuccessful', 'Cannot run until dependency "./procurement/spending/quarterly-contract-spending-report-uris" is successfully executed']]]]] From ./entities/entities :['Invalid dependency', 'Cannot run until dependency passes validation: ./entities/all'] From ./entities/entities :['From ./entities/all', ['Dependency unsuccessful', 'Cannot run until dependency "./entities/ottoman/ottoman-association-registry" is successfully executed']]
Pipeline ID: budgetkey/elasticsearch/index_entities
Invalid dependency :Cannot run until dependency passes validation: ./government_decisions/government_decisions From ./government_decisions/government_decisions :['Dependency unsuccessful', 'Cannot run until dependency "./government_decisions/scraper" is successfully executed']
Pipeline ID: budgetkey/elasticsearch/index_gov_decisions
Pipeline ID: budgetkey/elasticsearch/index_muni_budgets
Pipeline ID: budgetkey/elasticsearch/index_muni_tenders
Invalid dependency :Cannot run until dependency passes validation: ./budget/national/changes/processed/national-budget-changes-aggregated From ./budget/national/changes/processed/national-budget-changes-aggregated :['Dependency unsuccessful', 'Cannot run until dependency "./budget/national/changes/explanations/all" is successfully executed'] From ./budget/national/changes/processed/national-budget-changes-aggregated :['Dependency unsuccessful', 'Cannot run until dependency "./budget/national/changes/processed/transactions" is successfully executed']
Pipeline ID: budgetkey/elasticsearch/index_national_budget_changes
Invalid dependency :Cannot run until dependency passes validation: ./people/aggregated From ./people/aggregated :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/all'] From ./people/aggregated :['From ./people/all', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/all-appointments']] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/media/themarker/themarker" is successfully executed']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]]] From ./people/aggregated :['From ./people/all', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/procurement/procurement-individuals']] From ./people/aggregated :['From ./people/all', ['From ./people/procurement/procurement-individuals', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/latest-contract-spending']]] From ./people/aggregated :['From ./people/all', ['From ./people/procurement/procurement-individuals', ['From ./procurement/spending/latest-contract-spending', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports-data']]]] From ./people/aggregated :['From ./people/all', ['From ./people/procurement/procurement-individuals', ['From ./procurement/spending/latest-contract-spending', ['From ./procurement/spending/quarterly-contract-spending-reports-data', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports']]]]] From ./people/aggregated :['From ./people/all', ['From ./people/procurement/procurement-individuals', ['From ./procurement/spending/latest-contract-spending', ['From ./procurement/spending/quarterly-contract-spending-reports-data', ['From ./procurement/spending/quarterly-contract-spending-reports', ['Dependency unsuccessful', 'Cannot run until dependency "./procurement/spending/quarterly-contract-spending-report-uris" is successfully executed']]]]]] From ./people/aggregated :['From ./people/all', ['Dependency unsuccessful', 'Cannot run until dependency "./people/political_candidates/election_candidates21" is successfully executed']]
Pipeline ID: budgetkey/elasticsearch/index_people
Pipeline ID: budgetkey/elasticsearch/index_reports
Dependency unsuccessful :Cannot run until dependency "./supports/criteria/support-criteria" is successfully executed
Pipeline ID: budgetkey/elasticsearch/index_support_criteria
Pipeline ID: budgetkey/elasticsearch/index_support_programs
Pipeline ID: budgetkey/elasticsearch/index_supports
Pipeline ID: budgetkey/elasticsearch/index_tenders
Pipeline ID: budgetkey/elasticsearch/index_units
Dependency unsuccessful :Cannot run until dependency "./budgetkey/elasticsearch/index_budget" is successfully executed Invalid dependency :Cannot run until dependency passes validation: ./budgetkey/elasticsearch/index_national_budget_changes From ./budgetkey/elasticsearch/index_national_budget_changes :['Invalid dependency', 'Cannot run until dependency passes validation: ./budget/national/changes/processed/national-budget-changes-aggregated'] From ./budgetkey/elasticsearch/index_national_budget_changes :['From ./budget/national/changes/processed/national-budget-changes-aggregated', ['Dependency unsuccessful', 'Cannot run until dependency "./budget/national/changes/explanations/all" is successfully executed']] From ./budgetkey/elasticsearch/index_national_budget_changes :['From ./budget/national/changes/processed/national-budget-changes-aggregated', ['Dependency unsuccessful', 'Cannot run until dependency "./budget/national/changes/processed/transactions" is successfully executed']] Invalid dependency :Cannot run until dependency passes validation: ./budgetkey/elasticsearch/index_entities From ./budgetkey/elasticsearch/index_entities :['Invalid dependency', 'Cannot run until dependency passes validation: ./entities/entities'] From ./budgetkey/elasticsearch/index_entities :['From ./entities/entities', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/spending-by-entity']] From ./budgetkey/elasticsearch/index_entities :['From ./entities/entities', ['From ./procurement/spending/spending-by-entity', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/latest-contract-spending']]] From ./budgetkey/elasticsearch/index_entities :['From ./entities/entities', ['From ./procurement/spending/spending-by-entity', ['From ./procurement/spending/latest-contract-spending', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports-data']]]] From ./budgetkey/elasticsearch/index_entities :['From ./entities/entities', ['From ./procurement/spending/spending-by-entity', ['From ./procurement/spending/latest-contract-spending', ['From ./procurement/spending/quarterly-contract-spending-reports-data', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports']]]]] From ./budgetkey/elasticsearch/index_entities :['From ./entities/entities', ['From ./procurement/spending/spending-by-entity', ['From ./procurement/spending/latest-contract-spending', ['From ./procurement/spending/quarterly-contract-spending-reports-data', ['From ./procurement/spending/quarterly-contract-spending-reports', ['Dependency unsuccessful', 'Cannot run until dependency "./procurement/spending/quarterly-contract-spending-report-uris" is successfully executed']]]]]] From ./budgetkey/elasticsearch/index_entities :['From ./entities/entities', ['Invalid dependency', 'Cannot run until dependency passes validation: ./entities/all']] From ./budgetkey/elasticsearch/index_entities :['From ./entities/entities', ['From ./entities/all', ['Dependency unsuccessful', 'Cannot run until dependency "./entities/ottoman/ottoman-association-registry" is successfully executed']]] Invalid dependency :Cannot run until dependency passes validation: ./budgetkey/elasticsearch/index_contract_spending From ./budgetkey/elasticsearch/index_contract_spending :['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/latest-contract-spending'] From ./budgetkey/elasticsearch/index_contract_spending :['From ./procurement/spending/latest-contract-spending', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports-data']] From ./budgetkey/elasticsearch/index_contract_spending :['From ./procurement/spending/latest-contract-spending', ['From ./procurement/spending/quarterly-contract-spending-reports-data', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports']]] From ./budgetkey/elasticsearch/index_contract_spending :['From ./procurement/spending/latest-contract-spending', ['From ./procurement/spending/quarterly-contract-spending-reports-data', ['From ./procurement/spending/quarterly-contract-spending-reports', ['Dependency unsuccessful', 'Cannot run until dependency "./procurement/spending/quarterly-contract-spending-report-uris" is successfully executed']]]] Invalid dependency :Cannot run until dependency passes validation: ./budgetkey/elasticsearch/index_gov_decisions From ./budgetkey/elasticsearch/index_gov_decisions :['Invalid dependency', 'Cannot run until dependency passes validation: ./government_decisions/government_decisions'] From ./budgetkey/elasticsearch/index_gov_decisions :['From ./government_decisions/government_decisions', ['Dependency unsuccessful', 'Cannot run until dependency "./government_decisions/scraper" is successfully executed']]
Pipeline ID: budgetkey/elasticsearch/sitemaps
Pipeline ID: budgetkey/emails/emails
Dependency unsuccessful :Cannot run until dependency "./budget/national/changes/explanations/all" is successfully executed
Pipeline ID: budget/national/changes/processed/national-budget-changes-aggregated
Invalid dependency :Cannot run until dependency passes validation: ./procurement/calls_for_bids/calls-for-bids-all From ./procurement/calls_for_bids/calls-for-bids-all :['Dependency unsuccessful', 'Cannot run until dependency "./procurement/calls_for_bids/gov-publications" is successfully executed'] From ./procurement/calls_for_bids/calls-for-bids-all :['Dependency unsuccessful', 'Cannot run until dependency "./procurement/calls_for_bids/gov-publications2" is successfully executed'] From ./procurement/calls_for_bids/calls-for-bids-all :['Dependency unsuccessful', 'Cannot run until dependency "./procurement/calls_for_bids/gov-publications-ministry-of-immigration" is successfully executed']
Pipeline ID: budgetkey/elasticsearch/index_calls_for_bids
Invalid dependency :Cannot run until dependency passes validation: ./procurement/spending/latest-contract-spending From ./procurement/spending/latest-contract-spending :['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports-data'] From ./procurement/spending/latest-contract-spending :['From ./procurement/spending/quarterly-contract-spending-reports-data', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports']] From ./procurement/spending/latest-contract-spending :['From ./procurement/spending/quarterly-contract-spending-reports-data', ['From ./procurement/spending/quarterly-contract-spending-reports', ['Dependency unsuccessful', 'Cannot run until dependency "./procurement/spending/quarterly-contract-spending-report-uris" is successfully executed']]]
Pipeline ID: budgetkey/elasticsearch/index_contract_spending
Invalid dependency :Cannot run until dependency passes validation: ./entities/entities From ./entities/entities :['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/spending-by-entity'] From ./entities/entities :['From ./procurement/spending/spending-by-entity', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/latest-contract-spending']] From ./entities/entities :['From ./procurement/spending/spending-by-entity', ['From ./procurement/spending/latest-contract-spending', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports-data']]] From ./entities/entities :['From ./procurement/spending/spending-by-entity', ['From ./procurement/spending/latest-contract-spending', ['From ./procurement/spending/quarterly-contract-spending-reports-data', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports']]]] From ./entities/entities :['From ./procurement/spending/spending-by-entity', ['From ./procurement/spending/latest-contract-spending', ['From ./procurement/spending/quarterly-contract-spending-reports-data', ['From ./procurement/spending/quarterly-contract-spending-reports', ['Dependency unsuccessful', 'Cannot run until dependency "./procurement/spending/quarterly-contract-spending-report-uris" is successfully executed']]]]] From ./entities/entities :['Invalid dependency', 'Cannot run until dependency passes validation: ./entities/all'] From ./entities/entities :['From ./entities/all', ['Dependency unsuccessful', 'Cannot run until dependency "./entities/ottoman/ottoman-association-registry" is successfully executed']]
Pipeline ID: budgetkey/elasticsearch/index_entities
Invalid dependency :Cannot run until dependency passes validation: ./government_decisions/government_decisions From ./government_decisions/government_decisions :['Dependency unsuccessful', 'Cannot run until dependency "./government_decisions/scraper" is successfully executed']
Pipeline ID: budgetkey/elasticsearch/index_gov_decisions
Invalid dependency :Cannot run until dependency passes validation: ./budget/national/changes/processed/national-budget-changes-aggregated From ./budget/national/changes/processed/national-budget-changes-aggregated :['Dependency unsuccessful', 'Cannot run until dependency "./budget/national/changes/explanations/all" is successfully executed'] From ./budget/national/changes/processed/national-budget-changes-aggregated :['Dependency unsuccessful', 'Cannot run until dependency "./budget/national/changes/processed/transactions" is successfully executed']
Pipeline ID: budgetkey/elasticsearch/index_national_budget_changes
Invalid dependency :Cannot run until dependency passes validation: ./people/aggregated From ./people/aggregated :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/all'] From ./people/aggregated :['From ./people/all', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/all-appointments']] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/media/themarker/themarker" is successfully executed']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]]] From ./people/aggregated :['From ./people/all', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/procurement/procurement-individuals']] From ./people/aggregated :['From ./people/all', ['From ./people/procurement/procurement-individuals', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/latest-contract-spending']]] From ./people/aggregated :['From ./people/all', ['From ./people/procurement/procurement-individuals', ['From ./procurement/spending/latest-contract-spending', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports-data']]]] From ./people/aggregated :['From ./people/all', ['From ./people/procurement/procurement-individuals', ['From ./procurement/spending/latest-contract-spending', ['From ./procurement/spending/quarterly-contract-spending-reports-data', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports']]]]] From ./people/aggregated :['From ./people/all', ['From ./people/procurement/procurement-individuals', ['From ./procurement/spending/latest-contract-spending', ['From ./procurement/spending/quarterly-contract-spending-reports-data', ['From ./procurement/spending/quarterly-contract-spending-reports', ['Dependency unsuccessful', 'Cannot run until dependency "./procurement/spending/quarterly-contract-spending-report-uris" is successfully executed']]]]]] From ./people/aggregated :['From ./people/all', ['Dependency unsuccessful', 'Cannot run until dependency "./people/political_candidates/election_candidates21" is successfully executed']]
Pipeline ID: budgetkey/elasticsearch/index_people
Dependency unsuccessful :Cannot run until dependency "./supports/criteria/support-criteria" is successfully executed
Pipeline ID: budgetkey/elasticsearch/index_support_criteria
Dependency unsuccessful :Cannot run until dependency "./budgetkey/elasticsearch/index_budget" is successfully executed Invalid dependency :Cannot run until dependency passes validation: ./budgetkey/elasticsearch/index_national_budget_changes From ./budgetkey/elasticsearch/index_national_budget_changes :['Invalid dependency', 'Cannot run until dependency passes validation: ./budget/national/changes/processed/national-budget-changes-aggregated'] From ./budgetkey/elasticsearch/index_national_budget_changes :['From ./budget/national/changes/processed/national-budget-changes-aggregated', ['Dependency unsuccessful', 'Cannot run until dependency "./budget/national/changes/explanations/all" is successfully executed']] From ./budgetkey/elasticsearch/index_national_budget_changes :['From ./budget/national/changes/processed/national-budget-changes-aggregated', ['Dependency unsuccessful', 'Cannot run until dependency "./budget/national/changes/processed/transactions" is successfully executed']] Invalid dependency :Cannot run until dependency passes validation: ./budgetkey/elasticsearch/index_entities From ./budgetkey/elasticsearch/index_entities :['Invalid dependency', 'Cannot run until dependency passes validation: ./entities/entities'] From ./budgetkey/elasticsearch/index_entities :['From ./entities/entities', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/spending-by-entity']] From ./budgetkey/elasticsearch/index_entities :['From ./entities/entities', ['From ./procurement/spending/spending-by-entity', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/latest-contract-spending']]] From ./budgetkey/elasticsearch/index_entities :['From ./entities/entities', ['From ./procurement/spending/spending-by-entity', ['From ./procurement/spending/latest-contract-spending', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports-data']]]] From ./budgetkey/elasticsearch/index_entities :['From ./entities/entities', ['From ./procurement/spending/spending-by-entity', ['From ./procurement/spending/latest-contract-spending', ['From ./procurement/spending/quarterly-contract-spending-reports-data', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports']]]]] From ./budgetkey/elasticsearch/index_entities :['From ./entities/entities', ['From ./procurement/spending/spending-by-entity', ['From ./procurement/spending/latest-contract-spending', ['From ./procurement/spending/quarterly-contract-spending-reports-data', ['From ./procurement/spending/quarterly-contract-spending-reports', ['Dependency unsuccessful', 'Cannot run until dependency "./procurement/spending/quarterly-contract-spending-report-uris" is successfully executed']]]]]] From ./budgetkey/elasticsearch/index_entities :['From ./entities/entities', ['Invalid dependency', 'Cannot run until dependency passes validation: ./entities/all']] From ./budgetkey/elasticsearch/index_entities :['From ./entities/entities', ['From ./entities/all', ['Dependency unsuccessful', 'Cannot run until dependency "./entities/ottoman/ottoman-association-registry" is successfully executed']]] Invalid dependency :Cannot run until dependency passes validation: ./budgetkey/elasticsearch/index_contract_spending From ./budgetkey/elasticsearch/index_contract_spending :['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/latest-contract-spending'] From ./budgetkey/elasticsearch/index_contract_spending :['From ./procurement/spending/latest-contract-spending', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports-data']] From ./budgetkey/elasticsearch/index_contract_spending :['From ./procurement/spending/latest-contract-spending', ['From ./procurement/spending/quarterly-contract-spending-reports-data', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports']]] From ./budgetkey/elasticsearch/index_contract_spending :['From ./procurement/spending/latest-contract-spending', ['From ./procurement/spending/quarterly-contract-spending-reports-data', ['From ./procurement/spending/quarterly-contract-spending-reports', ['Dependency unsuccessful', 'Cannot run until dependency "./procurement/spending/quarterly-contract-spending-report-uris" is successfully executed']]]] Invalid dependency :Cannot run until dependency passes validation: ./budgetkey/elasticsearch/index_gov_decisions From ./budgetkey/elasticsearch/index_gov_decisions :['Invalid dependency', 'Cannot run until dependency passes validation: ./government_decisions/government_decisions'] From ./budgetkey/elasticsearch/index_gov_decisions :['From ./government_decisions/government_decisions', ['Dependency unsuccessful', 'Cannot run until dependency "./government_decisions/scraper" is successfully executed']]
Pipeline ID: budgetkey/elasticsearch/sitemaps
explanation_fetcher
Traceback (most recent call last):
File "/datapackage_pipelines_budgetkey/pipelines/budget/national/changes/explanations/explanation_fetcher.py", line 35, in get_explanations
assert resp.status_code < 400
AssertionError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/datapackage_pipelines_budgetkey/pipelines/budget/national/changes/explanations/explanation_fetcher.py", line 82, in <module>
spew(dp, itertools.chain(res_iter, [get_explanations(parameters['url'])]))
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew
for rec in res:
File "/datapackage_pipelines_budgetkey/pipelines/budget/national/changes/explanations/explanation_fetcher.py", line 45, in get_explanations
archive = gcl.download(url)
File "/datapackage_pipelines_budgetkey/common/google_chrome.py", line 165, in download
assert False, 'Failed to download file, %r' % downloads
AssertionError: Failed to download file, ['']
DEBUG :[chan 14] Max packet in: 32768 bytes
DEBUG :[chan 14] Max packet out: 32768 bytes
DEBUG :Secsh channel 14 opened.
DEBUG :[chan 14] Sesch channel 14 request ok
DEBUG :[chan 14] EOF received (14)
DEBUG :[chan 14] EOF sent (14)
DEBUG :Dropping user packet because connection is dead.
Pipeline ID: budget/national/changes/explanations/all
flow
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 13, in <module>
flow = flow_module.flow(parameters, datapackage, resources, ctx.stats)
File "/datapackage_pipelines_budgetkey/pipelines/budget/national/changes/original/committee-zipfile.py", line 14, in flow
archive_ = gcl.download(WEIRD_ZIP_FILE, outfile=f'p{YEAR}.zip')
File "/datapackage_pipelines_budgetkey/common/google_chrome.py", line 165, in download
assert False, 'Failed to download file, %r' % downloads
AssertionError: Failed to download file, ['']
DEBUG :[chan 14] Max packet in: 32768 bytes
DEBUG :[chan 14] Max packet out: 32768 bytes
DEBUG :Secsh channel 14 opened.
DEBUG :[chan 14] Sesch channel 14 request ok
DEBUG :[chan 14] EOF received (14)
DEBUG :[chan 14] EOF sent (14)
DEBUG :Dropping user packet because connection is dead.
Pipeline ID: budget/national/changes/original/committee-zipfile
dump.to_sql ERROR :Traceback (most recent call last): ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/dump/to_sql.py", line 15, in <module> ERROR :spew_flow(flow(ctx.parameters), ctx) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew ERROR :for rec in res: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/dumper_base.py", line 69, in row_counter ERROR :for row in iterator: ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_sql/writer.py", line 49, in write ERROR :ret = self.__update(connection, keyed_row) ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_sql/writer.py", line 100, in __update ERROR :res = connection.execute(expr) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1412, in execute ERROR :return meth( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 483, in _execute_on_connection ERROR :return connection._execute_clauseelement( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1627, in _execute_clauseelement ERROR :compiled_sql, extracted_params, cache_hit = elem._compile_w_cache( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 671, in _compile_w_cache ERROR :compiled_sql = self._compiler( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 288, in _compiler ERROR :return dialect.statement_compiler(dialect, self, **kw) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 1425, in __init__ ERROR :Compiled.__init__(self, dialect, statement, **kwargs) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 866, in __init__ ERROR :self.string = self.process(self.statement, **compile_kwargs) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 911, in process ERROR :return obj._compiler_dispatch(self, **kwargs) ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/visitors.py", line 143, in _compiler_dispatch ERROR :return meth(self, **kw) # type: ignore # noqa: E501 ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/compiler.py", line 6040, in visit_update ERROR :crud_params_struct = crud._get_crud_params( ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/crud.py", line 335, in _get_crud_params ERROR :raise exc.CompileError( ERROR :sqlalchemy.exc ERROR :. ERROR :CompileError ERROR :: ERROR :Unconsumed column names: code_level6, code_level4, title_level4, code_level2, code_level8, title_level2, title_level6, budget_kind_title, title_level8, budget_kind_code
Pipeline ID: budgetkey/elasticsearch/index_budget
Pipeline ID: budget/municipal/fetcher
Pipeline ID: budget/national/changes/original/current-year-fixes
Pipeline ID: budget/national/changes/original/national-budget-changes
Pipeline ID: budget/national/changes/processed/transactions
We get the data the Ministry of Finance publishes in data.gov.il every year. It comes in XLS format, with one row per TAKANA and phase (original, approved, executed). In this pipeline we create from each triplet a single row that has all the data.
Pipeline ID: budget/national/original/national-budgets
This pipeline joins the different phases of the budget (allocated, revised and executed). In the original file there's a separate row for each of the phases. We like it better as a single row with all phase info. Another thing this pipeline does is to rename the column titles - to more friendly English names. Also, it create rows for all hierarchies - where upper hierarchies (2, 4 & 6 digits) are plain aggregations of the 8-digit items they contain.
Pipeline ID: budget/national/processed/aggregated-yearly
Pipeline ID: budget/national/processed/category-explanations
This pipeline joins budget items that span across years.
Pipeline ID: budget/national/processed/connected-items-explained
This pipeline joins budget items that span across years.
Pipeline ID: budget/national/processed/connected-national-budgets
Pipeline ID: budget/national/processed/just-the-total
Pipeline ID: budget/national/processed/roof-names
This pipeline joins the budget data to itself so that each item has a list of its immediate children.
Pipeline ID: budget/national/processed/with-extras
Pipeline ID: budgetkey/analysis/spending/publisher_entity_analysis
Pipeline ID: budgetkey/analysis/spending/publisher_foa_analysis
Pipeline ID: budgetkey/analysis/spending/united
Pipeline ID: budgetkey/budget-functional-aggregates
Pipeline ID: budgetkey/documents
Pipeline ID: budgetkey/elasticsearch/index_activities
Pipeline ID: budgetkey/elasticsearch/index_muni_budgets
Pipeline ID: budgetkey/elasticsearch/index_muni_tenders
Pipeline ID: budgetkey/elasticsearch/index_reports
Pipeline ID: budgetkey/elasticsearch/index_support_programs
Pipeline ID: budgetkey/elasticsearch/index_supports
Pipeline ID: budgetkey/elasticsearch/index_tenders
Pipeline ID: budgetkey/elasticsearch/index_units
Pipeline ID: budgetkey/emails/emails