Pipeline ID: activities/social_services/social_service_reports
Pipeline ID: activities/social_services/social_service_suppliers
flow ERROR :Output pipe disappeared!
Pipeline ID: activities/social_services/social_service_tenders
Pipeline ID: activities/social_services/social_services
Pipeline ID: bank_of_israel/bank_of_israel
Pipeline ID: budget/municipal/fetcher
Pipeline ID: budget/national/changes/explanations/all
Pipeline ID: budget/national/changes/original/current-year-fixes
fix_values ERROR :Traceback (most recent call last): ERROR :File "/datapackage_pipelines_budgetkey/pipelines/budget/national/changes/original/fix_values.py", line 49, in <module> ERROR :process(process_row=process_row) ERROR :File "/usr/local/lib/python3.7/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 232, in process ERROR :process_row) ERROR :File "/usr/local/lib/python3.7/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats) ERROR :File "/usr/local/lib/python3.7/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew ERROR :for rec in res: ERROR :File "/usr/local/lib/python3.7/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 201, in generic_process_resource ERROR :parameters, stats) ERROR :File "/datapackage_pipelines_budgetkey/pipelines/budget/national/changes/original/fix_values.py", line 44, in process_row ERROR :row[amount] *= 1000 ERROR :TypeError ERROR :: ERROR :unsupported operand type(s) for *=: 'NoneType' and 'int'
Pipeline ID: budget/national/changes/original/national-budget-changes
Pipeline ID: budget/national/changes/processed/national-budget-changes-aggregated
Pipeline ID: budget/national/changes/processed/transactions
We get the data the Ministry of Finance publishes in data.gov.il every year. It comes in XLS format, with one row per TAKANA and phase (original, approved, executed). In this pipeline we create from each triplet a single row that has all the data.
Pipeline ID: budget/national/original/national-budgets
This pipeline joins the different phases of the budget (allocated, revised and executed). In the original file there's a separate row for each of the phases. We like it better as a single row with all phase info. Another thing this pipeline does is to rename the column titles - to more friendly English names. Also, it create rows for all hierarchies - where upper hierarchies (2, 4 & 6 digits) are plain aggregations of the 8-digit items they contain.
Pipeline ID: budget/national/processed/aggregated-yearly
Pipeline ID: budget/national/processed/category-explanations
This pipeline joins budget items that span across years.
Pipeline ID: budget/national/processed/connected-items-explained
This pipeline joins budget items that span across years.
Pipeline ID: budget/national/processed/connected-national-budgets
Pipeline ID: budget/national/processed/just-the-total
Pipeline ID: budget/national/processed/roof-names
This pipeline joins the budget data to itself so that each item has a list of its immediate children.
Pipeline ID: budget/national/processed/with-extras
Pipeline ID: budgetkey/analysis/spending/publisher_entity_analysis
Pipeline ID: budgetkey/analysis/spending/publisher_foa_analysis
Pipeline ID: budgetkey/analysis/spending/united
Pipeline ID: budgetkey/budget-functional-aggregates
Pipeline ID: budgetkey/documents
Pipeline ID: budgetkey/elasticsearch/index_activities
Pipeline ID: budgetkey/elasticsearch/index_budget
Pipeline ID: budgetkey/elasticsearch/index_calls_for_bids
Pipeline ID: budgetkey/elasticsearch/index_contract_spending
Pipeline ID: budgetkey/elasticsearch/index_entities
Pipeline ID: budgetkey/elasticsearch/index_gov_decisions
Pipeline ID: budgetkey/elasticsearch/index_muni_budgets
Pipeline ID: budgetkey/elasticsearch/index_national_budget_changes
Invalid dependency :Cannot run until dependency passes validation: ./people/aggregated From ./people/aggregated :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/all'] From ./people/aggregated :['From ./people/all', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/all-appointments']] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]]] From ./people/aggregated :['From ./people/all', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/procurement/procurement-individuals']] From ./people/aggregated :['From ./people/all', ['From ./people/procurement/procurement-individuals', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/latest-contract-spending']]] From ./people/aggregated :['From ./people/all', ['From ./people/procurement/procurement-individuals', ['From ./procurement/spending/latest-contract-spending', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports-data']]]] From ./people/aggregated :['From ./people/all', ['From ./people/procurement/procurement-individuals', ['From ./procurement/spending/latest-contract-spending', ['From ./procurement/spending/quarterly-contract-spending-reports-data', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports']]]]] From ./people/aggregated :['From ./people/all', ['From ./people/procurement/procurement-individuals', ['From ./procurement/spending/latest-contract-spending', ['From ./procurement/spending/quarterly-contract-spending-reports-data', ['From ./procurement/spending/quarterly-contract-spending-reports', ['Dependency unsuccessful', 'Cannot run until dependency "./procurement/spending/quarterly-contract-spending-report-uris" is successfully executed']]]]]]
Pipeline ID: budgetkey/elasticsearch/index_people
Pipeline ID: budgetkey/elasticsearch/index_reports
Dependency unsuccessful :Cannot run until dependency "./supports/criteria/support-criteria" is successfully executed
Pipeline ID: budgetkey/elasticsearch/index_support_criteria
Pipeline ID: budgetkey/elasticsearch/index_supports
Invalid dependency :Cannot run until dependency passes validation: ./procurement/tenders/processed From ./procurement/tenders/processed :['Dependency unsuccessful', 'Cannot run until dependency "./activities/social_services/social_service_tenders" is successfully executed']
Pipeline ID: budgetkey/elasticsearch/index_tenders
Pipeline ID: budgetkey/elasticsearch/index_units
Invalid dependency :Cannot run until dependency passes validation: ./budgetkey/elasticsearch/index_tenders From ./budgetkey/elasticsearch/index_tenders :['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/tenders/processed'] From ./budgetkey/elasticsearch/index_tenders :['From ./procurement/tenders/processed', ['Dependency unsuccessful', 'Cannot run until dependency "./activities/social_services/social_service_tenders" is successfully executed']] Invalid dependency :Cannot run until dependency passes validation: ./budgetkey/elasticsearch/index_support_criteria From ./budgetkey/elasticsearch/index_support_criteria :['Dependency unsuccessful', 'Cannot run until dependency "./supports/criteria/support-criteria" is successfully executed']
Pipeline ID: budgetkey/elasticsearch/sitemaps
Pipeline ID: budgetkey/emails/emails
Pipeline ID: donations/candidates
Pipeline ID: donations/parties
get_transactions
ERROR :Retrying in a bit (SSLError(MaxRetryError("HTTPSConnectionPool(host='statements.mevaker.gov.il', port=443): Max retries exceeded with url: /Handler/GuarantyDonationPublisherHandler.ashx (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)')))")))
DEBUG :Starting new HTTPS connection (1): statements.mevaker.gov.il:443
ERROR :Retrying in a bit (SSLError(MaxRetryError("HTTPSConnectionPool(host='statements.mevaker.gov.il', port=443): Max retries exceeded with url: /Handler/GuarantyDonationPublisherHandler.ashx (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)')))")))
DEBUG :Starting new HTTPS connection (1): statements.mevaker.gov.il:443
ERROR :Retrying in a bit (SSLError(MaxRetryError("HTTPSConnectionPool(host='statements.mevaker.gov.il', port=443): Max retries exceeded with url: /Handler/GuarantyDonationPublisherHandler.ashx (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)')))")))
DEBUG :Starting new HTTPS connection (1): statements.mevaker.gov.il:443
ERROR :Retrying in a bit (SSLError(MaxRetryError("HTTPSConnectionPool(host='statements.mevaker.gov.il', port=443): Max retries exceeded with url: /Handler/GuarantyDonationPublisherHandler.ashx (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)')))")))
DEBUG :Starting new HTTPS connection (1): statements.mevaker.gov.il:443
ERROR :Retrying in a bit (SSLError(MaxRetryError("HTTPSConnectionPool(host='statements.mevaker.gov.il', port=443): Max retries exceeded with url: /Handler/GuarantyDonationPublisherHandler.ashx (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)')))")))
DEBUG :Starting new HTTPS connection (1): statements.mevaker.gov.il:443
ERROR :Retrying in a bit (SSLError(MaxRetryError("HTTPSConnectionPool(host='statements.mevaker.gov.il', port=443): Max retries exceeded with url: /Handler/GuarantyDonationPublisherHandler.ashx (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)')))")))
DEBUG :Starting new HTTPS connection (1): statements.mevaker.gov.il:443
ERROR :Retrying in a bit (SSLError(MaxRetryError("HTTPSConnectionPool(host='statements.mevaker.gov.il', port=443): Max retries exceeded with url: /Handler/GuarantyDonationPublisherHandler.ashx (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)')))")))
DEBUG :Starting new HTTPS connection (1): statements.mevaker.gov.il:443
ERROR :Retrying in a bit (SSLError(MaxRetryError("HTTPSConnectionPool(host='statements.mevaker.gov.il', port=443): Max retries exceeded with url: /Handler/GuarantyDonationPublisherHandler.ashx (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)')))")))
DEBUG :Starting new HTTPS connection (1): statements.mevaker.gov.il:443
ERROR :Retrying in a bit (SSLError(MaxRetryError("HTTPSConnectionPool(host='statements.mevaker.gov.il', port=443): Max retries exceeded with url: /Handler/GuarantyDonationPublisherHandler.ashx (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)')))")))
DEBUG :Starting new HTTPS connection (1): statements.mevaker.gov.il:443
ERROR :Retrying in a bit (SSLError(MaxRetryError("HTTPSConnectionPool(host='statements.mevaker.gov.il', port=443): Max retries exceeded with url: /Handler/GuarantyDonationPublisherHandler.ashx (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)')))")))
Traceback (most recent call last):
File "/datapackage_pipelines_budgetkey/pipelines/donations/get_transactions.py", line 150, in <module>
spew(dp, process_resources(res_iter))
File "/usr/local/lib/python3.7/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew
for rec in res:
File "/datapackage_pipelines_budgetkey/pipelines/donations/get_transactions.py", line 109, in get_transactions
for resp in self.get_for_candidate(cid):
File "/datapackage_pipelines_budgetkey/pipelines/donations/get_transactions.py", line 95, in get_for_candidate
resp = self.get_for_range(cid, range_start, range_end)
File "/datapackage_pipelines_budgetkey/pipelines/donations/get_transactions.py", line 70, in get_for_range
assert len(resp) == 6
AssertionError
Pipeline ID: donations/transactions
Pipeline ID: education/tochniyot/tochniyot
Pipeline ID: entities/all
Pipeline ID: entities/associations/guidestar/guidestar
Pipeline ID: entities/associations/guidestar/guidestar-scraper
רשימת כלל העמותות הפעילות, מתעדכן שבועית.
Pipeline ID: entities/associations/registrar/registry
Pipeline ID: entities/companies/registrar/details
רשימת כלל החברות הפעילות, מתעדכן שבועית.
Pipeline ID: entities/companies/registrar/scraper
רשימת אגודות שיתופיות, מתוך ממשק נתונים של מאגר של משרד הכלכלה / האגף לאיגוד שיתופי, מתעדכן שבועית
Pipeline ID: entities/cooperatives/cooperatives
Pipeline ID: entities/endowments/endowments
Pipeline ID: entities/entities
Pipeline ID: entities/fingerprints
Pipeline ID: entities/foi_orgs/foi_orgs
scraper
Traceback (most recent call last):
File "/datapackage_pipelines_budgetkey/pipelines/entities/moital/scraper.py", line 79, in <module>
spew(datapackage, [scrape()])
File "/usr/local/lib/python3.7/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew
for rec in res:
File "/datapackage_pipelines_budgetkey/pipelines/entities/moital/scraper.py", line 32, in scrape
desired_capabilities=DesiredCapabilities.PHANTOMJS)
AttributeError: type object 'DesiredCapabilities' has no attribute 'PHANTOMJS'
Pipeline ID: entities/moital/moital_service_providers
רשימת אגודות עותומניות, מתוך תשובה לבקשת חופש מידע
Pipeline ID: entities/ottoman/ottoman-association-registry
Pipeline ID: entities/partnerships/partnerships
stream_remote_resources
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/datapackage_pipelines/specs/../lib/stream_remote_resources.py", line 245, in <module>
resource.pop('http_headers', None))
File "/usr/local/lib/python3.7/site-packages/datapackage_pipelines/specs/../lib/stream_remote_resources.py", line 199, in stream_reader
schema, headers, columns, stream, close = get_opener(url, _resource)()
File "/usr/local/lib/python3.7/site-packages/datapackage_pipelines/specs/../lib/stream_remote_resources.py", line 175, in opener
_stream.open()
File "/usr/local/lib/python3.7/site-packages/tabulator/stream.py", line 423, in open
self.__parser.open(source, encoding=self.__encoding)
File "/usr/local/lib/python3.7/site-packages/tabulator/parsers/xlsx.py", line 99, in open
self.__bytes, read_only=not self.__fill_merged_cells, data_only=True
File "/usr/local/lib/python3.7/site-packages/openpyxl/reader/excel.py", line 316, in load_workbook
data_only, keep_links)
File "/usr/local/lib/python3.7/site-packages/openpyxl/reader/excel.py", line 124, in __init__
self.archive = _validate_archive(fn)
File "/usr/local/lib/python3.7/site-packages/openpyxl/reader/excel.py", line 96, in _validate_archive
archive = ZipFile(filename, 'r')
File "/usr/local/lib/python3.7/zipfile.py", line 1258, in __init__
self._RealGetContents()
File "/usr/local/lib/python3.7/zipfile.py", line 1325, in _RealGetContents
raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file
Pipeline ID: entities/safety-orders/safety-orders
Dependency unsuccessful :Cannot run until dependency "./entities/safety-orders/safety-orders" is successfully executed
Pipeline ID: entities/safety-orders/safety-orders-by-entity
Pipeline ID: entities/special/registry
Pipeline ID: ezvonot/ezvonot_accounts
Pipeline ID: ezvonot/ezvonot_cases
Pipeline ID: ezvonot/ezvonot_crontab
Pipeline ID: ezvonot/ezvonot_decisions
Pipeline ID: ezvonot/ezvonot_decisions_join
Pipeline ID: ezvonot/ezvonot_deposits
Pipeline ID: ezvonot/ezvonot_purposes
Pipeline ID: government_decisions/government_decisions
Pipeline ID: government_decisions/scraper
Pipeline ID: lamas/municipal-data
Pipeline ID: lamas/municipality-to-city
Pipeline ID: maya/maya-bonds-and-options-holdings-changes
Pipeline ID: maya/maya-company-officer-list
Pipeline ID: maya/maya-company-stakeholder-list
flow
ERROR :Traceback (most recent call last):
ERROR :File "/usr/local/lib/python3.7/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 15, in <module>
ERROR :spew_flow(flow, ctx)
ERROR :File "/usr/local/lib/python3.7/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__
ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats)
ERROR :File "/usr/local/lib/python3.7/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew
ERROR :for rec in res:
ERROR :File "/usr/local/lib/python3.7/site-packages/dataflows/processors/delete_fields.py", line 7, in process_resource
ERROR :for row in rows:
ERROR :File "/usr/local/lib/python3.7/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource
ERROR :yield from self.func(resource)
ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/maya_parser_utils.py", line 52, in iter
ERROR :for row in rows:
ERROR :File "/usr/local/lib/python3.7/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource
ERROR :yield from self.func(resource)
ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/parse_maya_vip_appointments.py", line 111, in parse_document
ERROR :for row in rows:
ERROR :File "/usr/local/lib/python3.7/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource
ERROR :yield from self.func(resource)
ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/parse_maya_vip_appointments.py", line 88, in validate
ERROR :verify_row_values(row, 'IsIsraeli', {'אדם פרטי ללא אזרחות ישראלית', 'אדם פרטי עם אזרחות ישראלית'})
ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/maya_parser_utils.py", line 35, in verify_row_values
ERROR :raise Exception("Document {} failed validation. textAlias {} had illegal value: {}".format(url, field, doc[field]))
ERROR :Exception
ERROR ::
ERROR :Document https://mayafiles.tase.co.il/RHtm/1444001-1445000/H1444010.htm failed validation. textAlias IsIsraeli had illegal value: ['Individual who holds Israeli citizenship']
Pipeline ID: maya/maya-company-vip-appointments
Pipeline ID: maya/maya-complete-notification-list
flow
ERROR :Traceback (most recent call last):
ERROR :File "/usr/local/lib/python3.7/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 15, in <module>
ERROR :spew_flow(flow, ctx)
ERROR :File "/usr/local/lib/python3.7/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__
ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats)
ERROR :File "/usr/local/lib/python3.7/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew
ERROR :for rec in res:
ERROR :File "/usr/local/lib/python3.7/site-packages/dataflows/processors/delete_fields.py", line 7, in process_resource
ERROR :for row in rows:
ERROR :File "/usr/local/lib/python3.7/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource
ERROR :yield from self.func(resource)
ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/parse_maya_holdings_change.py", line 122, in parse_document
ERROR :for row in rows:
ERROR :File "/usr/local/lib/python3.7/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource
ERROR :yield from self.func(resource)
ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/parse_maya_holdings_change.py", line 118, in validate
ERROR :verify_row_values(row, 'EntityType', ENTITY_TYPES)
ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/maya_parser_utils.py", line 35, in verify_row_values
ERROR :raise Exception("Document {} failed validation. textAlias {} had illegal value: {}".format(url, field, doc[field]))
ERROR :Exception
ERROR ::
ERROR :Document https://mayafiles.tase.co.il/RHtm/1473001-1474000/H1473091.htm failed validation. textAlias EntityType had illegal value: ['Individual who does not hold Israeli citizenship', 'Individual who does not hold Israeli citizenship', 'Individual who does not hold Israeli citizenship']
Pipeline ID: maya/maya-holdings-changes
Pipeline ID: maya/maya-notification-parse-updates
flow
ERROR :Traceback (most recent call last):
ERROR :File "/usr/local/lib/python3.7/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 15, in <module>
ERROR :spew_flow(flow, ctx)
ERROR :File "/usr/local/lib/python3.7/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__
ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats)
ERROR :File "/usr/local/lib/python3.7/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew
ERROR :for rec in res:
ERROR :File "/usr/local/lib/python3.7/site-packages/dataflows/processors/delete_fields.py", line 7, in process_resource
ERROR :for row in rows:
ERROR :File "/usr/local/lib/python3.7/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource
ERROR :yield from self.func(resource)
ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/maya_parser_utils.py", line 52, in iter
ERROR :for row in rows:
ERROR :File "/usr/local/lib/python3.7/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource
ERROR :yield from self.func(resource)
ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/parse_reported_academic_degrees.py", line 67, in parse_document
ERROR :for row in rows:
ERROR :File "/usr/local/lib/python3.7/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource
ERROR :yield from self.func(resource)
ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/parse_reported_academic_degrees.py", line 51, in validate
ERROR :verify_row_values(row, 'SugMisparZihuy', SUG_MISPAR_ZIHUY_MAPPING)
ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/maya_parser_utils.py", line 35, in verify_row_values
ERROR :raise Exception("Document {} failed validation. textAlias {} had illegal value: {}".format(url, field, doc[field]))
ERROR :Exception
ERROR ::
ERROR :Document https://mayafiles.tase.co.il/RHtm/1444001-1445000/H1444010.htm failed validation. textAlias SugMisparZihuy had illegal value: ['ID no.']
Pipeline ID: maya/maya-reported-academic-degrees
flow
ERROR :Traceback (most recent call last):
ERROR :File "/usr/local/lib/python3.7/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 15, in <module>
ERROR :spew_flow(flow, ctx)
ERROR :File "/usr/local/lib/python3.7/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__
ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats)
ERROR :File "/usr/local/lib/python3.7/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew
ERROR :for rec in res:
ERROR :File "/usr/local/lib/python3.7/site-packages/dataflows/processors/delete_fields.py", line 7, in process_resource
ERROR :for row in rows:
ERROR :File "/usr/local/lib/python3.7/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource
ERROR :yield from self.func(resource)
ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/maya_parser_utils.py", line 52, in iter
ERROR :for row in rows:
ERROR :File "/usr/local/lib/python3.7/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource
ERROR :yield from self.func(resource)
ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/parse_reported_work_record.py", line 67, in parse_document
ERROR :for row in rows:
ERROR :File "/usr/local/lib/python3.7/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource
ERROR :yield from self.func(resource)
ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/parse_reported_work_record.py", line 52, in validate
ERROR :verify_row_values(row, 'SugMisparZihuy', SUG_MISPAR_ZIHUY_MAPPING)
ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/maya_parser_utils.py", line 35, in verify_row_values
ERROR :raise Exception("Document {} failed validation. textAlias {} had illegal value: {}".format(url, field, doc[field]))
ERROR :Exception
ERROR ::
ERROR :Document https://mayafiles.tase.co.il/RHtm/1444001-1445000/H1444010.htm failed validation. textAlias SugMisparZihuy had illegal value: ['ID no.']
Pipeline ID: maya/maya-reported-work-record
Pipeline ID: maya/maya-tase-companies-current-management
Pipeline ID: maya/scrape-maya-notification-list
Pipeline ID: maya/scrape-maya-tase-companies
Invalid dependency :Cannot run until dependency passes validation: ./people/all From ./people/all :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/all-appointments'] From ./people/all :['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations']] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]] From ./people/all :['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations']] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]] From ./people/all :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/procurement/procurement-individuals'] From ./people/all :['From ./people/procurement/procurement-individuals', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/latest-contract-spending']] From ./people/all :['From ./people/procurement/procurement-individuals', ['From ./procurement/spending/latest-contract-spending', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports-data']]] From ./people/all :['From ./people/procurement/procurement-individuals', ['From ./procurement/spending/latest-contract-spending', ['From ./procurement/spending/quarterly-contract-spending-reports-data', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/quarterly-contract-spending-reports']]]] From ./people/all :['From ./people/procurement/procurement-individuals', ['From ./procurement/spending/latest-contract-spending', ['From ./procurement/spending/quarterly-contract-spending-reports-data', ['From ./procurement/spending/quarterly-contract-spending-reports', ['Dependency unsuccessful', 'Cannot run until dependency "./procurement/spending/quarterly-contract-spending-report-uris" is successfully executed']]]]]
Pipeline ID: people/aggregated
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/all-appointments From ./people/company_appointments/all-appointments :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations'] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-nominations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]] From ./people/company_appointments/all-appointments :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations'] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-resignations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]
Pipeline ID: people/all
Pipeline ID: people/association_founders/association-founders
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations From ./people/company_appointments/maya/maya-nominations :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list'] From ./people/company_appointments/maya/maya-nominations :['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']] From ./people/company_appointments/maya/maya-nominations :['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]] Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations From ./people/company_appointments/maya/maya-resignations :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list'] From ./people/company_appointments/maya/maya-resignations :['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']] From ./people/company_appointments/maya/maya-resignations :['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]
Pipeline ID: people/company_appointments/all-appointments
Dependency unsuccessful :Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed
Pipeline ID: people/company_appointments/maya/collect-maya-forms-debug-stats
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations From ./people/company_appointments/maya/maya-nominations :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list'] From ./people/company_appointments/maya/maya-nominations :['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']] From ./people/company_appointments/maya/maya-nominations :['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]] Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations From ./people/company_appointments/maya/maya-resignations :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list'] From ./people/company_appointments/maya/maya-resignations :['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']] From ./people/company_appointments/maya/maya-resignations :['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]
Pipeline ID: people/company_appointments/maya/correlate
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates From ./people/company_appointments/maya/maya-notification-updates :['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']
Pipeline ID: people/company_appointments/maya/maya-list
Name, job and company description.
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list From ./people/company_appointments/maya/maya-list :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates'] From ./people/company_appointments/maya/maya-list :['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]
Pipeline ID: people/company_appointments/maya/maya-nominations
scrape-notification-list
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 910, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/datapackage_pipelines_budgetkey/pipelines/people/company_appointments/maya/scrape-notification-list.py", line 55, in _maya_api_call
return res.json()
File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 917, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Expecting value] <html style="height:100%"><head><META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"><meta name="format-detection" content="telephone=no"><meta name="viewport" content="initial-scale=1.0"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"><script type="text/javascript" src="/_Incapsula_Resource?SWJIYLWA=719d34d31c8e3a6e6fffd425f7e032f3"></script></head><body style="margin:0px;height:100%"><iframe id="main-iframe" src="/_Incapsula_Resource?CWUDNSAI=23&xinfo=13-27951820-0%200NNN%20RT%281664151111160%2014%29%20q%280%20-1%20-1%200%29%20r%280%20-1%29%20B15%2811%2c1795525%2c0%29%20U6&incident_id=352000840044364939-142777417455375629&edet=15&cinfo=0b000000&rpinfo=0&mth=POST" frameborder=0 width="100%" height="100%" marginheight="0px" marginwidth="0px">Request unsuccessful. Incapsula incident ID: 352000840044364939-142777417455375629</iframe></body></html>: 0
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/datapackage_pipelines_budgetkey/pipelines/people/company_appointments/maya/scrape-notification-list.py", line 130, in <module>
spew(datapackage, [collect()])
File "/usr/local/lib/python3.7/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew
for rec in res:
File "/datapackage_pipelines_budgetkey/pipelines/people/company_appointments/maya/scrape-notification-list.py", line 113, in collect
yield from _collect_date_range(year_start, year_end)
File "/datapackage_pipelines_budgetkey/pipelines/people/company_appointments/maya/scrape-notification-list.py", line 75, in _collect_date_range
res = _maya_api_call(date_from, date_to, current_page)
File "/datapackage_pipelines_budgetkey/pipelines/people/company_appointments/maya/scrape-notification-list.py", line 57, in _maya_api_call
raise Exception("Failed to Call Maya API for date_from:{} date_to:{} page_num:{}".format(date_from, date_to, page_num)) from e
Exception: Failed to Call Maya API for date_from:2006-01-01 date_to:2007-01-01 page_num:0
Pipeline ID: people/company_appointments/maya/maya-notification-list
id, date, company, notification_type, s3_object_name, url
Dependency unsuccessful :Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed
Pipeline ID: people/company_appointments/maya/maya-notification-updates
Name, job and company description.
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list From ./people/company_appointments/maya/maya-list :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates'] From ./people/company_appointments/maya/maya-list :['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]
Pipeline ID: people/company_appointments/maya/maya-resignations
רשימת כל המינויים והחברות מאתר כלכליסט
Pipeline ID: people/company_appointments/media/calcalist/calcalist
Name, job and company description.
Pipeline ID: people/company_appointments/media/themarker/themarker
Pipeline ID: people/company_appointments/state_owned/correlate
Pipeline ID: people/company_appointments/state_owned/parse_pdf_resources
columns: date, url
Pipeline ID: people/company_appointments/state_owned/reports
Pipeline ID: people/company_appointments/state_owned/state-owned-nominations
Pipeline ID: people/municipal_appointments/municipal_representitives
Pipeline ID: people/political_candidates/election_candidates21
Pipeline ID: people/political_donations/political_donations
Pipeline ID: people/procurement/procurement-individuals
Pipeline ID: procurement/calls_for_bids/calls-for-bids-all
Pipeline ID: procurement/calls_for_bids/calls-for-bids-moh
Pipeline ID: procurement/calls_for_bids/gov-publications
Pipeline ID: procurement/calls_for_bids/gov-publications-ministry-of-immigration
Pipeline ID: procurement/calls_for_bids/gov-publications2
Pipeline ID: procurement/calls_for_bids/jobiz-all
Pipeline ID: procurement/calls_for_bids/jobiz-calls-for-bids
Pipeline ID: procurement/calls_for_bids/jobiz-scraper
אנו סוכמים מידע מתוך הדוחות הרבעוניים לתוך טבלה המכילה את המידע העדכני ביותר.
Pipeline ID: procurement/spending/latest-contract-spending
Pipeline ID: procurement/spending/quarterly-contract-spending-report-uris
כל משרדי הממשלה ויחידות הסמך (שמחוברות למערכת מרכבה) צריכים לפרסם אחת לרבעון (תוך 45 יום מסוף הרבעון) דו״ח התקשרויות לפי פורמט מוגדר היטב הממומש כדו״ח מובנה במערכת המרכבה. כל המשרדים מדווחים עצמאית והדו״חות עולים לאתר היחידה לחופש המידע, משם אנו אוספים אותם. התוצר של השרשר הזה הוא פירוט כל הדיווחים של כל המשרדים.
Pipeline ID: procurement/spending/quarterly-contract-spending-reports
Pipeline ID: procurement/spending/quarterly-contract-spending-reports-data
Pipeline ID: procurement/spending/spending-by-entity
Pipeline ID: procurement/tenders/all
Dependency unsuccessful :Cannot run until dependency "./activities/social_services/social_service_tenders" is successfully executed
Pipeline ID: procurement/tenders/processed
Pipeline ID: reports/all
Pipeline ID: reports/ngos/ngo-activity-report
Pipeline ID: reports/ngos/ngo-district-report
Dependency unsuccessful :Cannot run until dependency "./supports/supports-2021" is successfully executed
Pipeline ID: supports/all
Pipeline ID: supports/by-payment-year
Pipeline ID: supports/by-request-year
sample
Traceback (most recent call last):
File "/datapackage_pipelines_budgetkey/processors/sample.py", line 11, in <module>
res_name = parameters.get('resource', datapackage['resources'][0]['name'])
IndexError: list index out of range
Pipeline ID: supports/criteria/support-criteria
Pipeline ID: supports/criteria/support-criteria-doj
Pipeline ID: supports/supports-2004
Pipeline ID: supports/supports-2005
Pipeline ID: supports/supports-2006
Pipeline ID: supports/supports-2007