Pipeline ID: activities/social_services/social_service_reports
Pipeline ID: activities/social_services/social_service_suppliers
Pipeline ID: activities/social_services/social_service_tenders
Pipeline ID: activities/social_services/social_services
Pipeline ID: bank_of_israel/bank_of_israel
Pipeline ID: budget/municipal/fetcher
Pipeline ID: budget/national/changes/explanations/all
Pipeline ID: budget/national/changes/original/current-year-fixes
Pipeline ID: budget/national/changes/original/national-budget-changes
Pipeline ID: budget/national/changes/processed/national-budget-changes-aggregated
Pipeline ID: budget/national/changes/processed/transactions
We get the data the Ministry of Finance publishes in data.gov.il every year. It comes in XLS format, with one row per TAKANA and phase (original, approved, executed). In this pipeline we create from each triplet a single row that has all the data.
Pipeline ID: budget/national/original/national-budgets
This pipeline joins the different phases of the budget (allocated, revised and executed). In the original file there's a separate row for each of the phases. We like it better as a single row with all phase info. Another thing this pipeline does is to rename the column titles - to more friendly English names. Also, it create rows for all hierarchies - where upper hierarchies (2, 4 & 6 digits) are plain aggregations of the 8-digit items they contain.
Pipeline ID: budget/national/processed/aggregated-yearly
Pipeline ID: budget/national/processed/category-explanations
This pipeline joins budget items that span across years.
Pipeline ID: budget/national/processed/connected-items-explained
This pipeline joins budget items that span across years.
Pipeline ID: budget/national/processed/connected-national-budgets
Pipeline ID: budget/national/processed/just-the-total
Pipeline ID: budget/national/processed/roof-names
This pipeline joins the budget data to itself so that each item has a list of its immediate children.
Pipeline ID: budget/national/processed/with-extras
Pipeline ID: budgetkey/analysis/spending/publisher_entity_analysis
Pipeline ID: budgetkey/analysis/spending/publisher_foa_analysis
Pipeline ID: budgetkey/analysis/spending/united
Pipeline ID: budgetkey/budget-functional-aggregates
Pipeline ID: budgetkey/documents
Pipeline ID: budgetkey/elasticsearch/index_activities
Pipeline ID: budgetkey/elasticsearch/index_budget
Dependency unsuccessful :Cannot run until dependency "./procurement/calls_for_bids/calls-for-bids-all" is successfully executed
Pipeline ID: budgetkey/elasticsearch/index_calls_for_bids
Pipeline ID: budgetkey/elasticsearch/index_contract_spending
Invalid dependency :Cannot run until dependency passes validation: ./entities/entities From ./entities/entities :['Invalid dependency', 'Cannot run until dependency passes validation: ./entities/all'] From ./entities/entities :['From ./entities/all', ['Dependency unsuccessful', 'Cannot run until dependency "./entities/cooperatives/cooperatives" is successfully executed']]
Pipeline ID: budgetkey/elasticsearch/index_entities
Pipeline ID: budgetkey/elasticsearch/index_gov_decisions
Pipeline ID: budgetkey/elasticsearch/index_muni_budgets
Pipeline ID: budgetkey/elasticsearch/index_muni_tenders
Pipeline ID: budgetkey/elasticsearch/index_national_budget_changes
Invalid dependency :Cannot run until dependency passes validation: ./people/aggregated From ./people/aggregated :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/all'] From ./people/aggregated :['From ./people/all', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/all-appointments']] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/media/themarker/themarker" is successfully executed']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/state_owned/state-owned-nominations']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/state_owned/state-owned-nominations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/state_owned/correlate']]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/state_owned/state-owned-nominations', ['From ./people/company_appointments/state_owned/correlate', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/state_owned/parse_pdf_resources']]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/state_owned/state-owned-nominations', ['From ./people/company_appointments/state_owned/correlate', ['From ./people/company_appointments/state_owned/parse_pdf_resources', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/state_owned/reports" is successfully executed']]]]]] From ./people/aggregated :['From ./people/all', ['Dependency unsuccessful', 'Cannot run until dependency "./people/association_founders/association-founders" is successfully executed']] From ./people/aggregated :['From ./people/all', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/procurement/procurement-individuals']] From ./people/aggregated :['From ./people/all', ['From ./people/procurement/procurement-individuals', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/latest-contract-spending']]] From ./people/aggregated :['From ./people/all', ['From ./people/procurement/procurement-individuals', ['From ./procurement/spending/latest-contract-spending', ['Dependency unsuccessful', 'Cannot run until dependency "./procurement/spending/quarterly-contract-spending-reports-data" is successfully executed']]]] From ./people/aggregated :['From ./people/all', ['Dependency unsuccessful', 'Cannot run until dependency "./people/political_donations/political_donations" is successfully executed']]
Pipeline ID: budgetkey/elasticsearch/index_people
Invalid dependency :Cannot run until dependency passes validation: ./reports/all From ./reports/all :['Dependency unsuccessful', 'Cannot run until dependency "./reports/ngos/ngo-district-report" is successfully executed'] From ./reports/all :['Dependency unsuccessful', 'Cannot run until dependency "./reports/ngos/ngo-activity-report" is successfully executed']
Pipeline ID: budgetkey/elasticsearch/index_reports
Dependency unsuccessful :Cannot run until dependency "./supports/criteria/support-criteria" is successfully executed
Pipeline ID: budgetkey/elasticsearch/index_support_criteria
Pipeline ID: budgetkey/elasticsearch/index_supports
Pipeline ID: budgetkey/elasticsearch/index_tenders
Pipeline ID: budgetkey/elasticsearch/index_units
Invalid dependency :Cannot run until dependency passes validation: ./budgetkey/elasticsearch/index_entities From ./budgetkey/elasticsearch/index_entities :['Invalid dependency', 'Cannot run until dependency passes validation: ./entities/entities'] From ./budgetkey/elasticsearch/index_entities :['From ./entities/entities', ['Invalid dependency', 'Cannot run until dependency passes validation: ./entities/all']] From ./budgetkey/elasticsearch/index_entities :['From ./entities/entities', ['From ./entities/all', ['Dependency unsuccessful', 'Cannot run until dependency "./entities/cooperatives/cooperatives" is successfully executed']]] Invalid dependency :Cannot run until dependency passes validation: ./budgetkey/elasticsearch/index_calls_for_bids From ./budgetkey/elasticsearch/index_calls_for_bids :['Dependency unsuccessful', 'Cannot run until dependency "./procurement/calls_for_bids/calls-for-bids-all" is successfully executed'] Invalid dependency :Cannot run until dependency passes validation: ./budgetkey/elasticsearch/index_support_criteria From ./budgetkey/elasticsearch/index_support_criteria :['Dependency unsuccessful', 'Cannot run until dependency "./supports/criteria/support-criteria" is successfully executed'] Invalid dependency :Cannot run until dependency passes validation: ./budgetkey/elasticsearch/index_reports From ./budgetkey/elasticsearch/index_reports :['Invalid dependency', 'Cannot run until dependency passes validation: ./reports/all'] From ./budgetkey/elasticsearch/index_reports :['From ./reports/all', ['Dependency unsuccessful', 'Cannot run until dependency "./reports/ngos/ngo-district-report" is successfully executed']] From ./budgetkey/elasticsearch/index_reports :['From ./reports/all', ['Dependency unsuccessful', 'Cannot run until dependency "./reports/ngos/ngo-activity-report" is successfully executed']]
Pipeline ID: budgetkey/elasticsearch/sitemaps
Pipeline ID: budgetkey/emails/emails
Dependency unsuccessful :Cannot run until dependency "./donations/parties" is successfully executed
Pipeline ID: donations/candidates
get_parties
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 414, in connect
self.sock = ssl_wrap_socket(
File "/usr/local/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
File "/usr/local/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/local/lib/python3.9/ssl.py", line 501, in wrap_socket
return self.sslsocket_class._create(
File "/usr/local/lib/python3.9/ssl.py", line 1041, in _create
self.do_handshake()
File "/usr/local/lib/python3.9/ssl.py", line 1310, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='statements.mevaker.gov.il', port=443): Max retries exceeded with url: /Handler/GuarantyDonationPublisherHandler.ashx (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/datapackage_pipelines_budgetkey/pipelines/donations/get_parties.py", line 38, in <module>
spew(dp, [get_parties()])
File "/datapackage_pipelines_budgetkey/pipelines/donations/get_parties.py", line 17, in get_parties
resp = requests.post('https://statements.mevaker.gov.il/Handler/GuarantyDonationPublisherHandler.ashx',
File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 115, in post
return request("post", url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 563, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='statements.mevaker.gov.il', port=443): Max retries exceeded with url: /Handler/GuarantyDonationPublisherHandler.ashx (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))
Pipeline ID: donations/parties
Invalid dependency :Cannot run until dependency passes validation: ./donations/candidates From ./donations/candidates :['Dependency unsuccessful', 'Cannot run until dependency "./donations/parties" is successfully executed']
Pipeline ID: donations/transactions
flow
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 13, in <module>
flow = flow_module.flow(parameters, datapackage, resources, ctx.stats)
File "/datapackage_pipelines_budgetkey/pipelines/education/tochniyot/scraper.py", line 103, in flow
return Flow(scrape(),
File "/datapackage_pipelines_budgetkey/pipelines/education/tochniyot/scraper.py", line 65, in scrape
education_programs_count = get_education_programs_count()
File "/datapackage_pipelines_budgetkey/pipelines/education/tochniyot/scraper.py", line 56, in get_education_programs_count
res = send_tochniyot_request()
File "/datapackage_pipelines_budgetkey/pipelines/education/tochniyot/scraper.py", line 80, in send_tochniyot_request
resp.raise_for_status()
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://apps.education.gov.il/TyhNet/ClientWs/TochnitCh.asmx/IturTochnitChByMeafyenim
Pipeline ID: education/tochniyot/tochniyot
Dependency unsuccessful :Cannot run until dependency "./entities/cooperatives/cooperatives" is successfully executed
Pipeline ID: entities/all
Pipeline ID: entities/associations/guidestar/guidestar
Pipeline ID: entities/associations/guidestar/guidestar-scraper
רשימת כלל העמותות הפעילות, מתעדכן שבועית.
Pipeline ID: entities/associations/registrar/registry
Pipeline ID: entities/companies/registrar/details
רשימת כלל החברות הפעילות, מתעדכן שבועית.
Pipeline ID: entities/companies/registrar/scraper
רשימת אגודות שיתופיות, מתוך ממשק נתונים של מאגר של משרד הכלכלה / האגף לאיגוד שיתופי, מתעדכן שבועית
dump.to_sql
ERROR :Traceback (most recent call last):
ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1880, in _execute_context
ERROR :self.dialect.do_executemany(
ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 982, in do_executemany
ERROR :context._psycopg2_fetched_rows = xtras.execute_values(
ERROR :File "/usr/local/lib/python3.9/site-packages/psycopg2/extras.py", line 1299, in execute_values
ERROR :cur.execute(b''.join(parts))
ERROR :psycopg2.errors
ERROR :.
ERROR :UniqueViolation
ERROR ::
ERROR :duplicate key value violates unique constraint "cooperatives_pkey"
DETAIL: Key (id)=(0) already exists.
ERROR :The above exception was the direct cause of the following exception:
ERROR :Traceback (most recent call last):
ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/dump/to_sql.py", line 15, in <module>
ERROR :spew_flow(flow(ctx.parameters), ctx)
ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__
ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats)
ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew
ERROR :for rec in res:
ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/dumpers/dumper_base.py", line 69, in row_counter
ERROR :for row in iterator:
ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_sql/writer.py", line 52, in write
ERROR :for wr in self.__insert():
ERROR :File "/usr/local/lib/python3.9/site-packages/tableschema_sql/writer.py", line 83, in __insert
ERROR :statement.execute(self.__buffer)
ERROR :File "<string>", line 2, in execute
ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/deprecations.py", line 402, in warned
ERROR :return fn(*args, **kwargs)
ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/sql/base.py", line 965, in execute
ERROR :return e._execute_clauseelement(
ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
ERROR :ret = self._execute_context(
ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
ERROR :self._handle_dbapi_exception(
ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception
ERROR :util.raise_(
ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
ERROR :raise exception
ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1880, in _execute_context
ERROR :self.dialect.do_executemany(
ERROR :File "/usr/local/lib/python3.9/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 982, in do_executemany
ERROR :context._psycopg2_fetched_rows = xtras.execute_values(
ERROR :File "/usr/local/lib/python3.9/site-packages/psycopg2/extras.py", line 1299, in execute_values
ERROR :cur.execute(b''.join(parts))
ERROR :sqlalchemy.exc
ERROR :.
ERROR :IntegrityError
ERROR ::
ERROR :(psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "cooperatives_pkey"
DETAIL: Key (id)=(0) already exists.
[SQL: INSERT INTO cooperatives (id, name, cooperative_registration_date, phone, primary_type_id, primary_type, legal_status, municipality_id, inspector, address) VALUES (%(id)s, %(name)s, %(cooperative_registration_date)s, %(phone)s, %(primary_type_id)s, %(primary_type)s, %(legal_status)s, %(municipality_id)s, %(inspector)s, %(address)s)]
[parameters: ({'id': '570014712', 'name': 'אלסביל - אגודה שיתופית לאספקת מים בריחניה בע"מ', 'cooperative_registration_date': datetime.datetime(1958, 2, 26, 0, 0), 'phone': None, 'primary_type_id': 5, 'primary_type': 'חקלאית', 'legal_status': 'סגורה - בוטלה לאחר פירוק', 'municipality_id': 540, 'inspector': 'יעקב חליל', 'address': 'ריחאנייה 13818'}, {'id': '570025502', 'name': 'אלסד - אגודה שיתופית חקלאית למי השקאה בגוש חלב בע"מ', 'cooperative_registration_date': datetime.datetime(1978, 10, 1, 0, 0), 'phone': '04-6989217', 'primary_type_id': 5, 'primary_type': 'חקלאית', 'legal_status': 'פעילה - מאושרת', 'municipality_id': 487, 'inspector': 'יעקב חליל', 'address': "ג'ש (גוש חלב) 13872 ת.ד: 5"}, {'id': '570019901', 'name': 'אלסהלה - אגודה שיתופית למי שתיה בעין-אלסהלה בע"מ', 'cooperative_registration_date': datetime.datetime(1968, 6, 9, 0, 0), 'phone': '050-6240762', 'primary_type_id': 5, 'primary_type': 'חקלאית', 'legal_status': 'סגורה - בוטלה לאחר פירוק', 'municipality_id': 657, 'inspector': 'יעקב חליל', 'address': 'עין א-סהלה 30026'}, {'id': '570034785', 'name': 'אלסואנה - אגודה שיתופית להשקיה חקלאית בע"מ', 'cooperative_registration_date': datetime.datetime(1998, 5, 4, 0, 0), 'phone': None, 'primary_type_id': 5, 'primary_type': 'חקלאית', 'legal_status': 'פעילה - מאושרת', 'municipality_id': 487, 'inspector': 'יעקב חליל', 'address': "ג'ש (גוש חלב) 13872 ת.ד: 232"}, {'id': '570027508', 'name': 'אלסוסן - אגודה שיתופית חקלאית למי השקאה באבו סנאן בע"מ', 'cooperative_registration_date': datetime.datetime(1981, 10, 6, 0, 0), 'phone': None, 'primary_type_id': 5, 'primary_type': 'חקלאית', 'legal_status': 'סגורה - בוטלה לאחר פירוק', 'municipality_id': 473, 'inspector': 'יעקב חליל', 'address': 'אבו סנאן 24905'}, {'id': '570018911', 'name': 'אלסוקנה - אגודה שיתופית לשכון בנצרת בע"מ', 'cooperative_registration_date': datetime.datetime(1967, 3, 10, 0, 0), 'phone': None, 'primary_type_id': 12, 'primary_type': 'שיכון', 'legal_status': 'סגורה - בוטלה לאחר פירוק', 'municipality_id': 7300, 'inspector': None, 'address': '33 נצרת'}, {'id': '570035147', 'name': 'אלסחאב - אגודה שיתופית חקלאית למי השקאה בע"מ', 'cooperative_registration_date': datetime.datetime(1998, 12, 13, 0, 0), 'phone': '04-9944259', 'primary_type_id': 5, 'primary_type': 'חקלאית', 'legal_status': 'פעילה - מאושרת', 'municipality_id': 8900, 'inspector': 'יעקב חליל', 'address': 'טמרה טמרה 3081100 ת.ד: 4082 מיקוד ת.ד: 3081100'}, {'id': '570017004', 'name': 'אלסלאם - אגודה שיתופית חקלאית לאספקת מים בג\'ת בע"מ', 'cooperative_registration_date': datetime.datetime(1962, 4, 10, 0, 0), 'phone': '04-8578432', 'primary_type_id': 5, 'primary_type': 'חקלאית', 'legal_status': 'פעילה - מאושרת', 'municipality_id': 628, 'inspector': 'יעקב חליל', 'address': "ג'ת 3009100 ת.ד: 621 מיקוד ת.ד: 3009100"} ... displaying 10 of 1001 total bound parameter sets ... {'id': '570052746', 'name': 'דגן גלבוע - אגודה שיתופית חקלאית בע"מ', 'cooperative_registration_date': datetime.datetime(2009, 11, 23, 0, 0), 'phone': '04-6798851', 'primary_type_id': 5, 'primary_type': 'חקלאית', 'legal_status': 'פעילה - שינוי שם אגודה', 'municipality_id': 262, 'inspector': 'מזל שרעבי', 'address': 'גינוסר 1498000'}, {'id': '570052381', 'name': 'דגן גנוסר - אגודה שיתופית חקלאית בע"מ', 'cooperative_registration_date': datetime.datetime(2009, 7, 22, 0, 0), 'phone': '04-6798860', 'primary_type_id': 5, 'primary_type': 'חקלאית', 'legal_status': 'פעילה - מאושרת', 'municipality_id': 262, 'inspector': 'מזל שרעבי', 'address': 'גינוסר 1498000'})]
(Background on this error at: https://sqlalche.me/e/14/gkpj)
Pipeline ID: entities/cooperatives/cooperatives
Pipeline ID: entities/endowments/endowments
Invalid dependency :Cannot run until dependency passes validation: ./entities/all From ./entities/all :['Dependency unsuccessful', 'Cannot run until dependency "./entities/cooperatives/cooperatives" is successfully executed']
Pipeline ID: entities/entities
Invalid dependency :Cannot run until dependency passes validation: ./entities/all From ./entities/all :['Invalid dependency', 'Cannot run until dependency passes validation: ./entities/companies/registrar/details'] From ./entities/all :['From ./entities/companies/registrar/details', ['Dependency unsuccessful', 'Cannot run until dependency "./entities/companies/registrar/scraper" is successfully executed']] From ./entities/all :['Invalid dependency', 'Cannot run until dependency passes validation: ./entities/associations/guidestar/guidestar'] From ./entities/all :['From ./entities/associations/guidestar/guidestar', ['Dependency unsuccessful', 'Cannot run until dependency "./entities/associations/guidestar/guidestar-scraper" is successfully executed']] From ./entities/all :['Dependency unsuccessful', 'Cannot run until dependency "./entities/special/registry" is successfully executed'] From ./entities/all :['Dependency unsuccessful', 'Cannot run until dependency "./entities/cooperatives/cooperatives" is successfully executed']
Pipeline ID: entities/fingerprints
Pipeline ID: entities/foi_orgs/foi_orgs
scraper
Traceback (most recent call last):
File "/datapackage_pipelines_budgetkey/pipelines/entities/moital/scraper.py", line 79, in <module>
spew(datapackage, [scrape()])
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew
for rec in res:
File "/datapackage_pipelines_budgetkey/pipelines/entities/moital/scraper.py", line 32, in scrape
desired_capabilities=DesiredCapabilities.PHANTOMJS)
AttributeError: type object 'DesiredCapabilities' has no attribute 'PHANTOMJS'
Pipeline ID: entities/moital/moital_service_providers
רשימת אגודות עותומניות, מתוך תשובה לבקשת חופש מידע
Pipeline ID: entities/ottoman/ottoman-association-registry
Pipeline ID: entities/partnerships/partnerships
stream_remote_resources
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/stream_remote_resources.py", line 244, in <module>
rows = stream_reader(resource, url, ignore_missing or url == "", limit_rows,
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/stream_remote_resources.py", line 199, in stream_reader
schema, headers, columns, stream, close = get_opener(url, _resource)()
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/stream_remote_resources.py", line 175, in opener
_stream.open()
File "/usr/local/lib/python3.9/site-packages/tabulator/stream.py", line 423, in open
self.__parser.open(source, encoding=self.__encoding)
File "/usr/local/lib/python3.9/site-packages/tabulator/parsers/xlsx.py", line 98, in open
self.__book = openpyxl.load_workbook(
File "/usr/local/lib/python3.9/site-packages/openpyxl/reader/excel.py", line 315, in load_workbook
reader = ExcelReader(filename, read_only, keep_vba,
File "/usr/local/lib/python3.9/site-packages/openpyxl/reader/excel.py", line 124, in __init__
self.archive = _validate_archive(fn)
File "/usr/local/lib/python3.9/site-packages/openpyxl/reader/excel.py", line 96, in _validate_archive
archive = ZipFile(filename, 'r')
File "/usr/local/lib/python3.9/zipfile.py", line 1266, in __init__
self._RealGetContents()
File "/usr/local/lib/python3.9/zipfile.py", line 1333, in _RealGetContents
raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file
Pipeline ID: entities/safety-orders/safety-orders
Dependency unsuccessful :Cannot run until dependency "./entities/safety-orders/safety-orders" is successfully executed
Pipeline ID: entities/safety-orders/safety-orders-by-entity
Pipeline ID: entities/special/registry
Pipeline ID: ezvonot/ezvonot_accounts
Pipeline ID: ezvonot/ezvonot_cases
Pipeline ID: ezvonot/ezvonot_crontab
Pipeline ID: ezvonot/ezvonot_decisions
Pipeline ID: ezvonot/ezvonot_decisions_join
Pipeline ID: ezvonot/ezvonot_deposits
Pipeline ID: ezvonot/ezvonot_purposes
Pipeline ID: government_decisions/government_decisions
scraper
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 449, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 444, in _make_request
httplib_response = conn.getresponse()
File "/usr/local/lib/python3.9/http/client.py", line 1377, in getresponse
response.begin()
File "/usr/local/lib/python3.9/http/client.py", line 320, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.9/http/client.py", line 281, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/lib/python3.9/socket.py", line 704, in readinto
return self._sock.recv_into(b)
File "/usr/local/lib/python3.9/ssl.py", line 1242, in recv_into
return self.read(nbytes, buffer)
File "/usr/local/lib/python3.9/ssl.py", line 1100, in read
return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.9/site-packages/urllib3/packages/six.py", line 770, in reraise
raise value
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 451, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 340, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='www.gov.il', port=443): Read timed out. (read timeout=60)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/datapackage_pipelines_budgetkey/pipelines/government_decisions/scraper.py", line 128, in <module>
spew(datapackage, chain(res_iter, [get_decision_list()]))
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew
for rec in res:
File "/datapackage_pipelines_budgetkey/pipelines/government_decisions/scraper.py", line 90, in get_decision_list
response = session.get(SEARCH_PAGE_RESULTS_URL.format(skip=count), timeout=TIMEOUT).json()
File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 578, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='www.gov.il', port=443): Read timed out. (read timeout=60)
Pipeline ID: government_decisions/scraper
Pipeline ID: lamas/municipal-data
Pipeline ID: lamas/municipality-to-city
Pipeline ID: maya/maya-bonds-and-options-holdings-changes
Pipeline ID: maya/maya-company-officer-list
Pipeline ID: maya/maya-company-stakeholder-list
flow
ERROR :Traceback (most recent call last):
ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 15, in <module>
ERROR :spew_flow(flow, ctx)
ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__
ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats)
ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew
ERROR :for rec in res:
ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/delete_fields.py", line 7, in process_resource
ERROR :for row in rows:
ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource
ERROR :yield from self.func(resource)
ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/maya_parser_utils.py", line 52, in iter
ERROR :for row in rows:
ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource
ERROR :yield from self.func(resource)
ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/parse_maya_vip_appointments.py", line 139, in parse_document
ERROR :for row in rows:
ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource
ERROR :yield from self.func(resource)
ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/parse_maya_vip_appointments.py", line 118, in validate
ERROR :verify_row_values(row, 'EmployedAtAnotherJobConnectedToTheCompany', EMPLOYED_IN_CONNECTED_COMPANY)
ERROR :File "/datapackage_pipelines_budgetkey/pipelines/maya/maya_parser_utils.py", line 35, in verify_row_values
ERROR :raise Exception("Document {} failed validation. textAlias {} had illegal value: {}".format(url, field, doc[field]))
ERROR :Exception
ERROR ::
ERROR :Document https://mayafiles.tase.co.il/RHtm/1497001-1498000/H1497830.htm failed validation. textAlias EmployedAtAnotherJobConnectedToTheCompany had illegal value: ['Does not hold']
Pipeline ID: maya/maya-company-vip-appointments
Pipeline ID: maya/maya-complete-notification-list
Pipeline ID: maya/maya-holdings-changes
Pipeline ID: maya/maya-notification-parse-updates
Pipeline ID: maya/maya-reported-academic-degrees
Pipeline ID: maya/maya-reported-work-record
Pipeline ID: maya/maya-tase-companies-current-management
Pipeline ID: maya/scrape-maya-notification-list
Pipeline ID: maya/scrape-maya-tase-companies
Invalid dependency :Cannot run until dependency passes validation: ./people/all From ./people/all :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/all-appointments'] From ./people/all :['From ./people/company_appointments/all-appointments', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/media/themarker/themarker" is successfully executed']] From ./people/all :['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations']] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]] From ./people/all :['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations']] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]] From ./people/all :['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/state_owned/state-owned-nominations']] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/state_owned/state-owned-nominations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/state_owned/correlate']]] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/state_owned/state-owned-nominations', ['From ./people/company_appointments/state_owned/correlate', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/state_owned/parse_pdf_resources']]]] From ./people/all :['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/state_owned/state-owned-nominations', ['From ./people/company_appointments/state_owned/correlate', ['From ./people/company_appointments/state_owned/parse_pdf_resources', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/state_owned/reports" is successfully executed']]]]] From ./people/all :['Dependency unsuccessful', 'Cannot run until dependency "./people/association_founders/association-founders" is successfully executed'] From ./people/all :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/procurement/procurement-individuals'] From ./people/all :['From ./people/procurement/procurement-individuals', ['Invalid dependency', 'Cannot run until dependency passes validation: ./procurement/spending/latest-contract-spending']] From ./people/all :['From ./people/procurement/procurement-individuals', ['From ./procurement/spending/latest-contract-spending', ['Dependency unsuccessful', 'Cannot run until dependency "./procurement/spending/quarterly-contract-spending-reports-data" is successfully executed']]] From ./people/all :['Dependency unsuccessful', 'Cannot run until dependency "./people/political_donations/political_donations" is successfully executed']
Pipeline ID: people/aggregated
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/all-appointments From ./people/company_appointments/all-appointments :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations'] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-nominations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]] From ./people/company_appointments/all-appointments :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations'] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-resignations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]] From ./people/company_appointments/all-appointments :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/state_owned/state-owned-nominations'] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/state_owned/state-owned-nominations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/state_owned/correlate']] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/state_owned/state-owned-nominations', ['From ./people/company_appointments/state_owned/correlate', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/state_owned/parse_pdf_resources']]] From ./people/company_appointments/all-appointments :['From ./people/company_appointments/state_owned/state-owned-nominations', ['From ./people/company_appointments/state_owned/correlate', ['From ./people/company_appointments/state_owned/parse_pdf_resources', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/state_owned/reports" is successfully executed']]]] Dependency unsuccessful :Cannot run until dependency "./people/political_donations/political_donations" is successfully executed
Pipeline ID: people/all
Pipeline ID: people/association_founders/association-founders
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations From ./people/company_appointments/maya/maya-nominations :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list'] From ./people/company_appointments/maya/maya-nominations :['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']] From ./people/company_appointments/maya/maya-nominations :['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]] Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations From ./people/company_appointments/maya/maya-resignations :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list'] From ./people/company_appointments/maya/maya-resignations :['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']] From ./people/company_appointments/maya/maya-resignations :['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]] Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/state_owned/state-owned-nominations From ./people/company_appointments/state_owned/state-owned-nominations :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/state_owned/correlate'] From ./people/company_appointments/state_owned/state-owned-nominations :['From ./people/company_appointments/state_owned/correlate', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/state_owned/parse_pdf_resources']] From ./people/company_appointments/state_owned/state-owned-nominations :['From ./people/company_appointments/state_owned/correlate', ['From ./people/company_appointments/state_owned/parse_pdf_resources', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/state_owned/reports" is successfully executed']]]
Pipeline ID: people/company_appointments/all-appointments
Dependency unsuccessful :Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed
Pipeline ID: people/company_appointments/maya/collect-maya-forms-debug-stats
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations From ./people/company_appointments/maya/maya-nominations :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list'] From ./people/company_appointments/maya/maya-nominations :['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']] From ./people/company_appointments/maya/maya-nominations :['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]] Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations From ./people/company_appointments/maya/maya-resignations :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list'] From ./people/company_appointments/maya/maya-resignations :['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']] From ./people/company_appointments/maya/maya-resignations :['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]
Pipeline ID: people/company_appointments/maya/correlate
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates From ./people/company_appointments/maya/maya-notification-updates :['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']
Pipeline ID: people/company_appointments/maya/maya-list
Name, job and company description.
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list From ./people/company_appointments/maya/maya-list :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates'] From ./people/company_appointments/maya/maya-list :['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]
Pipeline ID: people/company_appointments/maya/maya-nominations
scrape-notification-list
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/datapackage_pipelines_budgetkey/pipelines/people/company_appointments/maya/scrape-notification-list.py", line 55, in _maya_api_call
return res.json()
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/datapackage_pipelines_budgetkey/pipelines/people/company_appointments/maya/scrape-notification-list.py", line 130, in <module>
spew(datapackage, [collect()])
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew
for rec in res:
File "/datapackage_pipelines_budgetkey/pipelines/people/company_appointments/maya/scrape-notification-list.py", line 113, in collect
yield from _collect_date_range(year_start, year_end)
File "/datapackage_pipelines_budgetkey/pipelines/people/company_appointments/maya/scrape-notification-list.py", line 75, in _collect_date_range
res = _maya_api_call(date_from, date_to, current_page)
File "/datapackage_pipelines_budgetkey/pipelines/people/company_appointments/maya/scrape-notification-list.py", line 57, in _maya_api_call
raise Exception("Failed to Call Maya API for date_from:{} date_to:{} page_num:{}".format(date_from, date_to, page_num)) from e
Exception: Failed to Call Maya API for date_from:2006-01-01 date_to:2007-01-01 page_num:0
Pipeline ID: people/company_appointments/maya/maya-notification-list
id, date, company, notification_type, s3_object_name, url
Dependency unsuccessful :Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed
Pipeline ID: people/company_appointments/maya/maya-notification-updates
Name, job and company description.
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list From ./people/company_appointments/maya/maya-list :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates'] From ./people/company_appointments/maya/maya-list :['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]
Pipeline ID: people/company_appointments/maya/maya-resignations
רשימת כל המינויים והחברות מאתר כלכליסט
Pipeline ID: people/company_appointments/media/calcalist/calcalist
Name, job and company description.
Pipeline ID: people/company_appointments/media/themarker/themarker
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/state_owned/parse_pdf_resources From ./people/company_appointments/state_owned/parse_pdf_resources :['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/state_owned/reports" is successfully executed']
Pipeline ID: people/company_appointments/state_owned/correlate
parse_pdf
Traceback (most recent call last):
File "/datapackage_pipelines_budgetkey/pipelines/people/company_appointments/state_owned/parse_pdf.py", line 19, in <module>
import tabula
File "/usr/local/lib/python3.9/site-packages/tabula/__init__.py", line 3, in <module>
from .io import convert_into, convert_into_by_batch, read_pdf, read_pdf_with_template
File "/usr/local/lib/python3.9/site-packages/tabula/io.py", line 33, in <module>
import pandas as pd
File "/usr/local/lib/python3.9/site-packages/pandas/__init__.py", line 48, in <module>
from pandas.core.api import (
File "/usr/local/lib/python3.9/site-packages/pandas/core/api.py", line 22, in <module>
from pandas.core.algorithms import (
File "/usr/local/lib/python3.9/site-packages/pandas/core/algorithms.py", line 40, in <module>
from pandas.core.dtypes.cast import (
File "/usr/local/lib/python3.9/site-packages/pandas/core/dtypes/cast.py", line 23, in <module>
from dateutil.parser import ParserError
ImportError: cannot import name 'ParserError' from 'dateutil.parser' (/usr/local/lib/python3.9/site-packages/dateutil/parser/__init__.py)
Pipeline ID: people/company_appointments/state_owned/parse_pdf_resources
columns: date, url
Pipeline ID: people/company_appointments/state_owned/reports
Invalid dependency :Cannot run until dependency passes validation: ./people/company_appointments/state_owned/correlate From ./people/company_appointments/state_owned/correlate :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/state_owned/parse_pdf_resources'] From ./people/company_appointments/state_owned/correlate :['From ./people/company_appointments/state_owned/parse_pdf_resources', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/state_owned/reports" is successfully executed']]
Pipeline ID: people/company_appointments/state_owned/state-owned-nominations
Pipeline ID: people/municipal_appointments/municipal_representitives
Pipeline ID: people/political_candidates/election_candidates21
flow
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.9/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/datapackage/helpers.py", line 59, in retrieve_descriptor
the_descriptor = req.json()
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/dataflows/processors/load.py", line 121, in process_datapackage
return self.safe_process_datapackage(dp)
File "/usr/local/lib/python3.9/site-packages/dataflows/processors/load.py", line 159, in safe_process_datapackage
self.load_dp = Package(self.load_source)
File "/usr/local/lib/python3.9/site-packages/datapackage/package.py", line 102, in __init__
descriptor = helpers.retrieve_descriptor(descriptor)
File "/usr/local/lib/python3.9/site-packages/datapackage/helpers.py", line 62, in retrieve_descriptor
six.raise_from(exceptions.DataPackageException(message), error)
File "<string>", line 3, in raise_from
datapackage.exceptions.DataPackageException: Unable to load JSON at "https://next.obudget.org/datapackages/donations/transactions/datapackage.json"
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 79, in _process
self.datapackage = self.process_datapackage(self.datapackage)
File "/usr/local/lib/python3.9/site-packages/dataflows/processors/load.py", line 123, in process_datapackage
raise SourceLoadError('Failed to load source {!r} and options {!r}: {}'
dataflows.base.exceptions.SourceLoadError: Failed to load source 'https://next.obudget.org/datapackages/donations/transactions/datapackage.json' and options {}: Unable to load JSON at "https://next.obudget.org/datapackages/donations/transactions/datapackage.json"
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 15, in <module>
spew_flow(flow, ctx)
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/utilities/flow_utils.py", line 46, in spew_flow
datastream = flow.datastream()
File "/usr/local/lib/python3.9/site-packages/dataflows/base/flow.py", line 18, in datastream
return self._chain(ds)._process()
File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process
datastream = self.source._process()
File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process
datastream = self.source._process()
File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 86, in _process
self.raise_exception(exception)
File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 96, in raise_exception
raise error from cause
dataflows.base.exceptions.ProcessorError: Errored in processor load in position #1: Failed to load source 'https://next.obudget.org/datapackages/donations/transactions/datapackage.json' and options {}: Unable to load JSON at "https://next.obudget.org/datapackages/donations/transactions/datapackage.json"
Pipeline ID: people/political_donations/political_donations
Pipeline ID: people/procurement/procurement-individuals
flow
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 79, in _process
self.datapackage = self.process_datapackage(self.datapackage)
File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/iterable_loader.py", line 112, in process_datapackage
raise self.exc
File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/iterable_loader.py", line 60, in describe
sample = list(itertools.islice(self.iterable, self.SAMPLE_SIZE))
File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/iterable_loader.py", line 89, in handle_iterable
for x in self.iterable:
File "/datapackage_pipelines_budgetkey/pipelines/procurement/calls_for_bids/class_action.py", line 34, in scrape
ret['claim_date'] = ret['claim_date'].split(' ')[0] if ret['claim_date'] else None
KeyError: 'claim_date'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 15, in <module>
spew_flow(flow, ctx)
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/utilities/flow_utils.py", line 46, in spew_flow
datastream = flow.datastream()
File "/usr/local/lib/python3.9/site-packages/dataflows/base/flow.py", line 18, in datastream
return self._chain(ds)._process()
File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process
datastream = self.source._process()
File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process
datastream = self.source._process()
File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process
datastream = self.source._process()
[Previous line repeated 1 more time]
File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 86, in _process
self.raise_exception(exception)
File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 96, in raise_exception
raise error from cause
dataflows.base.exceptions.ProcessorError: Errored in processor iterable_loader in position #1: 'claim_date'
DEBUG :[chan 2] Max packet in: 32768 bytes
DEBUG :[chan 2] Max packet out: 32768 bytes
DEBUG :Secsh channel 2 opened.
DEBUG :[chan 2] Sesch channel 2 request ok
DEBUG :[chan 2] EOF received (2)
DEBUG :[chan 2] EOF sent (2)
DEBUG :Dropping user packet because connection is dead.
Pipeline ID: procurement/calls_for_bids/calls-for-bids-all
flow
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 79, in _process
self.datapackage = self.process_datapackage(self.datapackage)
File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/iterable_loader.py", line 112, in process_datapackage
raise self.exc
File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/iterable_loader.py", line 60, in describe
sample = list(itertools.islice(self.iterable, self.SAMPLE_SIZE))
File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/iterable_loader.py", line 89, in handle_iterable
for x in self.iterable:
File "/datapackage_pipelines_budgetkey/pipelines/procurement/calls_for_bids/m_tmicha.py", line 47, in m_tmicha_scraper
assert total > 0
AssertionError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 15, in <module>
spew_flow(flow, ctx)
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/utilities/flow_utils.py", line 46, in spew_flow
datastream = flow.datastream()
File "/usr/local/lib/python3.9/site-packages/dataflows/base/flow.py", line 18, in datastream
return self._chain(ds)._process()
File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process
datastream = self.source._process()
File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 75, in _process
datastream = self.source._process()
File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 86, in _process
self.raise_exception(exception)
File "/usr/local/lib/python3.9/site-packages/dataflows/base/datastream_processor.py", line 96, in raise_exception
raise error from cause
dataflows.base.exceptions.ProcessorError: Errored in processor iterable_loader in position #1:
Pipeline ID: procurement/calls_for_bids/calls-for-bids-moh
Pipeline ID: procurement/calls_for_bids/gov-publications
Pipeline ID: procurement/calls_for_bids/gov-publications-ministry-of-immigration
Pipeline ID: procurement/calls_for_bids/gov-publications2
Pipeline ID: procurement/calls_for_bids/jobiz-all
Pipeline ID: procurement/calls_for_bids/jobiz-calls-for-bids
Pipeline ID: procurement/calls_for_bids/jobiz-scraper
Pipeline ID: procurement/municipal/fetcher
אנו סוכמים מידע מתוך הדוחות הרבעוניים לתוך טבלה המכילה את המידע העדכני ביותר.
Pipeline ID: procurement/spending/latest-contract-spending
Pipeline ID: procurement/spending/quarterly-contract-spending-report-uris
כל משרדי הממשלה ויחידות הסמך (שמחוברות למערכת מרכבה) צריכים לפרסם אחת לרבעון (תוך 45 יום מסוף הרבעון) דו״ח התקשרויות לפי פורמט מוגדר היטב הממומש כדו״ח מובנה במערכת המרכבה. כל המשרדים מדווחים עצמאית והדו״חות עולים לאתר היחידה לחופש המידע, משם אנו אוספים אותם. התוצר של השרשר הזה הוא פירוט כל הדיווחים של כל המשרדים.
Pipeline ID: procurement/spending/quarterly-contract-spending-reports
Pipeline ID: procurement/spending/quarterly-contract-spending-reports-data
Pipeline ID: procurement/spending/spending-by-entity
Pipeline ID: procurement/tenders/all
Pipeline ID: procurement/tenders/processed
Dependency unsuccessful :Cannot run until dependency "./reports/ngos/ngo-district-report" is successfully executed Dependency unsuccessful :Cannot run until dependency "./reports/ngos/ngo-activity-report" is successfully executed
Pipeline ID: reports/all
filter ERROR :Traceback (most recent call last): ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/filter.py", line 18, in <module> ERROR :spew_flow(flow(ctx.parameters), ctx) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew ERROR :for rec in res: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/filter_rows.py", line 20, in process_resource ERROR :if condition(row): ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/filter_rows.py", line 6, in func ERROR :return any( ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/filter_rows.py", line 7, in <genexpr> ERROR :(row[k] == v) ERROR :KeyError ERROR :: ERROR :'association_has_article_46'
Pipeline ID: reports/ngos/ngo-activity-report
filter ERROR :Traceback (most recent call last): ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/filter.py", line 18, in <module> ERROR :spew_flow(flow(ctx.parameters), ctx) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__ ERROR :spew(self.datapackage, self.resource_iterator, stats=self.stats) ERROR :File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew ERROR :for rec in res: ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/filter_rows.py", line 20, in process_resource ERROR :if condition(row): ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/filter_rows.py", line 6, in func ERROR :return any( ERROR :File "/usr/local/lib/python3.9/site-packages/dataflows/processors/filter_rows.py", line 7, in <genexpr> ERROR :(row[k] == v) ERROR :KeyError ERROR :: ERROR :'association_has_article_46'
Pipeline ID: reports/ngos/ngo-district-report
Pipeline ID: supports/all
Pipeline ID: supports/by-payment-year
Pipeline ID: supports/by-request-year