Pipeline ID: budgetkey/elasticsearch/index_activities
Pipeline ID: budgetkey/elasticsearch/index_budget
Invalid dependency :Cannot run until dependency passes validation: ./procurement/calls_for_bids/calls-for-bids-all From ./procurement/calls_for_bids/calls-for-bids-all :['Dependency unsuccessful', 'Cannot run until dependency "./procurement/calls_for_bids/gov-publications" is successfully executed'] From ./procurement/calls_for_bids/calls-for-bids-all :['Dependency unsuccessful', 'Cannot run until dependency "./procurement/calls_for_bids/gov-publications2" is successfully executed'] From ./procurement/calls_for_bids/calls-for-bids-all :['Dependency unsuccessful', 'Cannot run until dependency "./procurement/calls_for_bids/gov-publications-ministry-of-immigration" is successfully executed']
Pipeline ID: budgetkey/elasticsearch/index_calls_for_bids
Pipeline ID: budgetkey/elasticsearch/index_contract_spending
Pipeline ID: budgetkey/elasticsearch/index_entities
Pipeline ID: budgetkey/elasticsearch/index_gov_decisions
Pipeline ID: budgetkey/elasticsearch/index_muni_budgets
Pipeline ID: budgetkey/elasticsearch/index_muni_tenders
flow
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/openai/_base_client.py", line 1113, in request
response.raise_for_status()
File "/usr/local/lib/python3.9/site-packages/httpx/_models.py", line 829, in raise_for_status
raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://api.openai.com/v1/embeddings'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400
DEBUG :Not retrying
DEBUG :Re-raising status error
ERROR :Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 15, in <module>
spew_flow(flow, ctx)
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__
spew(self.datapackage, self.resource_iterator, stats=self.stats)
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew
for rec in res:
File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource
yield from self.func(resource)
File "/datapackage_pipelines_budgetkey/pipelines/budgetkey/elasticsearch/chunker.py", line 47, in func
hit, embedding = embed(chunk)
File "/datapackage_pipelines_budgetkey/common/cached_openai.py", line 42, in embed
embedding = client.embeddings.create(
File "/usr/local/lib/python3.9/site-packages/openai/resources/embeddings.py", line 136, in create
return self._post(
File "/usr/local/lib/python3.9/site-packages/openai/_base_client.py", line 1360, in post
return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
File "/usr/local/lib/python3.9/site-packages/openai/_base_client.py", line 1133, in request
raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid 'input': maximum context length is 8192 tokens.", 'type': 'invalid_request_error', 'param': None, 'code': None}}
DEBUG :close.started
DEBUG :close.complete
Pipeline ID: budgetkey/elasticsearch/index_national_budget_changes
Invalid dependency :Cannot run until dependency passes validation: ./people/aggregated From ./people/aggregated :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/all'] From ./people/aggregated :['From ./people/all', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/all-appointments']] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/media/themarker/themarker" is successfully executed']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]]] From ./people/aggregated :['From ./people/all', ['Dependency unsuccessful', 'Cannot run until dependency "./people/political_candidates/election_candidates21" is successfully executed']]
Pipeline ID: budgetkey/elasticsearch/index_people
Pipeline ID: budgetkey/elasticsearch/index_reports
Dependency unsuccessful :Cannot run until dependency "./supports/criteria/support-criteria" is successfully executed
Pipeline ID: budgetkey/elasticsearch/index_support_criteria
Pipeline ID: budgetkey/elasticsearch/index_support_programs
Pipeline ID: budgetkey/elasticsearch/index_supports
Pipeline ID: budgetkey/elasticsearch/index_tenders
Pipeline ID: budgetkey/elasticsearch/index_units
Dependency unsuccessful :Cannot run until dependency "./budgetkey/elasticsearch/index_national_budget_changes" is successfully executed Dependency unsuccessful :Cannot run until dependency "./budgetkey/elasticsearch/index_entities" is successfully executed
Pipeline ID: budgetkey/elasticsearch/sitemaps
Invalid dependency :Cannot run until dependency passes validation: ./procurement/calls_for_bids/calls-for-bids-all From ./procurement/calls_for_bids/calls-for-bids-all :['Dependency unsuccessful', 'Cannot run until dependency "./procurement/calls_for_bids/gov-publications" is successfully executed'] From ./procurement/calls_for_bids/calls-for-bids-all :['Dependency unsuccessful', 'Cannot run until dependency "./procurement/calls_for_bids/gov-publications2" is successfully executed'] From ./procurement/calls_for_bids/calls-for-bids-all :['Dependency unsuccessful', 'Cannot run until dependency "./procurement/calls_for_bids/gov-publications-ministry-of-immigration" is successfully executed']
Pipeline ID: budgetkey/elasticsearch/index_calls_for_bids
Invalid dependency :Cannot run until dependency passes validation: ./people/aggregated From ./people/aggregated :['Invalid dependency', 'Cannot run until dependency passes validation: ./people/all'] From ./people/aggregated :['From ./people/all', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/all-appointments']] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/media/themarker/themarker" is successfully executed']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-nominations']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-nominations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-resignations']]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-list']]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['Invalid dependency', 'Cannot run until dependency passes validation: ./people/company_appointments/maya/maya-notification-updates']]]]] From ./people/aggregated :['From ./people/all', ['From ./people/company_appointments/all-appointments', ['From ./people/company_appointments/maya/maya-resignations', ['From ./people/company_appointments/maya/maya-list', ['From ./people/company_appointments/maya/maya-notification-updates', ['Dependency unsuccessful', 'Cannot run until dependency "./people/company_appointments/maya/maya-notification-list" is successfully executed']]]]]] From ./people/aggregated :['From ./people/all', ['Dependency unsuccessful', 'Cannot run until dependency "./people/political_candidates/election_candidates21" is successfully executed']]
Pipeline ID: budgetkey/elasticsearch/index_people
Dependency unsuccessful :Cannot run until dependency "./supports/criteria/support-criteria" is successfully executed
Pipeline ID: budgetkey/elasticsearch/index_support_criteria
Dependency unsuccessful :Cannot run until dependency "./budgetkey/elasticsearch/index_national_budget_changes" is successfully executed Dependency unsuccessful :Cannot run until dependency "./budgetkey/elasticsearch/index_entities" is successfully executed
Pipeline ID: budgetkey/elasticsearch/sitemaps
Pipeline ID: budgetkey/elasticsearch/index_entities
flow
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/openai/_base_client.py", line 1113, in request
response.raise_for_status()
File "/usr/local/lib/python3.9/site-packages/httpx/_models.py", line 829, in raise_for_status
raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://api.openai.com/v1/embeddings'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400
DEBUG :Not retrying
DEBUG :Re-raising status error
ERROR :Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/specs/../lib/flow.py", line 15, in <module>
spew_flow(flow, ctx)
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 181, in __exit__
spew(self.datapackage, self.resource_iterator, stats=self.stats)
File "/usr/local/lib/python3.9/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 75, in spew
for rec in res:
File "/usr/local/lib/python3.9/site-packages/dataflows/helpers/rows_processor.py", line 11, in process_resource
yield from self.func(resource)
File "/datapackage_pipelines_budgetkey/pipelines/budgetkey/elasticsearch/chunker.py", line 47, in func
hit, embedding = embed(chunk)
File "/datapackage_pipelines_budgetkey/common/cached_openai.py", line 42, in embed
embedding = client.embeddings.create(
File "/usr/local/lib/python3.9/site-packages/openai/resources/embeddings.py", line 136, in create
return self._post(
File "/usr/local/lib/python3.9/site-packages/openai/_base_client.py", line 1360, in post
return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
File "/usr/local/lib/python3.9/site-packages/openai/_base_client.py", line 1133, in request
raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid 'input': maximum context length is 8192 tokens.", 'type': 'invalid_request_error', 'param': None, 'code': None}}
DEBUG :close.started
DEBUG :close.complete
Pipeline ID: budgetkey/elasticsearch/index_national_budget_changes
Pipeline ID: budgetkey/elasticsearch/index_activities
Pipeline ID: budgetkey/elasticsearch/index_budget
Pipeline ID: budgetkey/elasticsearch/index_contract_spending
Pipeline ID: budgetkey/elasticsearch/index_gov_decisions
Pipeline ID: budgetkey/elasticsearch/index_muni_budgets
Pipeline ID: budgetkey/elasticsearch/index_muni_tenders
Pipeline ID: budgetkey/elasticsearch/index_reports
Pipeline ID: budgetkey/elasticsearch/index_support_programs
Pipeline ID: budgetkey/elasticsearch/index_supports
Pipeline ID: budgetkey/elasticsearch/index_tenders
Pipeline ID: budgetkey/elasticsearch/index_units