Skip to content

feat: Onboard EPA historical air quality dataset#221

Merged
adlersantos merged 28 commits intoGoogleCloudPlatform:mainfrom
nlarge-google:EPA
Dec 8, 2021
Merged

feat: Onboard EPA historical air quality dataset#221
adlersantos merged 28 commits intoGoogleCloudPlatform:mainfrom
nlarge-google:EPA

Conversation

@nlarge-google
Copy link
Copy Markdown
Contributor

@nlarge-google nlarge-google commented Nov 1, 2021 •
edited
Loading

Description

Included:
Annual summaries
CO Daily Summary
CO Hourly Summary
HAP Daily Summary
HAP Hourly Summary
Lead Daily Summary
NO2 Daily Summary
NO2 Hourly Summary
NONOxNOy Daily Summary
NONOxNOy Hourly Summary
Ozone Daily Summary
Ozone Hourly Summary
PM 10 Daily Summary
PM10 Hourly Summary
PM25 Frm Hourly Summary
PM25 NonFrm Daily Summary
PM25 NonFrm Hourly Summary
PM25 Speciation Daily Summary
PM25 Speciation Hourly Summary
Pressure Daily Summary
Pressure Hourly Summary
RH and DP Daily Summary
RH and DP Hourly Summary
SO2 Daily Summary
SO2 Hourly Summary
Temperature Daily Summary
Temperature Hourly Summary
VOC Daily Summary
VOC Hourly Summary
Wind Daily Summary
Wind Hourly Summary

Checklist

Note: Delete items below that aren't applicable to your pull request.

  • Please merge this PR for me once it is approved.
  • If this PR adds or edits a dataset or pipeline, it was reviewed and approved by the Google Cloud Public Datasets team beforehand.
  • If this PR adds or edits a dataset or pipeline, I put all my code inside datasets/epa_historical_air_quality and nothing outside of that directory.
  • This PR is appropriately labeled.

@nlarge-google
feat: Added annual_summaries, tested locally. Errors in AF
b717e9d
@nlarge-google
feat: Added co_daily_summaries. Not ready for production
bb846aa
@nlarge-google
feat: Added co_hourly_summary. Not ready for production.
953d1cd
@nlarge-google
fix: Changed dataset name
e3966b4
@nlarge-google
fix: Attempt to resolve AF load_to_bq errors
71a9d26
@nlarge-google
fix: Resolves issues with AF failure to execute DAG, also, some datat…
5efa0b5 
…ype fixes.
@nlarge-google
feat: Added HAP Hourly Summary. Fixed schema issues in HAP Daily Summ…
7a64407 
…ary.
@nlarge-google
fix: datatype fixes
27d962e
@nlarge-google
feat: Added no2 daily and hourly.
38d4e6a
@nlarge-google
feat: Added NONOxNOy pipelines daily and hourly
ac99d56
@nlarge-google
feat: Added multiple pipelines and assigned destination table to a va…
f4c09c8 
…riable value for each.
@nlarge-google
fix: Added terraform files for new pipelines.
9625f2d
@nlarge-google
fix: Regenerated some dags.
c2fb025
@nlarge-google
fix: Resolved variable issues in pipeline.yaml files so that they ope…
9535847 
…rate as intended in AF
@nlarge-google
fix: clean-up code
54c3d7f
@nlarge-google
fix: Reduced CHUNKSIZE in order to prevent memory outage in AF, preve…
7d93681 
…nting some pipelines to complete successfully.
@nlarge-google
fix: Resolved incorrect path entry in ozone daily summary pipeline.ya…
7d0f21e 
…ml; Fixed flake8 issue in csv_transform
@google-cla google-cla Bot added the cla: yes label Nov 1, 2021
@nlarge-google
Copy link
Copy Markdown
Contributor Author

nlarge-google commented Nov 1, 2021

@adlersantos Ready for review. Thanks!

adlersantos
adlersantos requested changes Nov 3, 2021
View reviewed changes
Copy link
Copy Markdown
Member

@adlersantos adlersantos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good in general. Just a few comments.

Comment thread datasets/epa_historical_air_quality/_images/run_csv_transform_kub/csv_transform.py Outdated
Comment thread datasets/epa_historical_air_quality/_images/run_csv_transform_kub/csv_transform.py Outdated
Comment thread datasets/epa_historical_air_quality/_images/run_csv_transform_kub/csv_transform.py Outdated
Comment thread datasets/epa_historical_air_quality/_images/run_csv_transform_kub/csv_transform.py Outdated
Comment thread datasets/epa_historical_air_quality/_images/run_csv_transform_kub/requirements.txt Outdated
Comment thread datasets/epa_historical_air_quality/annual_summaries/pipeline.yaml Outdated
Comment thread datasets/epa_historical_air_quality/co_daily_summary/pipeline.yaml Outdated
Comment thread datasets/epa_historical_air_quality/co_hourly_summary/pipeline.yaml Outdated
@nlarge-google
Copy link
Copy Markdown
Contributor Author

nlarge-google commented Nov 4, 2021 •
edited
Loading

@adlersantos I have done research into trying to understand why this PR keeps failing at the Run Tests point. Is there some recent change to the code behind these checks? What changes do I need to make to ensure that these checks pass?
It looks like other developers PRs are experiencing the same issue and it is not just this one.

Thanks!

@nlarge-google nlarge-google reopened this Nov 4, 2021
@nlarge-google
Copy link
Copy Markdown
Contributor Author

nlarge-google commented Nov 4, 2021

@adlersantos I dont see any button in github UI that allows me to rerun the checks without making code changes, so I had to close and reopen the PR in order to trigger the tests. If you know a better way to re-trigger the checks, or if closing and re-opening the PR is acceptable then please let me know.

@nlarge-google nlarge-google reopened this Nov 4, 2021
@nlarge-google
Copy link
Copy Markdown
Contributor Author

nlarge-google commented Nov 5, 2021

@adlersantos Please review this code at your earliest convenience. Thanks!

@nlarge-google
Copy link
Copy Markdown
Contributor Author

nlarge-google commented Nov 23, 2021

@adlersantos I have made the changes. Please review. Thanks.

@nlarge-google
fix: Increase memory and CPU for pressure_daily_summary and reduced b…
5bf3062 
…atch size in order to resolve memory pressure resulting in DAG failure.
@nlarge-google
Copy link
Copy Markdown
Contributor Author

nlarge-google commented Nov 29, 2021

@adlersantos All requested changes are completed. Please review again. Thanks!

adlersantos
adlersantos approved these changes Dec 8, 2021
View reviewed changes
Copy link
Copy Markdown
Member

@adlersantos adlersantos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot!

@adlersantos adlersantos merged commit 6267b82 into GoogleCloudPlatform:main Dec 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adlersantos adlersantos adlersantos approved these changes

Assignees

@adlersantos adlersantos

Labels

cla: yes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nlarge-google @adlersantos