Update minimum required Terraform version

Updated minimum terraform version to 1.4.5, resolving deprecations.

The following import statements may need to be run before terraform apply (with GLUE_SCRIPTS_BUCKET_NAME and MANIFEST_BUCKET_NAME replaced with the actual S3 bucket names):

terraform import aws_s3_bucket_ownership_controls.glue_resources GLUE_SCRIPTS_BUCKET_NAME
terraform import aws_s3_bucket_lifecycle_configuration.glue_resources GLUE_SCRIPTS_BUCKET_NAME

terraform import -provider=aws.controlshift aws_s3_bucket_ownership_controls.manifest MANIFEST_BUCKET_NAME

Thanks @strangeways for updating!

Fix on column mappings

Use character varying for hstore and jsonb columns instead of the unsupported string data type. Thanks @sjwmoveon for the fix!

Update to use new version of aws-lambda-redshift-loader

Use version 2.8.2 of aws-lambda-redshift-loader

Update Lambdas runtime to Node 16.x

Node 12.x runtime is deprecated and will stop getting security updates on Nov. 4th 2022. Full update from AWS: https://aws.amazon.com/blogs/developer/announcing-the-end-of-support-for-node-js-12-x-in-the-aws-sdk-for-javascript-v3/

Dynamically set Glue job script column mappings on terraform apply

The signatures_job.py script is now generated with the column mappings between CSV files and Redshift based on the signatures table schema as returned by the /api/bulk_data/schema.json API endpoint.
This should make it easier to update the script whenever there are schema changes in the signatures table, just by running terraform apply.

Fixes on column mappings

Fixed data types for multiple columns on the signatures_job.py script that prevented the values from being correctly imported to Redshift.

Update terraform AWS provider to v4

The expected changes on existing resources are the following:

• New aws_s3_bucket_acl.glue_resources resource for setting ACL on the glue scripts bucket. ACL hasn't changed, bucket is still private.
• New aws_s3_bucket_acl.manifest resource for setting ACL on the manifests bucket. ACL hasn't changed, bucket is still private.
• New aws_s3_bucket_lifecycle_configuration.manifest resource for setting the life cycle policy for files in the manifests bucket. Life cycle policy hasn't change, files will still be removed after 5 days.
• New aws_s3_bucket_server_side_encryption_configuration.glue_resources resource for setting server-side encryption on the glue scripts bucket.
• New aws_s3_bucket_server_side_encryption_configuration.manifest resource for setting server-side encryption on the manifests bucket.
• Resource aws_s3_bucket_object.signatures_script is replaced by aws_s3_object.signatures_script. This effectively creates a new file in S3 with the script run by Glue, the script itself hasn't changed though.

Handle CSVs compression

Adds support for setting the compress value for controlshift-redshift-loader Lambda function based on the value returned from the ControlShift instance API.

Upgrade Job to Glue 3.0

Glue 3.0 introduced several improvements, most notably, performance has been improved noticeably (you can see the release notes here).

The upgrade should be transparent if you haven't modified the Job's script, otherwise you can check out the documentation on migrating from AWS Glue 1.0 to AWS Glue 3.0.

Fix VPC endpoint for S3

Fixes a bug where the VPC Endpoint for S3 was created without an associated route table, making it unusable.