* Upgrade yarn packages to clear dependabot warnings
* Add prettier
* Run prettier
* Update docs
* Update babel config for moved plugins
* Add node_modules and similar to prettierignore, since on CI it's not run from same directory as gitignore
@ -6,7 +6,6 @@ If your household enters a new social housing tenancy or purchases a social hous
The information is provided via ‘<%= t('service_name') %>’, a service funded and managed by MHCLG. It collects information on the tenants or residents, tenancy or sale, and the dwelling itself. Some of this data is personal and sensitive, so MHCLG is responsible for ensuring it’s processed in line with data protection legislation.
## Why do we share this information?
Information collected via CORE is shared with other government departments and agencies. It’s shared with the Greater London Authority and the Regulator of Social Housing. Data providers can also access data for their organisations via CORE. Data is only shared for research and statistical purposes.
@ -27,16 +26,16 @@ Information collected via CORE relates to your tenancy, the dwelling you are liv
Collected data will be held for as long as necessary for research and statistical purposes. When no longer needed, data will be deleted in a safe manner. We’re aware some collected data is particularly sensitive. For example:
* ethnic group
* if previous tenure is a hospital, prison or approved probation hostel support
* if household left last settled home because discharged from prison, a long stay hospital or other institution
* if referral source is probation or prison, youth offending or community mental health team, or health service
- ethnic group
- if previous tenure is a hospital, prison or approved probation hostel support
- if household left last settled home because discharged from prison, a long stay hospital or other institution
- if referral source is probation or prison, youth offending or community mental health team, or health service
MHCLG publishes data annually, in aggregate form, as part of a report and complementary tables.
* For annual lettings data, visit: [https://www.gov.uk/government/collections/rents-lettings-and-tenancies](https://www.gov.uk/government/collections/rents-lettings-and-tenancies)
- For annual lettings data, visit: [https://www.gov.uk/government/collections/rents-lettings-and-tenancies](https://www.gov.uk/government/collections/rents-lettings-and-tenancies)
* For annual sales data, visit: [https://www.gov.uk/government/collections/social-housing-sales-including-right-to-buy-and-transfers](https://www.gov.uk/government/collections/social-housing-sales-including-right-to-buy-and-transfers)
- For annual sales data, visit: [https://www.gov.uk/government/collections/social-housing-sales-including-right-to-buy-and-transfers](https://www.gov.uk/government/collections/social-housing-sales-including-right-to-buy-and-transfers)
Detail-level data is anonymised and protected, minimising identification risk. It's held with the UK Data Archive.
@ -18,6 +18,6 @@ There are a few reasons we have opted to save the values directly, they are as f
- Changing the wording/casing of the answers could result in discrepancies in the database.
- There is a small risk that if the database is accessed by someone unauthorised they would have access to personally identifiable information if we were to collect Any. We will be mitigating this risk by encrypting the production database.
- There is a small risk that if the database is accessed by someone unauthorised they would have access to personally identifiable information if we were to collect Any. We will be mitigating this risk by encrypting the production database.
This decision is not too difficult to change and can be revisited in the future if there is sufficient reason to switch to storing keys/numbers and using enums and active record to convert those to the appropriate values.
- With [ESBuild](https://esbuild.github.io/) - fast and does some transpiling but [doesn't support ES5/IE11](https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/203)
- With [Rollup](https://www.rollupjs.org/guide/en/) - similar to ESBuild, node rather than Go based, doesn't have the big speed benefits
- With [Webpack](https://webpack.js.org/) - rather than the old approach of using Webpacker as a opinionated wrapper around webpack, this approach uses webpack directly.
- With [ESBuild](https://esbuild.github.io/) - fast and does some transpiling but [doesn't support ES5/IE11](https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/203)
- With [Rollup](https://www.rollupjs.org/guide/en/) - similar to ESBuild, node rather than Go based, doesn't have the big speed benefits
- With [Webpack](https://webpack.js.org/) - rather than the old approach of using Webpacker as a opinionated wrapper around webpack, this approach uses webpack directly.
3. [Shakapacker](https://github.com/shakacode/shakapacker) - the "official" community maintained fork of Webpacker 6 RC. Requires upgrading current install since breaking changes happened between Webpacker 5 & 6
4. [Vite](https://vite-ruby.netlify.app/) - Webpack alternative
@ -14,4 +14,4 @@ For **Stimulus** we were able to do this and are continuing to use it. This is a
- Adding the StimulusJS NPM package path to our [webpack config](https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/blob/main/webpack.config.js#L23) rules to be transpiled
- Adding the required Babel plugins to our [Babel config](https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/blob/main/babel.config.js#L34)
For **Turbo** the same approach was attempted but proved [unsuccessful](https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/430). As a result we decided to [remove Turbo](https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/406) until we can drop support for Internet Explorer. This does have a perceptible impact on UX/speed but provides the most browser compatibility.
For **Turbo** the same approach was attempted but proved [unsuccessful](https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/430). As a result we decided to [remove Turbo](https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/406) until we can drop support for Internet Explorer. This does have a perceptible impact on UX/speed but provides the most browser compatibility.
@ -12,6 +12,7 @@ There might be short extensions to the deadline, so shortly after the last day t
Also, if incorrect data is found during QA process, data providers might be asked to correct it. Once the data has been through its first QA processes and is as present and correct as possible, the ability to edit and delete logs is closed. This is typically in late summer/autumn, but it depends on the statistical analysis.
To accommodate the different end dates, we will now store 3 different dates on the form definition:
- Submission deadline (submission_deadline) - this is the date displayed at the top of a completed log in lettings and sales - "You can review and make changes to this log until 9 June 2024.". Nothing happens on this date
- New logs end date (new_logs_end_date) - no new logs for that collection year can be submitted, but logs can be edited
- Edit and delete logs end date (edit_end_date) - logs can no longer be edited or deleted. Completed logs can still be viewed. Materials / references to the collection year are removed.
In order to use the app as an API, you will need to configure requests to the API as so:
* Configure your request with Basic Auth. Set the username to be the same as `API_USER` and password as the `API_KEY` (`API_USER` and `API_KEY` are environment variables that should be set for the application)
* Check the endpoint you are calling is an action that is `create`, `show` or `update`
* Check you are setting the following request headers:
*`Content-Type = application/json`
* `Action = application/json` N.B. If you use `*/*` instead, the request won't be recognised as an API request`
- Configure your request with Basic Auth. Set the username to be the same as `API_USER` and password as the `API_KEY` (`API_USER` and `API_KEY` are environment variables that should be set for the application)
- Check the endpoint you are calling is an action that is `create`, `show` or `update`
- Check you are setting the following request headers:
-`Content-Type = application/json`
- `Action = application/json` N.B. If you use `*/*` instead, the request won't be recognised as an API request`
Currently only the logs controller is configured to accept and authenticate API requests, when the above API environment variables are set.
@ -31,7 +31,7 @@ An example question might look something like this:
}
```
In the above example the the question has the id `postcode_known`.
In the above example the the question has the id `postcode_known`.
The `check_answer_label` contains the text that will be displayed in the label of the table on the check answers page.
@ -85,6 +85,6 @@ The answer the data inputter provides to some questions allows us to infer the v
In the above example the width is an optional attribute and can be provided for text type questions to determine the width of the text box on the page when when the question is displayed to a user (this allows you to match the width of the text box on the page to that of the design for a question).
The above example links to the first example as both of these questions would be on the same page. The `inferred_check_answers_value` is what should be displayed on the check answers page for this question if we infer it. If the value of `postcode_known` was given as `0` (which is a no), as seen in the condition part of `inferred_check_answers_value`then we can infer that the data inputter does not know the postcode and so we would display the value of `Not known` on the check answers page for the postcode.
The above example links to the first example as both of these questions would be on the same page. The `inferred_check_answers_value` is what should be displayed on the check answers page for this question if we infer it. If the value of `postcode_known` was given as `0` (which is a no), as seen in the condition part of `inferred_check_answers_value` then we can infer that the data inputter does not know the postcode and so we would display the value of `Not known` on the check answers page for the postcode.
In the above example the `inferred_answers` refers to a question where we can infer the answer based on the answer of this question. In this case the `la` question can be inferred from the postcode value given by the data inputter as we are able to lookup the local authority based on the postcode given. We then set a property on the lettings log `is_la_inferred` to true to indicate that this is an answer we've inferred.
Currently, there are four environments with infrastructure:
- Meta
- Development (Review Apps)
- Staging
- Production
### Meta
This holds the Terraform “backend” and the ECR(s).
The Terraform “backend” consists of:
- S3 buckets - for storing Terraform state files. One for all non-production environments (including the meta environment itself), and another just for production.
- DynamoDB - for managing access and locking of all state files.
The ECR(s) are:
- core - holds the application Docker images.
- db-migration - holds the Docker images curated to help migrate a DB from PaaS to AWS.
- s3-migration - holds the Docker images curated to help migrate S3 files from PaaS to AWS.
N.B. the migration ECRs may or may not be present, depending on if the Terraform has been configured to create migration infrastructure. The migration infrastructure is only used to help migrate the DB and S3 from PaaS to AWS, so is usually therefore only temporarily present.
N.B. the migration ECRs may or may not be present, depending on if the Terraform has been configured to create migration infrastructure. The migration infrastructure is only used to help migrate the DB and S3 from PaaS to AWS, so is usually therefore only temporarily present.
### Development / Staging / Production
These are the main environments holding the “application” infrastructure.
These are the main environments holding the “application” infrastructure.
Though not exhaustive, each of them will generally contain the following key components:
- ECS Fargate cluster
- RDS (PostgreSQL database)
- ElastiCache (Redis data store)
- S3 buckets
- One for Bulk upload (sometimes also to referred to as the CSV bucket)
- One for CDS Export
- One for Bulk upload (sometimes also to referred to as the CSV bucket)
- One for CDS Export
- VPC
- Private subnets
- Public subnets
@ -43,11 +49,12 @@ Though not exhaustive, each of them will generally contain the following key com
- WAF (Firewall)
### Development / Review Apps
The development environment is used for Review Apps, and has some infrastructure that is created per-review-app and some that is shared by all apps.
The development environment is used for Review Apps, and has some infrastructure that is created per-review-app and some that is shared by all apps.
In general, each review app has its own ECS Fargate cluster and Redis instances (plus any infrastructure to enable this), while the rest is shared.
Where to find the Infrastructure?
The infrastructure is managed as code.
The infrastructure is managed as code.
In the terraform folder of the codebase, there will be dedicated sub-folders for each of the aforementioned environments, where all the infrastructure for them is defined.
## Deployment (Pipeline — Recommended)
@ -64,7 +71,6 @@ To deploy you need to:
6. Post success message on Slack.
7. Tag tickets as ‘Released’ and move tickets to done on JIRA.
## CI/CD
When a commit is made to `main` the following GitHub action jobs are triggered:
@ -88,26 +94,27 @@ After a sucessful deployment a comment will be added to the pull request with th
Once a pull request has been closed the review app infrastructure will be tore down to save on any costs. Should you wish to re-open a closed pull request the review app will be spun up again.
### Review app deployment failures
### Review app deployment failures
One reason a review app deployment might fail is that it is attempting to run migrations which conflict with data in the database. For example you might have introduced a unique constraint, but the database associated with the review app has duplicate data in it that would violate this constraint, and so the migration cannot be run.
## Destroying/recreating infrastructure
Things to watch out for when destroying/creating infra:
- All resources
- The lifecycle meta-argument prevent_destroy will stop you destroying things. Best to set this to false before trying to destroy!
- The lifecycle meta-argument prevent_destroy will stop you destroying things. Best to set this to false before trying to destroy!
- Database
- skip_final_snapshot being false will prevent you from destroying the db without creating a final snapshot.
- skip_final_snapshot being false will prevent you from destroying the db without creating a final snapshot.
- Load Balancer
- Sometimes when creating infra, you may see the error message: failure configuring LB attributes: InvalidConfigurationRequest: Access Denied for bucket: <load-balancer-access-log-bucket-name>. Please check S3bucket permission during a terraform apply. To get around this you may have wait a few minutes and try applying again to ensure everything is fully updated (the error shouldn’t appear on the second attempt). It’s unclear what the exact cause is, but as this is related to infra that enables load balancer access logging, it is suspected there might be a delay with the S3 bucket permissions being realised or the load balancer recognising it can access the bucket.
- Sometimes when creating infra, you may see the error message: failure configuring LB attributes: InvalidConfigurationRequest: Access Denied for bucket: <load-balancer-access-log-bucket-name>. Please check S3bucket permission during a terraform apply. To get around this you may have wait a few minutes and try applying again to ensure everything is fully updated (the error shouldn’t appear on the second attempt). It’s unclear what the exact cause is, but as this is related to infra that enables load balancer access logging, it is suspected there might be a delay with the S3 bucket permissions being realised or the load balancer recognising it can access the bucket.
- S3
- Terraform won’t let you delete buckets that have objects in them.
- Terraform won’t let you delete buckets that have objects in them.
- Secrets
- If you destroy secrets, they will actually be marked as ‘scheduled to delete’ which will take effect after a minimum of 7 days. You can’t recreate secrets with the same name during this period. If you want to destroy immediately, you need to do it from the command line (using your staging developer role, rather than your MHCLG-wide role used to apply Terraform) with this command: aws secretsmanager delete-secret --force-delete-without-recovery --secret-id <secret-arn>. (Note that if a secret is marked as scheduled to delete, you can undo this in the console to make it an ‘active’ secret again.)
- You may need to manually re-enter secret values into Secrets Manager at some point. When you do, just paste the secret value as plain text (don’t enter a key name, or format it as JSON).
- If you destroy secrets, they will actually be marked as ‘scheduled to delete’ which will take effect after a minimum of 7 days. You can’t recreate secrets with the same name during this period. If you want to destroy immediately, you need to do it from the command line (using your staging developer role, rather than your MHCLG-wide role used to apply Terraform) with this command: aws secretsmanager delete-secret --force-delete-without-recovery --secret-id <secret-arn>. (Note that if a secret is marked as scheduled to delete, you can undo this in the console to make it an ‘active’ secret again.)
- You may need to manually re-enter secret values into Secrets Manager at some point. When you do, just paste the secret value as plain text (don’t enter a key name, or format it as JSON).
- ECS
- Sometimes task definitions don’t get deleted. You may need to manually delete them.
- After destroying the db, you’ll need to make sure the ad hoc ECS task which seeds the database gets run in order to set up the database correctly.
- Sometimes task definitions don’t get deleted. You may need to manually delete them.
- After destroying the db, you’ll need to make sure the ad hoc ECS task which seeds the database gets run in order to set up the database correctly.
- SNS
- When creating an email subscription in an environment, Terraform will look up the email to use as the subscription endpoint from Secrets Manager. If you haven’t already created this (e.g. by running terraform apply -target="module.monitoring" -var="create_secrets_first=true") then this will lead to the subscription creation erroring, because it can’t retrieve the value of the secret (because it doesn’t exist yet). If this happens, remember you’ll need to go to Secrets Manager in the console and enter the desired email (as plaintext, no quotation marks or anything else required) as the value of the secret (which is most likely called MONITORING_EMAIL). Then run another apply with Terraform and this time it should succeed.
- When creating an email subscription in an environment, Terraform will look up the email to use as the subscription endpoint from Secrets Manager. If you haven’t already created this (e.g. by running terraform apply -target="module.monitoring" -var="create_secrets_first=true") then this will lead to the subscription creation erroring, because it can’t retrieve the value of the secret (because it doesn’t exist yet). If this happens, remember you’ll need to go to Secrets Manager in the console and enter the desired email (as plaintext, no quotation marks or anything else required) as the value of the secret (which is most likely called MONITORING_EMAIL). Then run another apply with Terraform and this time it should succeed.
- AWS CloudWatch (for general application / infrastructure logging)
- Sentry (for application error logging)
### CloudWatch
The CloudWatch service can be accessed from the AWS Console. You should authenticate onto the infrastructure environment whose logs you want to check.
From CloudWatch, navigate to the desired log group (e.g. for the app task running on ECS) and open the desired log stream, in order to read its log “events”.
Alternatively, you can also navigate to a specific AWS service / resource in question (e.g. ECS tasks), selecting the instance of interest (e.g. a specific ECS task), and finding the “logs” tab (or similar) to view the log “events”.
### Sentry
To access Sentry, ensure you have been added to the MHCLG account.
Generally error logs in Sentry will also be present somewhere in the CloudWatch logs, but they will be easier to assess here (e.g. number of occurrences over a time period). The logs in Sentry are created by the application when it makes Rails.logger.error calls.
## Debugging
### Application infrastructure
For debugging / investigating infrastructure issues you can use the AWS CloudWatch automatic dashboards. (e.g. is there a lack of physical space on the database, how long has the ECS had very high compute usage for etc.)
They can be found in the CloudWatch service on AWS console, by going to dashboards → automatic dashboards, and selecting the desired dashboard (e.g. Elastic Container Service).
They can be found in the CloudWatch service on AWS console, by going to dashboards → automatic dashboards, and selecting the desired dashboard (e.g. Elastic Container Service).
Alternatively, you can also navigate to the AWS resource in question (e.g. RDS database), selecting the instance of interest, and selecting the “monitoring” / ”metrics” tab (or similar), as this can provide alternate useful information also.
### Exec into a container
You can open a terminal directly on a running container / app, in order to run some commands that may help with debugging an issue.
You can open a terminal directly on a running container / app, in order to run some commands that may help with debugging an issue.
To do this, you will need to “exec” into the container.
#### Prerequisites
- AWS CLI
- AWS Session manager plugin Install the Session Manager plugin for the AWS CLI - AWS Systems Manager
- AWS Session manager plugin Install the Session Manager plugin for the AWS CLI - AWS Systems Manager
- AWS access
#### Accessing the rails console
Prerequisite:
Configure AWS auth following the [documentation in the infra repo](https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data-infrastructure/blob/main/docs/development_setup.md). This also details how to enter a subshell with suitable AWS credentials.
You can then use `aws ecs execute-command --cluster "core-$env-app" --task <taskid> --interactive --command <command>` to run the relevant command on a specific task.
You can then use `aws ecs execute-command --cluster "core-$env-app" --task <taskid> --interactive --command <command>` to run the relevant command on a specific task.
### Database
In order to investigate or look more closely at the database, you can exec into a container as above, and use the rails console to query the database.
## Factories for Lettings Log, Sales Log, Organisation, and User
## Factories for Lettings Log, Sales Log, Organisation, and User
Each of these factories has nested relationships and callbacks that ensure associated objects are created and linked properly. For instance, creating a `lettings_log` involves creating or associating with a `user`, which in turn is linked to an `organisation`, potentially leading to creating `organisation_rent_periods` and a `data_protection_confirmation`.
This documentation outlines the objects that are created and/or persisted to the database when using FactoryBot to create or build models for LettingsLog, SalesLog, Organisation, and User. There are other factories, but they are simpler, less frequently used and don't have as much resource hierarchy.
### Lettings Log
Objects Created/Persisted:
- **User**: The `assigned_to` user is created.
- **Organisation**: The `assigned_to` user’s organisation created by `User` factory.
- **DataProtectionConfirmation**: If `organisation` does not have DSA signed, `DataProtectionConfirmation` gets created with `assigned_to` user as a `data_protection_officer`
- **OrganisationRentPeriod**: If `log.period` is present and the `managing_organisation` does not have an `OrganisationRentPeriod` for that period, a new `OrganisationRentPeriod` is created and associated with `managing_organisation`.
Example Usage:
```
let(:lettings_log) { create(:lettings_log) }
```
### Sales Log
Objects Created/Persisted:
- **User**: The `assigned_to` user is created.
- **Organisation**: The `assigned_to` user’s organisation created by `User` factory.
- **DataProtectionConfirmation**: If `organisation` does not have DSA signed, `DataProtectionConfirmation` gets created with `assigned_to` user as a `data_protection_officer`
Example Usage:
```
let(:sales_log) { create(:sales_log) }
```
### Organisation
Objects Created/Persisted:
- **OrganisationRentPeriod**: For each rent period in transient attribute `rent_periods`, an `OrganisationRentPeriod` is created.
- **DataProtectionConfirmation**: If `with_dsa` is `true` (default), a `DataProtectionConfirmation` is created with a `data_protection_officer`
- **User**: Data protection officer that signs the data protection confirmation
- **DataProtectionConfirmation**: If `organisation` does not have DSA signed, `DataProtectionConfirmation` gets created with this user as a `data_protection_officer`
Rachael Booth
4 months ago
committed by
GitHub