Browse Source
* Update documentation (content, presentation and images) * Add some links * Revert image change for now Co-authored-by: baarkerlounger <db@slothlife.xyz>pull/703/head
Paul Robert Lloyd
3 years ago
committed by
GitHub
19 changed files with 415 additions and 269 deletions
@ -1,15 +1,17 @@
|
||||
# CDS exports |
||||
# Exporting to CDS |
||||
|
||||
All data collected by the application needs to be exported to the Consolidated Data Store (CDS) which is a data warehouse based on MS SQL running in the DAP (Data Analytics Platform). |
||||
|
||||
This is done via XML exports saved in an S3 bucket located in the DAP VPC using dedicated credentials shared out of band. The data mapping for this export can be found in `app/services/exports/case_log_export_service.rb`. Initially the application database field names and field types were chosen to match the existing CDS data as closely as possible to minimise the amount of transformation needed. This has led to a less than optimal data model though and increasingly we should look to transform at the mapping layer where beneficial for our application. |
||||
This is done via XML exports saved in an S3 bucket located in the DAP VPC using dedicated credentials shared out of band. The data mapping for this export can be found in `app/services/exports/case_log_export_service.rb` |
||||
|
||||
Initially the application database field names and field types were chosen to match the existing CDS data as closely as possible to minimise the amount of transformation needed. This has led to a less than optimal data model though and increasingly we should look to transform at the mapping layer where beneficial for our application. |
||||
|
||||
The export service is triggered nightly using [Gov PaaS tasks](https://docs.cloudfoundry.org/devguide/using-tasks.html). These tasks are triggered from a Github action, as Gov PaaS does not currently support the Cloud Foundry Task Scheduler. |
||||
|
||||
The S3 bucket is located in the DAP VPC rather than the application VPC as DAP runs in an AWS account directly so access to the S3 bucket can be restricted to only the IPs used by the application. This is not possible the other way around as Gov PaaS does not support restricting S3 access by IP (https://github.com/alphagov/paas-roadmap/issues/107). |
||||
The S3 bucket is located in the DAP VPC rather than the application VPC as DAP runs in an AWS account directly so access to the S3 bucket can be restricted to only the IPs used by the application. This is not possible the other way around as [Gov PaaS does not support restricting S3 access by IP](https://github.com/alphagov/paas-roadmap/issues/107). |
||||
|
||||
## Other options previously considered: |
||||
## Other options previously considered |
||||
|
||||
- CDC replication using a managed service such as [AWS DMS](https://aws.amazon.com/dms/) |
||||
- Would require VPC peering which Gov PaaS does not currently support (https://github.com/alphagov/paas-roadmap/issues/105) |
||||
- Would require VPC peering which [Gov PaaS does not currently support](https://github.com/alphagov/paas-roadmap/issues/105) |
||||
- Would require CDS to make changes to their ingestion model |
||||
|
@ -1,19 +1,21 @@
|
||||
# Form Runner |
||||
|
||||
The form runner is composed of: |
||||
The Form Runner is composed of: |
||||
|
||||
Ruby Classes: |
||||
- A singleton form handler that instantiates an instances of each form definition (config file we have) combined with the "setup" section that is common to all forms. This is created at rails boot time. (`app/models/form_handler.rb`) |
||||
- A Form class that is the entry point for parsing a form definition and handles most of the associated logic (`app/models/form.rb`) |
||||
- Section, Subsection, Page and Question classes (`app/models/form/`) |
||||
- Setup subsection specific instances (subclasses) of Section, Subsection, Pages and Questions (`app/form/setup/`) |
||||
|
||||
ERB Templates: |
||||
- A singleton form handler that instantiates an instances of each form definition (config file we have) combined with the setup section that is common to all forms. This is created at rails boot time. (`app/models/form_handler.rb`) |
||||
- A `Form` class that is the entry point for parsing a form definition and handles most of the associated logic (`app/models/form.rb`) |
||||
- `Section`, `Subsection`, `Page` and `Question` classes (`app/models/form/`) |
||||
- Setup subsection specific instances (subclasses) of `Section`, `Subsection`, `Pages` and `Questions` (`app/form/setup/`) |
||||
|
||||
ERB templates: |
||||
|
||||
- The page view which is the main view for each form page (`app/views/form/page.html.erb`) |
||||
- Partials for each question type (radio, checkbox, select, text, numeric, date) (`app/views/form/`) |
||||
- Partials for specific question guidance (`app/views/form/guidance`) |
||||
- The check answers page which is the view for the answer summary page of each section (`app/views/form/check_answers.html.erb`) |
||||
|
||||
Routes for each form page are generated by looping over each Page instance in each Form instance held by the Form Handler and defining a "Get" path. The corresponding controller method is also auto-generated with meta-programming via the same looping in `app/controllers/form_controller.rb` |
||||
Routes for each form page are generated by looping over each Page instance in each Form instance held by the form handler and defining a `GET` path. The corresponding controller method is also auto-generated with meta-programming via the same looping in `app/controllers/form_controller.rb` |
||||
|
||||
All form pages submit to the same controller method (`app/controllers/form_controller.rb#submit_form`) which validates and persists the data, and then redirects to the next form page that identifies as "routed_to" given the current case log state. |
||||
All form pages submit to the same controller method (`app/controllers/form_controller.rb#submit_form`) which validates and persists the data, and then redirects to the next form page that identifies as `routed_to` given the current case log state. |
||||
|
Before Width: | Height: | Size: 286 KiB |
Before Width: | Height: | Size: 147 KiB After Width: | Height: | Size: 88 KiB |
After Width: | Height: | Size: 404 KiB |
Before Width: | Height: | Size: 197 KiB After Width: | Height: | Size: 118 KiB |
@ -1,13 +1,17 @@
|
||||
# Infrastructure Metric monitoring |
||||
# Monitoring |
||||
|
||||
We use self-hosted Prometheus and Grafana for monitoring infrastructure metrics. These are run in a dedicated Gov PaaS space called "monitoring" and are deployed as Docker images using Github action pipelines. The repository for these and more information is here: [dluhc-data-collection-monitoring](https://github.com/communitiesuk/dluhc-data-collection-monitoring). |
||||
|
||||
# Application & Performance monitoring & alerting |
||||
## Performance monitoring and alerting |
||||
|
||||
For application error and performance monitoring we use managed [Sentry](https://sentry.io/organizations/dluhc-core). You will need to be added to the DLUHC account to access this. It triggers slack notifications to the #team-data-collection-alerts channel for all application errors in staging and production and for any controller endpoints that have a P95 transaction duration > 250ms over a 24 hour period. |
||||
|
||||
# Logs |
||||
## Logs |
||||
|
||||
For log persistence we use a managed ELK (Elasticsearch, Logstash, Kibana) stack provided by [Logit](https://logit.io/). You will need to be added to the DLUHC account to access this. Longs are retained for 14 days with a daily limit of 2GB. |
||||
|
||||
Logs are also available from Gov PaaS directly via cli `cf logs <gov-paas-space-name> --recent`. |
||||
Logs are also available from Gov PaaS directly via CLI: |
||||
|
||||
```bash |
||||
cf logs <gov-paas-space-name> --recent |
||||
``` |
||||
|
@ -1,23 +0,0 @@
|
||||
# Definitions |
||||
|
||||
- **Stock owning organisation** (parent): an organisation that owns housing stock (parent). It may manage the allocation of people in and out of their accommodation, or it may contract this out to a managing agent (child). |
||||
|
||||
- **Managing agent (child)**: These are about orgs. In scenarios where one organisation owns stock and another organisation is contracted to manage the stock and tenants, the latter organisation is often called a ‘managing agent’. A managing agent is the same as a child and is the term more commonly used by data providing organisations. Parent/child is what we call them internally but is not a term that should be used for external customers. Managing agents are responsible for the allocation of people in and out of the accommodation, and/or responsible for the services provided to support those people in the accommodation (in the case of Supported Housing). |
||||
|
||||
# Permissions |
||||
|
||||
## Organisational relationships: |
||||
|
||||
Organisations that own stock can contract out the management of that stock to another organisation. This relationship is often referred to as a parent/child relationship. This is a useful analogy as a parent can have multiple children, and a child can have many parents. A child organisation can also be a parent, and a parent organisation can also be a child organisation: |
||||
|
||||
![Organisational relationships](images/organisational_relationships.png) |
||||
|
||||
The case logs that a user can see depends on their role: |
||||
|
||||
- Customer Support users can access any case log |
||||
- Data coordinators can access any case log for which the organisation they work for is ultimately responsible for, meaning they can see logs managed by a child organisation |
||||
- Data providers can only access case logs for which their organisation manages (or directly owns) |
||||
|
||||
Taking the relationships from the above diagram, and looking at which logs each user can access: |
||||
|
||||
![User log access permissions](images/user_log_permissions.png) |
@ -0,0 +1,25 @@
|
||||
# Organisational relationships |
||||
|
||||
## Definitions |
||||
|
||||
- **Stock owning organisation**: An organisation that owns housing stock. It may manage the allocation of people in and out of their accommodation, or it may contract this out to managing agents. |
||||
|
||||
- **Managing agent**: In scenarios where one organisation owns stock and another organisation is contracted to manage the stock and tenants, the latter organisation is often called a ‘managing agent’. Managing agents are responsible for the allocation of people in and out of the accommodation, and/or responsible for the services provided to support those people in the accommodation (in the case of supported housing). |
||||
|
||||
## Permissions |
||||
|
||||
Organisations that own stock can contract out the management of that stock to another organisation. This relationship is often referred to as a parent/child relationship. This is a useful analogy as a parent can have multiple children, and a child can have many parents. A child organisation can also be a parent, and a parent organisation can also be a child organisation: |
||||
|
||||
![Organisational relationships](images/organisational_relationships.png) |
||||
|
||||
The case logs that a user can see depends on their role: |
||||
|
||||
- Customer support users can access any case log |
||||
|
||||
- Data coordinators can access any case log for which the organisation they work for is ultimately responsible for, meaning they can see logs managed by a child organisation |
||||
|
||||
- Data providers can only access case logs for which their organisation manages (or directly owns) |
||||
|
||||
Taking the relationships from the above diagram, and looking at which logs each user can access: |
||||
|
||||
![User log access permissions](images/user_log_permissions.png) |
@ -1,5 +1,9 @@
|
||||
# Supported housing schemes |
||||
|
||||
- **Schemes**: Groups of similar properties in the same location, intended for similar tenants with the same type of support needs, managed in the same way. As some of the information we need about a new tenancy is the same for all new tenancies in the ‘scheme’, users can set up a ‘scheme’ in the CORE system by completing the information once. In Supported Housing forms, the user just supplies the appropriate scheme. This means providers do not have to complete identical information multiple times in each CORE form. Effectively we model these as "templates" or "predefined answer sets" |
||||
## Schemes |
||||
|
||||
- **Management groups**: Schemes are often managed together as part of a ‘management group’. An organisation may have multiple management groups, and each management group may have multiple schemes. For Supported Housing logs, users must select the management group first, then select scheme. |
||||
Groups of similar properties in the same location, intended for similar tenants with the same type of support needs, managed in the same way. As some of the information we need about a new tenancy is the same for all new tenancies in the ‘scheme’, users can set up a ‘scheme’ in the CORE system by completing the information once. In Supported Housing forms, the user just supplies the appropriate scheme. This means providers do not have to complete identical information multiple times in each CORE form. Effectively we model these as templates or predefined answer sets. |
||||
|
||||
## Management groups |
||||
|
||||
Schemes are often managed together as part of a ‘management group’. An organisation may have multiple management groups, and each management group may have multiple schemes. For Supported Housing logs, users must select the management group first, then select scheme. |
||||
|
@ -1,8 +1,13 @@
|
||||
# Testing strategy |
||||
|
||||
- We use [RSpec](https://rspec.info/) and [Capybara](https://teamcapybara.github.io/capybara/) |
||||
|
||||
- Capybara is used for our feature tests. These use the Rack driver by default (faster) or the Gecko driver (installation required) when the `js: true` option is passed for a test. |
||||
|
||||
- Capybara is configured to run in headless mode but this can be toggled by commenting out `app/spec/rails_helper.rb#L14` |
||||
- Capybara is configured to use Gecko driver for JS tests as Chrome is more commonly used and so naturally more likely to be better tested but this can be switched to Chrome driver by changing `app/spec/rails_helper.rb#L13` |
||||
- Feature specs are generally written sparingly as they're also the slowest, where possible a request spec is preferred as this still tests a large surface area (route, controller, model, view) without the performance impact. They are not suitable for tests that need to run javascript or test that a specific set of UI events triggers a specific set of requests (with high confidence). |
||||
|
||||
- Capybara is configured to use Gecko driver for JavaScript tests as Chrome is more commonly used and so naturally more likely to be better tested but this can be switched to Chrome driver by changing `app/spec/rails_helper.rb#L13` |
||||
|
||||
- Feature specs are generally written sparingly as they’re also the slowest, where possible a request spec is preferred as this still tests a large surface area (route, controller, model, view) without the performance impact. They are not suitable for tests that need to run JavaScript or test that a specific set of interaction events that trigger a specific set of requests (with high confidence). |
||||
|
||||
- Test data is created with [FactoryBot](https://github.com/thoughtbot/factory_bot) where ever possible |
||||
|
@ -1,13 +0,0 @@
|
||||
# External Users |
||||
|
||||
The primary users of the system are external data providing organisations: Local Authorities and Private Registered Providers (Housing Associations). These have 2 main user type: |
||||
|
||||
- Data Coordinators - administrators for their own organisation, can also complete logs |
||||
- Data Providers - complete the logs |
||||
|
||||
Additionally there are Data Protection Officers (DPO) which at some organisations is a separate role, but in our codebase is modelled as an attribute of the user (i.e. a data coordinator or provider can additionally be a DPO). They are responsible for ensuring the organisation has signed the data sharing agreement. |
||||
|
||||
# Internal users |
||||
|
||||
- Customer support (helpdesk) - can administrate all organisations |
||||
- ADD statisticians - primary consumers of the data collected via CDS/DAP |
@ -0,0 +1,15 @@
|
||||
# User roles |
||||
|
||||
## External users |
||||
|
||||
The primary users of the system are external data providing organisations: Local Authorities and Private Registered Providers (Housing Associations). These have 2 main user types: |
||||
|
||||
- Data coordinators – administrators for their own organisation, can also complete logs |
||||
- Data providers – complete the logs |
||||
|
||||
Additionally there are Data Protection Officers (DPO), which for some organisations is a separate role, but in our codebase is modelled as an attribute of the user (i.e. a data coordinator or provider can additionally be a DPO). They are responsible for ensuring the organisation has signed the data sharing agreement. |
||||
|
||||
## Internal users |
||||
|
||||
- Customer support (help desk) – can administrate all organisations |
||||
- ADD statisticians – primary consumers of the data collected via CDS/DAP |
Loading…
Reference in new issue