From ba8816590318942fe9261ee4bbd77a7647740b8d Mon Sep 17 00:00:00 2001 From: kosiakkatrina <54268893+kosiakkatrina@users.noreply.github.com> Date: Fri, 22 Nov 2024 08:53:49 +0000 Subject: [PATCH] CLDC-3695 Update docs (#2795) * Update readme * lint * typo * Update app api doc * lint --- docs/Gemfile.lock | 1 + docs/app_api.md | 2 +- docs/exports.md | 16 ++-- docs/form/builder.md | 182 ++++++++++++++++++---------------------- docs/form/definition.md | 35 +++----- docs/form/index.md | 12 +-- docs/form/page.md | 67 +++++++-------- docs/form/question.md | 86 +++++++++---------- docs/form/section.md | 30 ++++--- docs/form/subsection.md | 32 ++++--- docs/setup.md | 14 +++- 11 files changed, 219 insertions(+), 258 deletions(-) diff --git a/docs/Gemfile.lock b/docs/Gemfile.lock index f8962318e..d102b14ef 100644 --- a/docs/Gemfile.lock +++ b/docs/Gemfile.lock @@ -255,6 +255,7 @@ GEM PLATFORMS arm64-darwin-21 + arm64-darwin-23 x86_64-darwin-22 x86_64-linux diff --git a/docs/app_api.md b/docs/app_api.md index e1987a897..627be0ad7 100644 --- a/docs/app_api.md +++ b/docs/app_api.md @@ -12,4 +12,4 @@ In order to use the app as an API, you will need to configure requests to the AP - `Content-Type = application/json` - `Action = application/json` N.B. If you use `*/*` instead, the request won't be recognised as an API request` -Currently only the logs controller is configured to accept and authenticate API requests, when the above API environment variables are set. +Currently, only the Logs Controller is configured to accept and authenticate API requests, provided that the specified API environment variables are set. Please note that the API has not been actively maintained for an extended period and may not function as expected. Additionally, the required environment variables are not configured on any of the environments deployed on AWS, rendering API requests to those environments non-functional. diff --git a/docs/exports.md b/docs/exports.md index f16efe893..4b971e29a 100644 --- a/docs/exports.md +++ b/docs/exports.md @@ -6,16 +6,14 @@ nav_order: 7 All data collected by the application needs to be exported to the Consolidated Data Store (CDS) which is a data warehouse based on MS SQL running in the DAP (Data Analytics Platform). -This is done via XML exports saved in an S3 bucket located in the DAP VPC using dedicated credentials shared out of band. The data mapping for this export can be found in `app/services/exports/lettings_log_export_service.rb` +This is done via XML exports saved in an S3 bucket. +We currently export lettings logs, users and organisations. +The data mapping for these exports can be found in: + +- Lettings logs `app/services/exports/lettings_log_export_service.rb` +- Organisations `app/services/exports/organisation_export_service.rb` +- Users `app/services/exports/user_export_service.rb` Initially the application database field names and field types were chosen to match the existing CDS data as closely as possible to minimise the amount of transformation needed. This has led to a less than optimal data model though and increasingly we should look to transform at the mapping layer where beneficial for our application. We have a cron job triggering the export service daily at 5am. - -The S3 bucket is located in the DAP VPC rather than the application VPC as DAP runs in an AWS account directly so access to the S3 bucket can be restricted to only the IPs used by the application. This is not possible the other way around as [Gov PaaS does not support restricting S3 access by IP](https://github.com/alphagov/paas-roadmap/issues/107). - -## Other options previously considered - -- CDC replication using a managed service such as [AWS DMS](https://aws.amazon.com/dms/) - - Would require VPC peering which [Gov PaaS does not currently support](https://github.com/alphagov/paas-roadmap/issues/105) - - Would require CDS to make changes to their ingestion model diff --git a/docs/form/builder.md b/docs/form/builder.md index 155abaed6..5d21b1c43 100644 --- a/docs/form/builder.md +++ b/docs/form/builder.md @@ -9,13 +9,11 @@ nav_order: 1 The setup this log section is treated slightly differently from the rest of the form. It is more accurately viewed as providing metadata about the form than as being part of the form itself. It also needs to know far more about the application specific context than other parts of the form such as who the current user is, what organisation they’re part of and what role they have etc. -As a result it’s not modelled as part of the config but rather as code. It still uses the same [Form Runner](/form/runner) components though. - -## Features the Form Config supports +## Features the Form supports - Defining sections, subsections, pages and questions that fit the GOV.UK task list pattern -- Auto-generated routes – URLs are automatically created from dasherized page names +- Auto-generated routes – URLs are automatically created from dasherized page names (ids) - Data persistence requires a database field to exist which matches the name/id for each question (and answer option for checkbox questions) @@ -39,63 +37,84 @@ As a result it’s not modelled as part of the config but rather as code. It sti - For complex HTML guidance partials can be referenced -## JSON Config - -The form for this is driven by a JSON file in `/config/forms/{start_year}_{end_year}.json` - -The JSON should follow the structure: - -```jsonc -{ - "form_type": "lettings" / "sales", - "start_year": Integer, // i.e. 2020 - "end_year": Integer, // i.e. 2021 - "sections": { - "[snake_case_section_name_string]": { - "label": String, - "description": String, - "subsections": { - "[snake_case_subsection_name_string]": { - "label": String, - "pages": { - "[snake_case_page_name_string]": { - "header": String, - "description": String, - "questions": { - "[snake_case_question_name_string]": { - "header": String, - "hint_text": String, - "check_answer_label": String, - "type": "text" / "numeric" / "radio" / "checkbox" / "date", - "min": Integer, // numeric only - "max": Integer, // numeric only - "step": Integer, // numeric only - "width": 2 / 3 / 4 / 5 / 10 / 20, // text and numeric only - "prefix": String, // numeric only - "suffix": String, //numeric only - "answer_options": { // checkbox and radio only - "0": String, - "1": String - }, - "conditional_for": { - "[snake_case_question_to_enable_1_name_string]": ["condition-that-enables"], - "[snake_case_question_to_enable_2_name_string]": ["condition-that-enables"] - }, - "inferred_answers": { "field_that_gets_inferred_from_current_field": { "is_that_field_inferred": true } }, - "inferred_check_answers_value": [{ - "condition": { "field_name_for_inferred_check_answers_condition": "field_value_for_inferred_check_answers_condition" }, - "value": "Inferred value that gets displayed if condition is met" - }] - } - }, - "depends_on": [{ "question_key": "answer_value_required_for_this_page_to_be_shown" }] - } - } - } - } - } - } -} +## Form definition + +The Form should follow the structure: + +``` +SECTIONS = [ + Form::Sales::Sections::Section +].freeze + +Form.new(nil, start_year, SECTIONS, form_type - "lettings" / "sales") + +class Form::Sales::Sections::Section < ::Form::Section + def initialize(id, hsh, form) + super + @id = [snake_case_section_name_string] + @label = [String] + @description = [String] + @subsections = [Form::Sales::Subsections::Subsection.new(nil, nil, self)] + end +end + +class Form::Sales::Subsections::Subsection < ::Form::Subsection + def initialize(id, hsh, section) + super + @id = [snake_case_subsection_name_string] + @label = [String] + @depends_on = [{ "question_key/method_key": "answer_value_required_for_this_subsection_to_be_shown" }] + end + + def pages + @pages ||= [Form::Sales::Pages::Page.new(nil, nil, self),] + end +end + +class Form::Sales::Pages::Page < ::Form::Page + def initialize(id, hsh, subsection) + super + @id = [snake_case_page_name_string] + @header = [String,] + @depends_on = [{ "question_key": "answer_value_required_for_this_page_to_be_shown" }] + end + + def questions + @questions ||= [ + Form::Sales::Questions::Question.new(nil, nil, self), + ] + end +end + +class Form::Sales::Questions::Question < ::Form::Question + def initialize(id, hsh, page) + super + @id = [snake_case_question_name_string] + @hint_text = [String,] + @check_answer_label = [String,] + @type = ["text" / "numeric" / "radio" / "checkbox" / "date",] + @min = [Integer, // numeric only] + @max = [Integer, // numeric only] + @step = [Integer, // numeric only] + @width = [2 / 3 / 4 / 5 / 10 / 20, // text and numeric only] + @prefix = [String, // numeric only] + @suffix = [String, //numeric only] + @answer_options = { // checkbox and radio only + "0": String, + "1": String + }, + @conditional_for = { + "[snake_case_question_to_enable_1_name_string]": ["condition-that-enables"], + "[snake_case_question_to_enable_2_name_string]": ["condition-that-enables"] + }, + @inferred_answers = { "field_that_gets_inferred_from_current_field": { "is_that_field_inferred": true } }, + @inferred_check_answers_value = [{ + "condition": { "field_name_for_inferred_check_answers_condition": "field_value_for_inferred_check_answers_condition" }, + "value": "Inferred value that gets displayed if condition is met" + }] + @question_number = Integer + end +end ``` Assumptions made by the format: @@ -127,47 +146,8 @@ Assumptions made by the format: Form navigation works by stepping sequentially through every page defined in the JSON form definition for the given subsection. For every page it checks if it has "depends_on" conditions. If it does, it evaluates them to determine whether that page should be show or not. -In this way we can build up whole branches by having: - -```jsonc -"page_1": { "questions": { "question_1: "answer_options": ["A", "B"] } }, -"page_2": { "questions": { "question_2: "answer_options": ["C", "D"] }, "depends_on": [{ "question_1": "A" }] }, -"page_3": { "questions": { "question_3: "answer_options": ["E", "F"] }, "depends_on": [{ "question_1": "A" }] }, -"page_4": { "questions": { "question_4: "answer_options": ["G", "H"] }, "depends_on": [{ "question_1": "B" }] }, -``` - -## JSON form validation against Schema - -To validate the form JSON against the schema you can run: - -```bash -rake form_definition:validate["config/forms/2021_2022.json"] -``` - -Note: you may have to escape square brackets in zsh: - -```bash -rake form_definition:validate\["config/forms/2021_2022.json"\] -``` - -This will validate the given form definition against the schema in `config/forms/schema/generic.json`. - -You can also run: - -```bash -rake form_definition:validate_all -``` - -This will validate all forms in directories `["config/forms", "spec/fixtures/forms"]` +We can also define custom `routed_to?` methods on pages for more complex routing logic. ## Form models and definition For information about the form model and related models (section, subsection, page, question) and how these relate to each other see [form definition](/form/definition). - -## Improvements that could be made - -- JSON schema definition could be expanded such that we can better automatically validate that a given config is valid and internally consistent - -- Generators could parse a given valid JSON form and generate the required database migrations to ensure all the expected fields exist and are of a compatible type - -- The parsed form could be visualised using something like GraphViz to help manually verify the coded config meets requirements diff --git a/docs/form/definition.md b/docs/form/definition.md index 3d27bb30e..8eb646284 100644 --- a/docs/form/definition.md +++ b/docs/form/definition.md @@ -6,26 +6,15 @@ nav_order: 3 # Form definition -The current system is built around a form definition written in JSON. At the top level every form will expect to have the following attributes: +The current system is built around a form definition constructed from various Form subclasses. At the top level every form will expect to have the following attributes: - Form type: this is to define whether the form is a lettings form or a sales form. The questions will differ between the types. - Start date: the start of the collection window for the form, this will usually be in April. -- End date: the end date of the collection window for the form, this will usually be in July, a year after the start date. +- Submission deadline: the official end date of the collection window for the form, this will usually be in July, a year after the start date. +- New logs end date: the end date for creating any new logs for this form +- Edit end date: the end date for editing any existing logs for this form - Sections: the sections in the form, this block is where the bulk of the form definition will be. -An example of this might look like the following: - -```json -{ - "form_type": "lettings", - "start_date": "2021-04-01T00:00:00.000+01:00", - "end_date": "2022-07-01T00:00:00.000+01:00", - "sections": { - ... - } -} -``` - Note that the end date of one form will overlap the start date of another to allow for late submissions. This means that every year there will be a period of time in which two forms are running simultaneously. A form is split up is as follows: @@ -39,24 +28,24 @@ Rails uses the model, view, controller (MVC) pattern which we follow. ## Form model -There is no need to manually initialise a form object as this is handled by the FormHandler class at boot time. If a new form needs to be added then a JSON file containing the form definition should be added to `config/forms` where the FormHandler will be able to locate it and instantiate it. +There is no need to manually initialise a form object as this is handled by the FormHandler class at boot time. A form has the following attributes: - `name`: The name of the form -- `setup_sections`: The setup section (this is not defined in the JSON, for more information see this) -- `form_definition`: The parsed form JSON -- `form_sections`: The sections found within the form definition JSON +- `setup_sections`: The setup section +- `form_sections`: The sections passed to form on init - `type`: The type of form (this is used to indicate if the form is for a sale or a letting) -- `sections`: The combination of the setup section with those found in the JSON definition +- `sections`: The combination of the setup section with form sections - `subsections`: The subsections of the form (these live under the sections) - `pages`: The pages of the form (these live under the subsections) - `questions`: The questions of the form (these live under the pages) - `start_date`: The start date of the form, in ISO 8601 format -- `end_date`: The end date of the form, in ISO 8601 format +- `submission_deadline`: The official end date of the form, in ISO 8601 format +- `new_logs_end_date`: The new logs end date of the form, in ISO 8601 format +- `edit_end_date`: The edit end date of the form, in ISO 8601 format -Each form has an `end_date` which for JSON forms is defined in the form definition JSON file and for code defined forms it is set to 1st July, 1 year after the start year. -Logs with a form that has `end_date` in the past can no longer be edited through the UI. +Logs with a form that has `edit_end_date` in the past can no longer be edited through the UI. ## Form views diff --git a/docs/form/index.md b/docs/form/index.md index 664b136c7..ed21e3b10 100644 --- a/docs/form/index.md +++ b/docs/form/index.md @@ -13,18 +13,12 @@ A paper form is produced for guidance and to help data providers collect the dat Data is accepted for a collection window for up to 3 months after it’s finished to allow for late data submission. This means that between April and July 2 versions of the form run simultaneously. -Other considerations that went into our design are being able to re-use as much of this solution for other data collections, and possibly having the ability to generate the form and/or form changes from a user interface. +Other initial considerations that went into our design are being able to re-use as much of this solution for other data collections, and possibly having the ability to generate the form and/or form changes from a user interface. -We haven’t used micro-services, preferring to deploy a single application but we have modelled the form itself as configuration in the form of a JSON structure that acts as a sort of DSL/form builder for the form. +Each form has historically been defined as a JSON configuration, but has since been replaced with subsection, page and question classes that contruct a form in code due to increased complexity. -The idea is to decouple the code that creates the required routes, controller methods, views etc to display the form from the actual wording of questions or order of pages such that it becomes possible to make changes to the form with little or no code changes. - -This should also mean that in the future it could be possible to create an interface that can construct the JSON config, which would open up the ability to make form changes to a wider audience. Doing this fully would require generating and running the necessary migrations for data storage, generating the required ActiveRecord methods to validate the data server side, and generating/updating API endpoints and documentation. All of this is likely to be beyond the scope of initial MVP but could be looked at in the future. - -Since initially the JSON config will not create database migrations or ActiveRecord model validations, it will instead assume that these have been correctly created for the config provided. The reasoning for this is the following assumptions: +To allow for easier content changes, the copy for questions has been extracted into translation files. The reasoning for this is the following assumptions: - The form will be tweaked regularly (amending questions wording, changing the order of questions or the page a question is displayed on) - The actual data collected will change very infrequently. Time series continuity is very important to ADD (Analysis and Data Directorate) so the actual data collected should stay largely consistent i.e. in general we can change the question wording in ways that makes the intent clearer or easier to understand, but not in ways that would make the data provider give a different answer. - -A form parser class will parse this config into ruby objects/methods that can be used as an API by the rest of the application, such that we could change the underlying config if needed (for example swap JSON for YAML or for DataBase objects) without needing to change the rest of the application. We’ll call this the Form Runner part of the application. diff --git a/docs/form/page.md b/docs/form/page.md index 47de24d98..7e0607a70 100644 --- a/docs/form/page.md +++ b/docs/form/page.md @@ -10,43 +10,44 @@ Pages sit below the [`Subsection`](subsection) level of a form definition. An example page might look something like this: -```json -"property_postcode": { - "header": "", - "description": "", - "questions": { - ... - }, - "depends_on": [ - { - "needstype": 1 +``` +class Form::Sales::Pages::PropertyPostcode < ::Form::Page + def initialize(id, hsh, subsection) + super + @id = property_postcode + @depends_on = [{ "needstype" => 1 }] + @title_text = { + "translation": "translation1", + "arguments": [ + { + "key": "some_general_field", + "label": true, + "i18n_template": "template1" + } + ] } - ], - "title_text": { - "translation": "translation1", - "arguments": [ - { - "key": "some_general_field", - "label": true, - "i18n_template": "template1" - } - ] - }, - "informative_text": { - "translation": "translation2", - "arguments": [ - { - "key": "some_currency_method", - "label": false, - "i18n_template": "template2", - "currency": true, - } + @informative_text": { + "translation": "translation2", + "arguments": [ + { + "key": "some_currency_method", + "label": false, + "i18n_template": "template2", + "currency": true, + } + ] + } + end + + def questions + @questions ||= [ + Form::Sales::Questions::Question.new(nil, nil, self), ] - }, -} + end +end ``` -In the above example the the subsection has the id `property_postcode`. This id is used for the url of the web page, but the underscore is replaced with a hash, so the url for this page would be `[environment-url]/logs/[log-id]/property-postcode` e.g. on staging this url might look like the following: `https://dluhc-core-staging.london.cloudapps.digital/logs/1234/property-postcode`. +In the above example the the subsection has the id `property_postcode`. This id is used for the url of the web page, but the underscore is replaced with a dash, so the url for this page would be `[environment-url]/logs/[log-id]/property-postcode` e.g. on staging this url might look like the following: `https://staging.submit-social-housing-data.communities.gov.uk/logs/1234/property-postcode`. The header is optional but if provided is used for the heading displayed on the page. diff --git a/docs/form/question.md b/docs/form/question.md index dd3254e72..7112596cf 100644 --- a/docs/form/question.md +++ b/docs/form/question.md @@ -10,25 +10,25 @@ Questions are under the page level of the form definition. An example question might look something like this: -```json -"postcode_known": { - "check_answer_label": "Do you know the property postcode?", - "header": "Do you know the property’s postcode?", - "hint_text": "", - "type": "radio", - "answer_options": { - "1": { - "value": "Yes" +``` +class Form::Sales::Questions::PostcodeKnown < ::Form::Question + def initialize(id, hsh, page) + super + @id = postcode_known + @hint_text = "" + @header = "Do you know the property postcode?" + @check_answer_label = "Do you know the property postcode?" + @type = "radio" + @answer_options = { + "1" => { "value" => "Yes" }, + "0" => { "value" => "No" } }, - "0": { - "value": "No" - } - }, - "conditional_for": { - "postcode_full": [1] - }, - "hidden_in_check_answers": true -} + @conditional_for = { + "postcode_full" => [1] + }, + @hidden_in_check_answers = true + end +end ``` In the above example the the question has the id `postcode_known`. @@ -45,15 +45,11 @@ The `conditional_for` contains the value needed to be selected by the data input the `hidden_in_check_answers` is used to hide a value from displaying on the check answers page. You only need to provide this if you want to set it to true in order to hide the value for some reason e.g. it's one of two questions appearing on a page and the other question is displayed on the check answers page. It's also worth noting that you can declare this as a with a `depends_on` which can be useful for conditionally displaying values on the check answers page. For example: -```json -"hidden_in_check_answers": { - "depends_on": [ - { - "age6_known": 0 - }, - { - "age6_known": 1 - } +``` +@hidden_in_check_answers = { + "depends_on" => [ + { "age6_known" => 0 }, + { "age6_known" => 1 } ] } ``` @@ -62,25 +58,25 @@ Would mean the question the above is attached to would be hidden in the check an The answer the data inputter provides to some questions allows us to infer the values of other questions we might have asked in the form, allowing us to save the data inputters some time. An example of how this might look is as follows: -```json -"postcode_full": { - "check_answer_label": "Postcode", - "header": "What is the property’s postcode?", - "hint_text": "", - "type": "text", - "width": 5, - "inferred_answers": { - "la": { - "is_la_inferred": true +``` +class Form::Sales::Questions::PostcodeFull < ::Form::Question + def initialize(id, hsh, page) + super + @id = postcode_full + @hint_text = "" + @header = "What is the property’s postcode?"" + @check_answer_label = "Postcode"" + @type = "text" + @width = 5 + @inferred_answers = { + "la" => { "is_la_inferred" => true } } - }, - "inferred_check_answers_value": [{ - "condition": { - "postcode_known": 0 - }, - "value": "Not known" - }] -} + @inferred_check_answers_value => [{ + "condition" => { "postcode_known" => 0 }, + "value": "Not known" + }] + end +end ``` In the above example the width is an optional attribute and can be provided for text type questions to determine the width of the text box on the page when when the question is displayed to a user (this allows you to match the width of the text box on the page to that of the design for a question). diff --git a/docs/form/section.md b/docs/form/section.md index 514842355..e4443af3b 100644 --- a/docs/form/section.md +++ b/docs/form/section.md @@ -10,24 +10,22 @@ Sections sit at the top level of a form definition. An example section might look something like this: -```json -"sections": { - "tenancy_and_property": { - "label": "Property and tenancy information", - "subsections": { - "property_information": { - ... - }, - "tenancy_information": { - ... - } - } - }, - ... -} +``` +class Form::Sales::Sections::TenancyAndProperty < ::Form::Section + def initialize(id, hsh, form) + super + @id = "tenancy_and_property" + @label = "Property and tenancy information" + @description = "" + @subsections = [ + Form::Sales::Subsections::PropertyInformation.new(nil, nil, self), + Form::Sales::Subsections::TenancyInformation.new(nil, nil, self) + ] + end +end ``` -In the above example the section id would be `tenancy_and_property` and its subsections would be `property_information` and `tenancy_information`. +In the above example the section id would be `tenancy_and_property` and its subsections would be `PropertyInformation` and `TenancyInformation`. The label contains the text that users will see for that section in the task list page of a lettings log. diff --git a/docs/form/subsection.md b/docs/form/subsection.md index aa81c0259..5e659836a 100644 --- a/docs/form/subsection.md +++ b/docs/form/subsection.md @@ -10,29 +10,25 @@ Subsections sit below the [`Section`](section) level of a form definition. An example subsection might look something like this: -```json -"property_information": { - "label": "Property information", - "depends_on": [ - { - "setup": "completed" - } - ], - "pages": { - "property_postcode": { - ... - }, - "property_local_authority": { - ... - } - } -} +``` +class Form::Sales::Subsections::PropertyInformation < ::Form::Subsection + def initialize(id, hsh, section) + super + @id = property_information + @depends_on = [{ "setup": "completed" }] + @label = "Property information" + end + + def pages + @pages ||= [Form::Sales::Pages::PropertyPostcode.new(nil, nil, self),Form::Sales::Pages::PropertyLocalAuthority.new(nil, nil, self)] + end +end ``` In the above example the the subsection has the id `property_information`. The `depends_on` contains the set of conditions that must be met for the section to be accessible to a data provider, in this example subsection depends on the completion of the setup section/subsection (note that this is a common condition as the answers provided to questions in the setup subsection often have an impact on what questions are asked of the data provider in later subsections of the form). The label contains the text that users will see for that subsection in the task list page of a lettings log. -The pages of the subsection in the example would be `property_postcode` and `property_local_authority`. +The pages of the subsection in the example would be `PropertyPostcode` and `PropertyLocalAuthority`. Subsections can contain one or more [pages](page). diff --git a/docs/setup.md b/docs/setup.md index d14fa58d9..4400a7ae2 100644 --- a/docs/setup.md +++ b/docs/setup.md @@ -79,15 +79,23 @@ We recommend using [nvm](https://github.com/nvm-sh/nvm) to manage NodeJS version macOS (using nvm): ```bash - nvm install 16 - nvm use 16 + nvm install 20 + nvm use 20 + brew install yarn + ``` + + or you could run it without specifying the version and it should use the version from .nvmrc + + ```bash + nvm install + nvm use brew install yarn ``` Linux (Debian): ```bash - curl -sL https://deb.nodesource.com/setup_16.x | sudo bash - + curl -sL https://deb.nodesource.com/setup_20.x | sudo bash - sudo apt -y install nodejs mkdir -p ~/.npm-packages npm config set prefix ~/.npm-packages