* create helper module to find all logs for a given user that have duplicates
write associated tests
update methods on user model to retrieve logs related to a given user
create a scope on logs to retrieve logs created by a given user
* check for duplicates and display notification banner on the index page where appropriate
tests to cover the desired behaviour
* alter duplicate logs helper to return duplicates in a format that can be passed through the params
update associated tests
* create user journey to view duplicate sets adn navigate to a given set
create a new route and add it to the link from the logs index page
create controller method and related view file
* update path for duplicate logs in routes, update factories, refactor duplicates scope to eliminate raw sql
* write tests for the duplicate logs index page
* correct linting errors
* minor amendments from code review.
* temp
* Refactor
* Create a scope for duplicate groups
* fixes
* lint
* Update controller specs
* Put duplicate sets count behind a feature flag
* refactor index
* Update duplicate_sets_count
* Update scopes
* tests
* pass in original log id
* tests
* Add duplicate index page content
---------
Co-authored-by: Kat <katrina@kosiak.co.uk>
* Update check answers to use the correct logs
* Display different link when correcting duplicates
* allow deduplicating logs by changing the answers
* Route back to the initial log if it still has duplicates
* Update back to log button
* Add banner
* Update success banner
* Fix a flaky test
* Rename param and update conditionals
* Add owning organisation to deduplication fields, only return back to the original log if it still belongs to the org
* Add start date validations for merging orgs
* Add errors for merged orgs
* tests
* don't run validations unless the date exists
* Adjust the dates in validations
* Send notification to old email address when changing it
* send email changed notification to old email
* Add specs
* send both emails
* Only send required confirmation emails
* Show flash notice
* Use reconfirmable template only if not confirmed
* Don't commit .env.development
* new email flow also when updating own email
* CLDC-2494: WIP
* CLDC-2494: wip
* CLDC-2494: page work in progress
* cleanup
* Add a path for duplicate logs
* Display all duplicate logs
* Move a test
* Display duplicate check answers for logs
* Add buttons to delete duplicates
* Add a route for sales logs duplicates
* Update duplicated page to work for sales logs
* lint
* Add auth
* Rebase updates
* Update displayed questions
* Add delete duplicates page
* Update page content for when there are multiple duplicates
* Add auth to the path
* Update delete button to use delete controller
* Update page for sales logs
* Update styling
* Update success banner for deleting duplicates
* Typo
* Render not found if there are no duplicate logs
* rebase changes
* Rebase changes
* Update back and cancel links
* Update the duplicate journey after deleting logs
* Update change button routing
* Refactor tests
* Add content for no duplicates
* Update params
* Refactor tests and paths
* Update params to include original_log_id from the beginning
* Rename file
* lint
* Add full stop after deletion messages
* Add caption and update duplicate log IDs in the banner
* lint
* Styling
---------
Co-authored-by: Aaron Spencer <aaron.spencer@softwire.com>
* CLDC-2439: min and max rent value check questions created
* CLDC-2439: Informative removed, check pages fixed.
* CLDC-2439: removed informative info tests and rent value check
* CLDC-2439: fixed informative missing test failure
* CLDC-2439: Text moved to en.yml
* CLDC-2439: tests added, copy moved to en.yml
* CLDC-2439: tests added, copy moved to en.yml
* CLDC-2439: content change for bulk upload errors WIP
* CLDC-2439: informative rendering change, content change, test updated
* CLDC-2439: failing test fix.
* CLDC-2439: display_informative_text test added for strings
* CLDC-2439: string as var in test
* Change number validation to allow 0 at the beginning
* Update valid_phone_number?
* Move tests to request
* Make phone required for new users
* Allow + in the numbers
* Feature test
* add a button to the logs list to delete multiple logs
style and position of button
helpers for displaying the button conditionally depending on user role and what filters and search are active
* correct indentation from 4 spaces to 2 in view file
* test appearance of delete logs button on index page for lettings logs
* write a happy path feature test for the entire journey
* create basic tests for the view component for listing logs to delete
* create request tests for the GET delete-logs path
* create request tests for the GET delete-logs-confirmation path
* create request tests for the DELETE delete-logs path
* comprehensive reworking after code review
ensure that we are not passing lists of ids through params in the query string, risking overflowing the maximum URL length,
adjust tests accordingly,
do not attempt to reuse the same table for sales and lettings
* alter config to allow creating controllers from the command line with associated spec files that matches how we test
* extract controller methods and associated tests to do with the delete logs feature into their own controller,
amend routes accordingly
* implement same work for sales as for lettings
* implement the story for lettings and sales logs under the organisation tab
routing and controller methods
testing for deleting sales logs, lettings or sales logs for an organisation
move storage of relevant routes inside the form object as a comprehensive view model
* merge the delete pages for lettings logs and sales logs, add to the tests for the lettings page to test sales specific content
* minor refactor to delete logs controller: ensure session filters are only fetched from teh session when needed and extract discard logs method to private method
* extract tables for lettings and sales to own partials
* refactor delete logs controller after tech review
improve the private method that builds the form object so that it has the flexibility to do so for all controller methods
ensure that the search term is passed to the delete logs controller when navigating through the organisations tab
ensure that noly logs for that organisation are displayed when navigating to delete logs through the organisations tab
* remove unnecessary untested arguments
* test new helper methods
* implement dirty fiddle to get the checkboxes smaller and also not misaligned
* ensure delete logs button is always visible on log lists when in the organisations tab
* minor linting corrections
* revert change, causing errors and outside the scope of this ticket
* simplify tests for whether delete logs button appears on index page
* replicate request specs from lettings for sales and organisations controllers
* minor refactor of lettings log feature spec setup, replicate happy path for sales
* minor refactors after rebasing onto Nat's work
* temp
* write tests for the delete logs form object
* lint: add new line at end of file
* respond to PO feedback
the log id in the delte logs table should be a link to the log
the delete logs button should be visible when the user is in a bulk upload journey
updated associated tests
* Remove bulk_upload_lettings_logs flag
* Add spacing above actions in data sharing agreement
* Add banner
* Add model validation
* fix log component
* Fix factory
* Put validation behind feature flag
* start adapting to data protection confirmation
* Make tests more generic
* Add tests for display_organisation_attributes
* Use data_protection_confirmed? method
* Address code review
* rubocop
* Prevent access to bulk upload pages when DPC not signed
* Address PO review
* Validate that correct template for the year is used for sales (with headers)
* Check the correct template is being used without headers
* Check correct template for lettings
* Update csv parser on sales
* Remove redundant methods
* Extract form years
* Reverse year check mathod
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2305
- This is rework to add missing validations for sales setup fields for bulk upload
# Changes
- There is code that clears log fields if they are not a valid option. in order to generate errors for these we check certain fields for content prior to `log.valid?` being called. Unfortunately there does not appear to be an easy way to access to valid options therefore these values have been hard coded
- Privacy notice must be accepted other considered a setup error
- The `type` question can actually be one of three questions with the same identifier. these are excluded as part of `validate_valid_radio_option` and instead validated with a different mechanism. The rationale being that `log.valid?` will clear data before we get there
- Remove tests associated to upload threshold which are no longer required
# Context
- We're currently using a custom fork of devise
# Changes
- Switch back to original devise gem
- This gives us the benefit of being able to more easily upgrade when the time comes
- Minor fixes to validation copy during password reset process which were not correct
* Sends a correct email if there are only soft validations
* Add soft errors valid page
* Add soft errors confirm page
* Confirm the soft validations
* Reuse the how to fix template for check soft validations email
* Update email link
* Move soft validation confirmation to processor
* Correctly set the log status, remove redundant confirm_soft_validations
* Redirect successful upload to logs index and display a success banner
* Implement the soft validations only journey for sales logs
* Display the soft validation errors on the soft-errors-valid page
* Fix page alignment
* Fix tests by mapping housingneeds in csv hepler
* Add the sales soft validations to unpend_and_confirm_soft_validations
* Change naming
* Update method names
* refactor
* undo typo
* Only set the existing soft validations for correct types
* Fix path name
* Add missing error mappings for location fields
* Add missing tests and cancel button
* Change some wording
* Typos
* feat: add max value check for buyer 1
* feat: add tests and refactor
* feat: add buyer 2 value check
* feat: combined income value check
* feat: add combined validation everywhere and add tests
* feat: add max value check for buyer 1
* feat: add tests and refactor
* feat: add buyer 2 value check
* feat: combined income value check
* feat: add combined validation everywhere and add tests
* db:update
* refactor: lint
* db: update
* feat: update copy for single validations
* feat: add correct answer card numbers
* feat: add tests
* feat: fix tests
* feat: correct uprn address routing not to show when uprn_confirmed is nil
* refactor: lint
* feat: update tests
* feat: make combined_income nil safe
* feat: merge with main
* test: update
* feat: update to new soft val designs
* feat: replace la with uprn, postoce_full, la in all interruption_screen_question_ids
* feat: update tests
* feat: wip blank fields and dependent fields on upload tos ee if valid and can upload with missing info - this is not the exact ac on the ticket yet
* feat: update seed
* feat: wip commit
* feat: add postcodenk error so can clear on validation
* feat: add postcode validation back
* feat: move la vals to shared and move and add tests
* feat: add correct pluralisation to warning message
* feat: add blank compound invalid fields methods
* feat: update validations
* feat: update pluralisation
* refactor: lint
* feat: clear errors associated with blanked values so log status is set correctly on creation
* feat: validate instead
* feat: avoid duplicated errors
* feat: dont auto-refuse income, different to imports
* feat: validate after every blank method
* feat: delete la validator spec
* tests: update
* refactor: erblinting
* refactor: cleanup
* refactor: move pluralizer to helper
* feat: copy update
* feat: rename
* feat: refactor to avoid redundant re-validations and test
* refactor: rubocop
* test: update
* test: update
* test: update
* update sidekiq
* feat: clear errors
* feat: run clearing twice in case first clear creates different errors
* feat: remove moved file
* feat: undo validation file tweaks as shared/specific overlap could do with a more general refactor
* feat: update tests
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2316
- Implement bulk upload sales for new collection year 2023
- This is a first pass implementation and will probably have some bugs in it and we can address over time
# Changes
- Add CSV parser for sales 2023 to handle CSV structure
- Tweak collection window validation so error now contextual to year selected for upload
- Handle arbitrary ordering of CSV columns
- Fix ordering of errors in report and now ordered by cell
- Added `Upload your file again` link styled as button on error report to match lettings experience
- Update tooling to convert logs to 2023 csv rows with support for random column ordering
# Known issues
- There seem to be some issues with how UPRN is handled if the UPRN cannot be validated.
- For the above I think there is dependency on https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/1570 which should clear any errored fields so users can continue to create logs and fix within the service
* add a validation to prevent an inconsistent combination of values and tests for this validation
* show related method in diff
* remove comment
* extract reusable logic from SalesLogVariables to independent module
* update lettings log tests around derivations related to renewal
* refactor derivation logic to share functionality with sales logs where possible
* remove some tests which are now duplicated above, refactor tests using Jack's wonderful :change suggestion, add in tests about logic deriving vacdays that was not previously covered
* minor changes after tech review
* Add new organisation name column
* Update new organisation name page
* Refactor validate_response
* Add placeholder new_organisation_address page
* Add existing organisation name validation
* Refactor error messages
* Extract some validations to the model
* Update which field we add the errors to
* add several methods to the sales log to allow subsequent work to be human readable
* add a validation to prevent an inconsistent combination of values and tests for this validation
* handle derivation of values around buyers living in the property
- derive values where appropriate
- clear these values when the derived state no longer holds
- update the routing for pages holding questions about whether particular buyers will live in the property to reflect when they are derived
- test the deriving and associated clearing of values
- update tests on page routing and sales log factory
* update a page routing condition for human readability using an existing method and update test to reflect this change
* show related method in diff
* minor amendments after tech review
* simplify reset_derived_questions after tech review
* refactor on deriving and clearing invalid derived values on sales log
* correct linter complaints
* remove comment
* add validation to one more field with a new error message as it is in fact possible to tirgger the validation in the setup section
* CLDC-2056 Add absorbing organisation question
* Handle next page and previous_template with `page`
* Hardcode backlinks
* Refactor flow to store value when selecting new org option
* Handle unanswered question
* Update error copy
* feat: add validation for number of field labels
* feat: check field labels are numbers
* feat: remove validations for number of columns
* refactor: rename non_blank_fields_count to valid_field_numbers_count
* test: add functions to generate custom field labels/values
* test: that extra invalid field labels don't cause issues
* test: that removing a valid field label reduces count by 1
* test: that adding a valid field label increases count by 1
* refactor: rename validate_fields_count and wrong_field_count
* feat: add validation that max col count not exceeded when no headers
* fix: convert numbers to strings in default_field_numbers
* feat: add leniency to max cols count (in case of 1 extra col)
* test: explicitly set year of bulk upload to be 2022
* test: add/update tests in validator_spec
* chore: lint
* test: remove tests from csv_parser_spec that were moved to validator_spec
* feat: update 2022 csv_parser to work with new validations
* refactor: define number of valid 2022 fields in one place
* refactor: remove redundant headers definition
* feat: update 2022 csv parser to have col flexibility like 2023
* test: for validator 2022 as well as 2023
* test: simplify 2022/2023 logic and improve layout
* chore: lint
* test: ensure context descriptions start with "when"
* refactor: check fields/columns count within csv parser, not validator
* test: update 2022 csv parser tests to work like 2023
* chore: lint
* rename variables in test file
* write tests for new validations
* validate that answers to numeric questions are given to the correct level of accuracy and in normal format with only digits
* add error messages for new validations
* add or alter step on numeric questions in sales, amend one test given step changes
* copy change
* remove validation that is specific to number format and fold that functionality into accuracy validation, rejig tests accordingly
* fix various tests
* remove reference to currency in validation message that concerns step only and doesn't check whether firled is currency related
* alter copy to avoid redundant repetition in validaiton message
* save changes made in rebase
* changes after PO review
two step values had been missed in previous work
in various places there were custom validations applied that duplicated the functionality of the generic min max validation that is applied to all numeric questions, in these situations sometimes the min and max on the question class was inconsistent with the limit that triggered validations
these have been corrected and made consistent
various tests were affected by this and have been amended accordingly
* remove tests introduced by mistake in rebase
* amend step value on earnings
* Create merge_request table and paths
* Save merging organisations wip
* Add update organisations
* Add merging organisations validation
* merge fixes
* Update schema to have merge_request_organisations
* Add relationships between merge request and organisations
* Update validations and saving organisations
* Add ability to remove merging orgs from the list
* Allow support users to create merge request for any organisation
* Update wording in organisations view
* Allow adding other merging organisations
* Add back button, update content
* Add validation if the organisation is not selected
* Fix path
* Use generic update method
* Update validations
* fix remove organisation
* remove reloads
* Update routes
* Authenticate scope
* PR review changes
* Add status, run validations unless the status is unsubmitted and save requesting organisation as a merging organisation as well
* PR comments
* Display continue button when there are at least 2 merging organisations
* write tests that invalid answers are cleared when the start date of a log is changed
* clear invalid answers when the start date of a log is changed
* correct error
* undo previous work to take new direction
* write tests to cover updated functionality of method in form to reset values when they are not routed to or when answers to radio questions are no longer valid
* update method in form to improve readability and to ensure that when the answers to radio questions are no longer valid, taht they are cleared
* add back an "end" deleted by mistake and fix a linting issue
* make some minor copy and variable name corrections
* fix a broken test after updates
* remove bulk upload tests for adding validations to unpermitted values to radio questions, these are now being cleared before validation
* fix some tests broken after rebasing
* add step to age questions in sales
* remove list of question ids from the form that should not be cleared, this information should be held on the questions themselves
* rename variables
* rename instance variable to be nice and long
* minor changes after rebase
* feat: update scheme status so incomplete unless has active locations
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: show completed/incomplete locations and validate when completed == 0
* feat: improve copy
* feat: change active -> completed
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: show completed/incomplete locations and validate when completed == 0
* feat: improve copy
* feat: change to confirmed to add clarification in code
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: show completed/incomplete locations and validate when completed == 0
* feat: improve copy
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: only confirm locations if complete (old) AND save button clicked (new)
* feat: add unconfirmed scope
* refactor: complete -> confirm
* feat: fix tests
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: show completed/incomplete locations and validate when completed == 0
* feat: improve copy
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: only confirm locations if complete (old) AND save button clicked (new)
* feat: add unconfirmed scope
* refactor: complete -> confirm
* feat: fix tests
* feat: fix more tests
* feat: fix more tests
* refactor: rubocop
* refactor: rubocop
* refactor: rubocop
* feat: add tests for incomplete schemes
* refactor: linting
* feat: test incomplete count too
* feat: test unconfirmed locations aren't options in location_id
* feat: test validation
* feat: test when scheme is confirmed
* feat: test when scheme is confirmed for support user
* feat: test when location is confirmed
* refactor: linting
* refactor: consistent use of factorybot or not within a file
* feat: confirm locations when complete even if save button not clicked, and update tests
* refactor: simplify
* refactor: simplify
* refactor: move location helper methods to helpers
* refactor: move scheme helper methods to helpers
* refactor: formatting
* refactor: po response
* refactor: review response
* rename variables in test file
* write tests for new validations
* validate that answers to numeric questions are given to the correct level of accuracy and in normal format with only digits
* add error messages for new validations
* add or alter step on numeric questions in sales, amend one test given step changes
* copy change
* remove validation that is specific to number format and fold that functionality into accuracy validation, rejig tests accordingly
* fix various tests
* remove reference to currency in validation message that concerns step only and doesn't check whether firled is currency related
* alter copy to avoid redundant repetition in validaiton message
* save changes made in rebase
* changes after PO review
two step values had been missed in previous work
in various places there were custom validations applied that duplicated the functionality of the generic min max validation that is applied to all numeric questions, in these situations sometimes the min and max on the question class was inconsistent with the limit that triggered validations
these have been corrected and made consistent
various tests were affected by this and have been amended accordingly
* remove tests introduced by mistake in rebase
natdeanlewissoftwire
kosiakkatrina
Arthur Campbell
Jack
Aaron Spencer
Phil Lee
SamSeed-Softwire