# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2316
- Implement bulk upload sales for new collection year 2023
- This is a first pass implementation and will probably have some bugs in it and we can address over time
# Changes
- Add CSV parser for sales 2023 to handle CSV structure
- Tweak collection window validation so error now contextual to year selected for upload
- Handle arbitrary ordering of CSV columns
- Fix ordering of errors in report and now ordered by cell
- Added `Upload your file again` link styled as button on error report to match lettings experience
- Update tooling to convert logs to 2023 csv rows with support for random column ordering
# Known issues
- There seem to be some issues with how UPRN is handled if the UPRN cannot be validated.
- For the above I think there is dependency on https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/1570 which should clear any errored fields so users can continue to create logs and fix within the service
* Clear invalid format previous postcode
* Fix import with income 2 outside of london validation
* Remove mscharge if it is under minimum
* Clear frombeds if it's out of range
* create a method on the FormHandler that returns the sales form questions for all years in the order that they appear in the form
* update csv email job to accomodate sales log export as well as lettings
add to tests to reflec the changes made
* write tests to cover the desired functionality of the SalesLogCsvService
* create the SalesLogCsvService
create a necessary method on the log to enable submission method to be included on the csv
derive values for the two halves of previous postcode for export
* add relevant links in the UI and pipe everything together in controllers
amend organisations controller to have flexibility to download logs of either type
add necessary methods to sales log controller, raising shared method to logs controller
update routing for amendments and additions
extract helper method to build urls for downloading logs within an organisation
* correct various linter complaints and tech review suggestions
* minor amendment to add old_id and reorder early columns
* undo my 'clever' refactor that broke things
* refactoring of csv service after some tech review and some UI testing in review app
* update tests to include a test of a full export and all values in teh csv
* correct minor routing error to ensure correct url is shown and tab selected after requesting csv email
* update organisations controller requests spec file to cover new functionality and make a minor amendment to authentication scope in the controller after error found in testing
* write request tests for the new functionality in the sales log controller, define authorisation in the controller
* minor correction after rubocop's kind suggestion'
* various corrections from first pass at PO, tech review, linter, etc
* refactor :ordered_sales_questions_for_all_years
* first pass at implementing flexible code-based form fixtures for testing
* second pass
* refactor all tests of :ordered_sales_questions_for_all_years to use new factories
* some refactoring in the testing of the csv service
* use that fact that params is always available in controllers and don't pass it around, inline some methods calls
* correct minor bug to ensure that "Return to logs" link returns to the correct index page
* remove reminder comments
* write further tests on the manipulation of questions into the csv headers, update factories of form constituents to allow the creation of forms with richer questions
* fix linter complaints
* minor alterations after rebase to account for changes made on other branches
* refactor after code review
* tweak fixtures after rebase containing alterations to the factory defaults
* Clear all the charges if the error is on tcharge
* Round earnings value upon import
* Rounds savings to the nearest 10
* Clear income on over_hard_max_for_london validation
* Refactor sale import validations to be consistent with lettings
* Extract charges attributes into a variable
* add a validation to prevent an inconsistent combination of values and tests for this validation
* show related method in diff
* remove comment
* extract reusable logic from SalesLogVariables to independent module
* update lettings log tests around derivations related to renewal
* refactor derivation logic to share functionality with sales logs where possible
* remove some tests which are now duplicated above, refactor tests using Jack's wonderful :change suggestion, add in tests about logic deriving vacdays that was not previously covered
* minor changes after tech review
needstype is a field that contributes to the vcalidation and can therefore trigger the validation
relevant test file has also been updated in line with this change
* Add new organisation name column
* Update new organisation name page
* Refactor validate_response
* Add placeholder new_organisation_address page
* Add existing organisation name validation
* Refactor error messages
* Extract some validations to the model
* Update which field we add the errors to
* add several methods to the sales log to allow subsequent work to be human readable
* add a validation to prevent an inconsistent combination of values and tests for this validation
* handle derivation of values around buyers living in the property
- derive values where appropriate
- clear these values when the derived state no longer holds
- update the routing for pages holding questions about whether particular buyers will live in the property to reflect when they are derived
- test the deriving and associated clearing of values
- update tests on page routing and sales log factory
* update a page routing condition for human readability using an existing method and update test to reflect this change
* show related method in diff
* minor amendments after tech review
* simplify reset_derived_questions after tech review
* refactor on deriving and clearing invalid derived values on sales log
* correct linter complaints
* remove comment
* add validation to one more field with a new error message as it is in fact possible to tirgger the validation in the setup section
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2301
- Content for 2022 and 2023 bulk upload journey need to differ to give users the correct info they need to complete the journey
# Changes
- Update 2022 and 2023 `Prepare your file` and guidance pages
* [CLDC-2202] Allow coordinators to set created_by
* Scope user selection when data coordinator
* Remove managing_for_other_user_enabled
* Move sales created_by page and question out of common
* Address comments
- only select required users
- remove not needed CYA checks
* CLDC-2056 Add absorbing organisation question
* Handle next page and previous_template with `page`
* Hardcode backlinks
* Refactor flow to store value when selecting new org option
* Handle unanswered question
* Update error copy
* feat: unrelated currency formatting
* Revert "feat: unrelated currency formatting"
This reverts commit 75bd3efbf8.
* feat: add soft val pages to property information section (not to setup)
* feat: update tests
* feat: update tests
* feat: add validation to setpu section
* refactor: lint
* test: update
* feat: set nil as default check answers card no
* feat: use : to define more explicitly
* feat: make optional as affects setup
* feat: update tests
* feat: put la in nil card with rest of uprn/address qs
* feat: route to uprn confirmation from check answers changes to uprn even though not an interruption screen
* feat: update tests
* feat: update tests
* refactor: lint
* feat: update tests
* feat: update tests
* feat: wip blank fields and dependent fields on upload tos ee if valid and can upload with missing info - this is not the exact ac on the ticket yet
* Revert "feat: wip blank fields and dependent fields on upload tos ee if valid and can upload with missing info - this is not the exact ac on the ticket yet"
This reverts commit 136e5f6f12.
* feat: fix uprn hidden in check answers
* feat: fix tests
* feat: fix tests
* rename variables in test file
* write tests for new validations
* validate that answers to numeric questions are given to the correct level of accuracy and in normal format with only digits
* add error messages for new validations
* add or alter step on numeric questions in sales, amend one test given step changes
* copy change
* remove validation that is specific to number format and fold that functionality into accuracy validation, rejig tests accordingly
* fix various tests
* remove reference to currency in validation message that concerns step only and doesn't check whether firled is currency related
* alter copy to avoid redundant repetition in validaiton message
* save changes made in rebase
* changes after PO review
two step values had been missed in previous work
in various places there were custom validations applied that duplicated the functionality of the generic min max validation that is applied to all numeric questions, in these situations sometimes the min and max on the question class was inconsistent with the limit that triggered validations
these have been corrected and made consistent
various tests were affected by this and have been amended accordingly
* remove tests introduced by mistake in rebase
* amend step value on earnings
* Create merge_request table and paths
* Save merging organisations wip
* Add update organisations
* Add merging organisations validation
* merge fixes
* Update schema to have merge_request_organisations
* Add relationships between merge request and organisations
* Update validations and saving organisations
* Add ability to remove merging orgs from the list
* Allow support users to create merge request for any organisation
* Update wording in organisations view
* Allow adding other merging organisations
* Add back button, update content
* Add validation if the organisation is not selected
* Fix path
* Use generic update method
* Update validations
* fix remove organisation
* remove reloads
* Update routes
* Authenticate scope
* PR review changes
* Add status, run validations unless the status is unsubmitted and save requesting organisation as a merging organisation as well
* PR comments
* Display continue button when there are at least 2 merging organisations
* write tests that invalid answers are cleared when the start date of a log is changed
* clear invalid answers when the start date of a log is changed
* correct error
* undo previous work to take new direction
* write tests to cover updated functionality of method in form to reset values when they are not routed to or when answers to radio questions are no longer valid
* update method in form to improve readability and to ensure that when the answers to radio questions are no longer valid, taht they are cleared
* add back an "end" deleted by mistake and fix a linting issue
* make some minor copy and variable name corrections
* fix a broken test after updates
* remove bulk upload tests for adding validations to unpermitted values to radio questions, these are now being cleared before validation
* fix some tests broken after rebasing
* add step to age questions in sales
* remove list of question ids from the form that should not be cleared, this information should be held on the questions themselves
* rename variables
* rename instance variable to be nice and long
* minor changes after rebase
* feat: assign simultaneously
* feat: assign simultaneously
* feat: revert
* feat: fix back persistence behaviour and update similar issues elsewhere, also remove redundant back text in govuk back links
* feat: add previous_page_redirect_paths to avoid infinite loops
* feat: leave here to avoid possible loops
* feat: leave here to avoid possible loops
* feat: add check answers page behaviour
* refactor: tweak
* feat: make tests pass, simplify code and incorporate tasklist/check answers back routing
* feat: update tests
* refactor: lint
* feat: update scheme status so incomplete unless has active locations
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: show completed/incomplete locations and validate when completed == 0
* feat: improve copy
* feat: change active -> completed
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: show completed/incomplete locations and validate when completed == 0
* feat: improve copy
* feat: change to confirmed to add clarification in code
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: show completed/incomplete locations and validate when completed == 0
* feat: improve copy
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: only confirm locations if complete (old) AND save button clicked (new)
* feat: add unconfirmed scope
* refactor: complete -> confirm
* feat: fix tests
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: show completed/incomplete locations and validate when completed == 0
* feat: improve copy
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: only confirm locations if complete (old) AND save button clicked (new)
* feat: add unconfirmed scope
* refactor: complete -> confirm
* feat: fix tests
* feat: fix more tests
* feat: fix more tests
* refactor: rubocop
* refactor: rubocop
* refactor: rubocop
* feat: add tests for incomplete schemes
* refactor: linting
* feat: test incomplete count too
* feat: test unconfirmed locations aren't options in location_id
* feat: test validation
* feat: test when scheme is confirmed
* feat: test when scheme is confirmed for support user
* feat: test when location is confirmed
* refactor: linting
* refactor: consistent use of factorybot or not within a file
* feat: confirm locations when complete even if save button not clicked, and update tests
* refactor: simplify
* refactor: simplify
* refactor: move location helper methods to helpers
* refactor: move scheme helper methods to helpers
* refactor: formatting
* refactor: po response
* refactor: review response
* Set uprn_known to 1 when setting UPRN
* Move fake to dev+test gemfile block
So that we can use factories in local
* Move UPRN known and UPRN questions to single UPRN page
* add in a new option for the period question and reorder existing answer options
* ensure that the lettings log calculates values correctly that relate to the period
* ensure that depends on conditions related to period take account of the new option
relatedly create methods on LettingsLog to allow improved readability in these depends on conditions
refactor depends on conditions with these new method
* update the 22_23 json form to use the new option as well
* update tests to cover deriving variables that depend on period
update test fixture to allow this
* refactor some depends on and amend LEttingsLog method to improve readability
* remove forgotten focus in spec file found in linting check
* amend context descriptino in new tests
* refactor a couple more depends on for readability
* rename variables in test file
* write tests for new validations
* validate that answers to numeric questions are given to the correct level of accuracy and in normal format with only digits
* add error messages for new validations
* add or alter step on numeric questions in sales, amend one test given step changes
* copy change
* remove validation that is specific to number format and fold that functionality into accuracy validation, rejig tests accordingly
* fix various tests
* remove reference to currency in validation message that concerns step only and doesn't check whether firled is currency related
* alter copy to avoid redundant repetition in validaiton message
* save changes made in rebase
* changes after PO review
two step values had been missed in previous work
in various places there were custom validations applied that duplicated the functionality of the generic min max validation that is applied to all numeric questions, in these situations sometimes the min and max on the question class was inconsistent with the limit that triggered validations
these have been corrected and made consistent
various tests were affected by this and have been amended accordingly
* remove tests introduced by mistake in rebase
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2184
- bulk upload error reports were not sorting by cells correctly
# Changes
- there was an issue where `Z100` would come after `AA100` which is not correct
- fix ordering issue above
- also made change so this is now delegated to the database rather than sorting in ruby
* Allow interruption screens to format money values
* Remove formatting from saleslog method
* Use money flag
* Use pre exhisting method to format values
* Update config/locales/en.yml
Co-authored-by: Arthur Campbell <51094020+arfacamble@users.noreply.github.com>
---------
Co-authored-by: Arthur Campbell <51094020+arfacamble@users.noreply.github.com>
* Format errors on money amounts with 2 decimals
* CLDC-1975 Format money input and error messages
* format discounted_ownership_value
* Format more locales
* Update lettings_log tests
* Fix import service spec
* create method to test whether we are currently in the crossover period and associated tests
* update copy, use method for testing whether we are in the crossover period, remove focus from test file
* reuse existing method to determine whether we are in a collection period
* use the existing method and update validations
* fix test broken by changes
* update location in same way as scheme
* create method on FormHandler that finds the start date of the earliest collection period
* ensure that default deactivation and reactivation dates also reflect the changes
* create tests for the new validations
* lint correction
* minor copy change
* minor logic change
* amend naming error after rebase conflict
remove test that is no longer correct after change on another branch to functionality
update a test after a copy change
* add first page for bulk upload resume journey
* bulk upload resume handles upload again
* add confirm page to bulk upload resume journey
* replace placeholder count with correct value
* apply recommendation for bulk upload resume choice
* add how to fix bulk upload mailer
* integrate new bulk upload approve journey
* add missing bulk upload error mappings
* remove test
* prevent approve being called multiple times
* bulk upload creates invisible logs ahead of time
* work invisible logs into bulk upload flow
* sort errors so deterministic
* remove unused ensure
* remove expected_log_count and processed
- these fields are no longer used or needed
* introduce pending status
* swap visible for pending logs
* only show visible lettings logs
* hard code status filters
* remove unused model methods
* only show visible sales logs
* form controller ignores hidden logs
* locations and schemes only affect visible logs
kosiakkatrina
James Rose
Phil Lee
Arthur Campbell
Jack
natdeanlewissoftwire