# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2316
- Implement bulk upload sales for new collection year 2023
- This is a first pass implementation and will probably have some bugs in it and we can address over time
# Changes
- Add CSV parser for sales 2023 to handle CSV structure
- Tweak collection window validation so error now contextual to year selected for upload
- Handle arbitrary ordering of CSV columns
- Fix ordering of errors in report and now ordered by cell
- Added `Upload your file again` link styled as button on error report to match lettings experience
- Update tooling to convert logs to 2023 csv rows with support for random column ordering
# Known issues
- There seem to be some issues with how UPRN is handled if the UPRN cannot be validated.
- For the above I think there is dependency on https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/1570 which should clear any errored fields so users can continue to create logs and fix within the service
* create a method on the FormHandler that returns the sales form questions for all years in the order that they appear in the form
* update csv email job to accomodate sales log export as well as lettings
add to tests to reflec the changes made
* write tests to cover the desired functionality of the SalesLogCsvService
* create the SalesLogCsvService
create a necessary method on the log to enable submission method to be included on the csv
derive values for the two halves of previous postcode for export
* add relevant links in the UI and pipe everything together in controllers
amend organisations controller to have flexibility to download logs of either type
add necessary methods to sales log controller, raising shared method to logs controller
update routing for amendments and additions
extract helper method to build urls for downloading logs within an organisation
* correct various linter complaints and tech review suggestions
* minor amendment to add old_id and reorder early columns
* undo my 'clever' refactor that broke things
* refactoring of csv service after some tech review and some UI testing in review app
* update tests to include a test of a full export and all values in teh csv
* correct minor routing error to ensure correct url is shown and tab selected after requesting csv email
* update organisations controller requests spec file to cover new functionality and make a minor amendment to authentication scope in the controller after error found in testing
* write request tests for the new functionality in the sales log controller, define authorisation in the controller
* minor correction after rubocop's kind suggestion'
* various corrections from first pass at PO, tech review, linter, etc
* refactor :ordered_sales_questions_for_all_years
* first pass at implementing flexible code-based form fixtures for testing
* second pass
* refactor all tests of :ordered_sales_questions_for_all_years to use new factories
* some refactoring in the testing of the csv service
* use that fact that params is always available in controllers and don't pass it around, inline some methods calls
* correct minor bug to ensure that "Return to logs" link returns to the correct index page
* remove reminder comments
* write further tests on the manipulation of questions into the csv headers, update factories of form constituents to allow the creation of forms with richer questions
* fix linter complaints
* minor alterations after rebase to account for changes made on other branches
* refactor after code review
* tweak fixtures after rebase containing alterations to the factory defaults
* Add new organisation address fields to merge requests
* Add new organisation address page
* Add new organisation telephone number placeholder page
* close the div
* Add new organisation name column
* Update new organisation name page
* Refactor validate_response
* Add placeholder new_organisation_address page
* Add existing organisation name validation
* Refactor error messages
* Extract some validations to the model
* Update which field we add the errors to
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2301
- Content for 2022 and 2023 bulk upload journey need to differ to give users the correct info they need to complete the journey
# Changes
- Update 2022 and 2023 `Prepare your file` and guidance pages
* CLDC-2056 Add absorbing organisation question
* Handle next page and previous_template with `page`
* Hardcode backlinks
* Refactor flow to store value when selecting new org option
* Handle unanswered question
* Update error copy
* Create merge_request table and paths
* Save merging organisations wip
* Add update organisations
* Add merging organisations validation
* merge fixes
* Update schema to have merge_request_organisations
* Add relationships between merge request and organisations
* Update validations and saving organisations
* Add ability to remove merging orgs from the list
* Allow support users to create merge request for any organisation
* Update wording in organisations view
* Allow adding other merging organisations
* Add back button, update content
* Add validation if the organisation is not selected
* Fix path
* Use generic update method
* Update validations
* fix remove organisation
* remove reloads
* Update routes
* Authenticate scope
* PR review changes
* Add status, run validations unless the status is unsubmitted and save requesting organisation as a merging organisation as well
* PR comments
* Display continue button when there are at least 2 merging organisations
* write tests that invalid answers are cleared when the start date of a log is changed
* clear invalid answers when the start date of a log is changed
* correct error
* undo previous work to take new direction
* write tests to cover updated functionality of method in form to reset values when they are not routed to or when answers to radio questions are no longer valid
* update method in form to improve readability and to ensure that when the answers to radio questions are no longer valid, taht they are cleared
* add back an "end" deleted by mistake and fix a linting issue
* make some minor copy and variable name corrections
* fix a broken test after updates
* remove bulk upload tests for adding validations to unpermitted values to radio questions, these are now being cleared before validation
* fix some tests broken after rebasing
* add step to age questions in sales
* remove list of question ids from the form that should not be cleared, this information should be held on the questions themselves
* rename variables
* rename instance variable to be nice and long
* minor changes after rebase
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2279
# Changes
- split bulk upload feature toggle by log type ie lettings and sales
- this allows us to toggle with finer granularity
- moved `FeatureToggle` from initializer to service class. not sure why it was an initializer in the first place? this means its available to zeitwerk and will be reloaded when needed
* feat: assign simultaneously
* feat: assign simultaneously
* feat: revert
* feat: fix back persistence behaviour and update similar issues elsewhere, also remove redundant back text in govuk back links
* feat: add previous_page_redirect_paths to avoid infinite loops
* feat: leave here to avoid possible loops
* feat: leave here to avoid possible loops
* feat: add check answers page behaviour
* refactor: tweak
* feat: make tests pass, simplify code and incorporate tasklist/check answers back routing
* feat: update tests
* refactor: lint
* feat: update scheme status so incomplete unless has active locations
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: show completed/incomplete locations and validate when completed == 0
* feat: improve copy
* feat: change active -> completed
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: show completed/incomplete locations and validate when completed == 0
* feat: improve copy
* feat: change to confirmed to add clarification in code
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: show completed/incomplete locations and validate when completed == 0
* feat: improve copy
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: only confirm locations if complete (old) AND save button clicked (new)
* feat: add unconfirmed scope
* refactor: complete -> confirm
* feat: fix tests
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: show completed/incomplete locations and validate when completed == 0
* feat: improve copy
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: only confirm locations if complete (old) AND save button clicked (new)
* feat: add unconfirmed scope
* refactor: complete -> confirm
* feat: fix tests
* feat: fix more tests
* feat: fix more tests
* refactor: rubocop
* refactor: rubocop
* refactor: rubocop
* feat: add tests for incomplete schemes
* refactor: linting
* feat: test incomplete count too
* feat: test unconfirmed locations aren't options in location_id
* feat: test validation
* feat: test when scheme is confirmed
* feat: test when scheme is confirmed for support user
* feat: test when location is confirmed
* refactor: linting
* refactor: consistent use of factorybot or not within a file
* feat: confirm locations when complete even if save button not clicked, and update tests
* refactor: simplify
* refactor: simplify
* refactor: move location helper methods to helpers
* refactor: move scheme helper methods to helpers
* refactor: formatting
* refactor: po response
* refactor: review response
* Move and rename some tests
* Add a merge organisation link
* Add merge page
* Don't display a second nav bar for support
* Fix spacing
* fix spacing and typos
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2201
- Add copy to explain that the `username` field in new core accepts an email which differs from old core
# Changes
- Add copy to lettings and sales `Prepare your file` pages
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2184
- bulk upload error reports were not sorting by cells correctly
# Changes
- there was an issue where `Z100` would come after `AA100` which is not correct
- fix ordering issue above
- also made change so this is now delegated to the database rather than sorting in ruby
* Format errors on money amounts with 2 decimals
* CLDC-1975 Format money input and error messages
* format discounted_ownership_value
* Format more locales
* Update lettings_log tests
* Fix import service spec
* create method to test whether we are currently in the crossover period and associated tests
* update copy, use method for testing whether we are in the crossover period, remove focus from test file
* reuse existing method to determine whether we are in a collection period
* use the existing method and update validations
* fix test broken by changes
* update location in same way as scheme
* create method on FormHandler that finds the start date of the earliest collection period
* ensure that default deactivation and reactivation dates also reflect the changes
* create tests for the new validations
* lint correction
* minor copy change
* minor logic change
* amend naming error after rebase conflict
remove test that is no longer correct after change on another branch to functionality
update a test after a copy change
* show summary when on error threshold
* remove bulk upload mailer template
- this template is no longer used
* setup errors do not consider threshold
- return approprate intro test depending if there are setup errors or
not
* add first page for bulk upload resume journey
* bulk upload resume handles upload again
* add confirm page to bulk upload resume journey
* replace placeholder count with correct value
* apply recommendation for bulk upload resume choice
* add how to fix bulk upload mailer
* integrate new bulk upload approve journey
* add missing bulk upload error mappings
* remove test
* prevent approve being called multiple times
* bulk upload creates invisible logs ahead of time
* work invisible logs into bulk upload flow
* sort errors so deterministic
* remove unused ensure
* remove expected_log_count and processed
- these fields are no longer used or needed
* introduce pending status
* swap visible for pending logs
* only show visible lettings logs
* hard code status filters
* remove unused model methods
* only show visible sales logs
* form controller ignores hidden logs
* locations and schemes only affect visible logs
* Fix possible bulk upload years
* Initialise 2021/22 form routes for tests
* Return test routes from form handler
* Fix some tests
* fix bulk upload tests
- change from mocks to actual code calls
* update years for bulk upload form
* Fix some uprn tests
* Set the scheme as available from previous year if the collection is in crossover
* Update location available_from
* Update location and scheme tests
* Refactor and comment out a scheme test
* Some uprn and export tests
* Fix lettings logs controller tests and comment out previous collection message test
* fix tests for new collection year
* fix bulk upload tests
for some reason tenancylength must not be given for validation to pass
* update bulk upload tests for new year
* More fixes
* Undo location changes
* Fix remaining uprn
* further new collection year test fixes
---------
Co-authored-by: Phil Lee <asmega@users.noreply.github.com>
* feat: add inset text and old template (actual link to come)
* feat: add old spec work from CLDC-1788 to avoid future conflicts, typo fix
* feat: add new files and update copy
* feat: add 23/24 spec files and update copy
* feat: specify rack version
* feat: specify rack version
* refactor: lint
* refactor: lint
* feat: fix failing tests
* db: update
* refactor: typos
* feat: update designs and add inset text
* feat: pass year to guidance page
* feat: merge with new behaviour on main
* feat: add void date guidance partial
* feat: set guidance pos bottom for 22/23 log
* feat: add tests
* refactor: linting
* Update app/models/form/question.rb
Co-authored-by: Phil Lee <asmega@users.noreply.github.com>
* feat: smart quotes
* feat: respond to po comments
* feat: add voiddate/mrcdate validation to mrcdate too
* feat: add ten year validations to startdate too
* feat: fix tests
---------
Co-authored-by: Phil Lee <asmega@users.noreply.github.com>
* ensure the log review page has the correct links in the breadcrumbs
* ensure check your answers pages have the correct links in the breadcrumbs
* write feature tests to check that breadcrumbs have the correct links
* utilise rails url helpers
* when routing to a log show page, remove unnecessary :id call in url helper
* slight tweak of methods used to get name of an instance of the class
* tweak bulk upload email copy formatting
* change bulk upload summary error report
- this is now tabbed with the full report
- also summary report has now changed from single table to multiple
tables
* bulk upload cannot navigate to summary report
- as when sent to view full report the user does not have enough
errors that need summarising
* update seeds to add self in review env, change spec to reflect this, update config yml to allow csv exports in review
* update interface of relevant methods
EmailCsvJob, LettingsLog.to_csv and LettingsLogCsvService consume codes_only flag
* update tests including adding a new csv file to test against
* update LettingsLogCsvService to output codes only csv
* correct minor error and linting
* enable codes only download in UI
- add link on lettings log index page
- pass codes_only flag through params in relevant links and methods
- convert flag to boolean in controller methods
* ensure link displayed successfully for all renderings of logs_list and params passed through relevant methods in organisations controller
* fix existing tests
* correct linting thing
* correct linting error
* update tests for lettings log controller
* correct linting errors
* update organisations controller tests
* make minor changes after code review
* remove changes made for testing on review app
* make codes only download visible to support users only
* change variable names throughout after info on rauby/rails naming conventions, update tests for change in who can view codes only download link
* rework csv service for readability, remove delegating methods from lettings log to keep all code to do with mapping between our domain and desired export format in one place
* update test name
* correct a small typo and remove a duplicated method after clever git merge conflict suggestion
* point review app at staging csv bucket for csv download
* change variables named codes_only_export to codes_only to avoid inconsistency
* write tests to ensure that differetn user roles have the correct permissions around csv download
* ensure that non support users may not download codes only exports
* correct a small error in a previous commit
* correct minor linting error
* feat: add new guidance page with corresponding template links for sales and lettings log
* feat: add specification files and links for sales/lettings
* refactor: tidying
* feat: change pseudo csv to xlsx to reflect reality
* feat: add tests for guidance page routing and content
* feat: add tests for lettings too
* Display the correct hint text depending on joint purchase
* update validation for non joint purchase
* Add person 6 questions and refactor check_answers_card_title
* update hint text
* Reuse join purchase page
* bulk upload error summary min limit
- the errors summary table only show errors for column where there
at least 16 errors for that column
- if there are fewer than 16 errors that can see detailed report to pin
point and fix those specific issues
* add #errors? to summary component
* show full error report if not enough errors
- if errors summary table is empty as below threshold
- send users to full report
- remove the back button so they cannot access summary report
* add upload again cta otherwise a dead end
* change hardcoded date text to today's date formatted correctly
* update to show an example date in the tax year of the relevant log where that year is known
* move helper method and reuse some existing logic, adjust tests accordingly
* Update datails known numbering in the db
* Update check answers card headers
* Display same person pges for joint and non-joint purchase
* Only display person questions once
* Remove unused person class and methods
* Always display person known pages and update headers
* Fix depends on, show retirement validation for buyer 2
* test
* Add person 5 to test
* fix context description
* Display person known question for appropriate number of other household members
* tests
* update test
* Remove page id stubbing
* Remove redundant tests
* only put validation on the pages that are routed to
* Refactor tests
Phil Lee
Jack
Arthur Campbell
kosiakkatrina
natdeanlewissoftwire