# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2316
- Implement bulk upload sales for new collection year 2023
- This is a first pass implementation and will probably have some bugs in it and we can address over time
# Changes
- Add CSV parser for sales 2023 to handle CSV structure
- Tweak collection window validation so error now contextual to year selected for upload
- Handle arbitrary ordering of CSV columns
- Fix ordering of errors in report and now ordered by cell
- Added `Upload your file again` link styled as button on error report to match lettings experience
- Update tooling to convert logs to 2023 csv rows with support for random column ordering
# Known issues
- There seem to be some issues with how UPRN is handled if the UPRN cannot be validated.
- For the above I think there is dependency on https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/1570 which should clear any errored fields so users can continue to create logs and fix within the service
* create a method on the FormHandler that returns the sales form questions for all years in the order that they appear in the form
* update csv email job to accomodate sales log export as well as lettings
add to tests to reflec the changes made
* write tests to cover the desired functionality of the SalesLogCsvService
* create the SalesLogCsvService
create a necessary method on the log to enable submission method to be included on the csv
derive values for the two halves of previous postcode for export
* add relevant links in the UI and pipe everything together in controllers
amend organisations controller to have flexibility to download logs of either type
add necessary methods to sales log controller, raising shared method to logs controller
update routing for amendments and additions
extract helper method to build urls for downloading logs within an organisation
* correct various linter complaints and tech review suggestions
* minor amendment to add old_id and reorder early columns
* undo my 'clever' refactor that broke things
* refactoring of csv service after some tech review and some UI testing in review app
* update tests to include a test of a full export and all values in teh csv
* correct minor routing error to ensure correct url is shown and tab selected after requesting csv email
* update organisations controller requests spec file to cover new functionality and make a minor amendment to authentication scope in the controller after error found in testing
* write request tests for the new functionality in the sales log controller, define authorisation in the controller
* minor correction after rubocop's kind suggestion'
* various corrections from first pass at PO, tech review, linter, etc
* refactor :ordered_sales_questions_for_all_years
* first pass at implementing flexible code-based form fixtures for testing
* second pass
* refactor all tests of :ordered_sales_questions_for_all_years to use new factories
* some refactoring in the testing of the csv service
* use that fact that params is always available in controllers and don't pass it around, inline some methods calls
* correct minor bug to ensure that "Return to logs" link returns to the correct index page
* remove reminder comments
* write further tests on the manipulation of questions into the csv headers, update factories of form constituents to allow the creation of forms with richer questions
* fix linter complaints
* minor alterations after rebase to account for changes made on other branches
* refactor after code review
* tweak fixtures after rebase containing alterations to the factory defaults
* write tests that invalid answers are cleared when the start date of a log is changed
* clear invalid answers when the start date of a log is changed
* correct error
* undo previous work to take new direction
* write tests to cover updated functionality of method in form to reset values when they are not routed to or when answers to radio questions are no longer valid
* update method in form to improve readability and to ensure that when the answers to radio questions are no longer valid, taht they are cleared
* add back an "end" deleted by mistake and fix a linting issue
* make some minor copy and variable name corrections
* fix a broken test after updates
* remove bulk upload tests for adding validations to unpermitted values to radio questions, these are now being cleared before validation
* fix some tests broken after rebasing
* add step to age questions in sales
* remove list of question ids from the form that should not be cleared, this information should be held on the questions themselves
* rename variables
* rename instance variable to be nice and long
* minor changes after rebase
* Update datails known numbering in the db
* Update check answers card headers
* Display same person pges for joint and non-joint purchase
* Only display person questions once
* Remove unused person class and methods
* Always display person known pages and update headers
* Fix depends on, show retirement validation for buyer 2
* test
* Add person 5 to test
* fix context description
* Display person known question for appropriate number of other household members
* tests
* update test
* Remove page id stubbing
* Remove redundant tests
* only put validation on the pages that are routed to
* Refactor tests
* feat: add 22/23 year validation
* feat: wip commit
* feat: update i18n
* feat: add one year gap exchange/completion validations
* tests: add tests for new validations
* test: add setup validations tests
* test: update factory to pass saledate validation
* feat: update seeds
* feat: update tests to reflect sales logs shouldn't exist in new service before 2022
* feat: merge with main, improve date behaviour
* refactor: cleanup
* feat: update tests
* feat: enforce saledate not after 22/23 collection year end
* Add min and max to age2 + questions
* validate that under 16s are economic children
* Update factory
* Update test
* Update ecstat, relationship and age validation
* Add postcode fields
* Add postcode page and questions
* Add postcode page to the property information subsection
* Only display local authority question if the la is not inferred
* Fix migration
* Move postcode validation
* Infer LA fields from postcode
* infer partial postcodes
* feat: add gender identity questions for all 4 people, combine into single page/question
* feat: update schema and add tests
* tests: update tests
* feat: update to main
* feat: update db and test
* Mark subsection as completed if it is not displayed in the tasklist and hide it from the UI
* Add sale_information section
* add sales information subsections
* Cldc 1531 staircasing (#1109)
* Add staircase field to sales_logs table
* Add staircase question and page
* add staircase page to the shared ownership scheme subsection fix file name
* Cldc 1539 previous bedrooms (#1108)
* add frombeds field
* Add previous_bedrooms page and question
* add Previous Bedrooms page to the shared ownership subsection
* Cldc 1532 about staircasing (#1110)
* Add stairboughts and stairowned fields to the database
* Add about staircasing page and questions
* Add about staircasing page to the shared wnership scheme subsection
* Add a space before percent
* Cldc 1546 monthly rent (#1111)
* Add monthly rent column to sales logs table
* Add monthly rent question and page
* Add monthly rent page to the shared ownership scheme subsection
* Cldc 1535 exchange contracts (#1112)
* Add exdate to sales logs table
* Add exchange date page and question
* Add exchange contracts page to the shared ownership subsection
* derive exday, exmonth and exyear, tests and lint
* rebase tests
* Cldc 1538 la nominations (#1115)
* Add la nominations column to sales logs table
* Add La nominations page and questions
* Add la nominations to the shared ownership subsection
* Add accidentally removed files
* Cldc 1545 about the deposit (#1113)
* feat: add question(s) without depends_on behaviour
* feat: separate ids for diff sections
* test: update tests
* test: add tests
* tests: test tweaks
* refactor: linting
* Cldc 1538 la nominations (#1115)
* Add la nominations column to sales logs table
* Add La nominations page and questions
* Add la nominations to the shared ownership subsection
* Add accidentally removed files
* feat: add question(s) without depends_on behaviour
Co-authored-by: kosiakkatrina <54268893+kosiakkatrina@users.noreply.github.com>
* Cldc 1533 is resale (#1118)
* feat: add resale question and page
* tests: add new tests
* test: update previous tests
* refactor: linting
* refactor: linting
* Cldc 1576 buyer prp (#1117)
* Add soctenant field to sales logs
* Add buyer previous page and question
* Add buyer previous page to shared ownership scheme subsection
* rebase migrate
* rebase too
* Cldc 1540 about price (#1121)
* add price fields to the sales logs table
* Add about proce questions
* add about proce pages
* Add about price pages to sale information sections
* rebase changes
* happy new year
* Switch the order of sales log sections
* Cldc 1540 fixes (#1130)
* Update order of pages and remove wrong id's
* Add spacing to the hint text
* feat: add page, question, and update db
* Cldc 1521 living before purchase (#1129)
* feat: add new page and question and update db
* test: add and update tests
* feat: reset db
* feat: infer mscharge if mscharge_known == No
* test: add and update tests
* feat: add page, question, and update db
* feat: infer mscharge if mscharge_known == No
* test: add and update tests
* feat: update to main
* test: update tests
* tests: update tests
* faet: move mscharge derivation
Co-authored-by: Kat <katrina@kosiak.co.uk>
Co-authored-by: kosiakkatrina <54268893+kosiakkatrina@users.noreply.github.com>
* Add armed forces spouse column to the db
* Add armed forces spouse page and question
* Add armed forces spouse question to the subsection
* rebase test
* Mark subsection as completed if it is not displayed in the tasklist and hide it from the UI
* Add sale_information section
* add sales information subsections
* Cldc 1531 staircasing (#1109)
* Add staircase field to sales_logs table
* Add staircase question and page
* add staircase page to the shared ownership scheme subsection fix file name
* Cldc 1539 previous bedrooms (#1108)
* add frombeds field
* Add previous_bedrooms page and question
* add Previous Bedrooms page to the shared ownership subsection
* Cldc 1532 about staircasing (#1110)
* Add stairboughts and stairowned fields to the database
* Add about staircasing page and questions
* Add about staircasing page to the shared wnership scheme subsection
* Add a space before percent
* Cldc 1546 monthly rent (#1111)
* Add monthly rent column to sales logs table
* Add monthly rent question and page
* Add monthly rent page to the shared ownership scheme subsection
* Cldc 1535 exchange contracts (#1112)
* Add exdate to sales logs table
* Add exchange date page and question
* Add exchange contracts page to the shared ownership subsection
* derive exday, exmonth and exyear, tests and lint
* rebase tests
* Cldc 1538 la nominations (#1115)
* Add la nominations column to sales logs table
* Add La nominations page and questions
* Add la nominations to the shared ownership subsection
* Add accidentally removed files
* Cldc 1545 about the deposit (#1113)
* feat: add question(s) without depends_on behaviour
* feat: separate ids for diff sections
* test: update tests
* test: add tests
* tests: test tweaks
* refactor: linting
* Cldc 1538 la nominations (#1115)
* Add la nominations column to sales logs table
* Add La nominations page and questions
* Add la nominations to the shared ownership subsection
* Add accidentally removed files
* feat: add question(s) without depends_on behaviour
Co-authored-by: kosiakkatrina <54268893+kosiakkatrina@users.noreply.github.com>
* Cldc 1533 is resale (#1118)
* feat: add resale question and page
* tests: add new tests
* test: update previous tests
* refactor: linting
* refactor: linting
* Cldc 1576 buyer prp (#1117)
* Add soctenant field to sales logs
* Add buyer previous page and question
* Add buyer previous page to shared ownership scheme subsection
* rebase migrate
* rebase too
* Cldc 1540 about price (#1121)
* add price fields to the sales logs table
* Add about proce questions
* add about proce pages
* Add about price pages to sale information sections
* rebase changes
* happy new year
* Switch the order of sales log sections
* Cldc 1540 fixes (#1130)
* Update order of pages and remove wrong id's
* Add spacing to the hint text
* Cldc 1521 living before purchase (#1129)
* feat: add new page and question and update db
* test: add and update tests
* feat: reset db
* Only display from beds for social housing (#1131)
* Cldc 1544 mortgage amount (#1136)
* feat: add sales log seeds
* feat: add question and page and update db
* tests: add and update tests
* Cldc 1540 fixes part two (#1135)
* Change about_price_social_housing to about_price_shared_ownership and display it in all shared ownership cases
* Change about_price to purchase_price and display it for outright sale
* lint
* Move exchange contracts date page (#1143)
* feat: PO fix and tests (#1144)
* feat: PO fix and tests
* feat: simmplification
* refactor: linting
* feat: copy tweak
Co-authored-by: natdeanlewissoftwire <94526761+natdeanlewissoftwire@users.noreply.github.com>
Co-authored-by: natdeanlewissoftwire <nat.dean-lewis@softwire.com>
* Add person1 working situation question
* Add ecstat3 column to the sales log
* migration
* Add the rest of ecstat questions
* Update schema
* tests
* Reuse pages and questions
* Reuse the same page
* refactor indexes
* Add other ecstat columns
* Extract person methods to parent class
* Include shared validations in sales log model and run validate_numeric_min_max method
* Add non london hard max income validation
* Add london income validation
* Remove child vlidation because buyer cannot be a child
* Add income1_value_check column
* Add buyer_1_income_value_check question and page
* Update income1_under_soft_min?
* Add mortgage and value check column
* Add mortgage value check page
* Update mortgage_over_soft_max? in soft validateions
* Remove unused error message
* Display person 2, 3, 4 age question if their details are known
* Infer check answers value for age questions as 'Not known'
* Hide age known question in check answers if it is answered yes or no
* Add card numbers to household characteristics question
* Update gender identity depends on
* lint
* Add a field to completed sales log in factory
* Update person 1 gender identity to depend on the people count
* Do not show buyer 2 questions if it is not a joint purchase
Phil Lee
Arthur Campbell
kosiakkatrina
Jack
natdeanlewissoftwire
David May-Miller
Sam