# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2316
- Implement bulk upload sales for new collection year 2023
- This is a first pass implementation and will probably have some bugs in it and we can address over time
# Changes
- Add CSV parser for sales 2023 to handle CSV structure
- Tweak collection window validation so error now contextual to year selected for upload
- Handle arbitrary ordering of CSV columns
- Fix ordering of errors in report and now ordered by cell
- Added `Upload your file again` link styled as button on error report to match lettings experience
- Update tooling to convert logs to 2023 csv rows with support for random column ordering
# Known issues
- There seem to be some issues with how UPRN is handled if the UPRN cannot be validated.
- For the above I think there is dependency on https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/1570 which should clear any errored fields so users can continue to create logs and fix within the service
* create a method on the FormHandler that returns the sales form questions for all years in the order that they appear in the form
* update csv email job to accomodate sales log export as well as lettings
add to tests to reflec the changes made
* write tests to cover the desired functionality of the SalesLogCsvService
* create the SalesLogCsvService
create a necessary method on the log to enable submission method to be included on the csv
derive values for the two halves of previous postcode for export
* add relevant links in the UI and pipe everything together in controllers
amend organisations controller to have flexibility to download logs of either type
add necessary methods to sales log controller, raising shared method to logs controller
update routing for amendments and additions
extract helper method to build urls for downloading logs within an organisation
* correct various linter complaints and tech review suggestions
* minor amendment to add old_id and reorder early columns
* undo my 'clever' refactor that broke things
* refactoring of csv service after some tech review and some UI testing in review app
* update tests to include a test of a full export and all values in teh csv
* correct minor routing error to ensure correct url is shown and tab selected after requesting csv email
* update organisations controller requests spec file to cover new functionality and make a minor amendment to authentication scope in the controller after error found in testing
* write request tests for the new functionality in the sales log controller, define authorisation in the controller
* minor correction after rubocop's kind suggestion'
* various corrections from first pass at PO, tech review, linter, etc
* refactor :ordered_sales_questions_for_all_years
* first pass at implementing flexible code-based form fixtures for testing
* second pass
* refactor all tests of :ordered_sales_questions_for_all_years to use new factories
* some refactoring in the testing of the csv service
* use that fact that params is always available in controllers and don't pass it around, inline some methods calls
* correct minor bug to ensure that "Return to logs" link returns to the correct index page
* remove reminder comments
* write further tests on the manipulation of questions into the csv headers, update factories of form constituents to allow the creation of forms with richer questions
* fix linter complaints
* minor alterations after rebase to account for changes made on other branches
* refactor after code review
* tweak fixtures after rebase containing alterations to the factory defaults
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-1888
- This is a continuation of https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/1277
- When bulk uploading we want to check users are not uploading data that already exists to prevent them submitting duplicate
# Changes
- This feature is behind a feature toggle. it has been disabled for staging for testing purposes but available in all other environments
- If a log already exists based off certain fields add errors to the associated fields
- We discount any hidden logs and only check "active" logs
- Added memoization to `#valid?` as an optimisation
* write tests that invalid answers are cleared when the start date of a log is changed
* clear invalid answers when the start date of a log is changed
* correct error
* undo previous work to take new direction
* write tests to cover updated functionality of method in form to reset values when they are not routed to or when answers to radio questions are no longer valid
* update method in form to improve readability and to ensure that when the answers to radio questions are no longer valid, taht they are cleared
* add back an "end" deleted by mistake and fix a linting issue
* make some minor copy and variable name corrections
* fix a broken test after updates
* remove bulk upload tests for adding validations to unpermitted values to radio questions, these are now being cleared before validation
* fix some tests broken after rebasing
* add step to age questions in sales
* remove list of question ids from the form that should not be cleared, this information should be held on the questions themselves
* rename variables
* rename instance variable to be nice and long
* minor changes after rebase
* Fix possible bulk upload years
* Initialise 2021/22 form routes for tests
* Return test routes from form handler
* Fix some tests
* fix bulk upload tests
- change from mocks to actual code calls
* update years for bulk upload form
* Fix some uprn tests
* Set the scheme as available from previous year if the collection is in crossover
* Update location available_from
* Update location and scheme tests
* Refactor and comment out a scheme test
* Some uprn and export tests
* Fix lettings logs controller tests and comment out previous collection message test
* fix tests for new collection year
* fix bulk upload tests
for some reason tenancylength must not be given for validation to pass
* update bulk upload tests for new year
* More fixes
* Undo location changes
* Fix remaining uprn
* further new collection year test fixes
---------
Co-authored-by: Phil Lee <asmega@users.noreply.github.com>
* add some methods on lettings log and use them and already existing methods to make depends on human readbale for pages in property information subsection
* reformat answer options for neatness
* ensure that property information subsection is conditionally not shown under correct conditions
* amend method to find the path for the next question to answer so that if no questions in the next section are currently routed to, it goes to the section after that
* make amendments for failing test and linting
* remove unnecessary method
* remove unnecessary conditions from depends on, the displayed in tasklist method override is sufficient
* test the change to the method creating a link to the next subsection
* minor refactor to displayed_is_tasklist condition and changes in testing
* implement the fix on 22/23 form also
* write a test covering the fix
* removal not_renewal? method and tweak depends on for all pages using that method
* linty fluff begone
* correct some errors in tests after changes to factories
* minor testing amendments after rebase
* Update datails known numbering in the db
* Update check answers card headers
* Display same person pges for joint and non-joint purchase
* Only display person questions once
* Remove unused person class and methods
* Always display person known pages and update headers
* Fix depends on, show retirement validation for buyer 2
* test
* Add person 5 to test
* fix context description
* Display person known question for appropriate number of other household members
* tests
* update test
* Remove page id stubbing
* Remove redundant tests
* only put validation on the pages that are routed to
* Refactor tests
* feat: add 22/23 year validation
* feat: wip commit
* feat: update i18n
* feat: add one year gap exchange/completion validations
* tests: add tests for new validations
* test: add setup validations tests
* test: update factory to pass saledate validation
* feat: update seeds
* feat: update tests to reflect sales logs shouldn't exist in new service before 2022
* feat: merge with main, improve date behaviour
* refactor: cleanup
* feat: update tests
* feat: enforce saledate not after 22/23 collection year end
* Add min and max to age2 + questions
* validate that under 16s are economic children
* Update factory
* Update test
* Update ecstat, relationship and age validation
* Add postcode fields
* Add postcode page and questions
* Add postcode page to the property information subsection
* Only display local authority question if the la is not inferred
* Fix migration
* Move postcode validation
* Infer LA fields from postcode
* infer partial postcodes
* able to view lettings bulk upload errors
* fix linting
* call service correctly in test
* add bulk upload sales questions mapping
* appease linter
* bulk upload error shows correct question
- depending on log type it will show relevant question for the field
concerned
* improve namespacing of classes
* add job to process bulk uploads
* move validation from parser to model
* add validations for field_1
* add validation for field_4
* pending tests for field_4
* convert field_mapping to array of hashes
* validate nulls based on form question
* actually load forms when toggling between forms
* validate null for startdate
* row parser has access to bulk upload
* csv upload validates first form section
* add postcode validation
* Refactor error mappings for row parser
* Add unittype question
* Fix null error setting and add builtype
* add wchair to bulk upload
* Add beds to bulk upload
* Add joint to bulk upload
* Add startertenancy to the bulk upload
* Add tenancy for bulk upload
* Add declaration to the bulk upload
* Add age1 and age1_known to bulk upload
* add ages to bulk upload
* add sex1 to bulk upload
* add ethnic_group and ethnic to bulk upload
* add national to bulk upload
* add ecstat1 to bulk upload
* add military related fields to bulk upload
* add preg_occ to bulk upload
* add housingneeds to bulk upload
* add illness to bulk upload
* add layear to bulk upload
* add waityear to bulk upload
* add reason to bulk upload
* add prevten to bulk upload
* add homeless to bulk upload
* add previous postcode to bulk upload
* add reasonable preferences to bulk upload
* add allocations system to bulk upload
* add referral to bulk upload
* add net_income_known to bulk upload
* add hb to bulk upload
* add benefits to bulk upload
* add rent fields to bulk upload
* add hhmemb to bulk upload
* use 2022 csv fixtures for bulk upload
* fix renewal mapping for bulk upload
* placeholder test for bulk upload validation
* fix bulk upload mapping for homeless field
* fix leftreg mapping for bulk upload
* fix user associations in bulk upload tests
* add gender fields for bulk upload
* add ecstatN fields to bulk upload
* add #relatN fields to bulk upload
* extract old_visible_id in factory to trait
* map net_income_known correctly for bulk upload
* fix income bugs for bulk upload
* add unitletas to bulk upload
* add #rsnvac to bulk upload
* add #sheltered to bulk upload
* add illness fields to bulk upload
* add #irproduct_other to bulk upload
* infer renewal from rsnvac for bulk upload
* add #tenancyother to bulk upload
* add #tenancylength to bulk upload
* bulk upload earnings accepts pennies but rounds
* add #reasonother to bulk upload
* fix mapping of #ppcodenk for bulk upload
* add #household_charge to bulk upload
* add #chcharge to bulk upload
* add #tcharge to bulk upload
* add #supcharg to bulk upload
* add pscharge to bulk upload
* add #scharge to bulk upload
* use case statement for bulk upload allocation
* add offered to bulk upload
* add propcode to bulk upload
* add major repair fields to bulk upload
* add #voiddate to bulk upload
* support YY year format for bulk upload
* test postcode strips whitespace for bulk upload
* add #la to bulk upload
* add previous la to bulk upload
* fix failing test
* remove duplicate line from rebase
* add first time social housing to bulk upload
* make methods private
* fix field_4 validation for bulk upload
- the null check was inverted by mistake
Co-authored-by: Kat <katrina@kosiak.co.uk>
* lockdown bulk upload routes
* able to view lettings bulk upload errors
* add error count to bulk upload results
* coverage for bulk upload filename on results
* group bulk upload errors by row on results
* able to view bulk upload sales results
* scope lettings and sales bulk upload results
* fix linting
* call service correctly in test
* add bulk upload sales questions mapping
* appease linter
* bulk upload error shows correct question
- depending on log type it will show relevant question for the field
concerned
* use local disk for bulk upload for dev env
- this saves the need to connect to S3 to play with bulk upload in dev
environment
* improve namespacing of classes
* add job to process bulk uploads
* use local disk storage for dev file upload
* fix test
* use inline active job queue_adapter for dev
* use test active job queue adapter for test env
* remove rubocop violation
* delete bulk upload from disk after processing
* populate errors with cell, row + metadata
* update error message with something meaningful
* shim in sales validator
* able to parse sales bulk uploads
* change migration to add purchase_code to errors
* bulk upload error component renders purchaser code
- when a sales log
* populate purchaser_code for bulk upload errors
- when log type is sales
* remove superfluous private method
* feat: add gender identity questions for all 4 people, combine into single page/question
* feat: update schema and add tests
* tests: update tests
* feat: update to main
* feat: update db and test
* Mark subsection as completed if it is not displayed in the tasklist and hide it from the UI
* Add sale_information section
* add sales information subsections
* Cldc 1531 staircasing (#1109)
* Add staircase field to sales_logs table
* Add staircase question and page
* add staircase page to the shared ownership scheme subsection fix file name
* Cldc 1539 previous bedrooms (#1108)
* add frombeds field
* Add previous_bedrooms page and question
* add Previous Bedrooms page to the shared ownership subsection
* Cldc 1532 about staircasing (#1110)
* Add stairboughts and stairowned fields to the database
* Add about staircasing page and questions
* Add about staircasing page to the shared wnership scheme subsection
* Add a space before percent
* Cldc 1546 monthly rent (#1111)
* Add monthly rent column to sales logs table
* Add monthly rent question and page
* Add monthly rent page to the shared ownership scheme subsection
* Cldc 1535 exchange contracts (#1112)
* Add exdate to sales logs table
* Add exchange date page and question
* Add exchange contracts page to the shared ownership subsection
* derive exday, exmonth and exyear, tests and lint
* rebase tests
* Cldc 1538 la nominations (#1115)
* Add la nominations column to sales logs table
* Add La nominations page and questions
* Add la nominations to the shared ownership subsection
* Add accidentally removed files
* Cldc 1545 about the deposit (#1113)
* feat: add question(s) without depends_on behaviour
* feat: separate ids for diff sections
* test: update tests
* test: add tests
* tests: test tweaks
* refactor: linting
* Cldc 1538 la nominations (#1115)
* Add la nominations column to sales logs table
* Add La nominations page and questions
* Add la nominations to the shared ownership subsection
* Add accidentally removed files
* feat: add question(s) without depends_on behaviour
Co-authored-by: kosiakkatrina <54268893+kosiakkatrina@users.noreply.github.com>
* Cldc 1533 is resale (#1118)
* feat: add resale question and page
* tests: add new tests
* test: update previous tests
* refactor: linting
* refactor: linting
* Cldc 1576 buyer prp (#1117)
* Add soctenant field to sales logs
* Add buyer previous page and question
* Add buyer previous page to shared ownership scheme subsection
* rebase migrate
* rebase too
* Cldc 1540 about price (#1121)
* add price fields to the sales logs table
* Add about proce questions
* add about proce pages
* Add about price pages to sale information sections
* rebase changes
* happy new year
* Switch the order of sales log sections
* Cldc 1540 fixes (#1130)
* Update order of pages and remove wrong id's
* Add spacing to the hint text
* feat: add page, question, and update db
* Cldc 1521 living before purchase (#1129)
* feat: add new page and question and update db
* test: add and update tests
* feat: reset db
* feat: infer mscharge if mscharge_known == No
* test: add and update tests
* feat: add page, question, and update db
* feat: infer mscharge if mscharge_known == No
* test: add and update tests
* feat: update to main
* test: update tests
* tests: update tests
* faet: move mscharge derivation
Co-authored-by: Kat <katrina@kosiak.co.uk>
Co-authored-by: kosiakkatrina <54268893+kosiakkatrina@users.noreply.github.com>
* Add armed forces spouse column to the db
* Add armed forces spouse page and question
* Add armed forces spouse question to the subsection
* rebase test
* feat: wip postcode page
* feat: wip add new flow
* feat: add check answers page, continue updating flow
* feat: add back behaviour
* feat: more flow work
* feat: further flow work and flash notice, name tidying
* feat: add check_answers referrer linking
* feat: add controller and details linking and lint
* refactor: erblinting
* feat: add local_authority page (currenlty separate from edit_local_authority to avoid conflict with add location to newly created scheme path, to be fixed later). also copy and routing updates
* feat: add validations to controller (could later be added to model)
* fix: correct date order
* feat: validate on model (except startdate)
* feat: copy update
* refactor: railsification
* feat: add date model validation
* feat: add updated availability text in check_answers
* feat: more updated dynamic text and linking
* refactor: erblinting
* refactor: cleanup of redundant code
* refactor: remove redundant create
* feat: use rails route
* test: wip tests
* test: wip tests
* feat: add new back behaviour for new scheme path and tests wip
* feat: add more back behaviour for new scheme path and fix local auth validation
* feat: further linking behaviour with routes and referrers, don't add new location automatically to new schemes
* feat: if location with no details added, route to normal path rather than check answers
* test: update model and helper location tests
* test: more test updates
* test: more test updates and remove redundant files
* test: remove redundant tests, add postcode test
* test: add new #new tests
* test: add test for all new controller methods
* test: add more startdate tests
* refactor: erblinting
* test: fix failing tests
* test: update scheme test
* feat: respond to PR comments
* feat: make postcode clear only if changed
* feat: add _update methods
* refactor: simplify validation and location_edit_path method
* refactor: add helper method for action text
* refactor: remove redundant logic
* refactor: simplify routing
* refactor: reintroduce location_params
* refactor: use presence validation for postcode
* refactor: use presence validation for la
* feat: validate confirmed
* tests: make tests pass with new startdate validation
* WIP
* refactor: simplify display attributes helpers and update tests
* refactor: linting
* feat: design change to save behaviour
* test: fix failing tests
* feat: location code copy tweak
* refactor: remove redundant function
* refactor: remove add_another_location and new location page
* feat: simplify startdate validations to move to model, use scheme available_from date for in range validation
* test: fix failing tests
* refactor: linting
* feat: remove name validation as a result of PO review
* test: update tests
* refactor: remove redundant context
Co-authored-by: James Rose <james@jbpr.net>
* Mark subsection as completed if it is not displayed in the tasklist and hide it from the UI
* Add sale_information section
* add sales information subsections
* Cldc 1531 staircasing (#1109)
* Add staircase field to sales_logs table
* Add staircase question and page
* add staircase page to the shared ownership scheme subsection fix file name
* Cldc 1539 previous bedrooms (#1108)
* add frombeds field
* Add previous_bedrooms page and question
* add Previous Bedrooms page to the shared ownership subsection
* Cldc 1532 about staircasing (#1110)
* Add stairboughts and stairowned fields to the database
* Add about staircasing page and questions
* Add about staircasing page to the shared wnership scheme subsection
* Add a space before percent
* Cldc 1546 monthly rent (#1111)
* Add monthly rent column to sales logs table
* Add monthly rent question and page
* Add monthly rent page to the shared ownership scheme subsection
* Cldc 1535 exchange contracts (#1112)
* Add exdate to sales logs table
* Add exchange date page and question
* Add exchange contracts page to the shared ownership subsection
* derive exday, exmonth and exyear, tests and lint
* rebase tests
* Cldc 1538 la nominations (#1115)
* Add la nominations column to sales logs table
* Add La nominations page and questions
* Add la nominations to the shared ownership subsection
* Add accidentally removed files
* Cldc 1545 about the deposit (#1113)
* feat: add question(s) without depends_on behaviour
* feat: separate ids for diff sections
* test: update tests
* test: add tests
* tests: test tweaks
* refactor: linting
* Cldc 1538 la nominations (#1115)
* Add la nominations column to sales logs table
* Add La nominations page and questions
* Add la nominations to the shared ownership subsection
* Add accidentally removed files
* feat: add question(s) without depends_on behaviour
Co-authored-by: kosiakkatrina <54268893+kosiakkatrina@users.noreply.github.com>
* Cldc 1533 is resale (#1118)
* feat: add resale question and page
* tests: add new tests
* test: update previous tests
* refactor: linting
* refactor: linting
* Cldc 1576 buyer prp (#1117)
* Add soctenant field to sales logs
* Add buyer previous page and question
* Add buyer previous page to shared ownership scheme subsection
* rebase migrate
* rebase too
* Cldc 1540 about price (#1121)
* add price fields to the sales logs table
* Add about proce questions
* add about proce pages
* Add about price pages to sale information sections
* rebase changes
* happy new year
* Switch the order of sales log sections
* Cldc 1540 fixes (#1130)
* Update order of pages and remove wrong id's
* Add spacing to the hint text
* Cldc 1521 living before purchase (#1129)
* feat: add new page and question and update db
* test: add and update tests
* feat: reset db
* Only display from beds for social housing (#1131)
* Cldc 1544 mortgage amount (#1136)
* feat: add sales log seeds
* feat: add question and page and update db
* tests: add and update tests
* Cldc 1540 fixes part two (#1135)
* Change about_price_social_housing to about_price_shared_ownership and display it in all shared ownership cases
* Change about_price to purchase_price and display it for outright sale
* lint
* Move exchange contracts date page (#1143)
* feat: PO fix and tests (#1144)
* feat: PO fix and tests
* feat: simmplification
* refactor: linting
* feat: copy tweak
Co-authored-by: natdeanlewissoftwire <94526761+natdeanlewissoftwire@users.noreply.github.com>
Co-authored-by: natdeanlewissoftwire <nat.dean-lewis@softwire.com>
* Add person1 working situation question
* Add ecstat3 column to the sales log
* migration
* Add the rest of ecstat questions
* Update schema
* tests
* Reuse pages and questions
* Reuse the same page
* refactor indexes
* Add other ecstat columns
* Extract person methods to parent class
* Include shared validations in sales log model and run validate_numeric_min_max method
* Add non london hard max income validation
* Add london income validation
* Remove child vlidation because buyer cannot be a child
* Add income1_value_check column
* Add buyer_1_income_value_check question and page
* Update income1_under_soft_min?
* Add mortgage and value check column
* Add mortgage value check page
* Update mortgage_over_soft_max? in soft validateions
* Remove unused error message
* Display person 2, 3, 4 age question if their details are known
* Infer check answers value for age questions as 'Not known'
* Hide age known question in check answers if it is answered yes or no
* Add card numbers to household characteristics question
* Update gender identity depends on
* lint
* Add a field to completed sales log in factory
* Update person 1 gender identity to depend on the people count
* Do not show buyer 2 questions if it is not a joint purchase
Phil Lee
Arthur Campbell
kosiakkatrina
natdeanlewissoftwire
Jack