* Add missing addresses csv job
* Update missing addresses csv service methods
* Add rake task
* Update the job to send missing town or city templates
* Update service to create mising town or city templates
* Add send missing town or city csv rake task
* Add IDs to the CSVs
* Put all log in the same csv
* Add issue type column
* Write wrong uprn logs to csv
* Add mailer methods
* Skip uprn issue for specified orgs
* Add sales csv rake task
* set SKIP_UPRN_ISSUE_ORG_IDS on review apps
* test
* Update notify template IDs for testing
* Initialize service with organisation instead of a hash
* Add expiration time to url
* Add optional tags and remove LA from csv
* Extract log to csv methods
* Update casing
* Update old IDs in factories
* Move constant
* Extract some repeating scopes
* Pass in organisations to skip instead of using an env var
* update template id for sales
* Update link expiry time and headers
* Lower the threshold for testing
* Add issue explanation to the email
* Add how to fix
* update emails
* CLDC-2810 Create all addresses CSV (#1953)
* Add rake tasks for creating all addresses CSV
* Write headers if logs don't exist, update header names
* Rename method
* CLDC-2810 Correct addresses from csv (#1957)
* Updating importing lettings addresses form csv
* Add import_sales_addresses_from_csv rake
* Allow correcting addresses from both templates
* escape .
* Reinfer LA if the postcode hasn't changed
* Update labels and email content
* Update missing addresses threshold
* Remove unused env var
* create helper module to find all logs for a given user that have duplicates
write associated tests
update methods on user model to retrieve logs related to a given user
create a scope on logs to retrieve logs created by a given user
* check for duplicates and display notification banner on the index page where appropriate
tests to cover the desired behaviour
* alter duplicate logs helper to return duplicates in a format that can be passed through the params
update associated tests
* create user journey to view duplicate sets adn navigate to a given set
create a new route and add it to the link from the logs index page
create controller method and related view file
* update path for duplicate logs in routes, update factories, refactor duplicates scope to eliminate raw sql
* write tests for the duplicate logs index page
* correct linting errors
* minor amendments from code review.
* temp
* Refactor
* Create a scope for duplicate groups
* fixes
* lint
* Update controller specs
* Put duplicate sets count behind a feature flag
* refactor index
* Update duplicate_sets_count
* Update scopes
* tests
* pass in original log id
* tests
* Add duplicate index page content
---------
Co-authored-by: Kat <katrina@kosiak.co.uk>
* CLDC-2494: WIP
* CLDC-2494: wip
* CLDC-2494: page work in progress
* cleanup
* Add a path for duplicate logs
* Display all duplicate logs
* Move a test
* Display duplicate check answers for logs
* Add buttons to delete duplicates
* Add a route for sales logs duplicates
* Update duplicated page to work for sales logs
* lint
* Add auth
* Rebase updates
* Update displayed questions
* Add delete duplicates page
* Update page content for when there are multiple duplicates
* Add auth to the path
* Update delete button to use delete controller
* Update page for sales logs
* Update styling
* Update success banner for deleting duplicates
* Typo
* Render not found if there are no duplicate logs
* rebase changes
* Rebase changes
* Update back and cancel links
* Update the duplicate journey after deleting logs
* Update change button routing
* Refactor tests
* Add content for no duplicates
* Update params
* Refactor tests and paths
* Update params to include original_log_id from the beginning
* Rename file
* lint
* Add full stop after deletion messages
* Add caption and update duplicate log IDs in the banner
* lint
* Styling
---------
Co-authored-by: Aaron Spencer <aaron.spencer@softwire.com>
* feat: setup to replicate failures to fix
* feat: wip test fixes
* feat: remainder of current state test fixes
* feat: set form date to past to trigger errors to fix
* feat: revert, don't want this in final diff
* feat: fix row_parser tests
* feat: sales log importer and validator fixes
* feat: remainder of test fixes
* refactor: lint
* call form to reset invalid values when saving a sales log
* fix tests failing after updating sales log functionality to reset invalid answers
* add a test to cover the new functionality added
* a further test
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2316
- Implement bulk upload sales for new collection year 2023
- This is a first pass implementation and will probably have some bugs in it and we can address over time
# Changes
- Add CSV parser for sales 2023 to handle CSV structure
- Tweak collection window validation so error now contextual to year selected for upload
- Handle arbitrary ordering of CSV columns
- Fix ordering of errors in report and now ordered by cell
- Added `Upload your file again` link styled as button on error report to match lettings experience
- Update tooling to convert logs to 2023 csv rows with support for random column ordering
# Known issues
- There seem to be some issues with how UPRN is handled if the UPRN cannot be validated.
- For the above I think there is dependency on https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/1570 which should clear any errored fields so users can continue to create logs and fix within the service
* create a method on the FormHandler that returns the sales form questions for all years in the order that they appear in the form
* update csv email job to accomodate sales log export as well as lettings
add to tests to reflec the changes made
* write tests to cover the desired functionality of the SalesLogCsvService
* create the SalesLogCsvService
create a necessary method on the log to enable submission method to be included on the csv
derive values for the two halves of previous postcode for export
* add relevant links in the UI and pipe everything together in controllers
amend organisations controller to have flexibility to download logs of either type
add necessary methods to sales log controller, raising shared method to logs controller
update routing for amendments and additions
extract helper method to build urls for downloading logs within an organisation
* correct various linter complaints and tech review suggestions
* minor amendment to add old_id and reorder early columns
* undo my 'clever' refactor that broke things
* refactoring of csv service after some tech review and some UI testing in review app
* update tests to include a test of a full export and all values in teh csv
* correct minor routing error to ensure correct url is shown and tab selected after requesting csv email
* update organisations controller requests spec file to cover new functionality and make a minor amendment to authentication scope in the controller after error found in testing
* write request tests for the new functionality in the sales log controller, define authorisation in the controller
* minor correction after rubocop's kind suggestion'
* various corrections from first pass at PO, tech review, linter, etc
* refactor :ordered_sales_questions_for_all_years
* first pass at implementing flexible code-based form fixtures for testing
* second pass
* refactor all tests of :ordered_sales_questions_for_all_years to use new factories
* some refactoring in the testing of the csv service
* use that fact that params is always available in controllers and don't pass it around, inline some methods calls
* correct minor bug to ensure that "Return to logs" link returns to the correct index page
* remove reminder comments
* write further tests on the manipulation of questions into the csv headers, update factories of form constituents to allow the creation of forms with richer questions
* fix linter complaints
* minor alterations after rebase to account for changes made on other branches
* refactor after code review
* tweak fixtures after rebase containing alterations to the factory defaults
* write tests that invalid answers are cleared when the start date of a log is changed
* clear invalid answers when the start date of a log is changed
* correct error
* undo previous work to take new direction
* write tests to cover updated functionality of method in form to reset values when they are not routed to or when answers to radio questions are no longer valid
* update method in form to improve readability and to ensure that when the answers to radio questions are no longer valid, taht they are cleared
* add back an "end" deleted by mistake and fix a linting issue
* make some minor copy and variable name corrections
* fix a broken test after updates
* remove bulk upload tests for adding validations to unpermitted values to radio questions, these are now being cleared before validation
* fix some tests broken after rebasing
* add step to age questions in sales
* remove list of question ids from the form that should not be cleared, this information should be held on the questions themselves
* rename variables
* rename instance variable to be nice and long
* minor changes after rebase
* Update datails known numbering in the db
* Update check answers card headers
* Display same person pges for joint and non-joint purchase
* Only display person questions once
* Remove unused person class and methods
* Always display person known pages and update headers
* Fix depends on, show retirement validation for buyer 2
* test
* Add person 5 to test
* fix context description
* Display person known question for appropriate number of other household members
* tests
* update test
* Remove page id stubbing
* Remove redundant tests
* only put validation on the pages that are routed to
* Refactor tests
* feat: add 22/23 year validation
* feat: wip commit
* feat: update i18n
* feat: add one year gap exchange/completion validations
* tests: add tests for new validations
* test: add setup validations tests
* test: update factory to pass saledate validation
* feat: update seeds
* feat: update tests to reflect sales logs shouldn't exist in new service before 2022
* feat: merge with main, improve date behaviour
* refactor: cleanup
* feat: update tests
* feat: enforce saledate not after 22/23 collection year end
* Add min and max to age2 + questions
* validate that under 16s are economic children
* Update factory
* Update test
* Update ecstat, relationship and age validation
* Add postcode fields
* Add postcode page and questions
* Add postcode page to the property information subsection
* Only display local authority question if the la is not inferred
* Fix migration
* Move postcode validation
* Infer LA fields from postcode
* infer partial postcodes
* feat: add gender identity questions for all 4 people, combine into single page/question
* feat: update schema and add tests
* tests: update tests
* feat: update to main
* feat: update db and test
* Mark subsection as completed if it is not displayed in the tasklist and hide it from the UI
* Add sale_information section
* add sales information subsections
* Cldc 1531 staircasing (#1109)
* Add staircase field to sales_logs table
* Add staircase question and page
* add staircase page to the shared ownership scheme subsection fix file name
* Cldc 1539 previous bedrooms (#1108)
* add frombeds field
* Add previous_bedrooms page and question
* add Previous Bedrooms page to the shared ownership subsection
* Cldc 1532 about staircasing (#1110)
* Add stairboughts and stairowned fields to the database
* Add about staircasing page and questions
* Add about staircasing page to the shared wnership scheme subsection
* Add a space before percent
* Cldc 1546 monthly rent (#1111)
* Add monthly rent column to sales logs table
* Add monthly rent question and page
* Add monthly rent page to the shared ownership scheme subsection
* Cldc 1535 exchange contracts (#1112)
* Add exdate to sales logs table
* Add exchange date page and question
* Add exchange contracts page to the shared ownership subsection
* derive exday, exmonth and exyear, tests and lint
* rebase tests
* Cldc 1538 la nominations (#1115)
* Add la nominations column to sales logs table
* Add La nominations page and questions
* Add la nominations to the shared ownership subsection
* Add accidentally removed files
* Cldc 1545 about the deposit (#1113)
* feat: add question(s) without depends_on behaviour
* feat: separate ids for diff sections
* test: update tests
* test: add tests
* tests: test tweaks
* refactor: linting
* Cldc 1538 la nominations (#1115)
* Add la nominations column to sales logs table
* Add La nominations page and questions
* Add la nominations to the shared ownership subsection
* Add accidentally removed files
* feat: add question(s) without depends_on behaviour
Co-authored-by: kosiakkatrina <54268893+kosiakkatrina@users.noreply.github.com>
* Cldc 1533 is resale (#1118)
* feat: add resale question and page
* tests: add new tests
* test: update previous tests
* refactor: linting
* refactor: linting
* Cldc 1576 buyer prp (#1117)
* Add soctenant field to sales logs
* Add buyer previous page and question
* Add buyer previous page to shared ownership scheme subsection
* rebase migrate
* rebase too
* Cldc 1540 about price (#1121)
* add price fields to the sales logs table
* Add about proce questions
* add about proce pages
* Add about price pages to sale information sections
* rebase changes
* happy new year
* Switch the order of sales log sections
* Cldc 1540 fixes (#1130)
* Update order of pages and remove wrong id's
* Add spacing to the hint text
* feat: add page, question, and update db
* Cldc 1521 living before purchase (#1129)
* feat: add new page and question and update db
* test: add and update tests
* feat: reset db
* feat: infer mscharge if mscharge_known == No
* test: add and update tests
* feat: add page, question, and update db
* feat: infer mscharge if mscharge_known == No
* test: add and update tests
* feat: update to main
* test: update tests
* tests: update tests
* faet: move mscharge derivation
Co-authored-by: Kat <katrina@kosiak.co.uk>
Co-authored-by: kosiakkatrina <54268893+kosiakkatrina@users.noreply.github.com>
* Add armed forces spouse column to the db
* Add armed forces spouse page and question
* Add armed forces spouse question to the subsection
* rebase test
* Mark subsection as completed if it is not displayed in the tasklist and hide it from the UI
* Add sale_information section
* add sales information subsections
* Cldc 1531 staircasing (#1109)
* Add staircase field to sales_logs table
* Add staircase question and page
* add staircase page to the shared ownership scheme subsection fix file name
* Cldc 1539 previous bedrooms (#1108)
* add frombeds field
* Add previous_bedrooms page and question
* add Previous Bedrooms page to the shared ownership subsection
* Cldc 1532 about staircasing (#1110)
* Add stairboughts and stairowned fields to the database
* Add about staircasing page and questions
* Add about staircasing page to the shared wnership scheme subsection
* Add a space before percent
* Cldc 1546 monthly rent (#1111)
* Add monthly rent column to sales logs table
* Add monthly rent question and page
* Add monthly rent page to the shared ownership scheme subsection
* Cldc 1535 exchange contracts (#1112)
* Add exdate to sales logs table
* Add exchange date page and question
* Add exchange contracts page to the shared ownership subsection
* derive exday, exmonth and exyear, tests and lint
* rebase tests
* Cldc 1538 la nominations (#1115)
* Add la nominations column to sales logs table
* Add La nominations page and questions
* Add la nominations to the shared ownership subsection
* Add accidentally removed files
* Cldc 1545 about the deposit (#1113)
* feat: add question(s) without depends_on behaviour
* feat: separate ids for diff sections
* test: update tests
* test: add tests
* tests: test tweaks
* refactor: linting
* Cldc 1538 la nominations (#1115)
* Add la nominations column to sales logs table
* Add La nominations page and questions
* Add la nominations to the shared ownership subsection
* Add accidentally removed files
* feat: add question(s) without depends_on behaviour
Co-authored-by: kosiakkatrina <54268893+kosiakkatrina@users.noreply.github.com>
* Cldc 1533 is resale (#1118)
* feat: add resale question and page
* tests: add new tests
* test: update previous tests
* refactor: linting
* refactor: linting
* Cldc 1576 buyer prp (#1117)
* Add soctenant field to sales logs
* Add buyer previous page and question
* Add buyer previous page to shared ownership scheme subsection
* rebase migrate
* rebase too
* Cldc 1540 about price (#1121)
* add price fields to the sales logs table
* Add about proce questions
* add about proce pages
* Add about price pages to sale information sections
* rebase changes
* happy new year
* Switch the order of sales log sections
* Cldc 1540 fixes (#1130)
* Update order of pages and remove wrong id's
* Add spacing to the hint text
* Cldc 1521 living before purchase (#1129)
* feat: add new page and question and update db
* test: add and update tests
* feat: reset db
* Only display from beds for social housing (#1131)
* Cldc 1544 mortgage amount (#1136)
* feat: add sales log seeds
* feat: add question and page and update db
* tests: add and update tests
* Cldc 1540 fixes part two (#1135)
* Change about_price_social_housing to about_price_shared_ownership and display it in all shared ownership cases
* Change about_price to purchase_price and display it for outright sale
* lint
* Move exchange contracts date page (#1143)
* feat: PO fix and tests (#1144)
* feat: PO fix and tests
* feat: simmplification
* refactor: linting
* feat: copy tweak
Co-authored-by: natdeanlewissoftwire <94526761+natdeanlewissoftwire@users.noreply.github.com>
Co-authored-by: natdeanlewissoftwire <nat.dean-lewis@softwire.com>
* Add person1 working situation question
* Add ecstat3 column to the sales log
* migration
* Add the rest of ecstat questions
* Update schema
* tests
* Reuse pages and questions
* Reuse the same page
* refactor indexes
* Add other ecstat columns
* Extract person methods to parent class
* Include shared validations in sales log model and run validate_numeric_min_max method
* Add non london hard max income validation
* Add london income validation
* Remove child vlidation because buyer cannot be a child
* Add income1_value_check column
* Add buyer_1_income_value_check question and page
* Update income1_under_soft_min?
* Add mortgage and value check column
* Add mortgage value check page
* Update mortgage_over_soft_max? in soft validateions
* Remove unused error message
* Display person 2, 3, 4 age question if their details are known
* Infer check answers value for age questions as 'Not known'
* Hide age known question in check answers if it is answered yes or no
* Add card numbers to household characteristics question
* Update gender identity depends on
* lint
* Add a field to completed sales log in factory
* Update person 1 gender identity to depend on the people count
* Do not show buyer 2 questions if it is not a joint purchase
kosiakkatrina
Arthur Campbell
natdeanlewissoftwire
Phil Lee
Jack
David May-Miller