* feat: validate nulls last in bu
* feat: add tests
* feat: update org errors and tests
* feat: update test copy
* feat: update tests
* CLDC-3328 Make bulk upload errors consistent between lettings and sales (#2341)
* feat: add validate_address_fields to lettings for 2024
* CLDC-3338: Add tolerance to discounted sale calculations (#2333)
* Rename method
* Update staircase/non staircase validations
* Add errors to type
* Remove validate_shared_ownership_deposit
* Don't add setup BU errors, deduplicate different sale type errors
* Add tolerance
* Reuse method
* Rename methods
* Skip type error completely in BU
* Update validation messages
* Update over tolerance method
* C:DC-3338: Add tolerance to grant calculations
---------
Co-authored-by: Kat <katrina@kosiak.co.uk>
* feat: add validate_address_fields to lettings for 2024
* feat: update tests
* Revert "CLDC-3338: Add tolerance to discounted sale calculations (#2333)"
This reverts commit cdecdfebfe.
---------
Co-authored-by: Robert Sullivan <Robert.Sullivan@Softwire.com>
Co-authored-by: Kat <katrina@kosiak.co.uk>
---------
Co-authored-by: Robert Sullivan <Robert.Sullivan@Softwire.com>
Co-authored-by: Kat <katrina@kosiak.co.uk>
* feat: mvp commit with address selector from address line 1 and postcode using OS places find endpoint
* feat: build out alternative routes through the address lookup flow
* refactor: lint
* refactor: remove commented lines
* feat: make nil safe
* feat: test address client
* feat: store address input separately
* feat: add migration
* feat: display inputs in answer label
* feat: update data presenter and test
* feat: revert precicions schema update
* feat: revert precision schema update
* feat: migrate new fields to sales logs
* feat: update shared log examples tests
* feat: use -1 for not listed address for extensibility
* feat: add service error to address_selection
* feat: handle case when no addresses are found
* feat: allow re-entering different uprns
* feat: improve error handling and don't accept "no match" precisions
* feat: add page tests
* feat: confirm uprn on address update
* feat: add question tests
* feat: don't set minimum match for initial search
* feat: mvp commit with address selector from address line 1 and postcode using OS places find endpoint
* feat: build out alternative routes through the address lookup flow
* refactor: lint
* refactor: remove commented lines
* feat: make nil safe
* feat: test address client
* feat: store address input separately
* feat: add migration
* feat: display inputs in answer label
* feat: update data presenter and test
* feat: revert precision schema update
* feat: migrate new fields to sales logs
* feat: update shared log examples tests
* feat: use -1 for not listed address for extensibility
* feat: add service error to address_selection
* feat: handle case when no addresses are found
* feat: allow re-entering different uprns
* feat: improve error handling and don't accept "no match" precisions
* feat: add page tests
* feat: confirm uprn on address update
* feat: add question tests
* feat: don't set minimum match for initial search
* feat: add no address found page
* feat: allow uprn known to be skipped, and set 0.4 min precision
* feat: accept best "good" match in bulk upload
* feat: update validations and test
* refactor: avoid stubbing using any_instance_of
* refactor: use change_table in migrations
* feat: add lettings behaviour to sales logs
* feat: update sales tests
* feat: xit test that is xited in lettings
* feat: stub os api in sales row parser
* feat: stub os api in sales row parser
* feat: add address line 1 to required attributes
* feat: update matching details and fix row parser validation
* refactor: improve readability
* feat: set uprns as option IDs to avoid multiple lookups
* feat: add sales uprn_selection
* feat: update schema
* feat: update tests
* feat: remove redundant methods
* feat: update error message and fix tests
* feat: update tests
* feat: update tests and error behaviour
* feat: update tests
* Empty commit
* CLDC-3224: Parse noint correctly in bulk upload row parsers
* CLDC-3224: Add task to fix previous noint data
* CLDC-3224: Remove nonsensical value for noint field in bu test
* CLDC-3224: Move where bulk upload noint fix status is set to ensure it matches processing version
* Put BU noint_fix_status setting before creating any logs
* feat: update header and test
* feat: update answers and hint for 24/25 and test
* feat: update tests
* refactor: lint
* feat: update buyer 2 as well
* refactor: lint
* refactor: lint
* Add new ethnic option to BU
---------
Co-authored-by: Kat <katrina@kosiak.co.uk>
* Add new nationality columns
* Add nationality questions to lettings
* Add nationality questions to sales
* Infer nationality where appropriate
* Add synonyms
* Update csv exports
* Export nationality_all for 24/25
* Update prefers not to say value
* Set 24/25 nationality fields from BU
* Undo the export (there's a separate ticket)
* Correctly display answers
* Rebase fixes, tests and lint
* Set nationality from lettings BU 24/25
* Rename files, add synonyms and accents
* Fix sales csvs
* Extract constant
* Copy 23 BU files to 24
* Renumber bulk upload fields for 2024
* Add prepare your file
* Update max columns
* Update fields in first_record_start_date
* Update managing org
* Rebase changes
* Blank commit for replication
* CLDC-3210: Check for R properly for sales BU savings field
* Sort refused check for income fields
* Add tests
* Allow any number of digits
* Empty commit
* feat: base lettings behaviour
* feat: formatting improvements
* Empty commit
* Empty commit
* Empty commit
* refactor: lint
* feat: more page styling
* feat: update tests
* refactor: lint
* feat: add requests spec
* refactor: simplify
* feat: add tests
* feat: copy change to reflect that the count in the header is not necessarily as large as the errors shown (it excludes duplicate errors on fields, and not_answered errors)
* feat: copy changes to sales as well
* feat: remove LegacyBulkUpload MVC
* feat: fix typo
* feat: update tests
* feat: mark additional not_answered sales q
* feat: update routing and only display warnign text when errors will be cleared
* feat: update copy and error rows
* refactor: lint
* feat: don't show soft validations in deletion report
* feat: update routing so deletion report is not shown once logs uploaded
* feat: update tests
* feat: make unique count track repeated errors across rows
* refactor: lint
* Correctly route to sales owning organisation page
* Display correct owning organisation options
* Set managing organisation as user organisation
* Add managing organisation question for sales
* Display managing organisation users in created_by
* set managing org to be same as owning org in BU
* Add managed by filter to sales
* Display reported by, correctly update owning org
* Update factory and 2022 BU
* refactor
* Merge managed sales logs
* test
* Update managing organisation for sales bulk upload
* Rebase tests
* skip a test as it is not relevant as the rake task would have already been ran
* PO fixes
* Remove old collection managed sales validation
* Update question header and filters conditions
* remove feature toggle
* feat: match email regardless of casing in bulk upload
* feat: allow null emails
* feat: set encoding by bom for file IO before parsing
* Revert "feat: set encoding by bom for file IO before parsing"
This reverts commit 03d2d17e7b.
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2305
- This is rework to add missing validations for sales setup fields for bulk upload
# Changes
- There is code that clears log fields if they are not a valid option. in order to generate errors for these we check certain fields for content prior to `log.valid?` being called. Unfortunately there does not appear to be an easy way to access to valid options therefore these values have been hard coded
- Privacy notice must be accepted other considered a setup error
- The `type` question can actually be one of three questions with the same identifier. these are excluded as part of `validate_valid_radio_option` and instead validated with a different mechanism. The rationale being that `log.valid?` will clear data before we get there
- Remove tests associated to upload threshold which are no longer required
* Do not allow 32 as an option for 22/23 sale type
* Correctly block log creation
* refactor test
* lint
* Add error to the correct field
* Add error as part of validating radio options
* Update row parser null check
* add a bespoke validation to the row parser
when buyer 1 is uploaded with working situation child, this should be validated with a custom message.
a test for this case was also created
* replicate the work from the last commit for the 2023 row parser and associated test file
* lint correction remove blank lines
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2316
- Implement bulk upload sales for new collection year 2023
- This is a first pass implementation and will probably have some bugs in it and we can address over time
# Changes
- Add CSV parser for sales 2023 to handle CSV structure
- Tweak collection window validation so error now contextual to year selected for upload
- Handle arbitrary ordering of CSV columns
- Fix ordering of errors in report and now ordered by cell
- Added `Upload your file again` link styled as button on error report to match lettings experience
- Update tooling to convert logs to 2023 csv rows with support for random column ordering
# Known issues
- There seem to be some issues with how UPRN is handled if the UPRN cannot be validated.
- For the above I think there is dependency on https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/1570 which should clear any errored fields so users can continue to create logs and fix within the service
natdeanlewissoftwire
kosiakkatrina
Robert Sullivan
Rachael Booth
Arthur Campbell
Phil Lee