# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-1888
- This is a continuation of https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/1277
- When bulk uploading we want to check users are not uploading data that already exists to prevent them submitting duplicate
# Changes
- This feature is behind a feature toggle. it has been disabled for staging for testing purposes but available in all other environments
- If a log already exists based off certain fields add errors to the associated fields
- We discount any hidden logs and only check "active" logs
- Added memoization to `#valid?` as an optimisation
* feat: add validation for number of field labels
* feat: check field labels are numbers
* feat: remove validations for number of columns
* refactor: rename non_blank_fields_count to valid_field_numbers_count
* test: add functions to generate custom field labels/values
* test: that extra invalid field labels don't cause issues
* test: that removing a valid field label reduces count by 1
* test: that adding a valid field label increases count by 1
* refactor: rename validate_fields_count and wrong_field_count
* feat: add validation that max col count not exceeded when no headers
* fix: convert numbers to strings in default_field_numbers
* feat: add leniency to max cols count (in case of 1 extra col)
* test: explicitly set year of bulk upload to be 2022
* test: add/update tests in validator_spec
* chore: lint
* test: remove tests from csv_parser_spec that were moved to validator_spec
* feat: update 2022 csv_parser to work with new validations
* refactor: define number of valid 2022 fields in one place
* refactor: remove redundant headers definition
* feat: update 2022 csv parser to have col flexibility like 2023
* test: for validator 2022 as well as 2023
* test: simplify 2022/2023 logic and improve layout
* chore: lint
* test: ensure context descriptions start with "when"
* refactor: check fields/columns count within csv parser, not validator
* test: update 2022 csv parser tests to work like 2023
* chore: lint
* feat: add validation to check UPRN exists if address doesn't
* test: add tests for UPRN/address fields
* test: make UPRN<=12 chars test more specific
* feat: show qu header on BU error template if check_answer_label missing
* test: that validate_nulls gets header if check_answer_label is missing
* fix: drop "known" from "You must answer UPRN known"
* fix: update uprn validation logic
* test: make UPRN being missing explicit
* test: improve test descriptions
* test: that errors added to address fields too when uprn & address fields all null
* refactor: standardise check for presence of needs type
* test: attempt to stub setup question to have nil check_answer_label
* Revert "test: attempt to stub setup question to have nil check_answer_label"
This reverts commit f5b3f6179b.
* test: put test within context block for clarity
* test: fix typo in test description
* test: ensure one other non-blank field exists in row
* feat: output "this question" if qu has no header or check_answer_label
* chore: lint
* fix: add .presence to question.header and remove & before .downcase
* fix: add :after_log context to uprn validation
* rename variables in test file
* write tests for new validations
* validate that answers to numeric questions are given to the correct level of accuracy and in normal format with only digits
* add error messages for new validations
* add or alter step on numeric questions in sales, amend one test given step changes
* copy change
* remove validation that is specific to number format and fold that functionality into accuracy validation, rejig tests accordingly
* fix various tests
* remove reference to currency in validation message that concerns step only and doesn't check whether firled is currency related
* alter copy to avoid redundant repetition in validaiton message
* save changes made in rebase
* changes after PO review
two step values had been missed in previous work
in various places there were custom validations applied that duplicated the functionality of the generic min max validation that is applied to all numeric questions, in these situations sometimes the min and max on the question class was inconsistent with the limit that triggered validations
these have been corrected and made consistent
various tests were affected by this and have been amended accordingly
* remove tests introduced by mistake in rebase
* amend step value on earnings
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2235
- Support invalid options for bulk upload. one use case when user for a new collection year supplies a value only valid in the previous collection year
# Changes
- this validation works before `log.valid?` clears any fields
- as a result there is the potential to get 2 errors on a field for when it becomes blanked and invalid option occur together
- bulk upload validations are now split so that they run before or after `log.valid?`. this is due to the fact that `log.valid?` heavily mutates the `log` object. so we want to validate both before and after the data mutates depending on what needs to be checked
- errors must be duplicated and merged as calling `valid?` clears any existing errors on the object
- all validations are assigned a specific context otherwise they are added to the default context and will also be called when a context is given
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2279
# Changes
- split bulk upload feature toggle by log type ie lettings and sales
- this allows us to toggle with finer granularity
- moved `FeatureToggle` from initializer to service class. not sure why it was an initializer in the first place? this means its available to zeitwerk and will be reloaded when needed
* rename variables in test file
* write tests for new validations
* validate that answers to numeric questions are given to the correct level of accuracy and in normal format with only digits
* add error messages for new validations
* add or alter step on numeric questions in sales, amend one test given step changes
* copy change
* remove validation that is specific to number format and fold that functionality into accuracy validation, rejig tests accordingly
* fix various tests
* remove reference to currency in validation message that concerns step only and doesn't check whether firled is currency related
* alter copy to avoid redundant repetition in validaiton message
* save changes made in rebase
* changes after PO review
two step values had been missed in previous work
in various places there were custom validations applied that duplicated the functionality of the generic min max validation that is applied to all numeric questions, in these situations sometimes the min and max on the question class was inconsistent with the limit that triggered validations
these have been corrected and made consistent
various tests were affected by this and have been amended accordingly
* remove tests introduced by mistake in rebase
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2223
- Change how `renewal` works for bulk upload in 2023 compared to 2022
# Changes
- `renewal` is mandatory and must be filled in otherwise there is an error
- it is no longer inferred from another field and we just use the value from the user
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2267
- Consider errors on owning and managing org fields as setup errors
- Otherwise the user is not shown the summary report to highlight these as blocking issues to resolve
# Changes
- if there are problems with owning or managing org fields these are now considered as setup errors
* show summary when on error threshold
* remove bulk upload mailer template
- this template is no longer used
* setup errors do not consider threshold
- return approprate intro test depending if there are setup errors or
not
* add first page for bulk upload resume journey
* bulk upload resume handles upload again
* add confirm page to bulk upload resume journey
* replace placeholder count with correct value
* apply recommendation for bulk upload resume choice
* add how to fix bulk upload mailer
* integrate new bulk upload approve journey
* add missing bulk upload error mappings
* remove test
* prevent approve being called multiple times
* bulk upload creates invisible logs ahead of time
* work invisible logs into bulk upload flow
* sort errors so deterministic
* remove unused ensure
* remove expected_log_count and processed
- these fields are no longer used or needed
* introduce pending status
* swap visible for pending logs
* only show visible lettings logs
* hard code status filters
* remove unused model methods
* only show visible sales logs
* form controller ignores hidden logs
* locations and schemes only affect visible logs
* feat: add inset text and old template (actual link to come)
* feat: add old spec work from CLDC-1788 to avoid future conflicts, typo fix
* feat: add new files and update copy
* feat: add 23/24 spec files and update copy
* feat: specify rack version
* feat: specify rack version
* refactor: lint
* refactor: lint
* feat: fix failing tests
* db: update
* refactor: typos
* feat: update designs and add inset text
* feat: pass year to guidance page
* feat: merge with new behaviour on main
* Set housingneeds to yes if field 59 is 0
* Only display errors on selected housing needs
* Add errors to relevant fields when no housingneeds are selected with housingneeds options
* Add errors to correct fields when housingneeds don't know is selected
* Refactor validations
* Update 2023 housingneeds validations
* clear age2 if it's out of range
* Clear beds if over max
* Clear charges if there are carehome charges given
* Clear charges if scharge is under 0
* Clear tshortfall if it is not positive
* Clear referral if it's an invalid temporary accommodation
* Clear charges if pscharge is outside the range
* Clear charges if supcharg is outside the range
* Refactor clearing charges
* Clear charges is scharge is outside the range
* Clear location and scheme if location is inactive during the tenancy start date
* Remove care home charge if it's outside the range
* Refactor validation error information infor a hash
* Clear age out of range for all people
* Remove unused constant
* Update import logging messages
* Extract earnings validation and remove to_set from hash
* Remove to_delete
* Update logging
* Update bulk upload errors
* feat: add void date guidance partial
* feat: set guidance pos bottom for 22/23 log
* feat: add tests
* refactor: linting
* Update app/models/form/question.rb
Co-authored-by: Phil Lee <asmega@users.noreply.github.com>
* feat: smart quotes
* feat: respond to po comments
* feat: add voiddate/mrcdate validation to mrcdate too
* feat: add ten year validations to startdate too
* feat: fix tests
---------
Co-authored-by: Phil Lee <asmega@users.noreply.github.com>
* Clear import fields failing non_temp_accommodation validation
* Clear joint tenancy if there is only 1 person in the household
* Clear offered if the value is above the maximum of 20
* Display error message on ecstat
* Remove ecstat if the income is too high for it
* Update service charge validations
* Create location deactivation periods instead of setting a scheme end date
* Add test for deactivation period
* Update ranges in tests
* Remove scheme validation test
* Check that the location gets confirmed
---------
Co-authored-by: Kat <katrina@kosiak.co.uk>
* Remove failing equity over max on import
* Remove failing equity under min on import
* Use LA instead of postcode if postcode is in invalid format
* Do not backfill information about people indexed 7 and higher
* Unskip test
Phil Lee
SamSeed-Softwire
kosiakkatrina
Arthur Campbell
James Rose
natdeanlewissoftwire
Jack