# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2316
- Implement bulk upload sales for new collection year 2023
- This is a first pass implementation and will probably have some bugs in it and we can address over time
# Changes
- Add CSV parser for sales 2023 to handle CSV structure
- Tweak collection window validation so error now contextual to year selected for upload
- Handle arbitrary ordering of CSV columns
- Fix ordering of errors in report and now ordered by cell
- Added `Upload your file again` link styled as button on error report to match lettings experience
- Update tooling to convert logs to 2023 csv rows with support for random column ordering
# Known issues
- There seem to be some issues with how UPRN is handled if the UPRN cannot be validated.
- For the above I think there is dependency on https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/1570 which should clear any errored fields so users can continue to create logs and fix within the service
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2287
- bulk upload errors on setup fields are not being categorised as setup errors
# Changes
- ensure errors on setup field are categorised as `setup` errors
- ordering of validations tweaked, so `validate_nulls` is further down the execution order. this is so any existing validation run first and `validate_nulls` only adds errors if there aren't any existing errors on a field
- removed old 16/60% tests are no longer required due to introduction of `how to fix` journey
* Add affected_question_ids to pregnancy check
* Update is_referrer_interruption_screen? check and naming
* Use interruption_screen_question_ids to set soft validation errors on relevant fields
* Add soft validations to sales bulk upload
* Add soft validations to lettings logs 23/24 bulk upload
* Add errors for optional soft validations
* Only add soft validations once
* Import helper methods
* Update test based on new validation messages
* Rebase fix
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-1888
- This is a continuation of https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/1277
- When bulk uploading we want to check users are not uploading data that already exists to prevent them submitting duplicate
# Changes
- This feature is behind a feature toggle. it has been disabled for staging for testing purposes but available in all other environments
- If a log already exists based off certain fields add errors to the associated fields
- We discount any hidden logs and only check "active" logs
- Added memoization to `#valid?` as an optimisation
* feat: add validation for number of field labels
* feat: check field labels are numbers
* feat: remove validations for number of columns
* refactor: rename non_blank_fields_count to valid_field_numbers_count
* test: add functions to generate custom field labels/values
* test: that extra invalid field labels don't cause issues
* test: that removing a valid field label reduces count by 1
* test: that adding a valid field label increases count by 1
* refactor: rename validate_fields_count and wrong_field_count
* feat: add validation that max col count not exceeded when no headers
* fix: convert numbers to strings in default_field_numbers
* feat: add leniency to max cols count (in case of 1 extra col)
* test: explicitly set year of bulk upload to be 2022
* test: add/update tests in validator_spec
* chore: lint
* test: remove tests from csv_parser_spec that were moved to validator_spec
* feat: update 2022 csv_parser to work with new validations
* refactor: define number of valid 2022 fields in one place
* refactor: remove redundant headers definition
* feat: update 2022 csv parser to have col flexibility like 2023
* test: for validator 2022 as well as 2023
* test: simplify 2022/2023 logic and improve layout
* chore: lint
* test: ensure context descriptions start with "when"
* refactor: check fields/columns count within csv parser, not validator
* test: update 2022 csv parser tests to work like 2023
* chore: lint
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2235
- Support invalid options for bulk upload. one use case when user for a new collection year supplies a value only valid in the previous collection year
# Changes
- this validation works before `log.valid?` clears any fields
- as a result there is the potential to get 2 errors on a field for when it becomes blanked and invalid option occur together
- bulk upload validations are now split so that they run before or after `log.valid?`. this is due to the fact that `log.valid?` heavily mutates the `log` object. so we want to validate both before and after the data mutates depending on what needs to be checked
- errors must be duplicated and merged as calling `valid?` clears any existing errors on the object
- all validations are assigned a specific context otherwise they are added to the default context and will also be called when a context is given
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2267
- Consider errors on owning and managing org fields as setup errors
- Otherwise the user is not shown the summary report to highlight these as blocking issues to resolve
# Changes
- if there are problems with owning or managing org fields these are now considered as setup errors
* add first page for bulk upload resume journey
* bulk upload resume handles upload again
* add confirm page to bulk upload resume journey
* replace placeholder count with correct value
* apply recommendation for bulk upload resume choice
* add how to fix bulk upload mailer
* integrate new bulk upload approve journey
* add missing bulk upload error mappings
* remove test
* prevent approve being called multiple times
* bulk upload creates invisible logs ahead of time
* work invisible logs into bulk upload flow
* sort errors so deterministic
* remove unused ensure
* remove expected_log_count and processed
- these fields are no longer used or needed
* introduce pending status
* swap visible for pending logs
* only show visible lettings logs
* hard code status filters
* remove unused model methods
* only show visible sales logs
* form controller ignores hidden logs
* locations and schemes only affect visible logs
* feat: add inset text and old template (actual link to come)
* feat: add old spec work from CLDC-1788 to avoid future conflicts, typo fix
* feat: add new files and update copy
* feat: add 23/24 spec files and update copy
* feat: specify rack version
* feat: specify rack version
* refactor: lint
* refactor: lint
* feat: fix failing tests
* db: update
* refactor: typos
* feat: update designs and add inset text
* feat: pass year to guidance page
* feat: merge with new behaviour on main
* Set housingneeds to yes if field 59 is 0
* Only display errors on selected housing needs
* Add errors to relevant fields when no housingneeds are selected with housingneeds options
* Add errors to correct fields when housingneeds don't know is selected
* Refactor validations
* Update 2023 housingneeds validations
* clear age2 if it's out of range
* Clear beds if over max
* Clear charges if there are carehome charges given
* Clear charges if scharge is under 0
* Clear tshortfall if it is not positive
* Clear referral if it's an invalid temporary accommodation
* Clear charges if pscharge is outside the range
* Clear charges if supcharg is outside the range
* Refactor clearing charges
* Clear charges is scharge is outside the range
* Clear location and scheme if location is inactive during the tenancy start date
* Remove care home charge if it's outside the range
* Refactor validation error information infor a hash
* Clear age out of range for all people
* Remove unused constant
* Update import logging messages
* Extract earnings validation and remove to_set from hash
* Remove to_delete
* Update logging
* Update bulk upload errors
* feat: add void date guidance partial
* feat: set guidance pos bottom for 22/23 log
* feat: add tests
* refactor: linting
* Update app/models/form/question.rb
Co-authored-by: Phil Lee <asmega@users.noreply.github.com>
* feat: smart quotes
* feat: respond to po comments
* feat: add voiddate/mrcdate validation to mrcdate too
* feat: add ten year validations to startdate too
* feat: fix tests
---------
Co-authored-by: Phil Lee <asmega@users.noreply.github.com>
* refactor questions from validator to row parser
* able to switch between bulk upload parsers
- depending on what year we are processing
* spec tooling to support bulk upload multi year
* row parser now has year in namespacing
* add static data for 2023 row parser
* add placeholder to log to csv for specs
* bulk upload aribtrary 23/34 column ordering works
* bulk upload supports 23/24 without headers
* bulk upload 23/24 supports BOM + invalid chars
* dupe tests
* port 23/24 attributes_for_log
* port 23/24 bulk upload validations
* force crossover period
* tweak max permitted columns
* able to return column for given field
* work out column for field for errors
* add field_4 as 23/24 setup field
* remove duplicate method
* map schemes and locations correctly
* handle arbitrary number of header rows
* add missing fields to bulk upload support
* bulk upload setup errors only for missing data
* bulk upload setup error for scheme
* bulk upload setup error for location
* ensure missing startdate is a setup error
* limit setup errors if errors already present
* bulk upload beter handles age validations
- limit the number of errors on a field. if the field already has an
existing error it does not add further errors to the field
- shim in extra validation for ages which take precedent over existing
log level validations which are lacking
* setup errors always added
- no longer observes if the grouping already has errors
- as we have fine grain control in this class of how errors should be
* fix missing pass through for renewal
* add placeholder tests for bulk upload mailer
* bulk upload fix setup errors email
- this plumbs in the condition so if any setup sections are incomplete we
send that partcular email and prevent the remaining flow
* tag bulk upload setup errors for downsteam use
* add category to bulk upload errors
* persist bulk upload error category
* populate bulk upload mailer with errors
* bulk upload considers housing needs fields
* bulk upload only permits one housing needs type
* add bulk upload validation
- no disabled needs cannot be selected in conjunction with a disabled need
* add bulk upload validation
- dont know disabled needs cannot be selected in conjunction with a disabled need
* add bulk upload validation
- no and don't know disabled access needs cannot be selected together
natdeanlewissoftwire
Phil Lee
kosiakkatrina
SamSeed-Softwire