* add placeholder tests for bulk upload mailer
* bulk upload fix setup errors email
- this plumbs in the condition so if any setup sections are incomplete we
send that partcular email and prevent the remaining flow
* tag bulk upload setup errors for downsteam use
* add category to bulk upload errors
* persist bulk upload error category
* populate bulk upload mailer with errors
* import LA if it cannot be inferred
* Assume mortgage is used if information about it is given
* Only import 22/23 logs
* remove comment
* Include the first day of collection window
* Only process sales logs in sales import job
* Only process lettings logs in lettings import job
* Add sales log import task to the full import
* change review app import bucket name
* Update mortgageused and joinmore
* Update jointmore and ownership scheme
* Fix ownershipsch so that it doesn't override to nil
* Set default relat2, update default household count
* het ownership from type if not given
* Improve logging
* Remove mortgageused method
* Look at Q16aProplensec2 column if Q16aProplen2 is empty
* Set default income used and pregblank
* Remove fields calculated internally
* Comment out sales import from full import task
* typo and change bucket name
This covers the following errors:
- Where the income is 0, set earnings and income to blank and set incref to refused
- Removing invalid tenancylength and tenancy values where tenancylength is invalid
- Removing prevten and age1 where incompatible
- We discovered earlier today (thanks Kat) that we have accidentally missed out ‘what are the monthly leasehold charges?’ in the ‘outright sales’ section of the 22/23 sales form on old CORE – the very last question on the form.
- We've decided that we will not add the question back in for 22/23. However we will add it back in for the upcoming 23/24 forms.
* Create test fixtures
* Add old id column
* Add sales logs importer
* Save and update completed discounted ownership log without postcode
* Update fixtures to be incomplete
* Import a completed shared ownership example
* Test with a non homebuy shared ownership example
* Add privacynotice
* Update hholdcount and confirmed fields
* Add buyer stilll serving mapping
* Add totadult/totchild and outright sale examples
* Update log fixtures
* Extract shared methods into logs import service and lint
* Add sales logs import rake task
* Update noint, xml examples and some mappings
* Add tests for checking that all required questions are answered
* Update tests, clean up import
* Map mortgage lender and mortgage lender other
* Infer Mscharge known as no for outright sale
* refactor setting default values
* when the armedforcesspouse is not answered set is as don't know
* Refactor tests: change log id names
* set savings to not known if not given
* Refactor tests: change nesting
* Backfill default household characteristics for completed log
* Add more default mapping
* Typo
* Improve logging and refactor tests
* Adjust test data to fit with the mappings that are known so far
* Rename fixture files
- We previously pushed logs into archives categorised by the quarter that they were created for.
- CDS requested that instead we push everything into a larger bucket seperated by FY.
* bulk upload considers housing needs fields
* bulk upload only permits one housing needs type
* add bulk upload validation
- no disabled needs cannot be selected in conjunction with a disabled need
* add bulk upload validation
- dont know disabled needs cannot be selected in conjunction with a disabled need
* add bulk upload validation
- no and don't know disabled access needs cannot be selected together
* bulk upload validate start date for given window
* use ActiveModel::Errors api correctly
- as opposed to calling methods manually
* refactor by combining lines
* bulk upload handles years with single digits
* feat: add 22/23 year validation
* feat: wip commit
* feat: update i18n
* feat: add one year gap exchange/completion validations
* tests: add tests for new validations
* test: add setup validations tests
* test: update factory to pass saledate validation
* feat: update seeds
* feat: update tests to reflect sales logs shouldn't exist in new service before 2022
* feat: merge with main, improve date behaviour
* refactor: cleanup
* feat: update tests
* feat: enforce saledate not after 22/23 collection year end
* feat: check date is valid
* feat: add hodate/saledate hard validation to saledate as well
* feat: add HandoverDate check to saledate question
* test: update
* db: update
* test: update
* feat: wip po updates
* feat: add soft_validation to setup
* feat: fix bug by making sale date soft validation optional
* feat: add tests
* test: update
* refactor: linting
* remove duplicate bulk validations
- validation was being performed both at CSV level and log level causing
a duplicate validation to appear
* bulk upload valiation errors now store the message
- previouly this was storing just the error type which we do not have a
mechanism to pipe these back to user readable error messages
* refactor bulk upload with extract method
* add bulk upload mailer test
* bulk upload mails users to fix logs on website
* bulk uplad email change resume link
* refactor local variable name
* CYA summary list tweaks for bulk upload
- When checking answers for a bulk upload related log the copy is
tweaked and made red as per designs
* different CYA treatment when for bulk upload
* bulk upload CYA missing answers always red
- whereas previosly this only happened if user was filtering logs via a
bulk upload
* create logs iff the log itself is valid
* do not create logs if a setup section not complete
* bulk upload with 60% errors will not create logs
* extract magic number to constant
* add bulk upload absolute threshold error rate
* refactor with extract method
* fix bulk upload age data types
* bulk upload handles both with and without headers
- headers are from the spreadsheet template
- otherwise assume cell A1 is start of the dataset
* add class to create logs from bulk upload
* create logs when processing bulk uploads
* remove bulk_upload_id from csv output
* create bulk upload logs only if all valid
- this will be changed later to allow for partial logs
- and only to create logs when a threshold has been met
* add method to blank invalid non setup fields
* bulk upload log creation blanks invalid fields
* fix incorrect logic for bulk upload renewal
* fix linting
* bulk upload log creation fail logs to sentry
* fix bulk upload line ending parsing
* extract bulk uploading csv parsing to class
* use csv parser in log creator
* change handle line endings mechanism
- we now strip all windows line endings for unix based line endings
- this normalises things making it simpler
* able to view lettings bulk upload errors
* fix linting
* call service correctly in test
* add bulk upload sales questions mapping
* appease linter
* bulk upload error shows correct question
- depending on log type it will show relevant question for the field
concerned
* improve namespacing of classes
* add job to process bulk uploads
* move validation from parser to model
* add validations for field_1
* add validation for field_4
* pending tests for field_4
* convert field_mapping to array of hashes
* validate nulls based on form question
* actually load forms when toggling between forms
* validate null for startdate
* row parser has access to bulk upload
* csv upload validates first form section
* add postcode validation
* Refactor error mappings for row parser
* Add unittype question
* Fix null error setting and add builtype
* add wchair to bulk upload
* Add beds to bulk upload
* Add joint to bulk upload
* Add startertenancy to the bulk upload
* Add tenancy for bulk upload
* Add declaration to the bulk upload
* Add age1 and age1_known to bulk upload
* add ages to bulk upload
* add sex1 to bulk upload
* add ethnic_group and ethnic to bulk upload
* add national to bulk upload
* add ecstat1 to bulk upload
* add military related fields to bulk upload
* add preg_occ to bulk upload
* add housingneeds to bulk upload
* add illness to bulk upload
* add layear to bulk upload
* add waityear to bulk upload
* add reason to bulk upload
* add prevten to bulk upload
* add homeless to bulk upload
* add previous postcode to bulk upload
* add reasonable preferences to bulk upload
* add allocations system to bulk upload
* add referral to bulk upload
* add net_income_known to bulk upload
* add hb to bulk upload
* add benefits to bulk upload
* add rent fields to bulk upload
* add hhmemb to bulk upload
* use 2022 csv fixtures for bulk upload
* fix renewal mapping for bulk upload
* placeholder test for bulk upload validation
* fix bulk upload mapping for homeless field
* fix leftreg mapping for bulk upload
* fix user associations in bulk upload tests
* add gender fields for bulk upload
* add ecstatN fields to bulk upload
* add #relatN fields to bulk upload
* extract old_visible_id in factory to trait
* map net_income_known correctly for bulk upload
* fix income bugs for bulk upload
* add unitletas to bulk upload
* add #rsnvac to bulk upload
* add #sheltered to bulk upload
* add illness fields to bulk upload
* add #irproduct_other to bulk upload
* infer renewal from rsnvac for bulk upload
* add #tenancyother to bulk upload
* add #tenancylength to bulk upload
* bulk upload earnings accepts pennies but rounds
* add #reasonother to bulk upload
* fix mapping of #ppcodenk for bulk upload
* add #household_charge to bulk upload
* add #chcharge to bulk upload
* add #tcharge to bulk upload
* add #supcharg to bulk upload
* add pscharge to bulk upload
* add #scharge to bulk upload
* use case statement for bulk upload allocation
* add offered to bulk upload
* add propcode to bulk upload
* add major repair fields to bulk upload
* add #voiddate to bulk upload
* support YY year format for bulk upload
* test postcode strips whitespace for bulk upload
* add #la to bulk upload
* add previous la to bulk upload
* fix failing test
* remove duplicate line from rebase
* add first time social housing to bulk upload
* make methods private
* fix field_4 validation for bulk upload
- the null check was inverted by mistake
Co-authored-by: Kat <katrina@kosiak.co.uk>
kosiakkatrina
Phil Lee
James Rose
natdeanlewissoftwire