* Set and clear duplicate log references
* Export duplicate_log_reference_id for lettings
* Export duplicate_log_reference_id for sales
* Export duplicate_log_reference_id in xml
* Update existing duplicate log references
* Add missing sales deduplication steps
* Set duplicate_set_id straight on logs
* Do not update updated_at to prevent export
* Increment duplicate_set_id
* Update duplicates when they change sets
* Remove duplicates when deleting logs
* Add archived forms to the form handler
* Remove archived form if the year after it is no longer editable
* Filter logs list by existing forms only
* Add archived_lettings if it's not defined in json
* Rebase changes
* Fix unrelated test
* Create DuplicateLogReference
* Set and clear duplicate log references
* Export duplicate_log_reference_id for lettings
* Export duplicate_log_reference_id for sales
* Export duplicate_log_reference_id in xml
* Update tests
* Update existing duplicate log references
* Rename a field
* Add missing sales deduplication steps
* Set duplicate_set_id straight on logs
* Do not update updated_at to prevent export
* Increment duplicate_set_id
* Update duplicates when they change sets
* Remove duplicates when deleting logs
* correctly update duplicate set IDs when answer is changed through normal form flow
* CLDC-3110: Update hard max income for ecstat 8 to 2000
* CLDC-3110: Update hard max income for ecstat 10 to match new highest max
* CLDC-3110: Base income validations on all tenants for lettings logs
* CLDC-3110: Tweak validation messages
* CLDC-3110: Use prefer not to say values when tenant ecstat is not specified
* feat: add schemes and locations download links and pages
* feat: update current path helper
* feat: update tests for different user visibility levels
* feat: update search caption tests
* refactor: lint tests
* refactor: lint tests
* git: revert unintentional inclusion
* feat: update tests
* refactor: lint
* feat: DRY up routing
* refactor: lint
* feat: add csv confirmation view
* feat: add scheme csv service
* feat: rename
* feat: update csv service
* feat: update csv service
* feat: update controller and rename view
* feat: update view
* refactor: lint
* feat: show correct headers in csv
* feat: add locations and combined csv behaviour
* feat: remove redundant user instance variable
* feat: add scheme csv service spec
* feat: add scheme email csv job tests
* feat: update filters in spec
* refactor: move scheme_email_csv_job_spec.rb
* feat: update spec
* refactor: remove blank line
* feat: add nowrap to all download links
* feat: update org schemes controller with org schemes (and rename for clarity)
* feat: update link indentation and spec
* feat: only include location LA name, and rename to location_local_authority
* feat: update seed locations with westminster local authorities to avoid similar confusion to some that arose in PO review
* feat: display multiple active periods on a single line
* feat: display multiple active periods on a single line
* feat: update line spacing in search captions
* feat: replace 2/3 with full column in download page
* feat: move scheme alphabeticising into manager
* feat: update tests now search/filterless copy has changed
* refactor: lint
* refactor: lint
* refactor: lint
* feat: add filter alphabeticising test
* feat: correct spacing
* Correctly route to sales owning organisation page
* Display correct owning organisation options
* Set managing organisation as user organisation
* Add managing organisation question for sales
* Display managing organisation users in created_by
* set managing org to be same as owning org in BU
* Add managed by filter to sales
* Display reported by, correctly update owning org
* Update factory and 2022 BU
* refactor
* Merge managed sales logs
* test
* Update managing organisation for sales bulk upload
* Rebase tests
* skip a test as it is not relevant as the rake task would have already been ran
* PO fixes
* Remove old collection managed sales validation
* Update question header and filters conditions
* remove feature toggle
* feat: update datetime format in csv services
* feat: update tests
* feat: keep system date fields in iso8601 format
* feat: merge fix
* feat: update tests
* Add missing addresses csv job
* Update missing addresses csv service methods
* Add rake task
* Update the job to send missing town or city templates
* Update service to create mising town or city templates
* Add send missing town or city csv rake task
* Add IDs to the CSVs
* Put all log in the same csv
* Add issue type column
* Write wrong uprn logs to csv
* Add mailer methods
* Skip uprn issue for specified orgs
* Add sales csv rake task
* set SKIP_UPRN_ISSUE_ORG_IDS on review apps
* test
* Update notify template IDs for testing
* Initialize service with organisation instead of a hash
* Add expiration time to url
* Add optional tags and remove LA from csv
* Extract log to csv methods
* Update casing
* Update old IDs in factories
* Move constant
* Extract some repeating scopes
* Pass in organisations to skip instead of using an env var
* update template id for sales
* Update link expiry time and headers
* Lower the threshold for testing
* Add issue explanation to the email
* Add how to fix
* update emails
* CLDC-2810 Create all addresses CSV (#1953)
* Add rake tasks for creating all addresses CSV
* Write headers if logs don't exist, update header names
* Rename method
* CLDC-2810 Correct addresses from csv (#1957)
* Updating importing lettings addresses form csv
* Add import_sales_addresses_from_csv rake
* Allow correcting addresses from both templates
* escape .
* Reinfer LA if the postcode hasn't changed
* Update labels and email content
* Update missing addresses threshold
* Remove unused env var
* feat: wip commit add person details to sales log csv service
* feat: add comments
* feat: update regex to include agen_known fields
* feat: revert
* feat: reject age_known fields and map codes and values to age fields instead
* feat: add age1_known
* feat: combine age not known and details not known fields
* feat: update tests
* refactor: lint
* feat: update tests
* feat: update lettings tests
* feat: update question test
* Adjust xml export to export refused details, update refused to take into account details_unknown
* Recalculate refused values for lettings logs with unknown person details
* Export refused values in the csv
* Update file names
* Confirm scharge, pscharge and supcharge soft validations for completed logs
* This test seems flaky, likely because of the order, so reducing #of logs
* Add generate_missing_answers_report task and method
* Create example report
* change task description
* refactor
* Report old form id instead of old id
* feat: wip update sales org select
* feat: update routing and hidden in check answers methods
* feat: set sales org id as derived
* feat: update tests
* refactor: linting
* feat: update tests and radio options
* feat: update test
* feat: add value helper test for non-constant value question
* refactor: lint
* feat: freeze time in unrelated test
* refactor: linting
* form handler to return all questions from lettings forms for all years with ordering interleaved
functionality and tests
* refactor lettings log csv service and all associated tests
remove methods on log models when we can call them directly on associated models
update job to call the service directly with the collection of logs rather
minor modifications to the sales log csv service
update many test files to test the appropriate logic in the appropriate place
* tidying
final amendments to tests
remove commented code
rename variable
* change the position of the rent value check field in the headers
* CLDC-2492 add creation method field to logs (#1738)
* create migrations to add creation method fields to both log types
* add enum definition to logs for creation method
* upadte csv export services to retrieve creation method values direct from the log, remove methods previously used from the log model
* run migrations to update schema
* ensure that logs created via bulk upload have this set correctly when created
* feat: setup to replicate failures to fix
* feat: wip test fixes
* feat: remainder of current state test fixes
* feat: set form date to past to trigger errors to fix
* feat: revert, don't want this in final diff
* feat: fix row_parser tests
* feat: sales log importer and validator fixes
* feat: remainder of test fixes
* refactor: lint
* Remove soft validations from sales setup and make them mandatory
* Remove soft validations from lettings setup and make them mandatory
* Fix remaining tests
* Add card numbers to pregnancy checks
* call form to reset invalid values when saving a sales log
* fix tests failing after updating sales log functionality to reset invalid answers
* add a test to cover the new functionality added
* a further test
* create a method on the FormHandler that returns the sales form questions for all years in the order that they appear in the form
* update csv email job to accomodate sales log export as well as lettings
add to tests to reflec the changes made
* write tests to cover the desired functionality of the SalesLogCsvService
* create the SalesLogCsvService
create a necessary method on the log to enable submission method to be included on the csv
derive values for the two halves of previous postcode for export
* add relevant links in the UI and pipe everything together in controllers
amend organisations controller to have flexibility to download logs of either type
add necessary methods to sales log controller, raising shared method to logs controller
update routing for amendments and additions
extract helper method to build urls for downloading logs within an organisation
* correct various linter complaints and tech review suggestions
* minor amendment to add old_id and reorder early columns
* undo my 'clever' refactor that broke things
* refactoring of csv service after some tech review and some UI testing in review app
* update tests to include a test of a full export and all values in teh csv
* correct minor routing error to ensure correct url is shown and tab selected after requesting csv email
* update organisations controller requests spec file to cover new functionality and make a minor amendment to authentication scope in the controller after error found in testing
* write request tests for the new functionality in the sales log controller, define authorisation in the controller
* minor correction after rubocop's kind suggestion'
* various corrections from first pass at PO, tech review, linter, etc
* refactor :ordered_sales_questions_for_all_years
* first pass at implementing flexible code-based form fixtures for testing
* second pass
* refactor all tests of :ordered_sales_questions_for_all_years to use new factories
* some refactoring in the testing of the csv service
* use that fact that params is always available in controllers and don't pass it around, inline some methods calls
* correct minor bug to ensure that "Return to logs" link returns to the correct index page
* remove reminder comments
* write further tests on the manipulation of questions into the csv headers, update factories of form constituents to allow the creation of forms with richer questions
* fix linter complaints
* minor alterations after rebase to account for changes made on other branches
* refactor after code review
* tweak fixtures after rebase containing alterations to the factory defaults
* feat: unrelated currency formatting
* Revert "feat: unrelated currency formatting"
This reverts commit 75bd3efbf8.
* feat: add soft val pages to property information section (not to setup)
* feat: update tests
* feat: update tests
* feat: add validation to setpu section
* refactor: lint
* test: update
* feat: set nil as default check answers card no
* feat: use : to define more explicitly
* feat: make optional as affects setup
* feat: update tests
* feat: put la in nil card with rest of uprn/address qs
* feat: route to uprn confirmation from check answers changes to uprn even though not an interruption screen
* feat: update tests
* feat: update tests
* refactor: lint
* feat: update tests
* feat: update tests
* feat: wip blank fields and dependent fields on upload tos ee if valid and can upload with missing info - this is not the exact ac on the ticket yet
* Revert "feat: wip blank fields and dependent fields on upload tos ee if valid and can upload with missing info - this is not the exact ac on the ticket yet"
This reverts commit 136e5f6f12.
* feat: fix uprn hidden in check answers
* feat: fix tests
* feat: fix tests
* add first page for bulk upload resume journey
* bulk upload resume handles upload again
* add confirm page to bulk upload resume journey
* replace placeholder count with correct value
* apply recommendation for bulk upload resume choice
* add how to fix bulk upload mailer
* integrate new bulk upload approve journey
* add missing bulk upload error mappings
* remove test
* prevent approve being called multiple times
* bulk upload creates invisible logs ahead of time
* work invisible logs into bulk upload flow
* sort errors so deterministic
* remove unused ensure
* remove expected_log_count and processed
- these fields are no longer used or needed
* introduce pending status
* swap visible for pending logs
* only show visible lettings logs
* hard code status filters
* remove unused model methods
* only show visible sales logs
* form controller ignores hidden logs
* locations and schemes only affect visible logs
* Add carehome charges value check field
* Add carehome charges soft validation to 23/24 forms
* Update care_home_charge_expected_not_provided?
* Add care_home_charges_value_check to 22/23
* Update csv
* Add care_home_charges_value_check to 21/22 form
* Update import to set value_chech to yes
* Add local authority links
* Display all local authorities for location
* set correct LA for log based on year
* Format local authorities for locations
* Rename variable
* Add import task for la links
* look at form start date because log start date might not be given
* Update app/models/lettings_log.rb
Co-authored-by: James Rose <james@jbpr.net>
* Update app/helpers/locations_helper.rb
Co-authored-by: James Rose <james@jbpr.net>
* Refactor
* Seed review app
* Change dates format
* Update the local authority link data
* Typo
* update mapping in Lettings logs
---------
Co-authored-by: James Rose <james@jbpr.net>
* update seeds to add self in review env, change spec to reflect this, update config yml to allow csv exports in review
* update interface of relevant methods
EmailCsvJob, LettingsLog.to_csv and LettingsLogCsvService consume codes_only flag
* update tests including adding a new csv file to test against
* update LettingsLogCsvService to output codes only csv
* correct minor error and linting
* enable codes only download in UI
- add link on lettings log index page
- pass codes_only flag through params in relevant links and methods
- convert flag to boolean in controller methods
* ensure link displayed successfully for all renderings of logs_list and params passed through relevant methods in organisations controller
* fix existing tests
* correct linting thing
* correct linting error
* update tests for lettings log controller
* correct linting errors
* update organisations controller tests
* make minor changes after code review
* remove changes made for testing on review app
* make codes only download visible to support users only
* change variable names throughout after info on rauby/rails naming conventions, update tests for change in who can view codes only download link
* rework csv service for readability, remove delegating methods from lettings log to keep all code to do with mapping between our domain and desired export format in one place
* update test name
* correct a small typo and remove a duplicated method after clever git merge conflict suggestion
* point review app at staging csv bucket for csv download
* change variables named codes_only_export to codes_only to avoid inconsistency
* write tests to ensure that differetn user roles have the correct permissions around csv download
* ensure that non support users may not download codes only exports
* correct a small error in a previous commit
* correct minor linting error
* Add local authorities table
* Seed initial local authorities
* set previous_location_only to false by default
* Add unique index and rake task
* Update field name and required fields
* Remove previous_location_only field
* tests
* add class to create logs from bulk upload
* create logs when processing bulk uploads
* remove bulk_upload_id from csv output
* create bulk upload logs only if all valid
- this will be changed later to allow for partial logs
- and only to create logs when a threshold has been met
* add method to blank invalid non setup fields
* bulk upload log creation blanks invalid fields
* fix incorrect logic for bulk upload renewal
* fix linting
* bulk upload log creation fail logs to sentry
* fix bulk upload line ending parsing
* extract bulk uploading csv parsing to class
* use csv parser in log creator
* change handle line endings mechanism
- we now strip all windows line endings for unix based line endings
- this normalises things making it simpler
* able to view lettings bulk upload errors
* fix linting
* call service correctly in test
* add bulk upload sales questions mapping
* appease linter
* bulk upload error shows correct question
- depending on log type it will show relevant question for the field
concerned
* improve namespacing of classes
* add job to process bulk uploads
* move validation from parser to model
* add validations for field_1
* add validation for field_4
* pending tests for field_4
* convert field_mapping to array of hashes
* validate nulls based on form question
* actually load forms when toggling between forms
* validate null for startdate
* row parser has access to bulk upload
* csv upload validates first form section
* add postcode validation
* Refactor error mappings for row parser
* Add unittype question
* Fix null error setting and add builtype
* add wchair to bulk upload
* Add beds to bulk upload
* Add joint to bulk upload
* Add startertenancy to the bulk upload
* Add tenancy for bulk upload
* Add declaration to the bulk upload
* Add age1 and age1_known to bulk upload
* add ages to bulk upload
* add sex1 to bulk upload
* add ethnic_group and ethnic to bulk upload
* add national to bulk upload
* add ecstat1 to bulk upload
* add military related fields to bulk upload
* add preg_occ to bulk upload
* add housingneeds to bulk upload
* add illness to bulk upload
* add layear to bulk upload
* add waityear to bulk upload
* add reason to bulk upload
* add prevten to bulk upload
* add homeless to bulk upload
* add previous postcode to bulk upload
* add reasonable preferences to bulk upload
* add allocations system to bulk upload
* add referral to bulk upload
* add net_income_known to bulk upload
* add hb to bulk upload
* add benefits to bulk upload
* add rent fields to bulk upload
* add hhmemb to bulk upload
* use 2022 csv fixtures for bulk upload
* fix renewal mapping for bulk upload
* placeholder test for bulk upload validation
* fix bulk upload mapping for homeless field
* fix leftreg mapping for bulk upload
* fix user associations in bulk upload tests
* add gender fields for bulk upload
* add ecstatN fields to bulk upload
* add #relatN fields to bulk upload
* extract old_visible_id in factory to trait
* map net_income_known correctly for bulk upload
* fix income bugs for bulk upload
* add unitletas to bulk upload
* add #rsnvac to bulk upload
* add #sheltered to bulk upload
* add illness fields to bulk upload
* add #irproduct_other to bulk upload
* infer renewal from rsnvac for bulk upload
* add #tenancyother to bulk upload
* add #tenancylength to bulk upload
* bulk upload earnings accepts pennies but rounds
* add #reasonother to bulk upload
* fix mapping of #ppcodenk for bulk upload
* add #household_charge to bulk upload
* add #chcharge to bulk upload
* add #tcharge to bulk upload
* add #supcharg to bulk upload
* add pscharge to bulk upload
* add #scharge to bulk upload
* use case statement for bulk upload allocation
* add offered to bulk upload
* add propcode to bulk upload
* add major repair fields to bulk upload
* add #voiddate to bulk upload
* support YY year format for bulk upload
* test postcode strips whitespace for bulk upload
* add #la to bulk upload
* add previous la to bulk upload
* fix failing test
* remove duplicate line from rebase
* add first time social housing to bulk upload
* make methods private
* fix field_4 validation for bulk upload
- the null check was inverted by mistake
Co-authored-by: Kat <katrina@kosiak.co.uk>
kosiakkatrina
Rachael Booth
natdeanlewissoftwire
Jack
Arthur Campbell
Phil Lee