CLDC-2718 Add vacdays to non support csvs (#1962)

* feat: remove vacdays from support only attributes

* feat: update tests

* CLDC-2810 Send missing addresses template (#1952)

* Add missing addresses csv job

* Update missing addresses csv service methods

* Add rake task

* Update the job to send missing town or city templates

* Update service to create mising town or city templates

* Add send missing town or city csv rake task

* Add IDs to the CSVs

* Put all log in the same csv

* Add issue type column

* Write wrong uprn logs to csv

* Add mailer methods

* Skip uprn issue for specified orgs

* Add sales csv rake task

* set SKIP_UPRN_ISSUE_ORG_IDS on review apps

* test

* Update notify template IDs for testing

* Initialize service with organisation instead of a hash

* Add expiration time to url

* Add optional tags and remove LA from csv

* Extract log to csv methods

* Update casing

* Update old IDs in factories

* Move constant

* Extract some repeating scopes

* Pass in organisations to skip instead of using an env var

* update template id for sales

* Update link expiry time and headers

* Lower the threshold for testing

* Add issue explanation to the email

* Add how to fix

* update emails

* CLDC-2810 Create all addresses CSV (#1953)

* Add rake tasks for creating all addresses CSV

* Write headers if logs don't exist, update header names

* Rename method

* CLDC-2810 Correct addresses from csv (#1957)

* Updating importing lettings addresses form csv

* Add import_sales_addresses_from_csv rake

* Allow correcting addresses from both templates

* escape .

* Reinfer LA if the postcode hasn't changed

* Update labels and email content

* Update missing addresses threshold

* Remove unused env var

* CLDC-2563: Update prod deployment pipeline (#1960)

* CLDC-2563: Update prod deployment pipeline

* CLDC-2563: remove redundant REPO_URL assignment

* CLDC-2563: use GITHUB_OUTPUT only where necessary

* CLDC-2563: don't push image if tag already exists

* CLDC-2563: actually use github.sha for production as well

* CLDC-2563: remove obsolete env check in push docker image job

* CLDC-2563: remove redundant main branch check

* Delete obsolete data_export_csv task (#1974)

* CLDC-2773 Update lettings (and sales) csv date format (#1954)

* feat: update datetime format in csv services

* feat: update tests

* feat: keep system date fields in iso8601 format

* feat: merge fix

* feat: update tests

* Bump postcss from 8.4.25 to 8.4.31 (#1969)

Bumps [postcss](https://github.com/postcss/postcss) from 8.4.25 to 8.4.31.
- [Release notes](https://github.com/postcss/postcss/releases)
- [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md)
- [Commits](https://github.com/postcss/postcss/compare/8.4.25...8.4.31)

---
updated-dependencies:
- dependency-name: postcss
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Correctly map equity ranges (#1964)

* CLDC-2723 Allow data coordinators access to stock owners' schemes (#1958)

* feat: allow data coordinators to create, edit and view stock owners' schemes

* feat: only show data coordinators their org and stock owners in scheme owner select, update location policy

* feat: update some tests

* feat: update tests

* feat: use zero? where appropriate and lint

* feat: update tests

* refactor: lint

* feat: use helper for answer options

* CLDC-2871 Fix bulk upload scheme lookup bug (#1971)

* feat: wip tests

* feat: update tests and functionality

* refactor: lint

* feat: update tests and don't add errors to scheme/location fields unless determined

* feat: update duplicate log behaviour and tests

* refactor: lint

* feat: use needstype helpers

* feat: use needstype helpers and test tweak

* feat: make tests explicit regarding needstype

* feat: make tests explicit regarding needstype

* Update schemes banner text (#1968)

* CLDC-2796 Display managed logs for support (#1966)

* Display managed logs

* Update tests

* feat: add schemes to navbar when org has stock owners and code refactoring (#1980)

* CLDC-2824 Update bulk upload header guidance (#1975)

* feat: update copy in 23/24 lettings and sales prepare file pages

* feat: add full stops in alignment with designs

* feat: update tests

* feat: update tests

* feat: update tests

* feat: update tests

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: kosiakkatrina <54268893+kosiakkatrina@users.noreply.github.com>
Co-authored-by: SamSeed-Softwire <63662292+SamSeed-Softwire@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

app/services/csv/lettings_log_csv_service.rb

@@ -221,7 +221,7 @@ module Csv
       "tshortfall" => %w[tshortfall wtshortfall],
     }.freeze
 
-    SUPPORT_ONLY_ATTRIBUTES = %w[hhmemb net_income_value_check first_time_property_let_as_social_housing renttype needstype postcode_known is_la_inferred totchild totelder totadult net_income_known is_carehome previous_la_known is_previous_la_inferred age1_known age2_known age3_known age4_known age5_known age6_known age7_known age8_known letting_allocation_unknown details_known_2 details_known_3 details_known_4 details_known_5 details_known_6 details_known_7 details_known_8 rent_type_detail wrent wscharge wpschrge wsupchrg wtcharge wtshortfall rent_value_check old_form_id old_id retirement_value_check tshortfall_known pregnancy_value_check hhtype new_old vacdays la prevloc updated_by_id bulk_upload_id uprn_confirmed].freeze
+    SUPPORT_ONLY_ATTRIBUTES = %w[hhmemb net_income_value_check first_time_property_let_as_social_housing renttype needstype postcode_known is_la_inferred totchild totelder totadult net_income_known is_carehome previous_la_known is_previous_la_inferred age1_known age2_known age3_known age4_known age5_known age6_known age7_known age8_known letting_allocation_unknown details_known_2 details_known_3 details_known_4 details_known_5 details_known_6 details_known_7 details_known_8 rent_type_detail wrent wscharge wpschrge wsupchrg wtcharge wtshortfall rent_value_check old_form_id old_id retirement_value_check tshortfall_known pregnancy_value_check hhtype new_old la prevloc updated_by_id bulk_upload_id uprn_confirmed].freeze
 
     def lettings_log_attributes
       ordered_questions = FormHandler.instance.ordered_lettings_questions_for_all_years

spec/fixtures/files/lettings_log_csv_export_non_support_codes.csv

@@ -1,2 +1,2 @@
-id,status,created_by,is_dpo,created_at,updated_by,updated_at,creation_method,collection_start_year,owning_organisation_name,managing_organisation_name,lettype,renewal,startdate,irproduct,irproduct_other,lar,tenancycode,propcode,uprn_known,uprn,address_line1,address_line2,town_or_city,county,postcode_full,la_label,unitletas,rsnvac,newprop,offered,unittype_gn,builtype,wchair,beds,voiddate,void_date_value_check,majorrepairs,mrcdate,major_repairs_date_value_check,joint,startertenancy,tenancy,tenancyother,tenancylength,sheltered,declaration,refused,age1,sex1,ethnic_group,ethnic,national,ecstat1,relat2,age2,sex2,ecstat2,relat3,age3,sex3,ecstat3,relat4,age4,sex4,ecstat4,relat5,age5,sex5,ecstat5,relat6,age6,sex6,ecstat6,relat7,age7,sex7,ecstat7,relat8,age8,sex8,ecstat8,armedforces,leftreg,reservist,preg_occ,housingneeds,housingneeds_type,housingneeds_a,housingneeds_b,housingneeds_c,housingneeds_f,housingneeds_g,housingneeds_h,housingneeds_other,illness,illness_type_4,illness_type_5,illness_type_2,illness_type_6,illness_type_7,illness_type_3,illness_type_9,illness_type_8,illness_type_1,illness_type_10,layear,waityear,reason,reasonother,prevten,homeless,ppcodenk,ppostcode_full,prevloc_label,reasonpref,rp_homeless,rp_insan_unsat,rp_medwel,rp_hardship,rp_dontknow,cbl,cap,chr,referral,referral_value_check,incref,earnings,incfreq,hb,has_benefits,benefits,household_charge,nocharge,period,chcharge,wchchrg,carehome_charges_value_check,brent,scharge,pscharge,supcharg,tcharge,scharge_value_check,pscharge_value_check,supcharg_value_check,hbrentshortfall,tshortfall,scheme_code,scheme_service_name,scheme_sensitive,SCHTYPE,scheme_registered_under_care_act,scheme_owning_organisation_name,scheme_primary_client_group,scheme_has_other_client_group,scheme_secondary_client_group,scheme_support_type,scheme_intended_stay,scheme_created_at,location_code,location_postcode,location_name,location_units,location_type_of_unit,location_mobility_type,location_admin_district,location_startdate
-,completed,choreographer@owtluk.com,false,2023-06-26T00:00:00+01:00,,2023-06-26T00:00:00+01:00,1,2023,DLUHC,DLUHC,7,0,2023-06-26,,,,HIJKLMN,ABCDEFG,0,,fake address,,London,,NW9 5LL,Barnet,2,6,2,2,7,1,1,3,2023-06-24,,1,2023-06-25,,3,1,4,,2,,1,1,35,F,0,2,13,0,P,32,M,6,R,-9,R,10,R,-9,R,10,,,,,,,,,,,,,,,,,1,4,1,2,1,0,1,0,0,0,0,0,0,1,0,0,1,0,0,0,0,0,0,0,2,7,4,,6,1,0,TN23 6LZ,Ashford,1,0,1,0,0,0,0,0,1,2,,0,68,1,6,1,1,,0,2,,,,200.0,50.0,40.0,35.0,325.0,,,,1,12.0,,,,,,,,,,,,,,,,,,,,
+id,status,created_by,is_dpo,created_at,updated_by,updated_at,creation_method,collection_start_year,owning_organisation_name,managing_organisation_name,lettype,renewal,startdate,irproduct,irproduct_other,lar,tenancycode,propcode,uprn_known,uprn,address_line1,address_line2,town_or_city,county,postcode_full,la_label,unitletas,rsnvac,newprop,offered,unittype_gn,builtype,wchair,beds,voiddate,vacdays,void_date_value_check,majorrepairs,mrcdate,major_repairs_date_value_check,joint,startertenancy,tenancy,tenancyother,tenancylength,sheltered,declaration,refused,age1,sex1,ethnic_group,ethnic,national,ecstat1,relat2,age2,sex2,ecstat2,relat3,age3,sex3,ecstat3,relat4,age4,sex4,ecstat4,relat5,age5,sex5,ecstat5,relat6,age6,sex6,ecstat6,relat7,age7,sex7,ecstat7,relat8,age8,sex8,ecstat8,armedforces,leftreg,reservist,preg_occ,housingneeds,housingneeds_type,housingneeds_a,housingneeds_b,housingneeds_c,housingneeds_f,housingneeds_g,housingneeds_h,housingneeds_other,illness,illness_type_4,illness_type_5,illness_type_2,illness_type_6,illness_type_7,illness_type_3,illness_type_9,illness_type_8,illness_type_1,illness_type_10,layear,waityear,reason,reasonother,prevten,homeless,ppcodenk,ppostcode_full,prevloc_label,reasonpref,rp_homeless,rp_insan_unsat,rp_medwel,rp_hardship,rp_dontknow,cbl,cap,chr,referral,referral_value_check,incref,earnings,incfreq,hb,has_benefits,benefits,household_charge,nocharge,period,chcharge,wchchrg,carehome_charges_value_check,brent,scharge,pscharge,supcharg,tcharge,scharge_value_check,pscharge_value_check,supcharg_value_check,hbrentshortfall,tshortfall,scheme_code,scheme_service_name,scheme_sensitive,SCHTYPE,scheme_registered_under_care_act,scheme_owning_organisation_name,scheme_primary_client_group,scheme_has_other_client_group,scheme_secondary_client_group,scheme_support_type,scheme_intended_stay,scheme_created_at,location_code,location_postcode,location_name,location_units,location_type_of_unit,location_mobility_type,location_admin_district,location_startdate
+,completed,choreographer@owtluk.com,false,2023-06-26T00:00:00+01:00,,2023-06-26T00:00:00+01:00,1,2023,DLUHC,DLUHC,7,0,2023-06-26,,,,HIJKLMN,ABCDEFG,0,,fake address,,London,,NW9 5LL,Barnet,2,6,2,2,7,1,1,3,2023-06-24,1,,1,2023-06-25,,3,1,4,,2,,1,1,35,F,0,2,13,0,P,32,M,6,R,-9,R,10,R,-9,R,10,,,,,,,,,,,,,,,,,1,4,1,2,1,0,1,0,0,0,0,0,0,1,0,0,1,0,0,0,0,0,0,0,2,7,4,,6,1,0,TN23 6LZ,Ashford,1,0,1,0,0,0,0,0,1,2,,0,68,1,6,1,1,,0,2,,,,200.0,50.0,40.0,35.0,325.0,,,,1,12.0,,,,,,,,,,,,,,,,,,,,

spec/fixtures/files/lettings_log_csv_export_non_support_labels.csv

@@ -1,2 +1,2 @@
-id,status,created_by,is_dpo,created_at,updated_by,updated_at,creation_method,collection_start_year,owning_organisation_name,managing_organisation_name,lettype,renewal,startdate,irproduct,irproduct_other,lar,tenancycode,propcode,uprn_known,uprn,address_line1,address_line2,town_or_city,county,postcode_full,la_label,unitletas,rsnvac,newprop,offered,unittype_gn,builtype,wchair,beds,voiddate,void_date_value_check,majorrepairs,mrcdate,major_repairs_date_value_check,joint,startertenancy,tenancy,tenancyother,tenancylength,sheltered,declaration,refused,age1,sex1,ethnic_group,ethnic,national,ecstat1,relat2,age2,sex2,ecstat2,relat3,age3,sex3,ecstat3,relat4,age4,sex4,ecstat4,relat5,age5,sex5,ecstat5,relat6,age6,sex6,ecstat6,relat7,age7,sex7,ecstat7,relat8,age8,sex8,ecstat8,armedforces,leftreg,reservist,preg_occ,housingneeds,housingneeds_type,housingneeds_a,housingneeds_b,housingneeds_c,housingneeds_f,housingneeds_g,housingneeds_h,housingneeds_other,illness,illness_type_4,illness_type_5,illness_type_2,illness_type_6,illness_type_7,illness_type_3,illness_type_9,illness_type_8,illness_type_1,illness_type_10,layear,waityear,reason,reasonother,prevten,homeless,ppcodenk,ppostcode_full,prevloc_label,reasonpref,rp_homeless,rp_insan_unsat,rp_medwel,rp_hardship,rp_dontknow,cbl,cap,chr,referral,referral_value_check,incref,earnings,incfreq,hb,has_benefits,benefits,household_charge,nocharge,period,chcharge,wchchrg,carehome_charges_value_check,brent,scharge,pscharge,supcharg,tcharge,scharge_value_check,pscharge_value_check,supcharg_value_check,hbrentshortfall,tshortfall,scheme_code,scheme_service_name,scheme_sensitive,SCHTYPE,scheme_registered_under_care_act,scheme_owning_organisation_name,scheme_primary_client_group,scheme_has_other_client_group,scheme_secondary_client_group,scheme_support_type,scheme_intended_stay,scheme_created_at,location_code,location_postcode,location_name,location_units,location_type_of_unit,location_mobility_type,location_admin_district,location_startdate
-,completed,choreographer@owtluk.com,false,2023-06-26T00:00:00+01:00,,2023-06-26T00:00:00+01:00,single log,2023,DLUHC,DLUHC,7,No,2023-06-26,,,,HIJKLMN,ABCDEFG,No,,fake address,,London,,NW9 5LL,Barnet,Affordable rent basis,Tenant abandoned property,2,2,House,Purpose built,Yes,3,2023-06-24,,Yes,2023-06-25,,Don’t know,Yes,Assured Shorthold Tenancy (AST) – Fixed term,,2,,1,1,35,Female,White,Irish,Tenant prefers not to say,Other,Partner,32,Male,Not seeking work,Prefers not to say,Not known,Prefers not to say,Prefers not to say,Person prefers not to say,Not known,Person prefers not to say,Person prefers not to say,,,,,,,,,,,,,,,,,Yes – the person is a current or former regular,No – they left up to and including 5 years ago,Yes,No,Yes,Fully wheelchair accessible housing,1,0,0,0,0,0,No,Yes,0,0,1,0,0,0,0,0,0,0,Less than 1 year,1 year but under 2 years,Loss of tied accommodation,,Other supported housing,No,Yes,TN23 6LZ,Ashford,Yes,0,1,0,0,0,0,0,1,Tenant applied directly (no referral or nomination),,0,68,Weekly,Universal Credit housing element,1,All,,0,Every 2 weeks,,,,200.0,50.0,40.0,35.0,325.0,,,,Yes,12.0,,,,,,,,,,,,,,,,,,,,
+id,status,created_by,is_dpo,created_at,updated_by,updated_at,creation_method,collection_start_year,owning_organisation_name,managing_organisation_name,lettype,renewal,startdate,irproduct,irproduct_other,lar,tenancycode,propcode,uprn_known,uprn,address_line1,address_line2,town_or_city,county,postcode_full,la_label,unitletas,rsnvac,newprop,offered,unittype_gn,builtype,wchair,beds,voiddate,vacdays,void_date_value_check,majorrepairs,mrcdate,major_repairs_date_value_check,joint,startertenancy,tenancy,tenancyother,tenancylength,sheltered,declaration,refused,age1,sex1,ethnic_group,ethnic,national,ecstat1,relat2,age2,sex2,ecstat2,relat3,age3,sex3,ecstat3,relat4,age4,sex4,ecstat4,relat5,age5,sex5,ecstat5,relat6,age6,sex6,ecstat6,relat7,age7,sex7,ecstat7,relat8,age8,sex8,ecstat8,armedforces,leftreg,reservist,preg_occ,housingneeds,housingneeds_type,housingneeds_a,housingneeds_b,housingneeds_c,housingneeds_f,housingneeds_g,housingneeds_h,housingneeds_other,illness,illness_type_4,illness_type_5,illness_type_2,illness_type_6,illness_type_7,illness_type_3,illness_type_9,illness_type_8,illness_type_1,illness_type_10,layear,waityear,reason,reasonother,prevten,homeless,ppcodenk,ppostcode_full,prevloc_label,reasonpref,rp_homeless,rp_insan_unsat,rp_medwel,rp_hardship,rp_dontknow,cbl,cap,chr,referral,referral_value_check,incref,earnings,incfreq,hb,has_benefits,benefits,household_charge,nocharge,period,chcharge,wchchrg,carehome_charges_value_check,brent,scharge,pscharge,supcharg,tcharge,scharge_value_check,pscharge_value_check,supcharg_value_check,hbrentshortfall,tshortfall,scheme_code,scheme_service_name,scheme_sensitive,SCHTYPE,scheme_registered_under_care_act,scheme_owning_organisation_name,scheme_primary_client_group,scheme_has_other_client_group,scheme_secondary_client_group,scheme_support_type,scheme_intended_stay,scheme_created_at,location_code,location_postcode,location_name,location_units,location_type_of_unit,location_mobility_type,location_admin_district,location_startdate
+,completed,choreographer@owtluk.com,false,2023-06-26T00:00:00+01:00,,2023-06-26T00:00:00+01:00,single log,2023,DLUHC,DLUHC,7,No,2023-06-26,,,,HIJKLMN,ABCDEFG,No,,fake address,,London,,NW9 5LL,Barnet,Affordable rent basis,Tenant abandoned property,2,2,House,Purpose built,Yes,3,2023-06-24,1,,Yes,2023-06-25,,Don’t know,Yes,Assured Shorthold Tenancy (AST) – Fixed term,,2,,1,1,35,Female,White,Irish,Tenant prefers not to say,Other,Partner,32,Male,Not seeking work,Prefers not to say,Not known,Prefers not to say,Prefers not to say,Person prefers not to say,Not known,Person prefers not to say,Person prefers not to say,,,,,,,,,,,,,,,,,Yes – the person is a current or former regular,No – they left up to and including 5 years ago,Yes,No,Yes,Fully wheelchair accessible housing,1,0,0,0,0,0,No,Yes,0,0,1,0,0,0,0,0,0,0,Less than 1 year,1 year but under 2 years,Loss of tied accommodation,,Other supported housing,No,Yes,TN23 6LZ,Ashford,Yes,0,1,0,0,0,0,0,1,Tenant applied directly (no referral or nomination),,0,68,Weekly,Universal Credit housing element,1,All,,0,Every 2 weeks,,,,200.0,50.0,40.0,35.0,325.0,,,,Yes,12.0,,,,,,,,,,,,,,,,,,,,