CLDC-1779 Bulk upload lettings validation (#1148)

* able to view lettings bulk upload errors

* fix linting

* call service correctly in test

* add bulk upload sales questions mapping

* appease linter

* bulk upload error shows correct question

- depending on log type it will show relevant question for the field
  concerned

* improve namespacing of classes

* add job to process bulk uploads

* move validation from parser to model

* add validations for field_1

* add validation for field_4

* pending tests for field_4

* convert field_mapping to array of hashes

* validate nulls based on form question

* actually load forms when toggling between forms

* validate null for startdate

* row parser has access to bulk upload

* csv upload validates first form section

* add postcode validation

* Refactor error mappings for row parser

* Add unittype question

* Fix null error setting and add builtype

* add wchair to bulk upload

* Add beds to bulk upload

* Add joint to bulk upload

* Add startertenancy to the bulk upload

* Add tenancy for bulk upload

* Add declaration to the bulk upload

* Add age1 and age1_known to bulk upload

* add ages to bulk upload

* add sex1 to bulk upload

* add ethnic_group and ethnic to bulk upload

* add national to bulk upload

* add ecstat1 to bulk upload

* add military related fields to bulk upload

* add preg_occ to bulk upload

* add housingneeds to bulk upload

* add illness to bulk upload

* add layear to bulk upload

* add waityear to bulk upload

* add reason to bulk upload

* add prevten to bulk upload

* add homeless to bulk upload

* add previous postcode to bulk upload

* add reasonable preferences to bulk upload

* add allocations system to bulk upload

* add referral to bulk upload

* add net_income_known to bulk upload

* add hb to bulk upload

* add benefits to bulk upload

* add rent fields to bulk upload

* add hhmemb to bulk upload

* use 2022 csv fixtures for bulk upload

* fix renewal mapping for bulk upload

* placeholder test for bulk upload validation

* fix bulk upload mapping for homeless field

* fix leftreg mapping for bulk upload

* fix user associations in bulk upload tests

* add gender fields for bulk upload

* add ecstatN fields to bulk upload

* add #relatN fields to bulk upload

* extract old_visible_id in factory to trait

* map net_income_known correctly for bulk upload

* fix income bugs for bulk upload

* add unitletas to bulk upload

* add #rsnvac to bulk upload

* add #sheltered to bulk upload

* add illness fields to bulk upload

* add #irproduct_other to bulk upload

* infer renewal from rsnvac for bulk upload

* add #tenancyother to bulk upload

* add #tenancylength to bulk upload

* bulk upload earnings accepts pennies but rounds

* add #reasonother to bulk upload

* fix mapping of #ppcodenk for bulk upload

* add #household_charge to bulk upload

* add #chcharge to bulk upload

* add #tcharge to bulk upload

* add #supcharg to bulk upload

* add pscharge to bulk upload

* add #scharge to bulk upload

* use case statement for bulk upload allocation

* add offered to bulk upload

* add propcode to bulk upload

* add major repair fields to bulk upload

* add #voiddate to bulk upload

* support YY year format for bulk upload

* test postcode strips whitespace for bulk upload

* add #la to bulk upload

* add previous la to bulk upload

* fix failing test

* remove duplicate line from rebase

* add first time social housing to bulk upload

* make methods private

* fix field_4 validation for bulk upload

- the null check was inverted by mistake

Co-authored-by: Kat <katrina@kosiak.co.uk>

app/models/form_handler.rb

@@ -78,6 +78,16 @@ class FormHandler
     forms.count { |form| form.start_date < now && now < form.end_date } > 1
   end
 
+  def use_fake_forms!
+    @directories = ["spec/fixtures/forms"]
+    @forms = get_all_forms
+  end
+
+  def use_real_forms!
+    @directories = ["config/forms"]
+    @forms = get_all_forms
+  end
+
 private
 
   def get_all_forms
@@ -85,6 +95,6 @@ private
   end
 
   def directories
-    Rails.env.test? ? ["spec/fixtures/forms"] : ["config/forms"]
+    @directories ||= Rails.env.test? ? ["spec/fixtures/forms"] : ["config/forms"]
   end
 end

app/models/lettings_log.rb

@@ -8,6 +8,7 @@ class LettingsLogValidator < ActiveModel::Validator
   include Validations::TenancyValidations
   include Validations::DateValidations
   include Validations::LocalAuthorityValidations
+
   def validate(record)
     validation_methods = public_methods.select { |method| method.starts_with?("validate_") }
     validation_methods.each { |meth| public_send(meth, record) }

app/services/bulk_upload/lettings/row_parser.rb

@@ -2,6 +2,8 @@ class BulkUpload::Lettings::RowParser
   include ActiveModel::Model
   include ActiveModel::Attributes
 
+  attribute :bulk_upload
+
   attribute :field_1, :integer
   attribute :field_2
   attribute :field_3
@@ -51,7 +53,7 @@ class BulkUpload::Lettings::RowParser
   attribute :field_47, :integer
   attribute :field_48, :integer
   attribute :field_49, :integer
-  attribute :field_50, :integer
+  attribute :field_50, :decimal
   attribute :field_51, :integer
   attribute :field_52, :integer
   attribute :field_53, :string
@@ -137,49 +139,647 @@ class BulkUpload::Lettings::RowParser
   attribute :field_133, :integer
   attribute :field_134, :integer
 
-  validates :field_1, presence: true, numericality: { in: (1..12) }
+  validates :field_1, presence: true, inclusion: { in: (1..12).to_a }
-  validates :field_4, numericality: { in: (1..999), allow_blank: true }
+  validates :field_4, presence: { if: proc { [2, 4, 6, 8, 10, 12].include?(field_1) } }
-  validates :field_4, presence: true, if: :field_4_presence_check
+
+  validates :field_96, presence: true
+  validates :field_97, presence: true
+  validates :field_98, presence: true
+
+  def valid?
+    errors.clear
+
+    super
+
+    validate_data_types
+    validate_nulls
 
-  validate :validate_possible_answers
+    log.valid?
 
-# delegate :valid?, to: :native_object
+    log.errors.each do |error|
-# delegate :errors, to: :native_object
+      fields = field_mapping_for_errors[error.attribute] || []
+      fields.each { |field| errors.add(field, error.type) }
+    end
+
+    errors.blank?
+  end
+
+  def log
+    @log ||= LettingsLog.new(attributes_for_log)
+  end
 
 private
 
-  def native_object
+  def attribute_set
-    @native_object ||= LettingsLog.new(attributes_for_log)
+    @attribute_set ||= instance_variable_get(:@attributes)
+  end
+
+  def validate_data_types
+    unless attribute_set["field_1"].value_before_type_cast&.match?(/\A\d+\z/)
+      errors.add(:field_1, :invalid)
+    end
+  end
+
+  def postcode_full
+    "#{field_108} #{field_109}" if field_108 && field_109
+  end
+
+  def postcode_known
+    if postcode_full.present?
+      1
+    elsif field_107.present?
+      0
+    end
+  end
+
+  def questions
+    log.form.subsections.flat_map { |ss| ss.applicable_questions(log) }
   end
 
-  def field_mapping
+  def validate_nulls
+    field_mapping_for_errors.each do |error_key, fields|
+      question_id = error_key.to_s
+      question = questions.find { |q| q.id == question_id }
+
+      next unless question
+      next if log.optional_fields.include?(question.id)
+      next if question.completed?(log)
+
+      fields.each { |field| errors.add(field, :blank) }
+    end
+  end
+
+  def field_mapping_for_errors
     {
-      field_134: :renewal,
+      lettype: [:field_1],
+      tenancycode: [:field_7],
+      postcode_known: %i[field_107 field_108 field_109],
+      postcode_full: %i[field_107 field_108 field_109],
+      la: %i[field_107],
+      owning_organisation_id: [:field_111],
+      managing_organisation_id: [:field_113],
+      renewal: [:field_134],
+      scheme: %i[field_4 field_5],
+      created_by: [],
+      needstype: [],
+      rent_type: %i[field_1 field_129 field_130],
+      startdate: %i[field_98 field_97 field_96],
+      unittype_gn: %i[field_102],
+      builtype: %i[field_103],
+      wchair: %i[field_104],
+      beds: %i[field_101],
+      joint: %i[field_133],
+      startertenancy: %i[field_8],
+      tenancy: %i[field_9],
+      tenancyother: %i[field_10],
+      tenancylength: %i[field_11],
+      declaration: %i[field_132],
+
+      age1_known: %i[field_12],
+      age1: %i[field_12],
+      age2_known: %i[field_13],
+      age2: %i[field_13],
+      age3_known: %i[field_14],
+      age3: %i[field_14],
+      age4_known: %i[field_15],
+      age4: %i[field_15],
+      age5_known: %i[field_16],
+      age5: %i[field_16],
+      age6_known: %i[field_17],
+      age6: %i[field_17],
+      age7_known: %i[field_18],
+      age7: %i[field_18],
+      age8_known: %i[field_19],
+      age8: %i[field_19],
+
+      sex1: %i[field_20],
+      sex2: %i[field_21],
+      sex3: %i[field_22],
+      sex4: %i[field_23],
+      sex5: %i[field_24],
+      sex6: %i[field_25],
+      sex7: %i[field_26],
+      sex8: %i[field_27],
+
+      ethnic_group: %i[field_43],
+      ethnic: %i[field_43],
+      national: %i[field_44],
+
+      relat2: %i[field_28],
+      relat3: %i[field_29],
+      relat4: %i[field_30],
+      relat5: %i[field_31],
+      relat6: %i[field_32],
+      relat7: %i[field_33],
+      relat8: %i[field_34],
+
+      ecstat1: %i[field_35],
+      ecstat2: %i[field_36],
+      ecstat3: %i[field_37],
+      ecstat4: %i[field_38],
+      ecstat5: %i[field_39],
+      ecstat6: %i[field_40],
+      ecstat7: %i[field_41],
+      ecstat8: %i[field_42],
+
+      armedforces: %i[field_45],
+      leftreg: %i[field_114],
+      reservist: %i[field_46],
+
+      preg_occ: %i[field_47],
+
+      housingneeds: %i[field_47],
+
+      illness: %i[field_118],
+
+      layear: %i[field_66],
+      waityear: %i[field_67],
+      reason: %i[field_52],
+      reasonother: %i[field_53],
+      prevten: %i[field_61],
+      homeless: %i[field_68],
+
+      prevloc: %i[field_62],
+      previous_la_known: %i[field_62],
+      ppcodenk: %i[field_65],
+      ppostcode_full: %i[field_63 field_64],
+
+      reasonpref: %i[field_69],
+      rp_homeless: %i[field_70],
+      rp_insan_unsat: %i[field_71],
+      rp_medwel: %i[field_72],
+      rp_hardship: %i[field_73],
+      rp_dontknow: %i[field_74],
+
+      cbl: %i[field_75],
+      chr: %i[field_76],
+      cap: %i[field_77],
+
+      referral: %i[field_78],
+
+      net_income_known: %i[field_51],
+      earnings: %i[field_50],
+      incfreq: %i[field_116],
+      hb: %i[field_48],
+      benefits: %i[field_49],
+
+      period: %i[field_79],
+      brent: %i[field_80],
+      scharge: %i[field_81],
+      pscharge: %i[field_82],
+      supcharg: %i[field_83],
+      tcharge: %i[field_84],
+      chcharge: %i[field_85],
+      household_charge: %i[field_86],
+      hbrentshortfall: %i[field_87],
+      tshortfall: %i[field_88],
+
+      unitletas: %i[field_105],
+      rsnvac: %i[field_106],
+      sheltered: %i[field_117],
+
+      illness_type_1: %i[field_119],
+      illness_type_2: %i[field_120],
+      illness_type_3: %i[field_121],
+      illness_type_4: %i[field_122],
+      illness_type_5: %i[field_123],
+      illness_type_6: %i[field_124],
+      illness_type_7: %i[field_125],
+      illness_type_8: %i[field_126],
+      illness_type_9: %i[field_127],
+      illness_type_10: %i[field_128],
+
+      irproduct_other: %i[field_131],
+
+      offered: %i[field_99],
+
+      propcode: %i[field_100],
+
+      majorrepairs: %i[field_92 field_93 field_94],
+      mrcdate: %i[field_92 field_93 field_94],
+
+      voiddate: %i[field_89 field_90 field_91],
     }
   end
 
-  def validate_possible_answers
+  def startdate
-    field_mapping.each do |field, attribute|
+    Date.new(field_98 + 2000, field_97, field_96) if field_98.present? && field_97.present? && field_96.present?
-      possible_answers = FormHandler.instance.current_lettings_form.questions.find { |q| q.id == attribute.to_s }.answer_options.keys
+  end
+
+  def renttype
+    case field_1
+    when 1, 2, 3, 4
+      :social
+    when 5, 6, 7, 8
+      :affordable
+    when 9, 10, 11, 12
+      :intermediate
+    end
+  end
 
-      unless possible_answers.include?(public_send(field))
+  def rent_type
-        errors.add(field, "Value supplied is not one of the permitted values")
+    case renttype
+    when :social
+      Imports::LettingsLogsImportService::RENT_TYPE[:social_rent]
+    when :affordable
+      if field_129 == 1
+        Imports::LettingsLogsImportService::RENT_TYPE[:london_affordable_rent]
+      else
+        Imports::LettingsLogsImportService::RENT_TYPE[:affordable_rent]
+      end
+    when :intermediate
+      case field_130
+      when 1
+        Imports::LettingsLogsImportService::RENT_TYPE[:rent_to_buy]
+      when 2
+        Imports::LettingsLogsImportService::RENT_TYPE[:london_living_rent]
+      when 3
+        Imports::LettingsLogsImportService::RENT_TYPE[:other_intermediate_rent_product]
       end
     end
   end
 
+  def owning_organisation_id
+    Organisation.find_by(old_visible_id: field_111)&.id
+  end
+
+  def managing_organisation_id
+    Organisation.find_by(old_visible_id: field_113)&.id
+  end
+
   def attributes_for_log
-    hash = field_mapping.invert
     attributes = {}
 
-    hash.map do |k, v|
+    attributes["lettype"] = field_1
-      attributes[k] = public_send(v)
+    attributes["tenancycode"] = field_7
-    end
+    attributes["la"] = field_107
+    attributes["postcode_known"] = postcode_known
+    attributes["postcode_full"] = postcode_full
+    attributes["owning_organisation_id"] = owning_organisation_id
+    attributes["managing_organisation_id"] = managing_organisation_id
+    attributes["renewal"] = renewal
+    attributes["scheme"] = scheme
+    attributes["created_by"] = bulk_upload.user
+    attributes["needstype"] = bulk_upload.needstype
+    attributes["rent_type"] = rent_type
+    attributes["startdate"] = startdate
+    attributes["unittype_gn"] = field_102
+    attributes["builtype"] = field_103
+    attributes["wchair"] = field_104
+    attributes["beds"] = field_101
+    attributes["joint"] = field_133
+    attributes["startertenancy"] = field_8
+    attributes["tenancy"] = field_9
+    attributes["tenancyother"] = field_10
+    attributes["tenancylength"] = field_11
+    attributes["declaration"] = field_132
+
+    attributes["age1_known"] = field_12.present? ? 0 : 1
+    attributes["age1"] = field_12
+    attributes["age2_known"] = field_13.present? ? 0 : 1
+    attributes["age2"] = field_13
+    attributes["age3_known"] = field_14.present? ? 0 : 1
+    attributes["age3"] = field_14
+    attributes["age4_known"] = field_15.present? ? 0 : 1
+    attributes["age4"] = field_15
+    attributes["age5_known"] = field_16.present? ? 0 : 1
+    attributes["age5"] = field_16
+    attributes["age6_known"] = field_17.present? ? 0 : 1
+    attributes["age6"] = field_17
+    attributes["age7_known"] = field_18.present? ? 0 : 1
+    attributes["age7"] = field_18
+    attributes["age8_known"] = field_19.present? ? 0 : 1
+    attributes["age8"] = field_19
+
+    attributes["sex1"] = field_20
+    attributes["sex2"] = field_21
+    attributes["sex3"] = field_22
+    attributes["sex4"] = field_23
+    attributes["sex5"] = field_24
+    attributes["sex6"] = field_25
+    attributes["sex7"] = field_26
+    attributes["sex8"] = field_27
+
+    attributes["ethnic_group"] = ethnic_group_from_ethnic
+    attributes["ethnic"] = field_43
+    attributes["national"] = field_44
+
+    attributes["relat2"] = field_28
+    attributes["relat3"] = field_29
+    attributes["relat4"] = field_30
+    attributes["relat5"] = field_31
+    attributes["relat6"] = field_32
+    attributes["relat7"] = field_33
+    attributes["relat8"] = field_34
+
+    attributes["ecstat1"] = field_35
+    attributes["ecstat2"] = field_36
+    attributes["ecstat3"] = field_37
+    attributes["ecstat4"] = field_38
+    attributes["ecstat5"] = field_39
+    attributes["ecstat6"] = field_40
+    attributes["ecstat7"] = field_41
+    attributes["ecstat8"] = field_42
+
+    attributes["details_known_2"] = details_known(2)
+    attributes["details_known_3"] = details_known(3)
+    attributes["details_known_4"] = details_known(4)
+    attributes["details_known_5"] = details_known(5)
+    attributes["details_known_6"] = details_known(6)
+    attributes["details_known_7"] = details_known(7)
+    attributes["details_known_8"] = details_known(8)
+
+    attributes["armedforces"] = field_45
+    attributes["leftreg"] = leftreg
+    attributes["reservist"] = field_46
+
+    attributes["preg_occ"] = field_47
+
+    attributes["housingneeds"] = housingneeds
+
+    attributes["illness"] = field_118
+
+    attributes["layear"] = field_66
+    attributes["waityear"] = field_67
+    attributes["reason"] = field_52
+    attributes["reasonother"] = field_53
+    attributes["prevten"] = field_61
+    attributes["homeless"] = homeless
+
+    attributes["prevloc"] = prevloc
+    attributes["previous_la_known"] = previous_la_known
+    attributes["ppcodenk"] = ppcodenk
+    attributes["ppostcode_full"] = ppostcode_full
+
+    attributes["reasonpref"] = field_69
+    attributes["rp_homeless"] = field_70
+    attributes["rp_insan_unsat"] = field_71
+    attributes["rp_medwel"] = field_72
+    attributes["rp_hardship"] = field_73
+    attributes["rp_dontknow"] = field_74
+
+    attributes["cbl"] = cbl
+    attributes["chr"] = chr
+    attributes["cap"] = cap
+    attributes["letting_allocation_unknown"] = letting_allocation_unknown
+
+    attributes["referral"] = field_78
+
+    attributes["net_income_known"] = net_income_known
+    attributes["earnings"] = earnings
+    attributes["incfreq"] = field_116
+    attributes["hb"] = field_48
+    attributes["benefits"] = field_49
+
+    attributes["period"] = field_79
+    attributes["brent"] = field_80
+    attributes["scharge"] = field_81
+    attributes["pscharge"] = field_82
+    attributes["supcharg"] = field_83
+    attributes["tcharge"] = field_84
+    attributes["chcharge"] = field_85
+    attributes["household_charge"] = field_86
+    attributes["hbrentshortfall"] = field_87
+    attributes["tshortfall_known"] = tshortfall_known
+    attributes["tshortfall"] = field_88
+
+    attributes["hhmemb"] = hhmemb
+
+    attributes["unitletas"] = field_105
+    attributes["rsnvac"] = rsnvac
+    attributes["sheltered"] = field_117
+
+    attributes["illness_type_1"] = field_119
+    attributes["illness_type_2"] = field_120
+    attributes["illness_type_3"] = field_121
+    attributes["illness_type_4"] = field_122
+    attributes["illness_type_5"] = field_123
+    attributes["illness_type_6"] = field_124
+    attributes["illness_type_7"] = field_125
+    attributes["illness_type_8"] = field_126
+    attributes["illness_type_9"] = field_127
+    attributes["illness_type_10"] = field_128
+
+    attributes["irproduct_other"] = field_131
+
+    attributes["offered"] = field_99
+
+    attributes["propcode"] = field_100
+
+    attributes["majorrepairs"] = majorrepairs
+
+    attributes["mrcdate"] = mrcdate
+
+    attributes["voiddate"] = voiddate
+
+    attributes["first_time_property_let_as_social_housing"] = first_time_property_let_as_social_housing
 
     attributes
   end
 
-  def field_4_presence_check
+  def first_time_property_let_as_social_housing
-    [1, 3, 5, 7, 9, 11].include?(field_1)
+    case rsnvac
+    when 15, 16, 17
+      1
+    end
+  end
+
+  def rsnvac
+    field_106
+  end
+
+  def voiddate
+    Date.new(field_91 + 2000, field_90, field_89) if field_91.present? && field_90.present? && field_89.present?
+  end
+
+  def majorrepairs
+    mrcdate.present? ? 1 : 0
+  end
+
+  def mrcdate
+    Date.new(field_94 + 2000, field_93, field_92) if field_94.present? && field_93.present? && field_92.present?
+  end
+
+  def prevloc
+    field_62
+  end
+
+  def previous_la_known
+    prevloc.present? ? 1 : 0
+  end
+
+  def ppcodenk
+    case field_65
+    when 1
+      1
+    when 2
+      0
+    end
+  end
+
+  def earnings
+    field_50.round if field_50.present?
+  end
+
+  def net_income_known
+    case field_51
+    when 1
+      0
+    when 2
+      1
+    when 3
+      1
+    when 4
+      2
+    end
+  end
+
+  def leftreg
+    case field_114
+    when 3
+      3
+    when 4
+      1
+    when 5
+      2
+    when 6
+      0
+    end
+  end
+
+  def homeless
+    case field_68
+    when 1
+      1
+    when 12
+      11
+    end
+  end
+
+  def renewal
+    case field_134
+    when 1
+      1
+    when 2
+      0
+    when nil
+      field_116 == 14 ? 1 : 0
+    end
+  end
+
+  def details_known(person_n)
+    send("person_#{person_n}_present?") ? 0 : 1
+  end
+
+  def hhmemb
+    [
+      person_2_present?,
+      person_3_present?,
+      person_4_present?,
+      person_5_present?,
+      person_6_present?,
+      person_7_present?,
+      person_8_present?,
+    ].count(true) + 1
+  end
+
+  def person_2_present?
+    field_13.present? && field_21.present? && field_28.present?
+  end
+
+  def person_3_present?
+    field_14.present? && field_22.present? && field_29.present?
+  end
+
+  def person_4_present?
+    field_15.present? && field_23.present? && field_30.present?
+  end
+
+  def person_5_present?
+    field_16.present? && field_24.present? && field_31.present?
+  end
+
+  def person_6_present?
+    field_17.present? && field_25.present? && field_32.present?
+  end
+
+  def person_7_present?
+    field_18.present? && field_26.present? && field_33.present?
+  end
+
+  def person_8_present?
+    field_19.present? && field_27.present? && field_34.present?
+  end
+
+  def tshortfall_known
+    field_87 == 1 ? 0 : 1
+  end
+
+  def letting_allocation_unknown
+    [cbl, chr, cap].all?(0) ? 1 : 0
+  end
+
+  def cbl
+    case field_75
+    when 2
+      0
+    when 1
+      1
+    end
+  end
+
+  def chr
+    case field_76
+    when 2
+      0
+    when 1
+      1
+    end
+  end
+
+  def cap
+    case field_77
+    when 2
+      0
+    when 1
+      1
+    end
+  end
+
+  def ppostcode_full
+    "#{field_63} #{field_64}".strip.gsub(/\s+/, " ")
+  end
+
+  def housingneeds
+    if field_59 == 1
+      1
+    elsif field_60 == 1
+      3
+    else
+      2
+    end
+  end
+
+  def ethnic_group_from_ethnic
+    return nil if field_43.blank?
+
+    case field_43
+    when 1, 2, 3, 18
+      0
+    when 4, 5, 6, 7
+      1
+    when 8, 9, 10, 11, 15
+      2
+    when 12, 13, 14
+      3
+    when 16, 19
+      4
+    when 17
+      17
+    end
+  end
+
+  def scheme
+    @scheme ||= Scheme.find_by(old_visible_id: field_4)
   end
 end

app/services/bulk_upload/lettings/validator.rb

@@ -151,9 +151,6 @@ class BulkUpload::Lettings::Validator
   end
 
   def call
-    row_offset = 6
-    col_offset = 0
-
     row_parsers.each_with_index do |row_parser, index|
       row_parser.valid?
 
@@ -166,7 +163,7 @@ class BulkUpload::Lettings::Validator
           tenant_code: row_parser.field_7,
           property_ref: row_parser.field_100,
           row:,
-          cell: "#{cols[field_number_for_attribute(error.attribute) + col_offset]}#{row}",
+          cell: "#{cols[field_number_for_attribute(error.attribute) - col_offset + 1]}#{row + 1}",
         )
       end
     end
@@ -178,6 +175,14 @@ class BulkUpload::Lettings::Validator
 
 private
 
+  def row_offset
+    5
+  end
+
+  def col_offset
+    1
+  end
+
   def field_number_for_attribute(attribute)
     attribute.to_s.split("_").last.to_i
   end
@@ -191,6 +196,7 @@ private
       stripped_row = row[1..]
       headers = ("field_1".."field_134").to_a
       hash = Hash[headers.zip(stripped_row)]
+      hash[:bulk_upload] = bulk_upload
 
       BulkUpload::Lettings::RowParser.new(hash)
     end
@@ -221,7 +227,7 @@ private
   end
 
   def body_rows
-    rows[6..]
+    rows[row_offset..]
   end
 
   def validate_file_not_empty

spec/factories/bulk_upload.rb

@@ -7,6 +7,7 @@ FactoryBot.define do
     year { 2022 }
     identifier { SecureRandom.uuid }
     sequence(:filename) { |n| "bulk-upload-#{n}.csv" }
+    needstype { 1 }
 
     trait(:sales) do
       log_type { BulkUpload.log_types[:sales] }

spec/factories/organisation.rb

@@ -9,6 +9,10 @@ FactoryBot.define do
     created_at { Time.zone.now }
     updated_at { Time.zone.now }
     holds_own_stock { true }
+
+    trait :with_old_visible_id do
+      old_visible_id { rand(9_999_999).to_s }
+    end
   end
 
   factory :organisation_rent_period do

spec/fixtures/files/2022_23_lettings_bulk_upload.csv

@@ -0,0 +1,72 @@
+Question,What is the letting type?,[BLANK],[BLANK],"Management group code
+
+If supported housing","Scheme code
+
+If supported housing",[BLANK],"What is the tenant code? 
+
+This is how you usually refer to this tenancy on your own systems",Is this a starter tenancy? ,What is the tenancy type?  ,If 'Other' what is the tenancy type?,"What is the length of the fixed-term tenancy to the nearest year? 
+
+If fixed-term tenancy",Age of Person 1,Age of Person 2,Age of Person 3,Age of Person 4,Age of Person 5,Age of Person 6,Age of Person 7,Age of Person 8,Gender identity of Person 1,Gender identity of Person 2,Gender identity of Person 3,Gender identity of Person 4,Gender identity of Person 5,Gender identity of Person 6,Gender identity of Person 7,Gender identity of Person 8,Person 2's relationship to lead tenant,Person 3's relationship to lead tenant,Person 4's relationship to lead tenant,Person 5's relationship to lead tenant,Person 6's relationship to lead tenant,Person 7's relationship to lead tenant,Person 8's relationship to lead tenant,Working situation of Person 1,Working situation of Person 2,Working situation of Person 3,Working situation of Person 4,Working situation of Person 5,Working situation of Person 6,Working situation of Person 7,Working situation of Person 8,What is the lead tenant’s ethnic group? ,What is the lead tenant’s nationality? ,Does anybody in the household have links to the UK armed forces? ,Was the person seriously injured or ill as a result of serving in the UK armed forces? ,Is anybody in the household pregnant? ,Is the tenant likely to be receiving benefits related to housing? ,"How much of the household's income is from Universal Credit, state pensions or benefits? ","How much income does the household have in total?
+
+Include any income after tax from employment, pensions, and Universal Credit 
+
+Don't include National Insurance (NI) contributions and tax, housing benefit, child benefit, or council tax support  ",Do you know the household's income?,What is the tenant's main reason for the household leaving their last settled home? ,"If 'other', what was the main reason for leaving their last settled home?",[BLANK],"Disabled access needs
+
+a) Fully wheelchair-accessible housing ","Disabled access needs
+
+b) Wheelchair access to essential rooms ","Disabled access needs
+
+c) Level access housing ","Disabled access needs
+
+f) Other disabled access needs ","Disabled access needs
+
+g) No disabled access needs ","Disabled access needs
+
+h) Don’t know",Where was the household immediately before this letting? ,What is the local authority of the household's last settled home?,Part 1 of postcode of last settled home,Part 2 of postcode of last settled home ,Do you know the postcode of the last settled home?,How long has the household continuously lived in the local authority area of the new letting? ,How long has the household been on the local authority waiting list for the new letting? ,Was the tenant homeless directly before this tenancy?,Was the household given 'reasonable preference' by the local authority?   ,"Reasonable Preference
+
+They were homeless or about to lose their home (within 56 days) ","Reasonable Preference
+
+They were living in unsanitary, overcrowded or unsatisfactory housing ","Reasonable Preference
+
+They needed to move due to medical and welfare reasons (including disability) ","Reasonable Preference
+
+They needed to move to avoid hardship to themselves or others ","Reasonable Preference
+
+Don't know ",Was the letting made under Choice-Based Lettings (CBL)? ,Was the letting made under the Common Housing Register (CHR)? ,Was the letting made under the Common Allocation Policy (CAP)? ,What was the source of referral for this letting? ,How often does the household pay rent and other charges? ,What is the basic rent? ,What is the service charge?  ,What is the personal service charge? ,What is the support charge? ,Total charge,"
+If this is a care home, 
+how much does the household pay every [time period]?","Does the household pay rent or other charges for the accommodation?
+
+If supported housing ","After the household has received any housing-related benefits, will they still need to pay basic rent and other charges? ",What do you expect the outstanding amount to be? ,What is the void or renewal day? - DD,What is the void or renewal month? - MM,What is the void or renewal year? - YYYY,What day were Major Repairs completed on? - DD,What month were Major Repairs completed on? - MM,What year were Major Repairs completed on? - YY,[BLANK],What day did the tenancy start? - DD,What month did the tenancy start? - MM,What year did the tenancy start? - YY,"Since becoming available, how many times has the property been previously offered? ","What is the property reference? 
+
+This is how you usually refer to this property on your own systems","How many bedrooms does the property have?
+
+If general needs","What type of unit is the property? 
+
+If general needs","Which type of building is the property?
+
+If general needs","Is the property built or adapted to wheelchair-user standards? 
+
+If general needs","What type was the property most recently let as? 
+
+If re-let",What is the reason for the property being vacant? ,"What is the local authority of the property?
+
+If general needs","Part 1 of postcode of property
+
+If general needs","Part 2 of postcode of property
+
+If general needs",[BLANK],"Which organisation owns this property? 
+
+Organisation's CORE ID",Username Field,"Which organisation manages this property? 
+
+Organisation's CORE ID",Is the person still serving in the UK armed forces? ,[BLANK],How often does the household receive income? ,"Is this letting in sheltered accommodation?
+
+If supported housing",Does anybody in the household have a physical or mental health condition (or other illness) expected to last 12 months or more? ,"Vision, for example  blindness or partial sight","Hearing, for example  deatness or partial hearing","Mobility, for example walking short distances or climbing stairs","Dexterity, for example  lifting and carrying objects, using a keyboard",Learning or understanding or concentrating,Memory,"Mental health, for example depression or anxiety",Stamina or breathing or fatigue,"Socially or behaviourally (e.g. associated with autism spectrum disorder (ASD) which includes Aspergers', or attention deficit hyperactivity disorder (ADHD))",Other illness or condition,Is this letting a London Affordable Rent letting?,Which type of Intermediate Rent is this letting?,Which 'Other' type of Intermediate Rent is this letting?,Has the tenant seen the DLUHC privacy notice? ,Is this a joint tenancy? ,Is this a renewal to the same tenant in the same property?,
+Values,1 - 12,,,1 - 999,,,max 13 digits,1 - 2,2 - 7,Text ,1 - 99,15 - 120 or R,1 - 120 or R,,,,,,,"M, F, X or R",,,,,,,,"P,C,X or R",,,,,,,0 - 11,,,,,,,,1 - 19,"12 - 13, 17 - 19",1 - 6,1 - 3,,"1, 3, 6, 9",1 - 4,0 - 99999,1 - 4,"1 - 2, 4,  7 - 14, 16 - 20, 28 - 31 or 34 - 47",Text,,1 or null ,,,,,,"3- 4,  6 - 7, 9 - 10, 13 - 14, 18 - 19, 21 or 23 - 35",ONS CODE - E + 8 Digits,XXX(X),XXX,1 - 2,1 - 2 or 5 - 10,2 or 5 - 11,1 or 12,1 - 3,1 or null,,,,,1 or 2,,,"1 - 4, 7 - 10, 12 - 17",1 - 9,xxxx.xx,,,,,,1 or null,1 - 3,0 - 9999,1 - 31,1 - 12,19 - 23,1 - 31,1 - 12,13 - 23,,1 - 31,1 - 12,22-23,0+,12 Digits,1 - 7,"1, 2, 4 or 6 - 10",1 or 2,,1 - 4,"5, 6, 8 - 14, ",ONS CODE E + 9 digits,XXX(X),XXX,,Up to 7 digits,Username of CORE account this letting log should be assigned to,Up to 7 digits,3 - 6,,1 - 3,1 - 4,1 - 3,1 or null,,,,,,,,,,1 - 3,,Text,1,1 - 3,1 - 2,
+Can be null?,No,,,"only if field 1 = 1, 3, 5, 7, 9 or 11",,,No,,,"Yes, if 9 is not 3","Yes, if 9 = 2, 3, 5 or 7",No,"Yes, if field 21, 28 and 36 are null","Yes, if 22, 29 and 37 are null","Yes, if 23, 30 and 38 are null","Yes, if 24, 31 and 39 are null","Yes, if 25, 32 and 40 are null","Yes, if 26, 33 and 41 are null","Yes, if  27, 34 and 42 are null", No,"Yes, if 13, 28 and 36 are null","Yes, if 14, 29 and 37 are null","Yes, if 15, 30 and 38 are null","Yes, if 16, 31 and 39 are null","Yes, if 17, 32 and 40 are null","Yes, if 18, 33 and 41 are null","Yes, if 19, 34 and 42 are null","Yes, if 13, 21 and 36 are null","Yes, if 14, 22 and 37 are null","Yes, if 15, 23 and 38 are null","Yes, if 16, 24 and 39 are null","Yes, if 17, 25 and 40 are null","Yes, if 18, 26 and 41 are null","Yes, if 19, 27 and 42 are null",No,"Yes, if 13, 21 and 28 are null","Yes, if 14, 22 and 29 are null","Yes, if 15, 23 and 30 are null","Yes, if 16, 24 and 31 are null","Yes, if 17, 25 and 32 are null","Yes, if 18, 26 and 33 are null","Yes, if 19, 27 and 34 are null",No,,,"Yes, must be null if 45 = 2 or 3;
+no, if 45 = 1, 4 or 5",No,,,If 51 = 4,No,No,"Yes, if 52 is not 20",,"Selections are  ((A or B  or C)) - ((A, or B or C and F)) - ((F)) - ((G)) - ((H))",,,,,,No,,"Yes, if 65 = 1",,"Yes, if 63 and 64 contain full and valid entries",No,,,,"  If 69 = 1, select at least one of the 5 categories;
+If 69 = 2 or 3, then Null.",,,,,No,,,,,Only if 85 or 86 = 1,Yes,,,Only if 85 or 86 = 1,Only if fields 80 - 84 and 86 are not null,Only if fields 80 - 85 are not null,Only if field 48 = 3 or 9 ,"If 87 = 2 or 3;
+if 87 = 1, then a value must be entered",No,,,Yes,,,,No,,,"If the property is being let for the first time, enter 0.","Only if 1 = 2, 4, 6, 8, 10 or 12",,,,No,"Only if 1 = 2, 4, 6, 8, 10 or 12;
+or 106 = 15 - 17",No,"Only if 1 = 2, 4, 6, 8, 10 or 12",,,,No,,No,"Yes, if 45 = 2, 3 or 6",,"Yes, if 50 = 1","Only if 1 = 1, 3, 5, 7, 9 or 11",No,Yes,,,,,,,,,,Only if 1 = 1 - 4 or 9 - 12.,Only if 1 = 1 - 8.,Only if 130 is not 3,No,No,Yes,
+Bulk upload format and duplicate check,All lettings,Question removed from 22/23 onwards,,Supported housing only,,Question Removed from 2020/21,,,,,,Duplicate check field,,,,,,,,Duplicate check field,,,,,,,,,,,,,,,Duplicate check field,,,,,,,,,,,,,,,,,,,Question removed from 22/23 onwards,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Duplicate check field,,,,,,,,,,,Question removed from 22/23 onwards,Duplicate check fields,,,,Duplicate check field,General Needs lettings only,,,All lettings,General Needs lettings only,All lettings,General Needs lettings only,,,Question removed from 2020/21,Duplicate check field, “Username does not exist”. ,,,Question removed from 21/22 onwards,,Supported Housing lettings only.,,,,,,,,,,,,Affordable Rent Lettings only,Intermediate Rent Lettings only,,All lettings,All lettings,,
+Bulk upload field number,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,
+,1,,,,,,123,1,2,,6,55,54,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

spec/services/bulk_upload/lettings/row_parser_spec.rb

@@ -3,77 +3,733 @@ require "rails_helper"
 RSpec.describe BulkUpload::Lettings::RowParser do
   subject(:parser) { described_class.new(attributes) }
 
+  let(:now) { Time.zone.today }
+
+  let(:attributes) { { bulk_upload: } }
+  let(:bulk_upload) { create(:bulk_upload, :lettings, user:) }
+  let(:user) { create(:user, organisation: owning_org) }
+  let(:owning_org) { create(:organisation) }
+  let(:managing_org) { create(:organisation) }
+  let(:setup_section_params) do
+    {
+      bulk_upload:,
+      field_1: "2",
+      field_111: owning_org.old_visible_id,
+      field_113: managing_org.old_visible_id,
+      field_96: now.day.to_s,
+      field_97: now.month.to_s,
+      field_98: now.strftime("%g"),
+      field_134: "2",
+    }
+  end
+
+  around do |example|
+    FormHandler.instance.use_real_forms!
+
+    example.run
+
+    FormHandler.instance.use_fake_forms!
+  end
+
   describe "validations" do
     before do
+      stub_request(:get, /api.postcodes.io/)
+      .to_return(status: 200, body: "{\"status\":200,\"result\":{\"admin_district\":\"Manchester\", \"codes\":{\"admin_district\": \"E08000003\"}}}", headers: {})
+
       parser.valid?
     end
 
-    describe "field_1" do
+    describe "#valid?" do
+      context "when calling the method multiple times" do
+        let(:attributes) { { bulk_upload:, field_134: 2 } }
+
+        it "does not add keep adding errors to the pile" do
+          expect { parser.valid? }.not_to change(parser.errors, :count)
+        end
+      end
+
+      context "when valid row" do
+        let(:attributes) do
+          {
+            bulk_upload:,
+            field_1: "1",
+            field_4: "1",
+            field_7: "123",
+            field_96: now.day.to_s,
+            field_97: now.month.to_s,
+            field_98: now.strftime("%g"),
+            field_108: "EC1N",
+            field_109: "2TD",
+            field_111: owning_org.old_visible_id,
+            field_113: managing_org.old_visible_id,
+            field_130: "1",
+            field_134: "2",
+            field_102: "2",
+            field_103: "1",
+            field_104: "1",
+            field_101: "1",
+            field_133: "2",
+            field_8: "1",
+            field_9: "2",
+            field_132: "1",
+
+            field_12: "42",
+            field_13: "41",
+            field_14: "20",
+            field_15: "18",
+            field_16: "16",
+            field_17: "14",
+            field_18: "12",
+            field_19: "20",
+
+            field_20: "F",
+            field_21: "M",
+            field_22: "F",
+            field_23: "M",
+            field_24: "F",
+            field_25: "M",
+            field_26: "F",
+            field_27: "M",
+
+            field_43: "17",
+            field_44: "18",
+
+            field_28: "P",
+            field_29: "C",
+            field_30: "X",
+            field_31: "R",
+            field_32: "C",
+            field_33: "C",
+            field_34: "X",
+
+            field_35: "1",
+            field_36: "2",
+            field_37: "6",
+            field_38: "7",
+            field_39: "8",
+            field_40: "9",
+            field_41: "0",
+            field_42: "10",
+
+            field_45: "1",
+            field_114: "4",
+            field_46: "1",
+
+            field_47: "1",
+
+            field_118: "2",
+
+            field_66: "5",
+            field_67: "2",
+            field_52: "31",
+            field_61: "3",
+            field_68: "12",
+
+            field_65: "1",
+            field_63: "EC1N",
+            field_64: "2TD",
+
+            field_69: "1",
+            field_70: "1",
+            field_71: "",
+            field_72: "1",
+            field_73: "",
+            field_74: "",
+
+            field_75: "1",
+            field_76: "2",
+            field_77: "2",
+
+            field_78: "2",
+
+            field_51: "1",
+            field_50: "2000",
+            field_116: "2",
+            field_48: "1",
+            field_49: "1",
+
+            field_79: "4",
+            field_80: "1234.56",
+            field_87: "1",
+            field_88: "234.56",
+          }
+        end
+
+        it "returns true" do
+          expect(parser).to be_valid
+        end
+
+        it "instantiates a log with everything completed", aggregate_failures: true do
+          questions = parser.send(:questions).reject do |q|
+            parser.send(:log).optional_fields.include?(q.id) || q.completed?(parser.send(:log))
+          end
+
+          expect(questions.map(&:id).size).to eq(0)
+          expect(questions.map(&:id)).to eql([])
+        end
+      end
+    end
+
+    describe "#field_1" do
       context "when null" do
-        let(:attributes) { { field_1: nil } }
+        let(:attributes) { { bulk_upload:, field_1: nil } }
+
+        it "returns an error" do
+          expect(parser.errors[:field_1]).to be_present
+        end
+      end
+
+      context "when incorrect data type" do
+        let(:attributes) { { bulk_upload:, field_1: "foo" } }
 
         it "returns an error" do
-          expect(parser.errors).to include(:field_1)
+          expect(parser.errors[:field_1]).to be_present
         end
       end
 
-      context "when outside permited range" do
+      context "when unpermitted value" do
-        let(:attributes) { { field_1: "13" } }
+        let(:attributes) { { bulk_upload:, field_1: "101" } }
 
         it "returns an error" do
-          expect(parser.errors).to include(:field_1)
+          expect(parser.errors[:field_1]).to be_present
         end
       end
 
       context "when valid" do
-        let(:attributes) { { field_1: 1 } }
+        let(:attributes) { { bulk_upload:, field_1: "1" } }
 
-        it "is valid" do
+        it "does not return any errors" do
-          expect(parser.errors).not_to include(:field_1)
+          expect(parser.errors[:field_1]).to be_blank
         end
       end
     end
 
-    describe "field_4" do
+    describe "#field_4" do
-      context "when text" do
+      context "when nullable not permitted" do
-        let(:attributes) { { field_4: "R" } }
+        let(:attributes) { { bulk_upload:, field_1: "2", field_4: nil } }
 
-        it "is not valid" do
+        it "cannot be nulled" do
-          expect(parser.errors).to include(:field_4)
+          expect(parser.errors[:field_4]).to be_present
         end
       end
 
-      context "when valid" do
+      context "when nullable permitted" do
-        let(:attributes) { { field_4: "3" } }
+        let(:attributes) { { bulk_upload:, field_1: "1", field_4: nil } }
+
+        it "can be nulled" do
+          expect(parser.errors[:field_4]).to be_blank
+        end
+      end
+
+      context "when matching scheme cannot be found" do
+        let(:attributes) { { bulk_upload:, field_1: "1", field_4: "123" } }
+
+        xit "returns an error" do
+          expect(parser.errors[:field_4]).to be_present
+        end
+      end
+    end
+
+    describe "#field_7" do
+      context "when null" do
+        let(:attributes) { { bulk_upload:, field_7: nil } }
 
-        it "is valid" do
+        xit "returns an error" do
-          expect(parser.errors).not_to include(:field_4)
+          expect(parser.errors[:field_7]).to be_present
         end
       end
+    end
 
-      context "when allowed to be null" do
+    describe "#field_10" do
-        let(:attributes) { { field_1: "2", field_4: "" } }
+      context "when field_9 is 3 aka other" do
+        let(:attributes) { { bulk_upload:, field_9: "3" } }
 
-        it "is valid" do
+        xit "returns an error" do
-          expect(parser.errors).not_to include(:field_4)
+          expect(parser.errors[:field_10]).to be_present
         end
       end
+    end
 
-      context "when not allowed to be null" do
+    describe "fields 96, 97, 98 => startdate" do
-        let(:attributes) { { field_1: "3", field_4: "" } }
+      context "when any one of these fields is blank" do
+        let(:attributes) { { bulk_upload:, field_96: nil, field_97: nil, field_98: nil } }
 
-        it "is not valid" do
+        it "returns an error" do
-          expect(parser.errors).to include(:field_4)
+          expect(parser.errors[:field_96]).to be_present
+          expect(parser.errors[:field_97]).to be_present
+          expect(parser.errors[:field_98]).to be_present
         end
       end
     end
 
     describe "#field_134" do
-      context "when not a possible value" do
+      context "when an unpermitted value" do
-        let(:attributes) { { field_134: "3" } }
+        let(:attributes) { { bulk_upload:, field_134: 3 } }
+
+        it "has errors on the field" do
+          expect(parser.errors[:field_134]).to be_present
+        end
+      end
+    end
+
+    describe "#field_103" do
+      context "when null" do
+        let(:attributes) { setup_section_params.merge({ field_103: nil }) }
+
+        it "returns an error" do
+          expect(parser.errors[:field_103]).to be_present
+        end
+      end
+
+      context "when unpermitted values" do
+        let(:attributes) { setup_section_params.merge({ field_103: "4" }) }
+
+        it "returns an error" do
+          expect(parser.errors[:field_103]).to be_present
+        end
+      end
+    end
+  end
+
+  describe "#log" do
+    describe "#cbl" do
+      context "when field_75 is yes ie 1" do
+        let(:attributes) { { bulk_upload:, field_75: 1 } }
+
+        it "sets value to 1" do
+          expect(parser.log.cbl).to be(1)
+        end
+      end
+
+      context "when field_75 is no ie 2" do
+        let(:attributes) { { bulk_upload:, field_75: 2 } }
+
+        it "sets value to 0" do
+          expect(parser.log.cbl).to be(0)
+        end
+      end
+    end
+
+    describe "#chr" do
+      context "when field_76 is yes ie 1" do
+        let(:attributes) { { bulk_upload:, field_76: 1 } }
+
+        it "sets value to 1" do
+          expect(parser.log.chr).to be(1)
+        end
+      end
+
+      context "when field_76 is no ie 2" do
+        let(:attributes) { { bulk_upload:, field_76: 2 } }
+
+        it "sets value to 0" do
+          expect(parser.log.chr).to be(0)
+        end
+      end
+    end
+
+    describe "#cap" do
+      context "when field_77 is yes ie 1" do
+        let(:attributes) { { bulk_upload:, field_77: 1 } }
+
+        it "sets value to 1" do
+          expect(parser.log.cap).to be(1)
+        end
+      end
+
+      context "when field_77 is no ie 2" do
+        let(:attributes) { { bulk_upload:, field_77: 2 } }
+
+        it "sets value to 0" do
+          expect(parser.log.cap).to be(0)
+        end
+      end
+    end
+
+    describe "#letting_allocation_unknown" do
+      context "when field_75, 76, 77 are no ie 2" do
+        let(:attributes) { { bulk_upload:, field_75: 2, field_76: 2, field_77: 2 } }
+
+        it "sets value to 1" do
+          expect(parser.log.letting_allocation_unknown).to be(1)
+        end
+      end
+
+      context "when any one of field_75, 76, 77 is yes ie 1" do
+        let(:attributes) { { bulk_upload:, field_75: 1 } }
+
+        it "sets value to 0" do
+          expect(parser.log.letting_allocation_unknown).to be(0)
+        end
+      end
+    end
+
+    describe "#renewal" do
+      context "when field_134 is no ie 2" do
+        let(:attributes) { { bulk_upload:, field_134: 2 } }
+
+        it "sets value to 0" do
+          expect(parser.log.renewal).to eq(0)
+        end
+      end
+
+      context "when field_134 is null but rsnvac/field_116 is 14" do
+        let(:attributes) { { bulk_upload:, field_134: "", field_116: "14" } }
+
+        it "sets renewal to 1" do
+          expect(parser.log.renewal).to eq(1)
+        end
+      end
+    end
+
+    describe "#sexN fields" do
+      let(:attributes) do
+        {
+          bulk_upload:,
+          field_20: "F",
+          field_21: "M",
+          field_22: "X",
+          field_23: "R",
+          field_24: "F",
+          field_25: "M",
+          field_26: "X",
+          field_27: "R",
+        }
+      end
+
+      it "sets value from correct mapping" do
+        expect(parser.log.sex1).to eql("F")
+        expect(parser.log.sex2).to eql("M")
+        expect(parser.log.sex3).to eql("X")
+        expect(parser.log.sex4).to eql("R")
+        expect(parser.log.sex5).to eql("F")
+        expect(parser.log.sex6).to eql("M")
+        expect(parser.log.sex7).to eql("X")
+        expect(parser.log.sex8).to eql("R")
+      end
+    end
+
+    describe "#ecstatN fields" do
+      let(:attributes) do
+        {
+          bulk_upload:,
+          field_35: "1",
+          field_36: "2",
+          field_37: "6",
+          field_38: "7",
+          field_39: "8",
+          field_40: "9",
+          field_41: "0",
+          field_42: "10",
+        }
+      end
+
+      it "sets value from correct mapping", aggregate_failures: true do
+        expect(parser.log.ecstat1).to eq(1)
+        expect(parser.log.ecstat2).to eq(2)
+        expect(parser.log.ecstat3).to eq(6)
+        expect(parser.log.ecstat4).to eq(7)
+        expect(parser.log.ecstat5).to eq(8)
+        expect(parser.log.ecstat6).to eq(9)
+        expect(parser.log.ecstat7).to eq(0)
+        expect(parser.log.ecstat8).to eq(10)
+      end
+    end
+
+    describe "#relatN fields" do
+      let(:attributes) do
+        {
+          bulk_upload:,
+          field_28: "P",
+          field_29: "C",
+          field_30: "X",
+          field_31: "R",
+          field_32: "P",
+          field_33: "C",
+          field_34: "X",
+        }
+      end
+
+      it "sets value from correct mapping", aggregate_failures: true do
+        expect(parser.log.relat2).to eq("P")
+        expect(parser.log.relat3).to eq("C")
+        expect(parser.log.relat4).to eq("X")
+        expect(parser.log.relat5).to eq("R")
+        expect(parser.log.relat6).to eq("P")
+        expect(parser.log.relat7).to eq("C")
+        expect(parser.log.relat8).to eq("X")
+      end
+    end
+
+    describe "#net_income_known" do
+      let(:attributes) { { bulk_upload:, field_51: "1" } }
+
+      it "sets value from correct mapping" do
+        expect(parser.log.net_income_known).to eq(0)
+      end
+    end
+
+    describe "#unitletas" do
+      let(:attributes) { { bulk_upload:, field_105: "1" } }
+
+      it "sets value from correct mapping" do
+        expect(parser.log.unitletas).to eq(1)
+      end
+    end
+
+    describe "#rsnvac" do
+      let(:attributes) { { bulk_upload:, field_106: "5" } }
+
+      it "sets value from correct mapping" do
+        expect(parser.log.rsnvac).to eq(5)
+      end
+    end
+
+    describe "#sheltered" do
+      let(:attributes) { { bulk_upload:, field_117: "1" } }
+
+      it "sets value from correct mapping" do
+        expect(parser.log.sheltered).to eq(1)
+      end
+    end
+
+    describe "illness fields" do
+      mapping = [
+        { attribute: :illness_type_1, field: :field_119 },
+        { attribute: :illness_type_2, field: :field_120 },
+        { attribute: :illness_type_3, field: :field_121 },
+        { attribute: :illness_type_4, field: :field_122 },
+        { attribute: :illness_type_5, field: :field_123 },
+        { attribute: :illness_type_6, field: :field_124 },
+        { attribute: :illness_type_7, field: :field_125 },
+        { attribute: :illness_type_8, field: :field_126 },
+        { attribute: :illness_type_9, field: :field_127 },
+        { attribute: :illness_type_10, field: :field_128 },
+      ]
+
+      mapping.each do |hash|
+        describe "##{hash[:attribute]}" do
+          context "when yes" do
+            let(:attributes) { { bulk_upload:, hash[:field] => "1" } }
+
+            it "sets value from correct mapping" do
+              expect(parser.log.public_send(hash[:attribute])).to eq(1)
+            end
+          end
+
+          context "when no" do
+            let(:attributes) { { bulk_upload:, hash[:field] => "" } }
+
+            it "sets value from correct mapping" do
+              expect(parser.log.public_send(hash[:attribute])).to be_nil
+            end
+          end
+        end
+      end
+    end
+
+    describe "#irproduct_other" do
+      let(:attributes) { { bulk_upload:, field_131: "some other product" } }
+
+      it "sets value to given free text string" do
+        expect(parser.log.irproduct_other).to eql("some other product")
+      end
+    end
+
+    describe "#tenancyother" do
+      let(:attributes) { { bulk_upload:, field_10: "some other tenancy" } }
+
+      it "sets value to given free text string" do
+        expect(parser.log.tenancyother).to eql("some other tenancy")
+      end
+    end
+
+    describe "#tenancylength" do
+      let(:attributes) { { bulk_upload:, field_11: "2" } }
+
+      it "sets value to given free text string" do
+        expect(parser.log.tenancylength).to eq(2)
+      end
+    end
+
+    describe "#earnings" do
+      let(:attributes) { { bulk_upload:, field_50: "104.50" } }
+
+      it "rounds to the nearest whole pound" do
+        expect(parser.log.earnings).to eq(105)
+      end
+    end
+
+    describe "#reasonother" do
+      let(:attributes) { { bulk_upload:, field_53: "some other reason" } }
+
+      it "sets value to given free text string" do
+        expect(parser.log.reasonother).to eql("some other reason")
+      end
+    end
+
+    describe "#ppcodenk" do
+      let(:attributes) { { bulk_upload:, field_65: "2" } }
+
+      it "sets correct value from mapping" do
+        expect(parser.log.ppcodenk).to eq(0)
+      end
+    end
+
+    describe "#household_charge" do
+      let(:attributes) { { bulk_upload:, field_86: "1" } }
+
+      it "sets correct value from mapping" do
+        expect(parser.log.household_charge).to eq(1)
+      end
+    end
+
+    describe "#chcharge" do
+      let(:attributes) { { bulk_upload:, field_85: "123.45" } }
+
+      it "sets value given" do
+        expect(parser.log.chcharge).to eq(123.45)
+      end
+    end
+
+    describe "#tcharge" do
+      let(:attributes) { { bulk_upload:, field_84: "123.45" } }
+
+      it "sets value given" do
+        expect(parser.log.tcharge).to eq(123.45)
+      end
+    end
+
+    describe "#supcharg" do
+      let(:attributes) { { bulk_upload:, field_83: "123.45" } }
+
+      it "sets value given" do
+        expect(parser.log.supcharg).to eq(123.45)
+      end
+    end
+
+    describe "#pscharge" do
+      let(:attributes) { { bulk_upload:, field_82: "123.45" } }
+
+      it "sets value given" do
+        expect(parser.log.pscharge).to eq(123.45)
+      end
+    end
+
+    describe "#scharge" do
+      let(:attributes) { { bulk_upload:, field_81: "123.45" } }
+
+      it "sets value given" do
+        expect(parser.log.scharge).to eq(123.45)
+      end
+    end
+
+    describe "#offered" do
+      let(:attributes) { { bulk_upload:, field_99: "3" } }
+
+      it "sets value given" do
+        expect(parser.log.offered).to eq(3)
+      end
+    end
+
+    describe "#propcode" do
+      let(:attributes) { { bulk_upload:, field_100: "abc123" } }
+
+      it "sets value given" do
+        expect(parser.log.propcode).to eq("abc123")
+      end
+    end
+
+    describe "#mrcdate" do
+      let(:attributes) { { bulk_upload:, field_92: "13", field_93: "12", field_94: "22" } }
+
+      it "sets value given" do
+        expect(parser.log.mrcdate).to eq(Date.new(2022, 12, 13))
+      end
+    end
+
+    describe "#majorrepairs" do
+      context "when mrcdate given" do
+        let(:attributes) { { bulk_upload:, field_92: "13", field_93: "12", field_94: "22" } }
+
+        it "sets #majorrepairs to 1" do
+          expect(parser.log.majorrepairs).to eq(1)
+        end
+      end
+
+      context "when mrcdate not given" do
+        let(:attributes) { { bulk_upload:, field_92: "", field_93: "", field_94: "" } }
+
+        it "sets #majorrepairs to 0" do
+          expect(parser.log.majorrepairs).to eq(0)
+        end
+      end
+    end
+
+    describe "#voiddate" do
+      let(:attributes) { { bulk_upload:, field_89: "13", field_90: "12", field_91: "22" } }
+
+      it "sets value given" do
+        expect(parser.log.voiddate).to eq(Date.new(2022, 12, 13))
+      end
+    end
+
+    describe "#startdate" do
+      let(:attributes) { { bulk_upload:, field_96: now.day.to_s, field_97: now.month.to_s, field_98: now.strftime("%g") } }
+
+      it "sets value given" do
+        expect(parser.log.startdate).to eq(now)
+      end
+    end
+
+    describe "#postcode_full" do
+      let(:attributes) { { bulk_upload:, field_108: " EC1N ", field_109: " 2TD " } }
+
+      it "strips whitespace" do
+        expect(parser.log.postcode_full).to eql("EC1N 2TD")
+      end
+    end
+
+    describe "#la" do
+      let(:attributes) { { bulk_upload:, field_107: "E07000223" } }
+
+      it "sets to given value" do
+        expect(parser.log.la).to eql("E07000223")
+      end
+    end
+
+    describe "#prevloc" do
+      let(:attributes) { { bulk_upload:, field_62: "E07000223" } }
+
+      it "sets to given value" do
+        expect(parser.log.prevloc).to eql("E07000223")
+      end
+    end
+
+    describe "#previous_la_known" do
+      context "when known" do
+        let(:attributes) { { bulk_upload:, field_62: "E07000223" } }
+
+        it "sets to 1" do
+          expect(parser.log.previous_la_known).to eq(1)
+        end
+      end
+
+      context "when not known" do
+        let(:attributes) { { bulk_upload:, field_62: "" } }
+
+        it "sets to 0" do
+          expect(parser.log.previous_la_known).to eq(0)
+        end
+      end
+    end
+
+    describe "#first_time_property_let_as_social_housing" do
+      context "when field_106 is 15, 16, or 17" do
+        let(:attributes) { { bulk_upload:, field_106: %w[15 16 17].sample } }
 
-        it "is not valid" do
+        it "sets to 1" do
-          expect(parser.errors).to include(:field_134)
+          expect(parser.log.first_time_property_let_as_social_housing).to eq(1)
         end
       end
     end

spec/services/bulk_upload/lettings/validator_spec.rb

@@ -30,7 +30,7 @@ RSpec.describe BulkUpload::Lettings::Validator do
   end
 
   context "when a valid csv" do
-    let(:path) { file_fixture("2021_22_lettings_bulk_upload.csv") }
+    let(:path) { file_fixture("2022_23_lettings_bulk_upload.csv") }
 
     it do
       validator.call

spec/services/bulk_upload/processor_spec.rb

@@ -11,7 +11,7 @@ RSpec.describe BulkUpload::Processor do
         instance_double(
           BulkUpload::Downloader,
           call: nil,
-          path: file_fixture("2021_22_lettings_bulk_upload.csv"),
+          path: file_fixture("2022_23_lettings_bulk_upload.csv"),
           delete_local_file!: nil,
         )
       end
@@ -19,7 +19,7 @@ RSpec.describe BulkUpload::Processor do
       it "persist the validation errors" do
         allow(BulkUpload::Downloader).to receive(:new).with(bulk_upload:).and_return(mock_downloader)
 
-        expect { processor.call }.to change(BulkUploadError, :count).by(9)
+        expect { processor.call }.to change(BulkUploadError, :count)
       end
 
       it "deletes the local file afterwards" do