submit-social-housing-lettings-and-sales-data

Commit Graph

Author	SHA1	Message	Date
Nat Dean-Lewis	1c6860f90b	CLDC-4140: update csv parser spec	4 days ago
Samuel Young	98c58cc19a	CLDC-4142: New question on gender same as sex - lettings (#3161 ) * CLDC-4142: Add new cols to database * CLDC-4142: Add questions to flow optimise household_characteristics as there's a lot of repeated code. should make the sections a bit easier to manage * CLDC-4142: Add conditional description question seems to largely just work without needing major tweaks * CLDC-4142: Ensure description if nil if not shown looks like there's no automated process for this. without this the description will be "" if not given * CLDC-4142: Add new question to log factory * CLDC-4142: Add new fields to BU parser update BU template file to match * CLDC-4142: Add new fields to BU file generator * CLDC-4142: Add new fields to xml export also improve xml export matcher to report differences * CLDC-4142: Add final question numbers * CLDC-4142: Add question specs * CLDC-4142: Add a PERSON_DETAILS override for gender_same_as_sex this ensures if details aren't known the field will be redacted * CLDC-4142: Update model tests * CLDC-4142: Update CSV export service tests * CLDC-4142: Update other tests * fixup! CLDC-4142: Update CSV export service tests update sales variable definitons too * fixup! CLDC-4142: Ensure description if nil if not shown update inferral method name and comment * CLDC-4142: Update CYA copy as requested cya for gender_same_as_sex 'no' instead of 'no, enter gender description' cya for gender_description has the label 'gender identity description' * CLDC-4142: Use label_from_value for custom CYA text this replaces @inferred_check_answers_value so it'll work in CSV export too	5 days ago
Nat Dean-Lewis	129033faf1	CLDC-4140: update csv-parser spec	5 days ago
Nat Dean-Lewis	8c79d152b2	CLDC-4141: New sex registered at birth question for lettings logs (#3171 ) * CLDC-4177: Create new sex-at-birth question * CLDC-4177: Update database * CLDC-4177: Adds SAB form for the primary buyer. * CLDC-4177: Adds sex-at-birth question for other household members * CLCD-4177: Unit tests * CLDC-4177: Misc inclusions where sex is also used * CLDC-4177: Lint fixes * CLDC-4177: Do not include sexRAB in pre-2026 exports * CLDC-4177: Fixes bug with copy key in new page * CLDC-4177: Changes logic used to decide which fields to export * CLDC-4177: Linting fixes * CLDC-4177: Converts new field to lowercase * CLDC-4177: Satisfy bulk upload tests (not tested manually) * CLDC-4177: Rename new fields to lowercase * CLDC-4177: Rename field in bulk upload. * CLDC-4141: Update database * CLDC-4141: New question and page models * CLDC-4141: Adds the new sab questions to the form * CLDC-4141: Update locales * CLDC-4141: Export sab questions * CLDC-4141: change buyer -> lead tenant for lettings log sexrab1 * CLDC-4141: enable sexrab questions for lettings log persons 2-8 * CLDC-4141: update row parser * CLDC-4141: update csv service * Fix db casing * CLDC-4141: update schema (nonfunctional change) * CLDC-4141: update row parser expected fields * CLDC-4141: use correct lettings q * CLDC-4141: update page and question tests * CLDC-4141: update subsection test * CLDC-4141: update row parser test * CLDC-4141: linting * CLDC-4141: remove frozen string literals * CLDC-4177: remove frozen string literals * CLDC-4141: fix person casing typo * Delete redundant migration * CLDC-4141: revert fields count change * CLDC-4141: update row parser spec * CLDC-4141: revert export service change * Revert "CLDC-4141: revert export service change" This reverts commit `c43ff40006`. * CLDC-4177: Create new sex-at-birth question * CLDC-4177: Update database * CLDC-4177: Adds SAB form for the primary buyer. * CLDC-4177: Adds sex-at-birth question for other household members * CLCD-4177: Unit tests * CLDC-4177: Misc inclusions where sex is also used * CLDC-4177: Lint fixes * CLDC-4177: Do not include sexRAB in pre-2026 exports * CLDC-4177: Fixes bug with copy key in new page * CLDC-4177: Changes logic used to decide which fields to export * CLDC-4177: Linting fixes * CLDC-4177: Converts new field to lowercase * CLDC-4177: Satisfy bulk upload tests (not tested manually) * CLDC-4177: Rename new fields to lowercase * CLDC-4177: Rename field in bulk upload. * Fix db casing * CLDC-4177: remove frozen string literals * Delete redundant migration * Remove extraneous lines from schema.rb * Add new sexrab fields to the :completed trait in the Sales Log factory * CLDC-4141: partial csv download test update * CLDC-4141: update csv conversion rake task * CLDC-4141: untrack temp csv export files * CLDC-4141: update remaining csv tests * CLDC-4141: fix schema since latest changes * CLDC-4141: update schema * CLDC-4141: revert breaking export service changes * CLDC-4141: update var def spec * CLDC-4141: update var def spec * CLDC-4141: update var def spec * CLDC-4141: update BU test file * CLDC-4141: update validator specs * CLDC-4141: update csv_parser columns * CLDC-4141: linting * CLDC-4141: linting * CLDC-4141: linting * CLDC-4141: update person_n_present methods * CLDC-4141: remove redundant MAX_COLUMNS check * CLDC-4141: avoid sales changes in this PR, will leave to CLDC-4177 * CLDC-4141: use FIELDS const where can be shared * CLDC-4141: revert FIELDS sharing * CLDC-4141: update export services tests * CLDC-4141: include fields rather than omitting in lettings_log_export_service.rb * CLDC-4141: share default 2026 field count * CLDC-4141: cleanup export field filtering * CLDC-4141: simplify export year filtering * CLDC-4141: use FIELDS to check max_columns * CLDC-4141: update 2026_27 manual test fixture * CLDC-4177: remove redundant inferred CYA vals * CLDC-4141: remove redundant inferred CYA vals * CLDC-4177: update bu files * CLDC-4177: update sales log export service * CLDC-4141: remove redundant field filtering * CLDC-4141: reinstate old comment * CLDC-4141: reinstate old comment * CLDC-4141: rename var def file * CLDC-4141: update variable definitions tests * CLDC-4141: update schema * CLDC-4177: update schema * CLDC-4177: linting * CLDC-4177: linting * CLDC-4141: update csv fixtures * CLDC-4177: update sales log export constants * CLDC-4177: update factories * CLDC-4177: lint * CLDC-4177: csv parser updates * CLDC-4177: bu field updates * CLDC-4177: update var def spec * CLDC-4141: Add final question numbers (CLDC-4202) * CLDC-4141: make to_2026_row explicit * CLDC-4177: make to_2026_row explicit * CLDC-4177: use shared field count * CLDC-4177: refactor to remove max cols going forwards * CLDC-4141: update schema * CLDC-4141: update schema * CLDC-4141: update schema * CLDC-4141: update schema * CLDC-4141: remove X from factory options * CLDC-4177: update log var defs * CLDC-4177: use correct shared field count * CLDC-4177: remove trailing commas * CLDC-4177: update log var defs * CLDC-4177: update bulk upload csv example * CLDC-4141: update bulk upload csv example * CLDC-4177: update bulk upload csv example * CLDC-4177: remove sexrab from expected sales export for now * CLDC-4141: correct csv service expectations * CLDC-4141: don't sample X for sexrab --------- Co-authored-by: Katherine Langford <katherine.langford@softwire.com> Co-authored-by: oscar-richardson-softwire <oscar.richardson@softwire.com> Co-authored-by: Samuel Young <samuel.young@softwire.com>	2 weeks ago
Samuel Young	a513984440	CLDC-4162: Set up Bulk Uploads for 2026 (#3134 ) * CLDC-4162: Add parsers for 2026 * CLDC-4162: Integrate 2026 parsers into codebase * CLDC-4162: Add tests * CLDC-4162: Update tests for existing years remove year dependencies where possible * CLDC-4162: Add example 2026 bulk upload files * CLDC-4162: Update BU test file generate buttons * fixup! CLDC-4162: Update BU test file generate buttons we only need the 14 days validation when the future forms feature toggle is off, otherwise any date can be input * fixup! CLDC-4162: Integrate 2026 parsers into codebase improve TODOs * fixup! CLDC-4162: Add parsers for 2026 add todos for unknown info * fixup! CLDC-4162: Update tests for existing years properly mark year specific tests * fixup! CLDC-4162: Update tests for existing years use 25 over 2025 * fixup! CLDC-4162 Update BU test file generate buttons --------- Co-authored-by: carolynbarker <8038496+carolynbarker@users.noreply.github.com>	1 month ago
kosiakkatrina	71c3f41bf9	CLDC-3803 Add lettings 25/26 BU (#2939 ) * Copy row and csv parser for 2025 lettings * Add 2025 BU translations file * Update row parser field number * Add "apecified accommodation" to rent type mapping * Update csv parser * Add prepare your file and connect the csv parser to log creator and validator * Update test file creation * Refactor some csv methods, add person relationship mapping * Update row parser specs * Further refactor some test helper methods * Merge update	12 months ago
kosiakkatrina	5d36a13e7c	Ignore irrelevant empty headers (#2800 )	1 year ago
kosiakkatrina	c3bfadfe7f	Empty bulk upload row parsing (#2638 ) * empty * Add stripped_row back * Skip empty rows	1 year ago
kosiakkatrina	8efef3bcf5	CLDC-3190 Add lettings BU for 24/25 (#2158 ) * Copy 23 BU files to 24 * Update field mapping * Add duplicate log error to charges * Check the sum of charges for duplicates * update stub * Only add charges to duplciate hash if they exist * Rebase change * Update columns * Add tcharge back to error mapping * Add prepare your file * Add csv parser 24/25 cases * Fix date check * Rebase fix	2 years ago
natdeanlewissoftwire	19335a2367	CLDC-2438 Fix carriage return csv bug (#1685 ) * feat: add extra subs to row_parsers * feat: add tests * refactor: gsub not tr * feat: remove redundant sub * feat: simplify replacement * feat: scrub before tr to fix invalid byte error * feat: add !	3 years ago
Phil Lee	cf9d9550e9	CLDC-2344 Remove bulk upload minimum column check (#1621 ) # Context - https://digital.dclg.gov.uk/jira/browse/CLDC-2344 - So the reported bug and these change do not match 100% - As the reported bug should have already been resolved in an earlier change as the error message that was reported has already been removed from the system - These changes remove the check from sales plus a few minor tidy ups # Changes - Remove dead code around error threshold which is no longer used or needed - Inverted negative predicate to positive one, `incorrect_field_count` => `correct_field_count` - Remove `validate_min_columns` from sales bulk upload	3 years ago
Phil Lee	45128b7250	Refactor with split of log to CSV by log type (#1593 ) # Context - Partially related to https://digital.dclg.gov.uk/jira/browse/CLDC-2316 - Comprehending sales or lettings bulk upload CSV is enough, comprehending both simultaneously is rather challenging # Changes - Split out test helper class by log type ie lettings/sales	3 years ago
SamSeed-Softwire	321ba71e0c	CLDC-2197 Improve column counting in bulk upload (#1557 ) * feat: add validation for number of field labels * feat: check field labels are numbers * feat: remove validations for number of columns * refactor: rename non_blank_fields_count to valid_field_numbers_count * test: add functions to generate custom field labels/values * test: that extra invalid field labels don't cause issues * test: that removing a valid field label reduces count by 1 * test: that adding a valid field label increases count by 1 * refactor: rename validate_fields_count and wrong_field_count * feat: add validation that max col count not exceeded when no headers * fix: convert numbers to strings in default_field_numbers * feat: add leniency to max cols count (in case of 1 extra col) * test: explicitly set year of bulk upload to be 2022 * test: add/update tests in validator_spec * chore: lint * test: remove tests from csv_parser_spec that were moved to validator_spec * feat: update 2022 csv_parser to work with new validations * refactor: define number of valid 2022 fields in one place * refactor: remove redundant headers definition * feat: update 2022 csv parser to have col flexibility like 2023 * test: for validator 2022 as well as 2023 * test: simplify 2022/2023 logic and improve layout * chore: lint * test: ensure context descriptions start with "when" * refactor: check fields/columns count within csv parser, not validator * test: update 2022 csv parser tests to work like 2023 * chore: lint	3 years ago
Phil Lee	35de06e324	CLDC-2067 Bulk upload for lettings 23/24 (#1417 ) * refactor questions from validator to row parser * able to switch between bulk upload parsers - depending on what year we are processing * spec tooling to support bulk upload multi year * row parser now has year in namespacing * add static data for 2023 row parser * add placeholder to log to csv for specs * bulk upload aribtrary 23/34 column ordering works * bulk upload supports 23/24 without headers * bulk upload 23/24 supports BOM + invalid chars * dupe tests * port 23/24 attributes_for_log * port 23/24 bulk upload validations * force crossover period * tweak max permitted columns * able to return column for given field * work out column for field for errors * add field_4 as 23/24 setup field * remove duplicate method * map schemes and locations correctly * handle arbitrary number of header rows * add missing fields to bulk upload support	3 years ago

4 Commits (1c6860f90b0cbc330883e297dcbfe3ea1aba8353)