submit-social-housing-lettings-and-sales-data

Commit Graph

Author	SHA1	Message	Date
Phil Lee	45128b7250	Refactor with split of log to CSV by log type (#1593 ) # Context - Partially related to https://digital.dclg.gov.uk/jira/browse/CLDC-2316 - Comprehending sales or lettings bulk upload CSV is enough, comprehending both simultaneously is rather challenging # Changes - Split out test helper class by log type ie lettings/sales	2 years ago
Phil Lee	cae731ed6f	CLDC-1888 Bulk upload handles duplicates (#1334 ) # Context - https://digital.dclg.gov.uk/jira/browse/CLDC-1888 - This is a continuation of https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/1277 - When bulk uploading we want to check users are not uploading data that already exists to prevent them submitting duplicate # Changes - This feature is behind a feature toggle. it has been disabled for staging for testing purposes but available in all other environments - If a log already exists based off certain fields add errors to the associated fields - We discount any hidden logs and only check "active" logs - Added memoization to `#valid?` as an optimisation	2 years ago
Phil Lee	5862e02d1b	bulk upload declaration validated as setup field (#1583 )	2 years ago
Phil Lee	454df8389e	CLDC-2135 Bulk upload resume with how fix (#1503 ) * add first page for bulk upload resume journey * bulk upload resume handles upload again * add confirm page to bulk upload resume journey * replace placeholder count with correct value * apply recommendation for bulk upload resume choice * add how to fix bulk upload mailer * integrate new bulk upload approve journey * add missing bulk upload error mappings * remove test * prevent approve being called multiple times * bulk upload creates invisible logs ahead of time * work invisible logs into bulk upload flow * sort errors so deterministic * remove unused ensure * remove expected_log_count and processed - these fields are no longer used or needed * introduce pending status * swap visible for pending logs * only show visible lettings logs * hard code status filters * remove unused model methods * only show visible sales logs * form controller ignores hidden logs * locations and schemes only affect visible logs	2 years ago
kosiakkatrina	ae9a4d0638	Fix collection year shift issues (#1506 ) * Fix possible bulk upload years * Initialise 2021/22 form routes for tests * Return test routes from form handler * Fix some tests * fix bulk upload tests - change from mocks to actual code calls * update years for bulk upload form * Fix some uprn tests * Set the scheme as available from previous year if the collection is in crossover * Update location available_from * Update location and scheme tests * Refactor and comment out a scheme test * Some uprn and export tests * Fix lettings logs controller tests and comment out previous collection message test * fix tests for new collection year * fix bulk upload tests for some reason tenancylength must not be given for validation to pass * update bulk upload tests for new year * More fixes * Undo location changes * Fix remaining uprn * further new collection year test fixes --------- Co-authored-by: Phil Lee <asmega@users.noreply.github.com>	2 years ago
Phil Lee	98cc162dda	bulk upload checks column count of CSV (#1423 )	2 years ago
Phil Lee	fa64694e0c	CLDC-1889 Bulk upload setup (#1399 ) * bulk upload setup errors only for missing data * bulk upload setup error for scheme * bulk upload setup error for location * ensure missing startdate is a setup error	2 years ago
Phil Lee	2aee5a80c5	CLDC-1768 Bulk upload file setup email (#1358 ) * add placeholder tests for bulk upload mailer * bulk upload fix setup errors email - this plumbs in the condition so if any setup sections are incomplete we send that partcular email and prevent the remaining flow * tag bulk upload setup errors for downsteam use * add category to bulk upload errors * persist bulk upload error category * populate bulk upload mailer with errors	2 years ago
Phil Lee	c481055f4d	CLDC-1965 Bulk upload correct again email (#1274 ) * tests for bulk upload mailer * bulk upload send upload again email	2 years ago
Phil Lee	a87dc14271	CLDC-1762 Bulk upload fix errors email (#1267 ) * refactor bulk upload with extract method * add bulk upload mailer test * bulk upload mails users to fix logs on website * bulk uplad email change resume link * refactor local variable name	2 years ago
Phil Lee	5077dc035b	CLDC-1767 Bulk upload fail email (#1264 ) * bulk upload file validations send failure email * handle errors during bulk upload parsing	2 years ago
Phil Lee	90e0bad2b7	bulk upload sends success email (#1261 ) - this happens if the csv is perfect and all logs are created as completed	2 years ago
Phil Lee	0009d535fe	CLDC-1896 Bulk upload line endings (#1208 ) * add class to create logs from bulk upload * create logs when processing bulk uploads * remove bulk_upload_id from csv output * create bulk upload logs only if all valid - this will be changed later to allow for partial logs - and only to create logs when a threshold has been met * add method to blank invalid non setup fields * bulk upload log creation blanks invalid fields * fix incorrect logic for bulk upload renewal * fix linting * bulk upload log creation fail logs to sentry * fix bulk upload line ending parsing * extract bulk uploading csv parsing to class * use csv parser in log creator * change handle line endings mechanism - we now strip all windows line endings for unix based line endings - this normalises things making it simpler	2 years ago
Phil Lee	90480d1af5	CLDC-1779 Bulk upload lettings validation (#1148 ) * able to view lettings bulk upload errors * fix linting * call service correctly in test * add bulk upload sales questions mapping * appease linter * bulk upload error shows correct question - depending on log type it will show relevant question for the field concerned * improve namespacing of classes * add job to process bulk uploads * move validation from parser to model * add validations for field_1 * add validation for field_4 * pending tests for field_4 * convert field_mapping to array of hashes * validate nulls based on form question * actually load forms when toggling between forms * validate null for startdate * row parser has access to bulk upload * csv upload validates first form section * add postcode validation * Refactor error mappings for row parser * Add unittype question * Fix null error setting and add builtype * add wchair to bulk upload * Add beds to bulk upload * Add joint to bulk upload * Add startertenancy to the bulk upload * Add tenancy for bulk upload * Add declaration to the bulk upload * Add age1 and age1_known to bulk upload * add ages to bulk upload * add sex1 to bulk upload * add ethnic_group and ethnic to bulk upload * add national to bulk upload * add ecstat1 to bulk upload * add military related fields to bulk upload * add preg_occ to bulk upload * add housingneeds to bulk upload * add illness to bulk upload * add layear to bulk upload * add waityear to bulk upload * add reason to bulk upload * add prevten to bulk upload * add homeless to bulk upload * add previous postcode to bulk upload * add reasonable preferences to bulk upload * add allocations system to bulk upload * add referral to bulk upload * add net_income_known to bulk upload * add hb to bulk upload * add benefits to bulk upload * add rent fields to bulk upload * add hhmemb to bulk upload * use 2022 csv fixtures for bulk upload * fix renewal mapping for bulk upload * placeholder test for bulk upload validation * fix bulk upload mapping for homeless field * fix leftreg mapping for bulk upload * fix user associations in bulk upload tests * add gender fields for bulk upload * add ecstatN fields to bulk upload * add #relatN fields to bulk upload * extract old_visible_id in factory to trait * map net_income_known correctly for bulk upload * fix income bugs for bulk upload * add unitletas to bulk upload * add #rsnvac to bulk upload * add #sheltered to bulk upload * add illness fields to bulk upload * add #irproduct_other to bulk upload * infer renewal from rsnvac for bulk upload * add #tenancyother to bulk upload * add #tenancylength to bulk upload * bulk upload earnings accepts pennies but rounds * add #reasonother to bulk upload * fix mapping of #ppcodenk for bulk upload * add #household_charge to bulk upload * add #chcharge to bulk upload * add #tcharge to bulk upload * add #supcharg to bulk upload * add pscharge to bulk upload * add #scharge to bulk upload * use case statement for bulk upload allocation * add offered to bulk upload * add propcode to bulk upload * add major repair fields to bulk upload * add #voiddate to bulk upload * support YY year format for bulk upload * test postcode strips whitespace for bulk upload * add #la to bulk upload * add previous la to bulk upload * fix failing test * remove duplicate line from rebase * add first time social housing to bulk upload * make methods private * fix field_4 validation for bulk upload - the null check was inverted by mistake Co-authored-by: Kat <katrina@kosiak.co.uk>	2 years ago
Phil Lee	197e2ae4cb	[1778] Bulk upload errors (#1091 ) * lockdown bulk upload routes * able to view lettings bulk upload errors * add error count to bulk upload results * coverage for bulk upload filename on results * group bulk upload errors by row on results * able to view bulk upload sales results * scope lettings and sales bulk upload results * fix linting * call service correctly in test * add bulk upload sales questions mapping * appease linter * bulk upload error shows correct question - depending on log type it will show relevant question for the field concerned * use local disk for bulk upload for dev env - this saves the need to connect to S3 to play with bulk upload in dev environment * improve namespacing of classes * add job to process bulk uploads * use local disk storage for dev file upload * fix test * use inline active job queue_adapter for dev * use test active job queue adapter for test env * remove rubocop violation * delete bulk upload from disk after processing * populate errors with cell, row + metadata * update error message with something meaningful * shim in sales validator * able to parse sales bulk uploads * change migration to add purchase_code to errors * bulk upload error component renders purchaser code - when a sales log * populate purchaser_code for bulk upload errors - when log type is sales * remove superfluous private method	2 years ago

15 Commits (751f8661db69276d4b12810fb99d1b6910755169)