submit-social-housing-lettings-and-sales-data

Commit Graph

Author	SHA1	Message	Date
kosiakkatrina	79c4f9b74b	CLDC-2334 Add journey to fix soft validation errors only (#1615 ) * Sends a correct email if there are only soft validations * Add soft errors valid page * Add soft errors confirm page * Confirm the soft validations * Reuse the how to fix template for check soft validations email * Update email link * Move soft validation confirmation to processor * Correctly set the log status, remove redundant confirm_soft_validations * Redirect successful upload to logs index and display a success banner * Implement the soft validations only journey for sales logs * Display the soft validation errors on the soft-errors-valid page * Fix page alignment * Fix tests by mapping housingneeds in csv hepler * Add the sales soft validations to unpend_and_confirm_soft_validations * Change naming * Update method names * refactor * undo typo * Only set the existing soft validations for correct types * Fix path name * Add missing error mappings for location fields * Add missing tests and cancel button * Change some wording * Typos	2 years ago
Phil Lee	d939e88435	CLDC-2316 Bulk upload sales 2023 (#1603 ) # Context - https://digital.dclg.gov.uk/jira/browse/CLDC-2316 - Implement bulk upload sales for new collection year 2023 - This is a first pass implementation and will probably have some bugs in it and we can address over time # Changes - Add CSV parser for sales 2023 to handle CSV structure - Tweak collection window validation so error now contextual to year selected for upload - Handle arbitrary ordering of CSV columns - Fix ordering of errors in report and now ordered by cell - Added `Upload your file again` link styled as button on error report to match lettings experience - Update tooling to convert logs to 2023 csv rows with support for random column ordering # Known issues - There seem to be some issues with how UPRN is handled if the UPRN cannot be validated. - For the above I think there is dependency on https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/1570 which should clear any errored fields so users can continue to create logs and fix within the service	2 years ago
Phil Lee	45128b7250	Refactor with split of log to CSV by log type (#1593 ) # Context - Partially related to https://digital.dclg.gov.uk/jira/browse/CLDC-2316 - Comprehending sales or lettings bulk upload CSV is enough, comprehending both simultaneously is rather challenging # Changes - Split out test helper class by log type ie lettings/sales	2 years ago
SamSeed-Softwire	321ba71e0c	CLDC-2197 Improve column counting in bulk upload (#1557 ) * feat: add validation for number of field labels * feat: check field labels are numbers * feat: remove validations for number of columns * refactor: rename non_blank_fields_count to valid_field_numbers_count * test: add functions to generate custom field labels/values * test: that extra invalid field labels don't cause issues * test: that removing a valid field label reduces count by 1 * test: that adding a valid field label increases count by 1 * refactor: rename validate_fields_count and wrong_field_count * feat: add validation that max col count not exceeded when no headers * fix: convert numbers to strings in default_field_numbers * feat: add leniency to max cols count (in case of 1 extra col) * test: explicitly set year of bulk upload to be 2022 * test: add/update tests in validator_spec * chore: lint * test: remove tests from csv_parser_spec that were moved to validator_spec * feat: update 2022 csv_parser to work with new validations * refactor: define number of valid 2022 fields in one place * refactor: remove redundant headers definition * feat: update 2022 csv parser to have col flexibility like 2023 * test: for validator 2022 as well as 2023 * test: simplify 2022/2023 logic and improve layout * chore: lint * test: ensure context descriptions start with "when" * refactor: check fields/columns count within csv parser, not validator * test: update 2022 csv parser tests to work like 2023 * chore: lint	2 years ago
kosiakkatrina	d1f21a5890	CLDC-2031 Sales bulk upload (#1574 ) * Add sales csv parser * Create log creator and update row parser for sales * Add validations * Add status_cache to sales logs * Parse completed log * Add basic validations to the row parser * Fix details known and proplen mapping * Add setup section errors * Update sales log validator * tests * Add sales resume and summary pages * send the correct emails	2 years ago
Phil Lee	454df8389e	CLDC-2135 Bulk upload resume with how fix (#1503 ) * add first page for bulk upload resume journey * bulk upload resume handles upload again * add confirm page to bulk upload resume journey * replace placeholder count with correct value * apply recommendation for bulk upload resume choice * add how to fix bulk upload mailer * integrate new bulk upload approve journey * add missing bulk upload error mappings * remove test * prevent approve being called multiple times * bulk upload creates invisible logs ahead of time * work invisible logs into bulk upload flow * sort errors so deterministic * remove unused ensure * remove expected_log_count and processed - these fields are no longer used or needed * introduce pending status * swap visible for pending logs * only show visible lettings logs * hard code status filters * remove unused model methods * only show visible sales logs * form controller ignores hidden logs * locations and schemes only affect visible logs	2 years ago
Phil Lee	0b973ceebd	add new uprn and address fields to bulk upload (#1462 )	2 years ago
Phil Lee	35de06e324	CLDC-2067 Bulk upload for lettings 23/24 (#1417 ) * refactor questions from validator to row parser * able to switch between bulk upload parsers - depending on what year we are processing * spec tooling to support bulk upload multi year * row parser now has year in namespacing * add static data for 2023 row parser * add placeholder to log to csv for specs * bulk upload aribtrary 23/34 column ordering works * bulk upload supports 23/24 without headers * bulk upload 23/24 supports BOM + invalid chars * dupe tests * port 23/24 attributes_for_log * port 23/24 bulk upload validations * force crossover period * tweak max permitted columns * able to return column for given field * work out column for field for errors * add field_4 as 23/24 setup field * remove duplicate method * map schemes and locations correctly * handle arbitrary number of header rows * add missing fields to bulk upload support	2 years ago
Phil Lee	e1b6e0f5a6	CLDC-2001 Bulk upload stacked errors (#1377 ) * limit setup errors if errors already present * bulk upload beter handles age validations - limit the number of errors on a field. if the field already has an existing error it does not add further errors to the field - shim in extra validation for ages which take precedent over existing log level validations which are lacking * setup errors always added - no longer observes if the grouping already has errors - as we have fine grain control in this class of how errors should be * fix missing pass through for renewal	2 years ago
Phil Lee	4e3919f80e	CLDC-1781 Bulk upload thresholds (#1252 ) * create logs iff the log itself is valid * do not create logs if a setup section not complete * bulk upload with 60% errors will not create logs * extract magic number to constant * add bulk upload absolute threshold error rate * refactor with extract method	2 years ago
Phil Lee	9513e1179d	CLDC-1764 Bulk upload filter (#1223 ) * refactor with extract method * can filter logs by bulk upload * hide log creation button when viewing bulk upload - this affects the logs index page filtering logs from a specific bulk upload * add info banner to bulk upload logs * placeholder for bulk upload logs header * when resuming bulk upload set filters * fill place holder with remaining logs to fix * add interstitial to resume if logs resolved * after resolving bulk upload logs show interstitial * fix linting error * extract view variable to helper method	2 years ago
Phil Lee	f0868b8173	CLDC-1898 Bulk upload headers (#1210 ) * fix bulk upload age data types * bulk upload handles both with and without headers - headers are from the spreadsheet template - otherwise assume cell A1 is start of the dataset	2 years ago
Phil Lee	0009d535fe	CLDC-1896 Bulk upload line endings (#1208 ) * add class to create logs from bulk upload * create logs when processing bulk uploads * remove bulk_upload_id from csv output * create bulk upload logs only if all valid - this will be changed later to allow for partial logs - and only to create logs when a threshold has been met * add method to blank invalid non setup fields * bulk upload log creation blanks invalid fields * fix incorrect logic for bulk upload renewal * fix linting * bulk upload log creation fail logs to sentry * fix bulk upload line ending parsing * extract bulk uploading csv parsing to class * use csv parser in log creator * change handle line endings mechanism - we now strip all windows line endings for unix based line endings - this normalises things making it simpler	2 years ago

13 Commits (0f456080e8ba85f5c2fb5c0774037de124a5b00c)