# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2316
- Implement bulk upload sales for new collection year 2023
- This is a first pass implementation and will probably have some bugs in it and we can address over time
# Changes
- Add CSV parser for sales 2023 to handle CSV structure
- Tweak collection window validation so error now contextual to year selected for upload
- Handle arbitrary ordering of CSV columns
- Fix ordering of errors in report and now ordered by cell
- Added `Upload your file again` link styled as button on error report to match lettings experience
- Update tooling to convert logs to 2023 csv rows with support for random column ordering
# Known issues
- There seem to be some issues with how UPRN is handled if the UPRN cannot be validated.
- For the above I think there is dependency on https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/1570 which should clear any errored fields so users can continue to create logs and fix within the service
* create a method on the FormHandler that returns the sales form questions for all years in the order that they appear in the form
* update csv email job to accomodate sales log export as well as lettings
add to tests to reflec the changes made
* write tests to cover the desired functionality of the SalesLogCsvService
* create the SalesLogCsvService
create a necessary method on the log to enable submission method to be included on the csv
derive values for the two halves of previous postcode for export
* add relevant links in the UI and pipe everything together in controllers
amend organisations controller to have flexibility to download logs of either type
add necessary methods to sales log controller, raising shared method to logs controller
update routing for amendments and additions
extract helper method to build urls for downloading logs within an organisation
* correct various linter complaints and tech review suggestions
* minor amendment to add old_id and reorder early columns
* undo my 'clever' refactor that broke things
* refactoring of csv service after some tech review and some UI testing in review app
* update tests to include a test of a full export and all values in teh csv
* correct minor routing error to ensure correct url is shown and tab selected after requesting csv email
* update organisations controller requests spec file to cover new functionality and make a minor amendment to authentication scope in the controller after error found in testing
* write request tests for the new functionality in the sales log controller, define authorisation in the controller
* minor correction after rubocop's kind suggestion'
* various corrections from first pass at PO, tech review, linter, etc
* refactor :ordered_sales_questions_for_all_years
* first pass at implementing flexible code-based form fixtures for testing
* second pass
* refactor all tests of :ordered_sales_questions_for_all_years to use new factories
* some refactoring in the testing of the csv service
* use that fact that params is always available in controllers and don't pass it around, inline some methods calls
* correct minor bug to ensure that "Return to logs" link returns to the correct index page
* remove reminder comments
* write further tests on the manipulation of questions into the csv headers, update factories of form constituents to allow the creation of forms with richer questions
* fix linter complaints
* minor alterations after rebase to account for changes made on other branches
* refactor after code review
* tweak fixtures after rebase containing alterations to the factory defaults
* Add new organisation address fields to merge requests
* Add new organisation address page
* Add new organisation telephone number placeholder page
* close the div
* Add new organisation name column
* Update new organisation name page
* Refactor validate_response
* Add placeholder new_organisation_address page
* Add existing organisation name validation
* Refactor error messages
* Extract some validations to the model
* Update which field we add the errors to
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2294
- When bulk upload returns a results page it would be useful if colleagues of the uploader can see this page to help fix errors
- It would also be useful if support users can see these reports to help diagnose bulk upload errors
# Changes
- Added `pundit` gem to handle authorization
- Bulk upload results previously only accessible to the bulk uploader. Now they can be seen by users in the same org as the uploader and also support users
* CLDC-2056 Add absorbing organisation question
* Handle next page and previous_template with `page`
* Hardcode backlinks
* Refactor flow to store value when selecting new org option
* Handle unanswered question
* Update error copy
* rename variables in test file
* write tests for new validations
* validate that answers to numeric questions are given to the correct level of accuracy and in normal format with only digits
* add error messages for new validations
* add or alter step on numeric questions in sales, amend one test given step changes
* copy change
* remove validation that is specific to number format and fold that functionality into accuracy validation, rejig tests accordingly
* fix various tests
* remove reference to currency in validation message that concerns step only and doesn't check whether firled is currency related
* alter copy to avoid redundant repetition in validaiton message
* save changes made in rebase
* changes after PO review
two step values had been missed in previous work
in various places there were custom validations applied that duplicated the functionality of the generic min max validation that is applied to all numeric questions, in these situations sometimes the min and max on the question class was inconsistent with the limit that triggered validations
these have been corrected and made consistent
various tests were affected by this and have been amended accordingly
* remove tests introduced by mistake in rebase
* amend step value on earnings
* Create merge_request table and paths
* Save merging organisations wip
* Add update organisations
* Add merging organisations validation
* merge fixes
* Update schema to have merge_request_organisations
* Add relationships between merge request and organisations
* Update validations and saving organisations
* Add ability to remove merging orgs from the list
* Allow support users to create merge request for any organisation
* Update wording in organisations view
* Allow adding other merging organisations
* Add back button, update content
* Add validation if the organisation is not selected
* Fix path
* Use generic update method
* Update validations
* fix remove organisation
* remove reloads
* Update routes
* Authenticate scope
* PR review changes
* Add status, run validations unless the status is unsubmitted and save requesting organisation as a merging organisation as well
* PR comments
* Display continue button when there are at least 2 merging organisations
* feat: update scheme status so incomplete unless has active locations
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: show completed/incomplete locations and validate when completed == 0
* feat: improve copy
* feat: change active -> completed
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: show completed/incomplete locations and validate when completed == 0
* feat: improve copy
* feat: change to confirmed to add clarification in code
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: show completed/incomplete locations and validate when completed == 0
* feat: improve copy
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: only confirm locations if complete (old) AND save button clicked (new)
* feat: add unconfirmed scope
* refactor: complete -> confirm
* feat: fix tests
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: show completed/incomplete locations and validate when completed == 0
* feat: improve copy
* feat: update scheme typeahead text
* feat: reject incomplete locations
* feat: only confirm locations if complete (old) AND save button clicked (new)
* feat: add unconfirmed scope
* refactor: complete -> confirm
* feat: fix tests
* feat: fix more tests
* feat: fix more tests
* refactor: rubocop
* refactor: rubocop
* refactor: rubocop
* feat: add tests for incomplete schemes
* refactor: linting
* feat: test incomplete count too
* feat: test unconfirmed locations aren't options in location_id
* feat: test validation
* feat: test when scheme is confirmed
* feat: test when scheme is confirmed for support user
* feat: test when location is confirmed
* refactor: linting
* refactor: consistent use of factorybot or not within a file
* feat: confirm locations when complete even if save button not clicked, and update tests
* refactor: simplify
* refactor: simplify
* refactor: move location helper methods to helpers
* refactor: move scheme helper methods to helpers
* refactor: formatting
* refactor: po response
* refactor: review response
* rename variables in test file
* write tests for new validations
* validate that answers to numeric questions are given to the correct level of accuracy and in normal format with only digits
* add error messages for new validations
* add or alter step on numeric questions in sales, amend one test given step changes
* copy change
* remove validation that is specific to number format and fold that functionality into accuracy validation, rejig tests accordingly
* fix various tests
* remove reference to currency in validation message that concerns step only and doesn't check whether firled is currency related
* alter copy to avoid redundant repetition in validaiton message
* save changes made in rebase
* changes after PO review
two step values had been missed in previous work
in various places there were custom validations applied that duplicated the functionality of the generic min max validation that is applied to all numeric questions, in these situations sometimes the min and max on the question class was inconsistent with the limit that triggered validations
these have been corrected and made consistent
various tests were affected by this and have been amended accordingly
* remove tests introduced by mistake in rebase
* Move and rename some tests
* Add a merge organisation link
* Add merge page
* Don't display a second nav bar for support
* Fix spacing
* fix spacing and typos
* add first page for bulk upload resume journey
* bulk upload resume handles upload again
* add confirm page to bulk upload resume journey
* replace placeholder count with correct value
* apply recommendation for bulk upload resume choice
* add how to fix bulk upload mailer
* integrate new bulk upload approve journey
* add missing bulk upload error mappings
* remove test
* prevent approve being called multiple times
* bulk upload creates invisible logs ahead of time
* work invisible logs into bulk upload flow
* sort errors so deterministic
* remove unused ensure
* remove expected_log_count and processed
- these fields are no longer used or needed
* introduce pending status
* swap visible for pending logs
* only show visible lettings logs
* hard code status filters
* remove unused model methods
* only show visible sales logs
* form controller ignores hidden logs
* locations and schemes only affect visible logs
* Fix possible bulk upload years
* Initialise 2021/22 form routes for tests
* Return test routes from form handler
* Fix some tests
* fix bulk upload tests
- change from mocks to actual code calls
* update years for bulk upload form
* Fix some uprn tests
* Set the scheme as available from previous year if the collection is in crossover
* Update location available_from
* Update location and scheme tests
* Refactor and comment out a scheme test
* Some uprn and export tests
* Fix lettings logs controller tests and comment out previous collection message test
* fix tests for new collection year
* fix bulk upload tests
for some reason tenancylength must not be given for validation to pass
* update bulk upload tests for new year
* More fixes
* Undo location changes
* Fix remaining uprn
* further new collection year test fixes
---------
Co-authored-by: Phil Lee <asmega@users.noreply.github.com>
* add a validation against the case where monthly leashold charges are zero, test this, remove unnecessary validation on cash discount as this is covered by validate_numeric_min_max
* remove focus: true from describe block
* minor changes for testing and linting
* correct minor bug found in PO
* make minor amendments to failing tests
* add some methods on lettings log and use them and already existing methods to make depends on human readbale for pages in property information subsection
* reformat answer options for neatness
* ensure that property information subsection is conditionally not shown under correct conditions
* amend method to find the path for the next question to answer so that if no questions in the next section are currently routed to, it goes to the section after that
* make amendments for failing test and linting
* remove unnecessary method
* remove unnecessary conditions from depends on, the displayed in tasklist method override is sufficient
* test the change to the method creating a link to the next subsection
* minor refactor to displayed_is_tasklist condition and changes in testing
* implement the fix on 22/23 form also
* write a test covering the fix
* removal not_renewal? method and tweak depends on for all pages using that method
* linty fluff begone
* correct some errors in tests after changes to factories
* minor testing amendments after rebase
* update seeds to add self in review env, change spec to reflect this, update config yml to allow csv exports in review
* update interface of relevant methods
EmailCsvJob, LettingsLog.to_csv and LettingsLogCsvService consume codes_only flag
* update tests including adding a new csv file to test against
* update LettingsLogCsvService to output codes only csv
* correct minor error and linting
* enable codes only download in UI
- add link on lettings log index page
- pass codes_only flag through params in relevant links and methods
- convert flag to boolean in controller methods
* ensure link displayed successfully for all renderings of logs_list and params passed through relevant methods in organisations controller
* fix existing tests
* correct linting thing
* correct linting error
* update tests for lettings log controller
* correct linting errors
* update organisations controller tests
* make minor changes after code review
* remove changes made for testing on review app
* make codes only download visible to support users only
* change variable names throughout after info on rauby/rails naming conventions, update tests for change in who can view codes only download link
* rework csv service for readability, remove delegating methods from lettings log to keep all code to do with mapping between our domain and desired export format in one place
* update test name
* correct a small typo and remove a duplicated method after clever git merge conflict suggestion
* point review app at staging csv bucket for csv download
* change variables named codes_only_export to codes_only to avoid inconsistency
* write tests to ensure that differetn user roles have the correct permissions around csv download
* ensure that non support users may not download codes only exports
* correct a small error in a previous commit
* correct minor linting error
* feat: add new guidance page with corresponding template links for sales and lettings log
* feat: add specification files and links for sales/lettings
* refactor: tidying
* feat: change pseudo csv to xlsx to reflect reality
* feat: add tests for guidance page routing and content
* feat: add tests for lettings too
* feat: add sales validation to check bedsits have <=1 beds
* test: sales validation to check bedsits have <=1 beds
* fix: typo in property validation method name
* feat: add same bedsit validation to number of bedrooms question
* test: fix typo propert -> property in property validations tests
* test: add test for validate_property_number_of_bedrooms (sales)
* feat: update wording for 'number of bedrooms' validation when bedsit
* test: condense sales property validations tests
* test: strengthen sales property validations error addition test
* refactor: simplify sales property validations into one method
* fix: update error message content to match Beth's choices
* chore: lint
* test: add requests test for invalid sales log params when posting
* refactor: use if rather than unless... not in property validation
* refactor: write method is_bedsit? on SalesLog
* test: check bedsit error not added if proptype or beds is nil
* lint: use update! not update
* feat: update validation messages to improve readability
* fix: provide valid date in request test for invalid proptype/beds
* feat: make is_bedsit and validate_bedsit_number_of_beds more readable
* refactor: make has_inferred_check_answers_value? do what it says it does
* refactor: combine method calls
* refactor: linting
* test: update
* refactor: copy
* db: update
* refactor: update db and lint
* feat: add present? back in to avoid 0 errors
* feat: flip ==
* refactor: flip ||
* refactor: flip name
* replace range validation with multiple to account for single-ended ranges
* add test for question with min but no max
* add min and max to household count question and remove bespoke validation. Remove code allowing max and no min given this situation does not seem to exist int eh form
* amend minor typo, add back a soft validation that had mysteriously gone missing
* feat: add 22/23 year validation
* feat: wip commit
* feat: update i18n
* feat: add one year gap exchange/completion validations
* tests: add tests for new validations
* test: add setup validations tests
* test: update factory to pass saledate validation
* feat: update seeds
* feat: update tests to reflect sales logs shouldn't exist in new service before 2022
* feat: merge with main, improve date behaviour
* refactor: cleanup
* feat: update tests
* feat: enforce saledate not after 22/23 collection year end
* refactor with extract method
* can filter logs by bulk upload
* hide log creation button when viewing bulk upload
- this affects the logs index page filtering logs from a specific bulk
upload
* add info banner to bulk upload logs
* placeholder for bulk upload logs header
* when resuming bulk upload set filters
* fill place holder with remaining logs to fix
* add interstitial to resume if logs resolved
* after resolving bulk upload logs show interstitial
* fix linting error
* extract view variable to helper method
* test: check managing org not gone from answer opts when relationship deleted
* feat: add current managing org to answer opts
* feat: check if managing org exists before trying to show it
* wip
* test: improve managing orgs opts test when not support
* test: improve managing orgs opts tests when support
* test: make relationship deletion test consistent with other tests
* test: add "(with hint)" to managing org opts test descriptions
* test: refactor managing orgs opts tests for support user case
* fix: don't call user in get_answer_label in CYA component
* style: reorder instance vars and remove old comments in managing_organisation.rb
* refactor: ensure label_from_value always accepts log & nil as args
* lint
* test: pass in log and user in housing provider opts test for support user
* test: update housing provider opts tests for non-support user
* feat: update housing provider answer opts to include current HP in db
* style: add space after user definition
* test: make context definition more human-readable
* test: refactor housing providers opts tests (not support user)
* test: check housing prov. still selectable after deleting relationship
* fix: define log and current_user instance vars in label_from_value (housing prov.)
* lint
* test: update lettings log feature tests
to differentiate between different numbers of stock owners when acting as a data coordinator
* test: check owning & managing orgs set correctly when a log is created
* test: add line breaks and start context descriptions with and (not if)
* test: artificially reference org_rel2 to avoid lint offense
* feat: don't set log owning org as user's org if that org doesn't hold stock
* test: improve test context descriptions in lettings_log_spec
* test: finish overhauling owning and managing org tests in lettings_log_spec
* test: change let! to let where possible in spec/features/lettings_log_spec.rb
* test: change let! to let where possible in spec/models/form/lettings/questions/managing_organisation_spec.rb
* test: change let! to let where possible in spec/models/form/lettings/questions/stock_owner_spec.rb
* test: remove if statement from "coordinator user's org doesn't hold stock" managing org test
* test: remove if statement from "coordinator user's org does hold stock" no managing orgs managing org test
* test: remove if statement from "coordinator user's org does hold stock" >=1 managing orgs managing org test
plus refactor previous test
* test: explicitly reference org rels in "coordinator user's org doesn't hold stock" managing org test
* test: don't create vars inside other vars (for tests edited/created in this branch)
* chore: save schema changes after migration
Co-authored-by: Phil Lee <asmega@users.noreply.github.com>
* lockdown bulk upload routes
* able to view lettings bulk upload errors
* add error count to bulk upload results
* coverage for bulk upload filename on results
* group bulk upload errors by row on results
* able to view bulk upload sales results
* scope lettings and sales bulk upload results
* fix linting
* call service correctly in test
* add bulk upload sales questions mapping
* appease linter
* bulk upload error shows correct question
- depending on log type it will show relevant question for the field
concerned
* use local disk for bulk upload for dev env
- this saves the need to connect to S3 to play with bulk upload in dev
environment
* improve namespacing of classes
* add job to process bulk uploads
* use local disk storage for dev file upload
* fix test
* use inline active job queue_adapter for dev
* use test active job queue adapter for test env
* remove rubocop violation
* delete bulk upload from disk after processing
* populate errors with cell, row + metadata
* update error message with something meaningful
* shim in sales validator
* able to parse sales bulk uploads
* change migration to add purchase_code to errors
* bulk upload error component renders purchaser code
- when a sales log
* populate purchaser_code for bulk upload errors
- when log type is sales
* remove superfluous private method
* feat: wip postcode page
* feat: wip add new flow
* feat: add check answers page, continue updating flow
* feat: add back behaviour
* feat: more flow work
* feat: further flow work and flash notice, name tidying
* feat: add check_answers referrer linking
* feat: add controller and details linking and lint
* refactor: erblinting
* feat: add local_authority page (currenlty separate from edit_local_authority to avoid conflict with add location to newly created scheme path, to be fixed later). also copy and routing updates
* feat: add validations to controller (could later be added to model)
* fix: correct date order
* feat: validate on model (except startdate)
* feat: copy update
* refactor: railsification
* feat: add date model validation
* feat: add updated availability text in check_answers
* feat: more updated dynamic text and linking
* refactor: erblinting
* refactor: cleanup of redundant code
* refactor: remove redundant create
* feat: use rails route
* test: wip tests
* test: wip tests
* feat: add new back behaviour for new scheme path and tests wip
* feat: add more back behaviour for new scheme path and fix local auth validation
* feat: further linking behaviour with routes and referrers, don't add new location automatically to new schemes
* feat: if location with no details added, route to normal path rather than check answers
* test: update model and helper location tests
* test: more test updates
* test: more test updates and remove redundant files
* test: remove redundant tests, add postcode test
* test: add new #new tests
* test: add test for all new controller methods
* test: add more startdate tests
* refactor: erblinting
* test: fix failing tests
* test: update scheme test
* feat: respond to PR comments
* feat: make postcode clear only if changed
* feat: add _update methods
* refactor: simplify validation and location_edit_path method
* refactor: add helper method for action text
* refactor: remove redundant logic
* refactor: simplify routing
* refactor: reintroduce location_params
* refactor: use presence validation for postcode
* refactor: use presence validation for la
* feat: validate confirmed
* tests: make tests pass with new startdate validation
* WIP
* refactor: simplify display attributes helpers and update tests
* refactor: linting
* feat: design change to save behaviour
* test: fix failing tests
* feat: location code copy tweak
* refactor: remove redundant function
* refactor: remove add_another_location and new location page
* feat: simplify startdate validations to move to model, use scheme available_from date for in range validation
* test: fix failing tests
* refactor: linting
* feat: remove name validation as a result of PO review
* test: update tests
* refactor: remove redundant context
Co-authored-by: James Rose <james@jbpr.net>
Phil Lee
Arthur Campbell
kosiakkatrina
Jack
natdeanlewissoftwire
SamSeed-Softwire