* Add in past end dates for testing
* Temporarily disable test marked to delete at year end
* Remove one off reinfer_local_authority task
* Update validator_spec
* Set 2024 date in form_handler_spec to be during crossover period as needed
* Use bulk_upload.year_combo for comparison in request tests to avoid year dependancy
* Update BU log creator specs for 2024
* Use year combo function in bulk upload mailer tests
* Refactor lettings validator_spec
* More fixes
* More work on bu validator specs - mostly sales
* Remove pre 2023 test
* More use of bulk_upload.year_combo in request tests
* Fix lint
* Tweak bulk upload error row component tests for year changes
* Further fixes
* Fix 2023 lettings row parser spec
* Sales log to csv fix
* Refactor BU processor tests
* Fix field number row identifier
* Fix linting
* More years in request spec
* fix
* Don't use db unnecessarily in financial validations spec
* Fix sale date changing 2024 -> 2023 test
* Add tests for bulk_upload.year_combo
* Update bu factory year specification
* Refactoring
* Linting
* Don't use helper in factory
* Remove new 2023 specific test
* Remove dummy end dates
* further fixes
* Fix date / crossover issues
* Try changing form end dates
* Revert "Try changing form end dates"
This reverts commit e027e29cf0.
* Make use of :ignore-validation-errors trait
* Try with end dates in past
* Revert "Try with end dates in past"
This reverts commit 6b87938d70.
---------
Co-authored-by: Rachael Booth <rachael.booth@softwire.com>
* Remove legacy user from user factory
* Build instead of creating in some model tests
* Do not create a new DPO for orgs created within user factories
* Add original setup doc
* Update documentation with updated factories
* Fix up DSA creations
* update factory completed traits using an after build hook so that trait does what it says regardless of start or sale date of log
* use completed trait rather than completed2024 trait from factories
skip validation on saving logs from previous years so that logs created too far in the past do not trigger validations
* remove sale and start dates from logs in tests where that should not be important
build rather than create to reduce db interactions
in order that the setup is complete, build the necessary user and organisation with ids
* fix a wide range of tests affected by the changes in log factories completed traits
* CLDC-3463: Fix bug with vacdays calculation
* CLDC-3463: Fix lettings csv export tests
* Lint
* Use build_stubbed where necessary for lettings logs in tests
* Update all logs in rake task but without validation
* Try validations in rake tasks and log when needing to skip them
* write helper method to support having the correct rent period checkboxes checked
* update new and create in organisations controller and view to enable creation of relevant organisation rent periods
write tests for this
* small changes to models
* enable editing rent period in the UI
display change button on org details page
display rent periods question on edit page
* enable updating org rent periods
update logic in #update
related tests
* alter redirect after support user creates an organisation
* adjust various UI elements: ordering of rows, copy changes, label size and associated tests
* rework the #rent_period_labels method to return All under the correct conditions, rework tests related to that. + fix assorted tests that were either flakey or breaking due to addition of rent periods logic to create and update
* amend failing tests and resolve linting complaints
* changes following review
* disable checkboxes for rent periods if they are in use so that users are not able to make existing logs invalid
hint text added to the question to explain this
I have also added all rent periods to a hidden field to remove the need to fetch them again form the db in the update method
* update validation to reflect the fact that an org having no associated rent periods no longer means they accept all rent periods
update tests adding both cases and removing unnecessary additional db additions
* rake task to create rent period associations for orgs that have none
* revert mistaken copy changes in designs
* create rent periods in factories as default, with an option to skip. skip automatic creation in tests specifically related to rent periods
* stub api call for factory value, update csv tests and fixtures accordingly
* extract a good chunk of tests out of lettings_log_spec and into a dedicated derived fields spec file. in many cases refactor tests
* remove before(:context) and associated patterns.
use assign_attributes in various places for cleaner code
* escape . in regex for API call stubs to satisfy codeQL
remove destroy_all call at the start of a test that was dealing with leftover modesl in the test db
* further refactoring of various tests to reduce database interactions and improve speed
* remove outdated distinction between unitletas mappings from before 23/24
* remove tests that seem to be testing active record and/or ruby Date class
* write helper method to support having the correct rent period checkboxes checked
* update new and create in organisations controller and view to enable creation of relevant organisation rent periods
write tests for this
* small changes to models
* enable editing rent period in the UI
display change button on org details page
display rent periods question on edit page
* enable updating org rent periods
update logic in #update
related tests
* alter redirect after support user creates an organisation
* adjust various UI elements: ordering of rows, copy changes, label size and associated tests
* rework the #rent_period_labels method to return All under the correct conditions, rework tests related to that. + fix assorted tests that were either flakey or breaking due to addition of rent periods logic to create and update
* amend failing tests and resolve linting complaints
* changes following review
* disable checkboxes for rent periods if they are in use so that users are not able to make existing logs invalid
hint text added to the question to explain this
I have also added all rent periods to a hidden field to remove the need to fetch them again form the db in the update method
* update validation to reflect the fact that an org having no associated rent periods no longer means they accept all rent periods
update tests adding both cases and removing unnecessary additional db additions
* rake task to create rent period associations for orgs that have none
* revert mistaken copy changes in designs
* create rent periods in factories as default, with an option to skip. skip automatic creation in tests specifically related to rent periods
* stub api call for factory value, update csv tests and fixtures accordingly
* extract a good chunk of tests out of lettings_log_spec and into a dedicated derived fields spec file. in many cases refactor tests
* remove before(:context) and associated patterns.
use assign_attributes in various places for cleaner code
* escape . in regex for API call stubs to satisfy codeQL
remove destroy_all call at the start of a test that was dealing with leftover modesl in the test db
* further refactoring of various tests to reduce database interactions and improve speed
* remove outdated distinction between unitletas mappings from before 23/24
* remove tests that seem to be testing active record and/or ruby Date class
* create presenter class to hold all the db calls and data for showing the home page
plus tests, some changes to factoreis and helpers necessary
this should avoid an overly bloated controller method or calling the DB from a view helper
* view partial changes
to accommodate design changes, decouple box with counter and blue link boxes
since various boxes show or not or in a different order depending on whether the org logs sales and whether we are in the crossover period I have also made grid and row partials to improve readability on the index page.
* redesign the home page using the new presenter and partials, deliver the presenter object to the view from the controller
* adjust tests and factories and address api call stub
* remove view helpers that are no longer being used and remove outdated tests for home page
* amend presenter and associated calls from view, tests, to ensure support user always sees sales data
* write request specs for the homepage
* undo unnecessary change to factory
* make corrections following rebase conflicts
* various minor changes following tech review
* logs assigned to a provider should be vivisble to them in the homepage count rather than logs created by them
* resolve rebase confusion in csv fixture files
* CLDC-3426: Ensure outright sale section can be completed with mortgage used unknown
* CLDC-3426: fix test
* CLDC-3426: Refactor tests
* Avoid unnecessarily creating users in sales log factory
* Fix test that inexplicably relies on creation
* Empty
* CLDC-3345 Rename created_by to assigned_to (#2372)
* Rename created_by to assigned_to
* Replace created_by with assigned_to
* Update created_by to assigned_to in exports, remove blank assigned to
* CLDC-3345 Add and set created_by fields (#2373)
* Add created_by
* Update existing created_by values
* Set created_by on single log
* Set created_by on BU
* Add created_by to exports
* feat: update since last merge
---------
Co-authored-by: natdeanlewissoftwire <nat.dean-lewis@softwire.com>
* Fix papertrail create version
---------
Co-authored-by: natdeanlewissoftwire <nat.dean-lewis@softwire.com>
* Update schemes from csv
* Handle validation errors
* update locations from csv
* Export logs using updated schemes and locations
* Refactor
* Only update owning org to related ort
* Update logs
* Log and not update old logs
* do not update the org with closed collection logs
* Do not export old logs with changed location
* Rename logs and refactor tests
* Handle rent ranges for open collection periods
* Log ids for logs that turn in progress
* Allow changing scheme to the one from related org
* log updates logs and don't mark them unesolved
* Clear scheme/location if location scheme changes
* Check soft rent validation for closed collection
* Clear location/scheme in remaining exportable logs
* Log updates before clearing locations and schemes
* feat: make charges soft validations check soft max range rather than just over soft max
* Update testst
* Allow soft max without a soft validation
* refactor tests
* Refactor test names
* feat: update unpend_and_confirm_soft_validations
* feat: refactor unpend_and_confirm_soft_validations and test the counts of value check fields to be confirmed are correct
* feat: further simplify unpend_and_confirm_soft_validations
* feat: further simplify unpend_and_confirm_soft_validations
---------
Co-authored-by: Kat <katrina@kosiak.co.uk>
* Empty commit
* CLDC-3224: Parse noint correctly in bulk upload row parsers
* CLDC-3224: Add task to fix previous noint data
* CLDC-3224: Remove nonsensical value for noint field in bu test
* CLDC-3224: Move where bulk upload noint fix status is set to ensure it matches processing version
* Put BU noint_fix_status setting before creating any logs
* CLDC-3227: Add soft validation for reason other when it's likely a standard category
* CLDC-3227: Add hard validation for homeless indicators in other reason for leaving last settled home
* CLDC-3227: Update hard validation regex to ignore all non alphabet characters surrounding a match
* CLDC-3227: Add unit tests
* Refactor household validations spec to remove dependancy on fake 2021 form
* Add card number to question
* Move informative text to translations as well
* CLDC-3227: Allow neighbouring non-alphabet characters in soft validation check
* Remove debug lines
* Remove blank line
* Remove bonus whitespace
* Deal with appearance in csv export
* Only do regexp.union once
* Fix informative text in value check
* CLDC-3110: Update hard max income for ecstat 8 to 2000
* CLDC-3110: Update hard max income for ecstat 10 to match new highest max
* CLDC-3110: Base income validations on all tenants for lettings logs
* CLDC-3110: Tweak validation messages
* CLDC-3110: Use prefer not to say values when tenant ecstat is not specified
* feat: add blank homepage, update routing and tests
* feat: add welcome message and thoroughly test routing
* refactor: lint
* feat: update tests
* CLDC-3076: Make example dates consistent (#2107)
* CLDC-3076: Make example dates consistent
* Use example date even when some hint text provided already
* Temp remove some date restrictions
* Update to 2 line breaks
* Revert "Temp remove some date restrictions"
This reverts commit cd7f18f9f1.
* Fix lettings log accessor in date question (#2117)
* Fix lettings log accessor in date question
* Remove hardcoded date example from mrcdate question (#2118)
---------
Co-authored-by: Rachael Booth <Rachael.Booth@softwire.com>
* CLDC-3061 Add guidance page (#2121)
* Add guidance page
* Link to guidance from start page
* feat: test home/start paths explicitly
* CLDC-2253 Add collection resources (#2120)
* Update collection resources, add to homepage
* Add guidance link to an empty page
* Update headings
* Rebase fix
* Update title
* Update file names
* Add section break
* CLDC-2593 Add upcoming deadlines section (#2119)
* Add upcoming deadlines section
* Update the content to use the correct dates
* Update content
* lint
* typos
* CLDC-2252 Add homepage task section (#2115)
* feat: wip add lettings, sales and schemes sections with correct text, counts, links and colouring
* feat: add flex styling to match designs
* CLDC-3076: Make example dates consistent (#2107)
* CLDC-3076: Make example dates consistent
* Use example date even when some hint text provided already
* Temp remove some date restrictions
* Update to 2 line breaks
* Revert "Temp remove some date restrictions"
This reverts commit cd7f18f9f1.
* Fix lettings log accessor in date question (#2117)
* Fix lettings log accessor in date question
* Remove hardcoded date example from mrcdate question (#2118)
---------
Co-authored-by: Rachael Booth <Rachael.Booth@softwire.com>
* feat: update breakpoints for responsive layout changes
* lint: use hash lookup where possible
* lint: erblinting
* feat: improve formatting
* Reuse govuk grid
* Revert "Reuse govuk grid"
This reverts commit 8c71f5d9ed.
* feat: test home page data boxes
* refactor: lint
* refactor: lint
* feat: test link behaviour is correct in all user scenarios
* refactor: lint
* feat: update tests
* feat: combine task, resources, deadlines sections
---------
Co-authored-by: Rachael Booth <Rachael.Booth@softwire.com>
Co-authored-by: kosiakkatrina <54268893+kosiakkatrina@users.noreply.github.com>
Co-authored-by: Kat <kosiak.katrina@gmail.com>
* CLDC-2255 Add homepage notifications (#2131)
* feat: add notification table
* feat: add notification banner, use unread gem for notification management
* feat: add notifications page and remove unread_notification.rb
* feat: add blank homepage, update routing and tests
* feat: add welcome message and thoroughly test routing
* refactor: lint
* feat: update tests
* CLDC-3061 Add guidance page (#2121)
* Add guidance page
* Link to guidance from start page
* feat: test home/start paths explicitly
* feat: add notification table
* feat: add notification banner, use unread gem for notification management
* feat: add notifications page and remove unread_notification.rb
* feat: default p tag around sanitized page content
* feat: add active scope
* feat: use newest active unread/unauthenticated notification and update start page
* feat: add tests of notification behaviour and routing and refactor
* refactor: lint
* feat: update Gemfile.lock
* feat: add timestamps to readmark table
* feat: update gemfile.lock
* refactor: lint
* feat: test notifications page doesn't show notifications and code simplification
* feat: move notification helper methods to notifications_helper.rb
---------
Co-authored-by: kosiakkatrina <54268893+kosiakkatrina@users.noreply.github.com>
* feat: clear all no-status filters on in progress links
* CLDC-2590 Add about this service section (#2124)
* Add about core section
* Move about core to the correct spot on start page
* Update content
* Add some margins
---------
Co-authored-by: Rachael Booth <Rachael.Booth@softwire.com>
Co-authored-by: kosiakkatrina <54268893+kosiakkatrina@users.noreply.github.com>
Co-authored-by: Kat <kosiak.katrina@gmail.com>
* Allow scheme editing for support users
* Allow editing locations
* refactor attribute list
* Add inset text
* Redirect change back to scheme path
* Reset secondary client group
* Correctly route to sales owning organisation page
* Display correct owning organisation options
* Set managing organisation as user organisation
* Add managing organisation question for sales
* Display managing organisation users in created_by
* set managing org to be same as owning org in BU
* Add managed by filter to sales
* Display reported by, correctly update owning org
* Update factory and 2022 BU
* refactor
* Merge managed sales logs
* test
* Update managing organisation for sales bulk upload
* Rebase tests
* skip a test as it is not relevant as the rake task would have already been ran
* PO fixes
* Remove old collection managed sales validation
* Update question header and filters conditions
* remove feature toggle
* Add missing addresses csv job
* Update missing addresses csv service methods
* Add rake task
* Update the job to send missing town or city templates
* Update service to create mising town or city templates
* Add send missing town or city csv rake task
* Add IDs to the CSVs
* Put all log in the same csv
* Add issue type column
* Write wrong uprn logs to csv
* Add mailer methods
* Skip uprn issue for specified orgs
* Add sales csv rake task
* set SKIP_UPRN_ISSUE_ORG_IDS on review apps
* test
* Update notify template IDs for testing
* Initialize service with organisation instead of a hash
* Add expiration time to url
* Add optional tags and remove LA from csv
* Extract log to csv methods
* Update casing
* Update old IDs in factories
* Move constant
* Extract some repeating scopes
* Pass in organisations to skip instead of using an env var
* update template id for sales
* Update link expiry time and headers
* Lower the threshold for testing
* Add issue explanation to the email
* Add how to fix
* update emails
* CLDC-2810 Create all addresses CSV (#1953)
* Add rake tasks for creating all addresses CSV
* Write headers if logs don't exist, update header names
* Rename method
* CLDC-2810 Correct addresses from csv (#1957)
* Updating importing lettings addresses form csv
* Add import_sales_addresses_from_csv rake
* Allow correcting addresses from both templates
* escape .
* Reinfer LA if the postcode hasn't changed
* Update labels and email content
* Update missing addresses threshold
* Remove unused env var
* Make has_other_client_group a mandatory question
* Add a rake task to update has_other_client_group
* test and lint
* Set has_other_client group on import
* create helper module to find all logs for a given user that have duplicates
write associated tests
update methods on user model to retrieve logs related to a given user
create a scope on logs to retrieve logs created by a given user
* check for duplicates and display notification banner on the index page where appropriate
tests to cover the desired behaviour
* alter duplicate logs helper to return duplicates in a format that can be passed through the params
update associated tests
* create user journey to view duplicate sets adn navigate to a given set
create a new route and add it to the link from the logs index page
create controller method and related view file
* update path for duplicate logs in routes, update factories, refactor duplicates scope to eliminate raw sql
* write tests for the duplicate logs index page
* correct linting errors
* minor amendments from code review.
* temp
* Refactor
* Create a scope for duplicate groups
* fixes
* lint
* Update controller specs
* Put duplicate sets count behind a feature flag
* refactor index
* Update duplicate_sets_count
* Update scopes
* tests
* pass in original log id
* tests
* Add duplicate index page content
---------
Co-authored-by: Kat <katrina@kosiak.co.uk>
* Add location filters to the controller
* Display the status filter and clear filters
* refactor status filter into scope
* Update incomplete, make reactivated scope work in conjunction with other scopes. Lint
* specs
* styling
* Update test name
* uncomment a test
* Move filters under navigation
* Filter schemes
* Add status filter to schemes pages
* Filter schemes deactivating in more than 6 monts as active
* Always serialize session filters for schemes
* Remove ordering in search, fix tests
* Update incomplete scope
* CLDC-2494: WIP
* CLDC-2494: wip
* CLDC-2494: page work in progress
* cleanup
* Add a path for duplicate logs
* Display all duplicate logs
* Move a test
* Display duplicate check answers for logs
* Add buttons to delete duplicates
* Add a route for sales logs duplicates
* Update duplicated page to work for sales logs
* lint
* Add auth
* Rebase updates
* Update displayed questions
* Add delete duplicates page
* Update page content for when there are multiple duplicates
* Add auth to the path
* Update delete button to use delete controller
* Update page for sales logs
* Update styling
* Update success banner for deleting duplicates
* Typo
* Render not found if there are no duplicate logs
* rebase changes
* Rebase changes
* Update back and cancel links
* Update the duplicate journey after deleting logs
* Update change button routing
* Refactor tests
* Add content for no duplicates
* Update params
* Refactor tests and paths
* Update params to include original_log_id from the beginning
* Rename file
* lint
* Add full stop after deletion messages
* Add caption and update duplicate log IDs in the banner
* lint
* Styling
---------
Co-authored-by: Aaron Spencer <aaron.spencer@softwire.com>
* protect against stack level errors in the case where a log has the status in progress but due to changes in the form should now have the status completed
further update the lettings log factory as the :completed trait was producing logs that were not copmlete
* add tests for calculate_status that implicitly also check if the factory traits for copmleted are working
also some minor renaming and refactoring
* correct linting errors
* remove status update but protect against error resurfacing in the future
* feat: setup to replicate failures to fix
* feat: wip test fixes
* feat: remainder of current state test fixes
* feat: set form date to past to trigger errors to fix
* feat: revert, don't want this in final diff
* feat: fix row_parser tests
* feat: sales log importer and validator fixes
* feat: remainder of test fixes
* refactor: lint
* Remove bulk_upload_lettings_logs flag
* Add spacing above actions in data sharing agreement
* Add banner
* Add model validation
* fix log component
* Fix factory
* Put validation behind feature flag
* start adapting to data protection confirmation
* Make tests more generic
* Add tests for display_organisation_attributes
* Use data_protection_confirmed? method
* Address code review
* rubocop
* Prevent access to bulk upload pages when DPC not signed
* Address PO review
* Add data sharing agreement
* Add data sharing agreement view
* Add controller actions
* Update details page
* Rubocop fix
* Fill in placeholders and persist data in table
* Update agreement template
* call form to reset invalid values when saving a sales log
* fix tests failing after updating sales log functionality to reset invalid answers
* add a test to cover the new functionality added
* a further test
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2316
- Implement bulk upload sales for new collection year 2023
- This is a first pass implementation and will probably have some bugs in it and we can address over time
# Changes
- Add CSV parser for sales 2023 to handle CSV structure
- Tweak collection window validation so error now contextual to year selected for upload
- Handle arbitrary ordering of CSV columns
- Fix ordering of errors in report and now ordered by cell
- Added `Upload your file again` link styled as button on error report to match lettings experience
- Update tooling to convert logs to 2023 csv rows with support for random column ordering
# Known issues
- There seem to be some issues with how UPRN is handled if the UPRN cannot be validated.
- For the above I think there is dependency on https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/1570 which should clear any errored fields so users can continue to create logs and fix within the service
* create a method on the FormHandler that returns the sales form questions for all years in the order that they appear in the form
* update csv email job to accomodate sales log export as well as lettings
add to tests to reflec the changes made
* write tests to cover the desired functionality of the SalesLogCsvService
* create the SalesLogCsvService
create a necessary method on the log to enable submission method to be included on the csv
derive values for the two halves of previous postcode for export
* add relevant links in the UI and pipe everything together in controllers
amend organisations controller to have flexibility to download logs of either type
add necessary methods to sales log controller, raising shared method to logs controller
update routing for amendments and additions
extract helper method to build urls for downloading logs within an organisation
* correct various linter complaints and tech review suggestions
* minor amendment to add old_id and reorder early columns
* undo my 'clever' refactor that broke things
* refactoring of csv service after some tech review and some UI testing in review app
* update tests to include a test of a full export and all values in teh csv
* correct minor routing error to ensure correct url is shown and tab selected after requesting csv email
* update organisations controller requests spec file to cover new functionality and make a minor amendment to authentication scope in the controller after error found in testing
* write request tests for the new functionality in the sales log controller, define authorisation in the controller
* minor correction after rubocop's kind suggestion'
* various corrections from first pass at PO, tech review, linter, etc
* refactor :ordered_sales_questions_for_all_years
* first pass at implementing flexible code-based form fixtures for testing
* second pass
* refactor all tests of :ordered_sales_questions_for_all_years to use new factories
* some refactoring in the testing of the csv service
* use that fact that params is always available in controllers and don't pass it around, inline some methods calls
* correct minor bug to ensure that "Return to logs" link returns to the correct index page
* remove reminder comments
* write further tests on the manipulation of questions into the csv headers, update factories of form constituents to allow the creation of forms with richer questions
* fix linter complaints
* minor alterations after rebase to account for changes made on other branches
* refactor after code review
* tweak fixtures after rebase containing alterations to the factory defaults
# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-1888
- This is a continuation of https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/1277
- When bulk uploading we want to check users are not uploading data that already exists to prevent them submitting duplicate
# Changes
- This feature is behind a feature toggle. it has been disabled for staging for testing purposes but available in all other environments
- If a log already exists based off certain fields add errors to the associated fields
- We discount any hidden logs and only check "active" logs
- Added memoization to `#valid?` as an optimisation
Rachael Booth
kosiakkatrina
Arthur Campbell
natdeanlewissoftwire
Jack
Phil Lee