# Context
- https://digital.dclg.gov.uk/jira/browse/CLDC-2316
- Implement bulk upload sales for new collection year 2023
- This is a first pass implementation and will probably have some bugs in it and we can address over time
# Changes
- Add CSV parser for sales 2023 to handle CSV structure
- Tweak collection window validation so error now contextual to year selected for upload
- Handle arbitrary ordering of CSV columns
- Fix ordering of errors in report and now ordered by cell
- Added `Upload your file again` link styled as button on error report to match lettings experience
- Update tooling to convert logs to 2023 csv rows with support for random column ordering
# Known issues
- There seem to be some issues with how UPRN is handled if the UPRN cannot be validated.
- For the above I think there is dependency on https://github.com/communitiesuk/submit-social-housing-lettings-and-sales-data/pull/1570 which should clear any errored fields so users can continue to create logs and fix within the service
* create a method on the FormHandler that returns the sales form questions for all years in the order that they appear in the form
* update csv email job to accomodate sales log export as well as lettings
add to tests to reflec the changes made
* write tests to cover the desired functionality of the SalesLogCsvService
* create the SalesLogCsvService
create a necessary method on the log to enable submission method to be included on the csv
derive values for the two halves of previous postcode for export
* add relevant links in the UI and pipe everything together in controllers
amend organisations controller to have flexibility to download logs of either type
add necessary methods to sales log controller, raising shared method to logs controller
update routing for amendments and additions
extract helper method to build urls for downloading logs within an organisation
* correct various linter complaints and tech review suggestions
* minor amendment to add old_id and reorder early columns
* undo my 'clever' refactor that broke things
* refactoring of csv service after some tech review and some UI testing in review app
* update tests to include a test of a full export and all values in teh csv
* correct minor routing error to ensure correct url is shown and tab selected after requesting csv email
* update organisations controller requests spec file to cover new functionality and make a minor amendment to authentication scope in the controller after error found in testing
* write request tests for the new functionality in the sales log controller, define authorisation in the controller
* minor correction after rubocop's kind suggestion'
* various corrections from first pass at PO, tech review, linter, etc
* refactor :ordered_sales_questions_for_all_years
* first pass at implementing flexible code-based form fixtures for testing
* second pass
* refactor all tests of :ordered_sales_questions_for_all_years to use new factories
* some refactoring in the testing of the csv service
* use that fact that params is always available in controllers and don't pass it around, inline some methods calls
* correct minor bug to ensure that "Return to logs" link returns to the correct index page
* remove reminder comments
* write further tests on the manipulation of questions into the csv headers, update factories of form constituents to allow the creation of forms with richer questions
* fix linter complaints
* minor alterations after rebase to account for changes made on other branches
* refactor after code review
* tweak fixtures after rebase containing alterations to the factory defaults
* add first page for bulk upload resume journey
* bulk upload resume handles upload again
* add confirm page to bulk upload resume journey
* replace placeholder count with correct value
* apply recommendation for bulk upload resume choice
* add how to fix bulk upload mailer
* integrate new bulk upload approve journey
* add missing bulk upload error mappings
* remove test
* prevent approve being called multiple times
* bulk upload creates invisible logs ahead of time
* work invisible logs into bulk upload flow
* sort errors so deterministic
* remove unused ensure
* remove expected_log_count and processed
- these fields are no longer used or needed
* introduce pending status
* swap visible for pending logs
* only show visible lettings logs
* hard code status filters
* remove unused model methods
* only show visible sales logs
* form controller ignores hidden logs
* locations and schemes only affect visible logs
* update seeds to add self in review env, change spec to reflect this, update config yml to allow csv exports in review
* update interface of relevant methods
EmailCsvJob, LettingsLog.to_csv and LettingsLogCsvService consume codes_only flag
* update tests including adding a new csv file to test against
* update LettingsLogCsvService to output codes only csv
* correct minor error and linting
* enable codes only download in UI
- add link on lettings log index page
- pass codes_only flag through params in relevant links and methods
- convert flag to boolean in controller methods
* ensure link displayed successfully for all renderings of logs_list and params passed through relevant methods in organisations controller
* fix existing tests
* correct linting thing
* correct linting error
* update tests for lettings log controller
* correct linting errors
* update organisations controller tests
* make minor changes after code review
* remove changes made for testing on review app
* make codes only download visible to support users only
* change variable names throughout after info on rauby/rails naming conventions, update tests for change in who can view codes only download link
* rework csv service for readability, remove delegating methods from lettings log to keep all code to do with mapping between our domain and desired export format in one place
* update test name
* correct a small typo and remove a duplicated method after clever git merge conflict suggestion
* point review app at staging csv bucket for csv download
* change variables named codes_only_export to codes_only to avoid inconsistency
* write tests to ensure that differetn user roles have the correct permissions around csv download
* ensure that non support users may not download codes only exports
* correct a small error in a previous commit
* correct minor linting error
* CYA summary list tweaks for bulk upload
- When checking answers for a bulk upload related log the copy is
tweaked and made red as per designs
* different CYA treatment when for bulk upload
* bulk upload CYA missing answers always red
- whereas previosly this only happened if user was filtering logs via a
bulk upload
* refactor with extract method
* can filter logs by bulk upload
* hide log creation button when viewing bulk upload
- this affects the logs index page filtering logs from a specific bulk
upload
* add info banner to bulk upload logs
* placeholder for bulk upload logs header
* when resuming bulk upload set filters
* fill place holder with remaining logs to fix
* add interstitial to resume if logs resolved
* after resolving bulk upload logs show interstitial
* fix linting error
* extract view variable to helper method
* Mark log as impacted by deactivation when location is deactivated
* Display affected logs in the table
* Route affected logs to tenancy start date question
* Update routes to get the tenancy start date page from form
* update next_page_redirect_path
* rename column
* Fix tests
* Add next_unresolved_page_id to pages
* Update unresulved when the log is corrected
* Mark logs as unresolved when scheme gets deactivated
* display correct content when there are no unresolved logs
* mark logs resolved after the user leaves check answers page
* Display link in success banner and reset banner when the link is clicked
* display inset hint text for unresolved log questions
* Display unresolved logs banner
* update banner message
* Persist the link in the banner
* update inset text
* Update success banner text
* Add unresolved and created_by scopes
* rename method
* add unresolved_log_redirect_page_id to form + typo and route
* Add UnresolvedLogHelper and extract flack notice message
* pluralize and return early
* remove flash[:notice] = nil
* to keep it consistent for sales log
* Extract unresolved path and fix a link
* extract resolve method and fix attribute nme
* update path
* typo
* Add abstract log class and sales log class
Created a parent log class for sales log and lettings log. Any bits common
to both sales and lettings can live in the parent class. As the sales log
functionality is built up any commonalities with lettings log can be extracted
into the parent log class. The sales log model is set up without a json form
and instead the form is defined in code - like the setup section of the lettings
log.
* update sales logs controller
* update lettings controller specs
* update filter method name
* update organisations controller
* use lettings method
* Add deleted tests back
* lint
Co-authored-by: Kat <katrina@madetech.com>
Co-authored-by: Kat <kosiak.katrina@gmail.com>
* Replaced log CSV direct download with email
* Tidy up authorization of organisations controller
- We already have a method to authenticate the scope of the user, so we can reuse that.
* Use Rails routes instead of absolute paths for CSV download links
* Introduce base NotifyMailer to to abstract away shared Notify mail functionality
* Fix mailer spec name
* Add worker instance to PaaS manifest
* Add CSV download bucket instance name into environment
* Update tests for improved search term handling
* Fix download mailer tests
* Clarifying comments
Co-authored-by: natdeanlewissoftwire <nat.dean-lewis@softwire.com>
Co-authored-by: James Rose <james@jbpr.net>
This renames the case_log to lettings_log as everything we've written so
far has been geared towards lettings of social housing so it makes sense to
have the name describe this. This is also a precursor to adding in stuff for
sales logs (whatever shape that takes)
Co-authored-by: James Rose <james@jbpr.net>
* remove irrelevant fields from csv based on the user
* display yes/no instead of true/false
* display organisation and user names instead of ids
* refactor tests
* Add an organisation question for the support user (WIP)
Co-authored-by: baarkerlounger <baarkerlounger@users.noreply.github.com>
* All the things are green
* Add user question
* Restrict shown answer options based on already answered questions
* Derive managing org
* Guard against missing DB connection
* Remove empty file
* Only select the fields we actually need
* All is redundant
* Fix spec description
* Fix comments
Co-authored-by: Kat <katrina@madetech.com>
Co-authored-by: baarkerlounger <baarkerlounger@users.noreply.github.com>
* Refactor depends on in the form json to specify the object for the dependency
Co-authored-by: baarkerlounger <baarkerlounger@users.noreply.github.com>
* Remove org section
* Enable user based depends on
* Add question
* Question is derived
* Log debug message when values get auto cleared
* Enable answer options to be created from a model lookup
* Make organisation fields optional since they won't immediately have one when created by support users
* Add question to check answers page
* 22/23 fix
* Rubocop
* Spec check answers label lookup
* Only set managing organisation if not already set
* Bump deps
* Rescue lack of database connection for CI
Co-authored-by: Kat <katrina@madetech.com>
Co-authored-by: baarkerlounger <baarkerlounger@users.noreply.github.com>
* test to download only search results
* testing I can download only logs matching both search and filter
* rubocop
* matching style for title
* ILIKE for search
* rubocop
* using ILIKE
* upated rack due to vulnerability
* refactoring
* refactoring
* refactored test
* refactored matching search results on multiple page tests for title
* no pagination title
* no pagination title for when no logs
* fixed all title related tests for filter and search
* rubocop
* removing extra space in the view - unneccesary
* better test when searching for postcode
* failing lint?
* refactored condition
* refactored content to links
* refactored content to links in case logs
* Get all organisations in controller
* Display organisations data in the table
* Route to logs for specific organisation
* add tests
* update spec
* lint fixes
* set up failing test for organisation logs page
* fix failing test
* write test for organisations support user page
* Update a organisation page test and lint
* added pagination test with next and previous links and total count for support user
* test for pagination in organisations title
* Added "Organisations" to to organisations page title
* add pagination test for organisations page 2, remove second before block
* Add the remaining pagination tests
* Redirect when accessing organisation logs by non support user
* Test for displaying logs for specific organisation
* Add test for org name
* Add a failing log filter test for specific org
* Extract filter methods into a helper
* Allow logs filtering for specific org
* Fix test, support user was creating an extra org, remove orgs filter for specific org
* Remove redundant test, lint
* Reuse primary navigation component and add sub navigation for support users
* allow support users edit or and add sub navigation to about this org
* allow support users to access the edit org page
* only allow to edit existing editable fields
* display correct values in the organisations table
* allow support user to update org
* user table component for organisations table
* use guard clause for organisation logs page
* remove create a new lettings log from organisation logs
* Move case logs filter from helpers to modules
* lint erb
* yarn lint
* bring back if statement in logs controller
* update modules import
* let!
* test for links first in the org cotroller spec
* interpolate number of orgs
* conditionally render sub navigation
Co-authored-by: Kat <katrina@madetech.com>
Co-authored-by: Dushan Despotovic <dushan@madetech.com>
Co-authored-by: JG <moarpheus@gmail.com>
* Filter values correctly
* Remove filter value if hidden
* Return empty string rather than nil for accessible autocomplete
* Additional test for accessible automcomplete defaulting
* Rename org filter
* Ensure filters work when ID is passed
* UI init
* Lint
* Fix filter nesting
* Reduce width
* Set checked status of filters correctly
* Test filter presence
* Add filter test
* Rubocop
* Tweak styles for autocomplete within filter
Co-authored-by: Paul Robert Lloyd <me+git@paulrobertlloyd.com>
* radio button
* radio button again
* rubocop versioning issue fixed
* linux and darwin
* Fix Gemfile
* rubocop
* erb lint
Co-authored-by: baarkerlounger <db@slothlife.xyz>
This commit fixes the tests that failed as a result of the introduction
of the created_by column in the case log table. The tests were failing as
it is now necessary to have a created_by for a case log
* add year filter
* update text and filter order
* fix typo
* extract partials
* add tests with 2 filters
* Allow OR filters within scope categories and AND filters across them
* Rubocop
* Remove some of the N+1 query madness by preloading required organisations
* Refactor scopes
* Fix new specs
* unticked checkboxes by deault
* fix test
Co-authored-by: baarkerlounger <db@slothlife.xyz>
Co-authored-by: baarkerlounger <5101747+baarkerlounger@users.noreply.github.com>
* Add a filter for status
* Add styling, separate filter tab into a file
* Add mobile styling
* use session instead of cookies
* Get statuses from case_log
* remove explicit builder set
* set filter from within get index
* style new log button
* refactor tests
* Link text
* Controller response
* Fix download link
* Check CSV only downloads collection
* Download value labels rather than values
* Set mime type in link markup
* Add Pagy pagination to logs index
* Add result count
* Show total count at top
* Replace view spec with request spec
* More logs context
* Make sure we have next/prev
* Add total count
* Generisize
* Remove pagination indicators when only 1 page
* Add pagination to title
* Whole pagination nav only shows if > 1 page
* Dry up paginated titles
* Update style namespace
* Iterate over all forms
* Re-lookup page to ensure we have the definition for the right form
* Match form naming convention for spec and real
* Add test for case log form
* Scope auth for org pages
* Scope user methods
* Refactor form UI controller
* More tests for access
* Consistently return not found for scoped auth
* Add ADR
* Authenticate bulk uploads
* Authenticate soft validations controller
* Scope bulk upload to user's org for now
* Pass through current user
* Update docs/adr/adr-012-controller-http-return-statuses.md
Co-authored-by: Dushan <47317567+dushan-madetech@users.noreply.github.com>
* Update spec/requests/case_log_controller_spec.rb
Co-authored-by: Dushan <47317567+dushan-madetech@users.noreply.github.com>
Co-authored-by: Dushan <47317567+dushan-madetech@users.noreply.github.com>
* Add organisation model and user association
* Add user admin panel
* email
* Update seeds
* Update spec
* Case logs belong to organisations
* Org case log association
* Add user case log association
* User case log helper methods
* Org case log relation helpers
* Case log index page only shows your organisations case logs
* No access to tasklist page for logs that aren't associated with your org
* No access to form pages for case logs that aren't owned or managed by your org
* Check answers access
* Submit form access
* Refactor out not found methods
* Allow user admin update without password
* Update feature specs
* Rubocop
* Update case log specs
* Test admin user update without password
* Spec grammar
Phil Lee
Arthur Campbell
kosiakkatrina
natdeanlewissoftwire
James Rose
natdeanlewissoftwire
Dushan
Sam
baarkerlounger
J G
JG
Ted-U
baarkerlounger
Dushan Despotovic
Paul Robert Lloyd